捉虫日记 | MySQL 5.7.20 try_acquire_lock_impl 异常导致mysql crash
背景
近期线上MySQL 5.7.20集群不定期(多则三周,短则一两天)出现主库mysql crash、触发主从切换问题,堆栈信息如下;
从堆栈信息可以明显看出,在调用 try_acquire_lock_impl
时触发的crash。
分析
在官方Bug库未搜到类似问题,转而从代码库入手,搜到对应的BUG —— 8bc828b982f678d6b57c1853bbe78080c8f84e84:
BUG#26502135: MYSQLD SEGFAULTS IN
MDL_CONTEXT::TRY_ACQUIRE_LOCK_IMPL
ANALYSIS:
=========
Server sometimes exited when multiple threads tried to
acquire and release metadata locks simultaneously (for
example, necessary to access a table). The same problem
could have occurred when new objects were registered/
deregistered in Performance Schema.
The problem was caused by a bug in LF_HASH - our lock free
hash implementation which is used by metadata locking
subsystem in 5.7 branch. In 5.5 and 5.6 we only use LF_HASH
in Performance Schema Instrumentation implementation. So
for these versions, the problem was limited to P_S.
The problem was in my_lfind() function, which searches for
the specific hash element by going through the elements
list. During this search it loads information about element
checked such as key pointer and hash value into local
variables. Then it confirms that they are not corrupted by
concurrent delete operation (which will set pointer to 0)
by checking if element is still in the list. The latter
check did not take into account that compiler (and
processor) can reorder reads in such a way that load of key
pointer will happen after it, making result of the check
invalid.
FIX:
====
This patch fixes the problem by ensuring that no such
reordering can take place. This is achieved by using
my_atomic_loadptr() which contains compiler and processor
memory barriers for the check mentioned above and other
similar places.
The default (for non-Windows systems) implementation of
my_atomic*() relies on old __sync intrisics and implements
my_atomic_loadptr() as read-modify operation. To avoid
scalability/performance penalty associated with addition of
my_atomic_loadptr()'s we change the my_atomic*() to use
newer __atomic intrisics when available. This new default
implementation doesn't have such a drawback.
大体含义是:
当多个线程分别同时获取、释放metadata locks时,或者在 Performance Schema 中注册/撤销新的object时,可能会触发该问题,导致 mysql server crash。
该问题是 LF_HASH(Lock-Free Extensible Hash Tables) 的BUG引起的,那么 LF_HASH 用在什么地方呢?
- 在5.5、5.6中只用在 Performance Schema Instrumentation 模块。
- 在5.7中也用于metadata加锁模块。
问题出在my_lfind() 函数中,该函数针对cursor->prev
的判断未考虑CAS,该patch通过使用 my_atomic_loadptr()
解决了该问题:
diff --git a/mysys/lf_hash.c b/mysys/lf_hash.c
index dc019b07bd9..3a3f665a4f1 100644
--- a/mysys/lf_hash.c
+++ b/mysys/lf_hash.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2006, 2016, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2006, 2017, Oracle and/or its affiliates. All rights reserved.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -83,7 +83,8 @@ retry:
do { /* PTR() isn't necessary below, head is a dummy node */
cursor->curr= (LF_SLIST *)(*cursor->prev);
_lf_pin(pins, 1, cursor->curr);
- } while (*cursor->prev != (intptr)cursor->curr && LF_BACKOFF);
+ } while (my_atomic_loadptr((void**)cursor->prev) != cursor->curr &&
+ LF_BACKOFF);
for (;;)
{
if (unlikely(!cursor->curr))
@@ -97,7 +98,7 @@ retry:
cur_hashnr= cursor->curr->hashnr;
cur_key= cursor->curr->key;
cur_keylen= cursor->curr->keylen;
- if (*cursor->prev != (intptr)cursor->curr)
+ if (my_atomic_loadptr((void**)cursor->prev) != cursor->curr)
{
(void)LF_BACKOFF;
goto retry;
解决
查看change log,该问题在5.7.22版本修复的:
A server exit could result from simultaneous attempts by multiple threads to register and deregister metadata Performance Schema objects, or to acquire and release metadata locks. (Bug #26502135)
升级内核版本到5.7.29,之后巡检1个月,该问题未再出现,问题解决。
PS:
篇幅有限,在后续文章中会单独分析 MDL、LF_HASH 源码,敬请关注。
欢迎关注我的微信公众号【MySQL数据库技术】。
知乎 - 数据库技术 专栏: https://zhuanlan.zhihu.com/mysqldb
思否/segmentfault: https://segmentfault.com/u/dbtech
开源中国/oschina: https://my.oschina.net/dbtech
掘金: https://juejin.im/user/5e9d3ed251882538083fed1f/posts
博客园/cnblogs: https://www.cnblogs.com/dbtech
捉虫日记 | MySQL 5.7.20 try_acquire_lock_impl 异常导致mysql crash的更多相关文章
- Linux 文件系统引起的云盘文件系统异常导致 MySQL 数据页损坏事故恢复复盘
事故的起因是因为当我访问某个数据库的某个表的时候,MySQL 立即出现崩溃并且去查看 MySQL 的错误日志出现类似信息 --09T05::.232564Z [ERROR] InnoDB: Space ...
- 记录一次更改服务器名称导致mysql 不能正常登录、启动
由于客户要求更改服务器的名称,以便区分多台服务器:修改前mysql 能正常登录,但是修改后,登录时报错: Enter password: ERROR 1524 (HY000): Plugin '*C6 ...
- 捉虫记(四)线程安全导致的HighCpu
一个朋友QQ群里说网站启动后会cpu很高,想要帮忙看一下dump. 1.打开windbg加载dump文件后第一个命令lmf,这个命令显示加载的dll以及路径,这样子可以找个dll来帮忙加载sos,(额 ...
- [MySQL Reference Manual] 20 分区
20 分区 20 分区 20.1 MySQL的分区概述 20.2 分区类型 20.2.1 RANGE分区 20.2.2 LIST分区 20.2.3 COLUMNS分区 20.2.3.1 RANGE C ...
- mysql 5.6.20的安装、配置服务、设置编码格式
一.安装 安装环境 系统:Window 32 版本:Mysql 5.6.20 1. 首先从官网上http://dev.mysql.com/downloads/mysql/ ...
- win 2012 安装mysql 5.7.20 及报错 This application requires Visual Studio 2013 Redistributable. Please install the Redistributable then run this installer again 的解决办法
本文地址:http://www.cnblogs.com/jying/p/7764147.html 转载请注明出处. 安装过程其实挺简单,基本上下一步下一步,可以参考我的另一篇mysql安装文章: ...
- Windows 64 位 mysql 5.7.20 安装教程
mysql 5.7以上版本包解压中没有data目录和my-default.ini和my.ini文件以及服务无法启动的解决办法以及修改初始密码的方法 mysql官网下载地址:https://dev.my ...
- Windows 下 MySql 5.7.20安装及data和my.ini文件的配置(转)
Windows 下 MySql 5.7.20安装及data和my.ini文件的配置 本文通过图文并茂的形式给大家介绍了MySql 5.7.20安装及data和my.ini文件的配置方法. my ...
- CentOS 7.4 使用源码包编译安装MySQL 5.7.20
使用yum安装的MySQL一般版本比较旧,但是运行稳定.如果想要尝试最新的功能或者需要指定特殊的功能的话,就需要手工进行编译安装了. 一.下载安装包 (一).先下载MySQL源码,网址为:https: ...
随机推荐
- python之学生信息管理系统
1 #!usr/bin/python 2 #encoding=utf-8 3 4 #1. 打印学生管理系统界面 5 def printStd(): 6 print ("*"*50) ...
- .Net Core微服务——服务发现:Consul(一)
先思考一些问题:它是做什么的.以及怎么使用它.带着这些问题往下走. consul是做什么的 consul用于微服务下的服务治理.服务治理是什么?它包含但不限于:服务发现.服务配置.健康检查.键值存储. ...
- 「BZOJ 2956」模积和
「BZOJ 2956」模积和 令 \(l=\min(n,m)\).这个 \(i\neq j\) 非常不优雅,所以我们考虑分开计算,即: \[\begin{aligned} &\sum_{i=1 ...
- Leetcode12. 整数转罗马数字Leetcode18. 四数之和
> 简洁易懂讲清原理,讲不清你来打我~ 输入整数,输出对应的罗马字符串
PAT乙级:1090危险品装箱(25分) 题干 集装箱运输货物时,我们必须特别小心,不能把不相容的货物装在一只箱子里.比如氧化剂绝对不能跟易燃液体同箱,否则很容易造成爆炸. 本题给定一张不相容物品的清 ...
- 单点登录详解(token简述)(七)
前言 为什么整理单点登录? 主要的原因还是自己以前学习的时候曾经用过,但是时间太久,忘记了里面用到了哪些技术.及如何实现的,每次想到单点登录总是感觉即会又不会,这次整理session时,又涉及到了单点 ...
- SAML 2.0 实例分析 idp向sp发送响应(4)
当idp与user建立起联系后,idp向sp发送响应 <samlp:Response xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol ...
- [POI2008]PER
很有思维的一道题 这个题的题面非常简单,出题人很友好,没有搞什么奇怪的背景,(卡农(P3214)的作者看看人家),所以理解题面就是: 一句话题意: 给定一个长度为 \(n\) 的数列,求这个数列是在其 ...
- GetOverlappedResult 函数
BOOL GetOverlappedResult( HANDLE hFile, LPOVERLAPPED lpOverlapped, LPDWORD lpNumberOfBytesTransferre ...