centos 7.0运行docker出现内核报错解决方法
目前我这里docker是运行在centos 7.0系统里,使用1.5版本docker,最近一台服务器总是不定期死机,通过查看日志发现属于内核bug导致,报错信息如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
|
May 11 03:43:08 ip-10-10-29-201 kernel: BUG: soft lockup - CPU #4 stuck for 22s! [handler20:1542]
May 11 03:43:08 ip-10-10-29-201 kernel: Modules linked in : iptable_nat nf_nat_ipv4 iptable_filter ip_tables binfmt_misc ipmi_si vfat fat usb_storage mpt3sas mpt2sas raid_
class scsi_transport_sas mptctl mptbase dell_rbu tcp_diag inet_diag veth bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop dm_mod openvswitch vxl an ip_tunnel gre libcrc32c xt_nat ipt_MASQUERADE xt_addrtype nf_nat xt_limit ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_conntrack sg nf_conntrack ipmi_de vintf iTCO_wdt iTCO_vendor_support dcdbas coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_ helper cryptd pcspkr sb_edac edac_core ses enclosure ipmi_msghandler tg3 wmi acpi_power_meter ptp pps_core mei_me mei ntb lpc_ich mperf mfd_core shpchp ext4 May 11 03:43:08 ip-10-10-29-201 kernel: mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_common mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ahci drm libahci libata i2c_core megaraid_sas [last unloaded: ip_tables]
May 11 03:43:08 ip-10-10-29-201 kernel: CPU: 4 PID: 1542 Comm: handler20 Tainted: G W -------------- 3.10.0-123.el7.x86_64 #1
May 11 03:43:08 ip-10-10-29-201 kernel: Hardware name: Dell Inc. PowerEdge R720 /0X6FFV , BIOS 1.6.0 03 /07/2013
May 11 03:43:08 ip-10-10-29-201 kernel: task: ffff880418adf1c0 ti: ffff8800c8d08000 task.ti: ffff8800c8d08000 May 11 03:43:08 ip-10-10-29-201 kernel: RIP: 0010:[<ffffffff815e90e7>] [<ffffffff815e90e7>] _raw_spin_lock+0x37 /0x50
May 11 03:43:08 ip-10-10-29-201 kernel: RSP: 0018:ffff88041fc43ac8 EFLAGS: 00000206 May 11 03:43:08 ip-10-10-29-201 kernel: RAX: 000000000000108b RBX: 0000000000000000 RCX: 0000000000000000 May 11 03:43:08 ip-10-10-29-201 kernel: RDX: 0000000000000002 RSI: 0000000000000002 RDI: ffff88081609c318 May 11 03:43:08 ip-10-10-29-201 kernel: RBP: ffff88041fc43ac8 R08: ffff8801049856d8 R09: ffff88041fc43a00 May 11 03:43:08 ip-10-10-29-201 kernel: R10: 0000000000000000 R11: 00000000e1bec8f9 R12: ffff88041fc43a38 May 11 03:43:08 ip-10-10-29-201 kernel: R13: ffffffff815f2d9d R14: ffff88041fc43ac8 R15: ffff88081609c300 May 11 03:43:08 ip-10-10-29-201 kernel: FS: 00007fb082b8b700(0000) GS:ffff88041fc40000(0000) knlGS:0000000000000000 May 11 03:43:08 ip-10-10-29-201 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 11 03:43:08 ip-10-10-29-201 kernel: CR2: 00007f2a743e6000 CR3: 00000008183c9000 CR4: 00000000000407e0 May 11 03:43:08 ip-10-10-29-201 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 May 11 03:43:08 ip-10-10-29-201 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 May 11 03:43:08 ip-10-10-29-201 kernel: Stack: May 11 03:43:08 ip-10-10-29-201 kernel: ffff88041fc43af8 ffffffffa042429f ffff88003714be00 ffffe8fbefc41540 May 11 03:43:08 ip-10-10-29-201 kernel: ffff880419070e80 ffff88041fc43b30 ffff88041fc43be0 ffffffffa04239a4 May 11 03:43:08 ip-10-10-29-201 kernel: 00000001b9ec8070 ffff88003714be00 ffff88041fc43b28 0000000000000246 May 11 03:43:08 ip-10-10-29-201 kernel: Call Trace: May 11 03:43:08 ip-10-10-29-201 kernel: <IRQ> May 11 03:43:08 ip-10-10-29-201 kernel: May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa042429f>] ovs_flow_stats_update+0x4f /0xd0 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa04239a4>] ovs_dp_process_received_packet+0x84 /0x120 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa042a01a>] ovs_vport_receive+0x2a /0x30 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa042b4cd>] vxlan_rcv+0x6d /0x90 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa037b228>] vxlan_udp_encap_recv+0xb8 /0x130 [vxlan]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff81538bc2>] udp_queue_rcv_skb+0x162 /0x3d0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff815394bd>] __udp4_lib_rcv+0x19d /0x690
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff815094d0>] ? ip_rcv_finish+0x350 /0x350
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff815399ca>] udp_rcv+0x1a /0x20
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff81509584>] ip_local_deliver_finish+0xb4 /0x1f0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff81509858>] ip_local_deliver+0x48 /0x80
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff815091fd>] ip_rcv_finish+0x7d /0x350
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff81509ac4>] ip_rcv+0x234 /0x380
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814cfdb6>] __netif_receive_skb_core+0x676 /0x870
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814cffc8>] __netif_receive_skb+0x18 /0x60
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814d0b7e>] process_backlog+0xae /0x180
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814d041a>] net_rx_action+0x15a /0x250
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff81067047>] __do_softirq+0xf7 /0x290
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff815f3a5c>] call_softirq+0x1c /0x30
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff81014d25>] do_softirq+0x55 /0x90
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff810673e5>] irq_exit+0x115 /0x120
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff815f4358>] do_IRQ+0x58 /0xf0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff815e94ad>] common_interrupt+0x6d /0x6d
May 11 03:43:08 ip-10-10-29-201 kernel: <EOI> May 11 03:43:08 ip-10-10-29-201 kernel: May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa0424465>] ? ovs_flow_stats_get+0x145 /0x180 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa0424453>] ? ovs_flow_stats_get+0x133 /0x180 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa04217b7>] ovs_flow_cmd_fill_info+0x1c7 /0x320 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa0421c5c>] ovs_flow_cmd_build_info.constprop.25+0x6c /0xa0 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffffa0422155>] ovs_flow_cmd_new_or_set+0x4c5 /0x520 [openvswitch]
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff8108ec58>] ? __wake_up_common+0x58 /0x90
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814ffcd8>] genl_family_rcv_msg+0x258 /0x3d0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814ffe50>] ? genl_family_rcv_msg+0x3d0 /0x3d0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814ffee1>] genl_rcv_msg+0x91 /0xd0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814fdf99>] netlink_rcv_skb+0xa9 /0xc0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814fe4c8>] genl_rcv+0x28 /0x40
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814fd5bd>] netlink_unicast+0xed /0x1b0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814fd9a7>] netlink_sendmsg+0x327 /0x760
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814fa874>] ? netlink_rcv_wake+0x44 /0x60
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814fb92b>] ? netlink_recvmsg+0x1cb /0x3e0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814b79b0>] sock_sendmsg+0xb0 /0xf0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814b807f>] ? sock_recvmsg+0xbf /0x100
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff8109b23e>] ? task_scan_min+0x3e /0x60
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff815e908b>] ? _raw_spin_unlock_bh+0x1b /0x40
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff814b7de9>] ___sys_sendmsg+0x3a9 /0x3c0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff811f7fa9>] ? ep_scan_ready_list.isra.9+0x1b9 /0x1f0
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff811f8123>] ? ep_poll+0x123 /0x370
May 11 03:43:08 ip-10-10-29-201 kernel: [<ffffffff81079af3>] ? getrusage+0x43 /0x70
May 11 03:43:09 ip-10-10-29-201 kernel: [<ffffffff814b8cd1>] __sys_sendmsg+0x51 /0x90
May 11 03:43:09 ip-10-10-29-201 kernel: [<ffffffff814b8d22>] SyS_sendmsg+0x12 /0x20
May 11 03:43:09 ip-10-10-29-201 kernel: [<ffffffff815f2119>] system_call_fastpath+0x16 /0x1b
May 11 03:43:09 ip-10-10-29-201 kernel: Code: 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 83 e8 01 7 4 0a <0f> b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66 90 eb da 66 0f |
通过在stackoverflow查询发现此问题属于内核bug,解决方法是升级内核。
下面是把centos 7.0默认3.10版本内核升级为4.0.2版本过程
1、导入yum源的认证key
1
|
rpm -- import https: //www .elrepo.org /RPM-GPG-KEY-elrepo .org
|
2、安装yum源
1
|
rpm -Uvh http: //www .elrepo.org /elrepo-release-7 .0-2.el7.elrepo.noarch.rpm
|
3、安装新内核
在yum的ELRepo源中,有mainline(4.0.2)这个内核版本
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
|
[root@ip-10-10-29-201 ~] # yum --enablerepo=elrepo-kernel install kernel-ml-devel kernel-ml
Loaded plugins: fastestmirror MooseFS | 951 B 00:00:00 base | 3.6 kB 00:00:00 elrepo | 2.9 kB 00:00:00 elrepo-kernel | 2.9 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1 /2 ): elrepo /primary_db | 233 kB 00:00:02
(2 /2 ): elrepo-kernel /primary_db | 782 kB 00:00:04
MooseFS /primary | 4.2 kB 00:00:00
Loading mirror speeds from cached hostfile * base: mirrors.yun-idc.com
* elrepo: repos.lax-noc.com
* elrepo-kernel: repos.lax-noc.com
* extras: mirror.bit.edu.cn
* updates: mirror.bit.edu.cn
MooseFS 30 /30
Resolving Dependencies --> Running transaction check ---> Package kernel-ml.x86_64 0:4.0.2-1.el7.elrepo will be installed ---> Package kernel-ml-devel.x86_64 0:4.0.2-1.el7.elrepo will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================== Package Arch Version Repository Size
========================================================================================================================================================================== Installing: kernel-ml x86_64 4.0.2-1.el7.elrepo elrepo-kernel 36 M
kernel-ml-devel x86_64 4.0.2-1.el7.elrepo elrepo-kernel 9.5 M
Transaction Summary ========================================================================================================================================================================== Install 2 Packages Total download size: 45 M Installed size: 199 M Is this ok [y /d/N ]: y
Downloading packages: (1 /2 ): kernel-ml-4.0.2-1.el7.elrepo.x86_64.rpm | 36 MB 00:00:11
(2 /2 ): kernel-ml-devel-4.0.2-1.el7.elrepo.x86_64.rpm | 9.5 MB 00:00:31
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 1.5 MB /s | 45 MB 00:00:31
Running transaction check Running transaction test
Transaction test succeeded
Running transaction Warning: RPMDB altered outside of yum. Installing : kernel-ml-devel-4.0.2-1.el7.elrepo.x86_64 1 /2
Installing : kernel-ml-4.0.2-1.el7.elrepo.x86_64 2 /2
Verifying : kernel-ml-4.0.2-1.el7.elrepo.x86_64 1 /2
Verifying : kernel-ml-devel-4.0.2-1.el7.elrepo.x86_64 2 /2
Installed: kernel-ml.x86_64 0:4.0.2-1.el7.elrepo kernel-ml-devel.x86_64 0:4.0.2-1.el7.elrepo
Complete! |
4、查看当前内核版本
1
2
|
[root@ip-10-10-29-201 ~] # uname -r
3.10.0-123.el7.x86_64 |
重要:目前内核还是默认的版本,如果在这一步完成后你就直接reboot了,重启后使用的内核版本还是默认的3.10,不会使用新的4.0.2,想修改启动的顺序,需要进行下一步
查看默认启动顺序
1
2
3
4
|
[root@ip-10-10-29-201 ~] # awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
CentOS Linux (4.0.2-1.el7.elrepo.x86_64) 7 (Core) CentOS Linux, with Linux 3.10.0-123.el7.x86_64 CentOS Linux, with Linux 0-rescue-18b184aa09434ecf9739a70c6b63638a |
默认启动的顺序是从0开始,但我们新内核是从头插入(目前位置在1,而4.0.2的是在0),所以需要选择0,如果想生效最新的内核,需要
1
|
[root@ip-10-10-29-201 ~] # grub2-set-default 0
|
5、重启
1
|
Reboot |
6、重启后查看内核
1
2
|
[root@ip-10-10-29-201 conf] # uname -r
4.0.2-1.el7.elrepo.x86_64 |
经过升级后,20天没有出现此问题,所以判断此次文件为内核bug引起,通过升级内核解决。
本文转自 reinxu 51CTO博客,原文链接:http://blog.51cto.com/dl528888/1655142,如需转载请自行联系原作者
centos 7.0运行docker出现内核报错解决方法的更多相关文章
- create-react-app创建项目后,运行npm run eject报错解决方法
运行npm run eject报错解决方法 主要问题是脚手架添加.gitgnore文件,但是却没有本地仓库,使用以下命令操作以下就可以了 git init git add . git commit - ...
- 【jmeter】NO-GUI模式运行的过程及报错解决方法(转)
1.什么都不需要设置(若非得想配置点什么的话,那请看备注) 2.将脚本抛到Linux里 3.执行 jmeter -n -t /opt/day7_buildbug.jmx -l result.jtl - ...
- Android Studio support 26.0.0-alpha1 Failed to resolve: com.android.support:appcompat-v7:27.+ 报错解决方法
AS下如何生成自定义的.jks签名文件, 以及如何生成数字签名 链接:http://www.cnblogs.com/smyhvae/p/4456420.html 链接:http://blog.csdn ...
- MyEclipse Server view报错解决方法
MyEclipse Server view报错解决方法 方法/步骤 启动MyEclipse,弹出一个框,报错. ---------------------------------------- ...
- eclipse创建的maven项目,pom.xml文件报错解决方法
[错误一:]maven 编译级别过低 [解决办法:] 使用 maven-compiler-plugin 将 maven 编译级别改为 jdk1.6 以上: <!-- java编译插件 --> ...
- Loadrunner参数化逗号报错解决方法
Loadrunner参数化逗号报错解决方法 介绍Loadrunner参数化时,参数中包含有逗号时出错的解决方法. 在Loadrunner进行参数化时,参数中如果含有逗号,编辑保存后会报错: 此 ...
- iOS url带中文下载时 报错解决方法
问题描述:下载文件时, 请求带中文的URL的资源时,比如:http://s237.sznews.com/pic/2010/11/23/e4fa5794926548ac953a8a525a23b6f2/ ...
- jquery jssdk分享报错解决方法
jssdk分享报错解决方法 一般都是参数传错了
- django.db.utils.InternalError: (1060, "Duplicate column name 'user_id'")迁移报错解决方法
django.db.utils.InternalError: (1060, "Duplicate column name 'user_id'")迁移报错解决方法 django.db ...
随机推荐
- 基于redis的订单号生成方案
目前,比较火的nosql数据库,如MongoDB,Redis,Riak都提供了类似incr原子行操作. 下面是PHP版的一种实现方式: <?php /** * 基于Redis的全局订单号id * ...
- Alpha测试与Beta测试
粗略说一下Alpha测试与beta测试 1.Alpha测试 α测试是由一个用户在开发环境下进行的测试,也可以是公司内部的用户在模拟实际操作环境下进行的测试.α测试的目的是评价软件产品的功能.局域化.可 ...
- 数据结构和算法(Golang实现)(17)常见数据结构-树
树 树是一种比较高级的基础数据结构,由n个有限节点组成的具有层次关系的集合. 树的定义: 有节点间的层次关系,分为父节点和子节点. 有唯一一个根节点,该根节点没有父节点. 除了根节点,每个节点有且只有 ...
- Python处理HTTP返回包遇到问题总结TypeError、keyError、SyntaxError、AttributeError
处理HTTP返回包包括对关键参数的校验,参数完整性检验,获取返回包参数的方法,返回包数据去重方法 在执行时遇到不少问题,部分问题记录如下: 1.报错信息:“TypeError: list indice ...
- Python - 利用词云wordcloud,jieba和中国地图制作四大名著的热词图
热词图很酷炫,也非常适合热点事件,抓住重点,以图文结合的方式表现出来,很有冲击力.下面这段代码是制作热词图的,用到了以下技术: jieba,把文本分词 wordcloud,制作热图 chardet,辨 ...
- 自己写的一个HTML的小网页
上次在上直播课的时候,教员提到了html这种标记语言.自己就在W3school上面学了一点点关于html的一些皮毛,自己动手写了一个小网页,同时自己对CTF这一块比较感兴趣,但是自己还是一个干干净净的 ...
- hashlib的md5计算
hashlib的md5计算 hashlib概述 涉及加密服务:Cryptographic Services 其中 hashlib是涉及 安全散列 和 消息摘要 ,提供多个不同的加密算法借口,如SHA1 ...
- VSCode六大通用插件真香合集
目录 一.background:设置心水背景图 安利理由: 安装及设置步骤: 设置过程中使用的代码: 成果展示: 注意: 二.Material Theme(VSCode主题)+Material Ico ...
- JS+Selenium+excel追加写入,使用python成功爬取京东任何商品~
之前一直是requests库做爬虫,这次尝试下使用selenium做爬虫,效率不高,但是却没有限制,文章是分别结合大牛的selenium爬虫以及excel追加写入操作而成,还有待优化,打算爬取更多信息 ...
- JasperReports入门教程(一):快速入门
JasperReports入门教程(一):快速入门 背景 现在公司的项目需要实现一个可以配置的报表,以便快速的适应客户的需求变化.后来在网上查资料发现可以使用JasperReports + Jaspe ...