Mongodb 故障分享 初始化时"errmsg" : "exception: new file allocation failure" 并且长时间处于STARTUP2
Hello,大家下午好。
近几天的项目有点赶,所以耽误了更新。现在给大家分享下,在安装mongodb的过程中,遇到的故障一则。其实很小白的问题,当时遇到这个问题的时候比较心慌,浪费了很多时间,跟大家分享下解决的思路吧。
先描述下我的环境,请参照前一篇博客,利用脚本初始化出4个节点(由于实验原因,我使用了单台的服务器)。
首先我们启动这四个初始化节点。(具体的文件配置,请参照前文,初始化的内容,这里仅用/etc/mongodb.conf为例)
[root@1]# cat /etc/mongodb.conf
dbpath=/usr/local/mongodb-3.0.2/data
logpath=/usr/local/mongodb-3.0.2/logs/mongo.log
pidfilepath=/var/run/mongodb/mongodb.pid
unixSocketPrefix=/usr/local/mongodb-3.0.2/socket
directoryperdb=true
replSet=picture
shardsvr=true
logappend=true
port = 27017
maxConns=20000
oplogSize=30720
fork=true
nohttpinterface=true
nojournal=true
[root@1]# mongod --config=/etc/mongodb.conf
about to fork child process, waiting until server is ready for connections.
forked process: 32560
child process started successfully, parent exiting
[root@1]# mongod --config=/etc/mongodb1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 32574
child process started successfully, parent exiting
[root@1]# mongod --config=/etc/mongodb2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 32588
child process started successfully, parent exiting
[root@1]# mongod --config=/etc/mongodb3.conf
about to fork child process, waiting until server is ready for connections.
forked process: 32602
child process started successfully, parent exiting
[root@1]# ps -ef | grep mongo //我们可以看到四个实例已经启动。下一步我们将进行角色分配(即副本集搭建)
root 32560 1 0 15:43 ? 00:00:00 mongod --config=/etc/mongodb.conf
root 32574 1 0 15:43 ? 00:00:00 mongod --config=/etc/mongodb1.conf
root 32588 1 0 15:43 ? 00:00:00 mongod --config=/etc/mongodb2.conf
root 32602 1 0 15:43 ? 00:00:00 mongod --config=/etc/mongodb3.conf
root 32616 28203 0 15:46 pts/0 00:00:00 grep mongo
[root@1]#
[root@1]# mongo
MongoDB shell version: 3.0.2
connecting to: test
Server has startup warnings:
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten]
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten]
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten]
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten]
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 65535 processes, 655350 files. Number of processes should be at least 327675 : 0.5 times number of files.
2016-01-25T15:43:46.645+0800 I CONTROL [initandlisten]
> use admin //首先切换到admindb,进行副本集配置
switched to db admin
> config = {_id: 'picture', members: [
... {_id: 0, host: '$IP:27017'},
... {_id: 1, host: '$IP:37017'},
... {_id: 2, host: '$IP:47017'},
... {_id: 3, host: '$IP:30000',arbiterOnly:true}] //此时配置将产生一个主节点,两个从节点,一个仲裁节点
... }
{
"_id" : "picture",
"members" : [
{
"_id" : 0,
"host" : "$IP:27017"
},
{
"_id" : 1,
"host" : "$IP:37017"
},
{
"_id" : 2,
"host" : "$IP:47017"
},
{
"_id" : 3,
"host" : "$IP:30000",
"arbiterOnly" : true
}
]
}
> rs.initiate(config);
{
"errmsg" : "exception: new file allocation failure", //大家可以看到初始化失败
"code" : 12520,
"ok" : 0
}
查询下各个节点状态,悲剧了,长时间停留在STARTUP2状态
picture:OTHER> rs.status()
{
"set" : "picture",
"date" : ISODate("2016-01-25T07:51:15.665Z"),
"myState" : 5,
"members" : [
{
"_id" : 0,
"name" : "$IP:27017",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 449,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"configVersion" : 1,
"self" : true
},
{
"_id" : 1,
"name" : "$IP:37017",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 154,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2016-01-25T07:51:15.326Z"),
"lastHeartbeatRecv" : ISODate("2016-01-25T07:51:15.326Z"),
"pingMs" : 0,
"configVersion" : 1
},
{
"_id" : 2,
"name" : "$IP:47017",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 154,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2016-01-25T07:51:15.326Z"),
"lastHeartbeatRecv" : ISODate("2016-01-25T07:51:15.324Z"),
"pingMs" : 0,
"configVersion" : 1
},
{
"_id" : 3,
"name" : "$IP:30000",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 154,
"lastHeartbeat" : ISODate("2016-01-25T07:51:15.324Z"),
"lastHeartbeatRecv" : ISODate("2016-01-25T07:51:15.324Z"),
"pingMs" : 0,
"configVersion" : 1
}
],
"ok" : 1
}
picture:OTHER> //大家发现本节点状态为other,………… ,很久很久以后 依然如此。
我们的解决思路是什么?
首先,想跟大家分享下mongodb的整个的启动和副本集初始化过程(可以关注下一章节的博客)。我们知道 STARTUP是其中的一个阶段。
然后,出现问题。其实第一个想法就是看log。那么我们看下log告诉我们什么了。
【$IP:27017-log】
2016-01-25T15:48:41.275+0800 I REPL [conn1] replSetInitiate admin command received from client
2016-01-25T15:48:41.277+0800 I REPL [conn1] replSet replSetInitiate config object with 4 members parses ok
2016-01-25T15:48:41.278+0800 I NETWORK [initandlisten] connection accepted from $IP:38296 #2 (2 connections now open)
2016-01-25T15:48:41.279+0800 I NETWORK [initandlisten] connection accepted from $IP:38299 #3 (3 connections now open)
2016-01-25T15:48:41.279+0800 I NETWORK [initandlisten] connection accepted from $IP:38300 #4 (4 connections now open)
2016-01-25T15:48:41.280+0800 I REPL [ReplicationExecutor] New replica set config in use: { _id: "picture", version: 1, members: [ { _id: 0, host: "$IP:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "$IP:37017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "$IP:47017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "$IP:30000", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
2016-01-25T15:48:41.280+0800 I REPL [ReplicationExecutor] This node is $IP:27017 in the config
2016-01-25T15:48:41.280+0800 I REPL [ReplicationExecutor] transition to STARTUP2
2016-01-25T15:48:41.280+0800 I REPL [conn1] ******
2016-01-25T15:48:41.280+0800 I REPL [conn1] creating replication oplog of size: 30720MB...
2016-01-25T15:48:41.281+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.1, filling with zeroes...
2016-01-25T15:48:41.281+0800 I REPL [ReplicationExecutor] Member $IP:47017 is now in state STARTUP
2016-01-25T15:48:41.281+0800 I REPL [ReplicationExecutor] Member $IP:37017 is now in state STARTUP
2016-01-25T15:48:41.281+0800 I REPL [ReplicationExecutor] Member $IP:30000 is now in state STARTUP
2016-01-25T15:48:41.283+0800 I STORAGE [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.1, size: 2047MB, took 0.002 secs
2016-01-25T15:48:41.283+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.2, filling with zeroes...
2016-01-25T15:48:41.286+0800 I STORAGE [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.2, size: 2047MB, took 0.002 secs
2016-01-25T15:48:41.286+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.3, filling with zeroes...
2016-01-25T15:48:41.288+0800 I STORAGE [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.3, size: 2047MB, took 0.002 secs
2016-01-25T15:48:41.288+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.4, filling with zeroes...
2016-01-25T15:48:41.291+0800 I STORAGE [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.4, size: 2047MB, took 0.002 secs
2016-01-25T15:48:41.291+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.5, filling with zeroes...
2016-01-25T15:48:41.293+0800 I STORAGE [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.5, size: 2047MB, took 0.002 secs
2016-01-25T15:48:41.293+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.6, filling with zeroes...
2016-01-25T15:48:41.298+0800 I STORAGE [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.6, size: 2047MB, took 0.004 secs
2016-01-25T15:48:41.298+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.7, filling with zeroes...
2016-01-25T15:48:41.301+0800 I STORAGE [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.7, size: 2047MB, took 0.002 secs
2016-01-25T15:48:41.301+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.8, filling with zeroes...
2016-01-25T15:48:41.303+0800 I STORAGE [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.8, size: 2047MB, took 0.002 secs
2016-01-25T15:48:41.304+0800 I STORAGE [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.9, filling with zeroes...
2016-01-25T15:48:41.318+0800 I STORAGE [FileAllocator] FileAllocator: posix_fallocate //发现日志至此不动了
【$IP:37017-log】
2016-01-25T15:51:59.127+0800 I STORAGE [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
//Congratulations!!!!!~~~
没有空间!!我们查看下系统的空间使用情况。
[root@1 logs]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 22G 21G 0 100% /
tmpfs tmpfs 8.4G 0 8.4G 0% /dev/shm
/dev/vdb ext4 22G 21G 0 100% /
/dev/vdb ext4 529G 208M 502G 1% /data
[root@1 logs]#
就是这么简单的问题。这个是一位好朋友在运行了我的脚本之后,反馈给我的。分享给大家。
其实在实践中,能够遇到各种预期外的状况,很开心。
希望大家能多多交流。
谢谢,反馈给我这个问题的张同学~~撒花~~
Mongodb 故障分享 初始化时"errmsg" : "exception: new file allocation failure" 并且长时间处于STARTUP2的更多相关文章
- mongodb new file allocation failure
话说那天正在向mongodb中写入数据,突然就蹦出了 new file allocation failure ,以为是数据有错误,就检查了一番,可没问题啊,看着像是mongo自己的问题,于是百度了一番 ...
- APP长时间处于后台,再次调用时提示用户重新登录
第一步:当应用被处于后台时,调用计时器的start()方法,开始计时 在所有Activity继承的BaseSwiBackAct中的 public void onStop() { EventBus.ge ...
- 微信分享网页时自定义缩略图和简介(.net版本)
要实现微信分享网页时自定义缩略图和简介,需开发者在公众平台网站中创建公众号.获取接口权限后,通过微信JS-SDK的分享接口,来实现微信分享功能. 下面来说明实现步骤. 第一部分 准备步骤 步骤一:注册 ...
- JQuery/JS插件 jsTree加载树,预先加载,初始化时加载前三级节点,当展开第三级节点时 就加载该节点下的所有子节点
jsTree加载树, 初始化时 加载前三级节点, 当展开第三级节点时 就加载该节点下的所有子节点 html: <!DOCTYPE html> <html> <head&g ...
- shareSDK实现分享操作时只显示英文字体
今天用shareSDK实现分享操作时,突然发现分享视图只显示英文字体,而之前是显示中文的,以下是自己找到的解决方法: ,首先,找到shareSDk里的“ShareSDKUI_Localizable.s ...
- EasyUI Accordion下的Panel面板初始化时全部折叠
EasyUI Accordion下的Panel面板有一个属性:selected,默认值为:false.初始化时,若设置'selected:true',则面板默认打开,效果如下: <div tit ...
- iOS UIimage初始化时的两种方法
第一种方式:UIImage *image = [UIImage imageNamed:@"image"]; 使用这种方式,第一次读取的时候,先把这个图片存到缓存里,下次再使用时直接 ...
- html---textarea初始化时就有个table空格以及tab键操作无效
1 初始化时就有一个tab空格 watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvRnJlZUFwZQ==/font/5a6L5L2T/fontsize/400 ...
- UIView(包括子类)的几个初始化时执行动作的时机
转载自:http://www.tqcto.com/article/mobile/56963.html 根据你需要执行的动作, 这里有几个方法: -(id)initWithFrame:(CGRect)f ...
随机推荐
- javascript高级程序设计第5章,引用类型
object类型: 创建object实列的方式有两种,一种是new()方法,一种是对象字面量表示法: 第一种法方: var obj = new object(); obj.name = 'name' ...
- 在SpringMVC框架下实现数据的国际化(即数据实现多国文字之间的转换)
在eclipse中javaEE环境下:导入必要的架包 web.xml配置文件: <?xml version="1.0" encoding="UTF-8"? ...
- 大数据 > 数据平台方案评估
分类 当前措施 说明 百度竞价如何进行数据分析(SEM工程师)数据来源: 1. 百度后台推广数据:api 总展现 总点击 点击率 总消费 点击均价 BDP功能点 1. 串联百度->网站商务通-& ...
- Android first --- 网络编程
网络编程 ###图片下载查看 1.发送http请求 URL url = new URL(address); //获取连接对象,并没有建立连接 HttpURLConnection conn = (Htt ...
- VM出现该虚拟机正在使用中的提示,让获取所有权限解决办法
今天打开虚拟机正要学习,结果说是虚拟机似乎正在使用中,让我重新获取权限.解决办法:打开提示的配置文件的位置,将一个以.lck结尾的文件夹删除或者保存为另外的文件名称,再打开虚拟机就OK了.
- css-position
值 描述 absolute 生成绝对定位的元素,相对于 static 定位以外的第一个父元素进行定位. 元素的位置通过 "left", "top", " ...
- 基于东北F4的设计模式情景剧——第一幕 装饰模式(Decorator Pattern)
第一场 难题未解 布景:铁岭,晴天,午后,风.在一幢还算气派的写字楼的三层外墙上,挂着一条红色横幅,上面用歪歪扭扭的毛笔字写着"东北F4软件外包工作室".大风中,那早已褪色的条幅剧 ...
- json接口
http://api.rottentomatoes.com/api/public/v1.0/lists/movies/in_theaters.json?apikey=7waqfqbprs7pajbz2 ...
- CocoaPods使用 主要带图。转载。
原文地址 快速更新,不需要查找更新库 命令:pod update --verbose --no-repo-update 虽然网上关于CocoaPods安装教程多不胜数,但是我在安装的过程中还是出现了 ...
- SpringMVC生成任意文件,访问链接即下载
原理上讲就是返回的 ResponseEntity<byte[]> 形式的值就可以了 @RequestMapping("/api/watermark_download") ...