Hello，大家下午好。

近几天的项目有点赶，所以耽误了更新。现在给大家分享下，在安装mongodb的过程中，遇到的故障一则。其实很小白的问题，当时遇到这个问题的时候比较心慌，浪费了很多时间，跟大家分享下解决的思路吧。

先描述下我的环境，请参照前一篇博客，利用脚本初始化出4个节点（由于实验原因，我使用了单台的服务器）。

首先我们启动这四个初始化节点。（具体的文件配置，请参照前文，初始化的内容，这里仅用/etc/mongodb.conf为例）

[root@1]# cat /etc/mongodb.conf
dbpath=/usr/local/mongodb-3.0.2/data
logpath=/usr/local/mongodb-3.0.2/logs/mongo.log
pidfilepath=/var/run/mongodb/mongodb.pid
unixSocketPrefix=/usr/local/mongodb-3.0.2/socket
directoryperdb=true
replSet=picture
shardsvr=true
logappend=true
port = 27017
maxConns=20000
oplogSize=30720
fork=true
nohttpinterface=true
nojournal=true

[root@1]# mongod --config=/etc/mongodb.conf
about to fork child process, waiting until server is ready for connections.
forked process: 32560
child process started successfully, parent exiting
[root@1]# mongod --config=/etc/mongodb1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 32574
child process started successfully, parent exiting
[root@1]# mongod --config=/etc/mongodb2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 32588
child process started successfully, parent exiting
[root@1]# mongod --config=/etc/mongodb3.conf
about to fork child process, waiting until server is ready for connections.
forked process: 32602
child process started successfully, parent exiting

[root@1]# ps -ef | grep mongo                                  //我们可以看到四个实例已经启动。下一步我们将进行角色分配（即副本集搭建）
root     32560     1  0 15:43 ?        00:00:00 mongod --config=/etc/mongodb.conf
root     32574     1  0 15:43 ?        00:00:00 mongod --config=/etc/mongodb1.conf
root     32588     1  0 15:43 ?        00:00:00 mongod --config=/etc/mongodb2.conf
root     32602     1  0 15:43 ?        00:00:00 mongod --config=/etc/mongodb3.conf
root     32616 28203  0 15:46 pts/0    00:00:00 grep mongo
[root@1]#

[root@1]# mongo
MongoDB shell version: 3.0.2
connecting to: test
Server has startup warnings:
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten]
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten]
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten]
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten]
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 65535 processes, 655350 files. Number of processes should be at least 327675 : 0.5 times number of files.
2016-01-25T15:43:46.645+0800 I CONTROL  [initandlisten]
> use admin                                            //首先切换到admindb，进行副本集配置
switched to db admin
> config = {_id: 'picture', members: [
...       {_id: 0, host: '$IP:27017'},
...       {_id: 1, host: '$IP:37017'},
...       {_id: 2, host: '$IP:47017'},
...       {_id: 3, host: '$IP:30000',arbiterOnly:true}]                //此时配置将产生一个主节点，两个从节点，一个仲裁节点
... }
{
    "_id" : "picture",
    "members" : [
        {
            "_id" : 0,
            "host" : "$IP:27017"
        },
        {
            "_id" : 1,
            "host" : "$IP:37017"
        },
        {
            "_id" : 2,
            "host" : "$IP:47017"
        },
        {
            "_id" : 3,
            "host" : "$IP:30000",
            "arbiterOnly" : true
        }
    ]
}
> rs.initiate(config);
{
    "errmsg" : "exception: new file allocation failure",                   //大家可以看到初始化失败
    "code" : 12520,
    "ok" : 0
}

查询下各个节点状态，悲剧了，长时间停留在STARTUP2状态

picture:OTHER> rs.status()
{
    "set" : "picture",
    "date" : ISODate("2016-01-25T07:51:15.665Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "$IP:27017",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 449,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "configVersion" : 1,
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "$IP:37017",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 154,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2016-01-25T07:51:15.326Z"),
            "lastHeartbeatRecv" : ISODate("2016-01-25T07:51:15.326Z"),
            "pingMs" : 0,
            "configVersion" : 1
        },
        {
            "_id" : 2,
            "name" : "$IP:47017",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 154,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2016-01-25T07:51:15.326Z"),
            "lastHeartbeatRecv" : ISODate("2016-01-25T07:51:15.324Z"),
            "pingMs" : 0,
            "configVersion" : 1
        },
        {
            "_id" : 3,
            "name" : "$IP:30000",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 154,
            "lastHeartbeat" : ISODate("2016-01-25T07:51:15.324Z"),
            "lastHeartbeatRecv" : ISODate("2016-01-25T07:51:15.324Z"),
            "pingMs" : 0,
            "configVersion" : 1
        }
    ],
    "ok" : 1
}
picture:OTHER> //大家发现本节点状态为other，………… ，很久很久以后  依然如此。

我们的解决思路是什么？

首先，想跟大家分享下mongodb的整个的启动和副本集初始化过程（可以关注下一章节的博客）。我们知道  STARTUP是其中的一个阶段。

然后，出现问题。其实第一个想法就是看log。那么我们看下log告诉我们什么了。

【$IP：27017-log】

2016-01-25T15:48:41.275+0800 I REPL     [conn1] replSetInitiate admin command received from client
2016-01-25T15:48:41.277+0800 I REPL     [conn1] replSet replSetInitiate config object with 4 members parses ok
2016-01-25T15:48:41.278+0800 I NETWORK  [initandlisten] connection accepted from $IP:38296 #2 (2 connections now open)
2016-01-25T15:48:41.279+0800 I NETWORK  [initandlisten] connection accepted from $IP:38299 #3 (3 connections now open)
2016-01-25T15:48:41.279+0800 I NETWORK  [initandlisten] connection accepted from $IP:38300 #4 (4 connections now open)
2016-01-25T15:48:41.280+0800 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "picture", version: 1, members: [ { _id: 0, host: "$IP:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "$IP:37017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "$IP:47017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "$IP:30000", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
2016-01-25T15:48:41.280+0800 I REPL     [ReplicationExecutor] This node is $IP:27017 in the config
2016-01-25T15:48:41.280+0800 I REPL     [ReplicationExecutor] transition to STARTUP2
2016-01-25T15:48:41.280+0800 I REPL     [conn1] ******
2016-01-25T15:48:41.280+0800 I REPL     [conn1] creating replication oplog of size: 30720MB...
2016-01-25T15:48:41.281+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.1, filling with zeroes...
2016-01-25T15:48:41.281+0800 I REPL     [ReplicationExecutor] Member $IP:47017 is now in state STARTUP
2016-01-25T15:48:41.281+0800 I REPL     [ReplicationExecutor] Member $IP:37017 is now in state STARTUP
2016-01-25T15:48:41.281+0800 I REPL     [ReplicationExecutor] Member $IP:30000 is now in state STARTUP
2016-01-25T15:48:41.283+0800 I STORAGE  [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.1, size: 2047MB,  took 0.002 secs
2016-01-25T15:48:41.283+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.2, filling with zeroes...
2016-01-25T15:48:41.286+0800 I STORAGE  [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.2, size: 2047MB,  took 0.002 secs
2016-01-25T15:48:41.286+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.3, filling with zeroes...
2016-01-25T15:48:41.288+0800 I STORAGE  [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.3, size: 2047MB,  took 0.002 secs
2016-01-25T15:48:41.288+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.4, filling with zeroes...
2016-01-25T15:48:41.291+0800 I STORAGE  [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.4, size: 2047MB,  took 0.002 secs
2016-01-25T15:48:41.291+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.5, filling with zeroes...
2016-01-25T15:48:41.293+0800 I STORAGE  [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.5, size: 2047MB,  took 0.002 secs
2016-01-25T15:48:41.293+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.6, filling with zeroes...
2016-01-25T15:48:41.298+0800 I STORAGE  [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.6, size: 2047MB,  took 0.004 secs
2016-01-25T15:48:41.298+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.7, filling with zeroes...
2016-01-25T15:48:41.301+0800 I STORAGE  [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.7, size: 2047MB,  took 0.002 secs
2016-01-25T15:48:41.301+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.8, filling with zeroes...
2016-01-25T15:48:41.303+0800 I STORAGE  [FileAllocator] done allocating datafile /usr/local/mongodb-3.0.2/data/local/local.8, size: 2047MB,  took 0.002 secs
2016-01-25T15:48:41.304+0800 I STORAGE  [FileAllocator] allocating new datafile /usr/local/mongodb-3.0.2/data/local/local.9, filling with zeroes...
2016-01-25T15:48:41.318+0800 I STORAGE  [FileAllocator] FileAllocator: posix_fallocate  //发现日志至此不动了

【$IP：37017-log】

2016-01-25T15:51:59.127+0800 I STORAGE  [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back

//Congratulations!!!!!~~~

没有空间！！我们查看下系统的空间使用情况。

[root@1 logs]# df -TH
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/vda1      ext4    22G   21G     0 100% /
tmpfs          tmpfs  8.4G     0  8.4G   0% /dev/shm
/dev/vdb       ext4    22G   21G     0 100% /
/dev/vdb       ext4   529G  208M  502G   1% /data
[root@1 logs]#

就是这么简单的问题。这个是一位好朋友在运行了我的脚本之后，反馈给我的。分享给大家。

其实在实践中，能够遇到各种预期外的状况，很开心。

希望大家能多多交流。

谢谢，反馈给我这个问题的张同学~~撒花~~

Mongodb 故障分享 初始化时"errmsg" : "exception: new file allocation failure" 并且长时间处于STARTUP2的更多相关文章

  1. mongodb new file allocation failure

    话说那天正在向mongodb中写入数据,突然就蹦出了 new file allocation failure ,以为是数据有错误,就检查了一番,可没问题啊,看着像是mongo自己的问题,于是百度了一番 ...

  2. APP长时间处于后台，再次调用时提示用户重新登录

    第一步:当应用被处于后台时,调用计时器的start()方法,开始计时 在所有Activity继承的BaseSwiBackAct中的 public void onStop() { EventBus.ge ...

  3. 微信分享网页时自定义缩略图和简介（.net版本）

    要实现微信分享网页时自定义缩略图和简介,需开发者在公众平台网站中创建公众号.获取接口权限后,通过微信JS-SDK的分享接口,来实现微信分享功能. 下面来说明实现步骤. 第一部分 准备步骤 步骤一:注册 ...

  4. JQuery/JS插件 jsTree加载树，预先加载，初始化时加载前三级节点，当展开第三级节点时 就加载该节点下的所有子节点

    jsTree加载树, 初始化时 加载前三级节点, 当展开第三级节点时 就加载该节点下的所有子节点 html: <!DOCTYPE html> <html> <head&g ...

  5. shareSDK实现分享操作时只显示英文字体

    今天用shareSDK实现分享操作时,突然发现分享视图只显示英文字体,而之前是显示中文的,以下是自己找到的解决方法: ,首先,找到shareSDk里的“ShareSDKUI_Localizable.s ...

  6. EasyUI Accordion下的Panel面板初始化时全部折叠

    EasyUI Accordion下的Panel面板有一个属性:selected,默认值为:false.初始化时,若设置'selected:true',则面板默认打开,效果如下: <div tit ...

  7. iOS UIimage初始化时的两种方法

    第一种方式:UIImage *image = [UIImage imageNamed:@"image"]; 使用这种方式,第一次读取的时候,先把这个图片存到缓存里,下次再使用时直接 ...

  8. html---textarea初始化时就有个table空格以及tab键操作无效

    1 初始化时就有一个tab空格 watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvRnJlZUFwZQ==/font/5a6L5L2T/fontsize/400 ...

  9. UIView（包括子类）的几个初始化时执行动作的时机

    转载自:http://www.tqcto.com/article/mobile/56963.html 根据你需要执行的动作, 这里有几个方法: -(id)initWithFrame:(CGRect)f ...

随机推荐

  1. 【转】Description Resource Path Location Type Java compiler level&n

    转载地址:http://blog.sina.com.cn/s/blog_ae96abfd0101qbq0.html 在项目上右键Properties->Project Facets,在打开的Pr ...

  2. 对c++ public、protected、private关键字的理解

    首先要明确一下: 1.这三个关键字在两种地方会用到,一个是对类的成员变量和成员函数修饰时(比如私有的成员变量,受保护的成员变量·,公有的函数),还有一种是对继承方式的修饰(比如公有继承,保护继承). ...

  3. 一个ubuntu phper的自我修养（workbench）

    workbench从此和navicat的激活码说再见 workbench是一个免费易用功能强大的mysql图形化管理软件,navicat上用到的功能,workbench上都能找到. 一.workben ...

  4. CSS2中的定位属性

    这也是一个重点!

  5. CAD2008 启动慢 卡住

    CAD2008打开很慢解决CAD2008打开很慢的方法: 第一个:1.[开始]―[控制面板]. 2.选择[网络和Internet连接]选项,再点[Internet选项]. 3.在Internet属性对 ...

  6. android 命名 数组 所有国家 String[] COUNTRIES

    static final String[] COUNTRIES = new String[] { "Afghanistan", "Albania", " ...

  7. JS 拼接字符串数组

    1.格式1 1.1例子 [ {name: '北京',value: Math.round(Math.random()*1000)}, {name: '天津',value: Math.round(Math ...

  8. 7. Swift 基于Xmpp和openfire实现一个简单的登录注册

    1. 基本步骤:首先导入Xmpp框架,配置环境 ->由于我们使用的是OC的Xmpp框架,再进行Swift开发时需要进行桥接. 具体方法就是创建一个基于c的.h的头文件,然后将我们需要编译OC的语 ...

  9. Building Websites in ASP.NET

    ASP.NET offers three frameworks for creating web applications: Web Forms, ASP.NET MVC, and ASP.NET W ...

  10. HMI与设计模式

    设计模式是做一个好的架构的一个基础.那么设计模式具体的概念是啥呢?百度百科曰:设计模式(Design pattern)是一套被反复使用.多数人知晓的.经过分类编目的.代码设计经验的总结.使用设计模式是 ...