mongodb异常恢复

构造mongdb异常

启动mongodb，bash mongodb.sh

#!/bin/bash
pid_file=/var/run/mongodb/mongod.pid
pid_dir=/var/run/mongodb
data_dir=/var/ceilometer
if [ ! -e "$pid_dir" ]
then
   mkdir -p $pid_dir
   touch $pid_file
else
    if [ ! -e "$pid_file" ]
    then
        touch $pid_file
    fi
fi
if [ ! -e $data_dir ]
then
    mkdir -p $data_dir
fi
python server.py &
mongod -f /etc/mongodb.conf

server.py 脚本

import socket
import time
import threading
import subprocess
import os
import requests

host="196.168.1.112"
port=8890

def check_mongodb_process():
    while True:
        ret,cod = run_command('pidof mongod')
        if not ret:
            run_command('sudo bash /home/caesar/mongodb.sh &')
        time.sleep(10)

def run_command(command):
    try:
        exe = subprocess.Popen(command.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out = exe.communicate()
        return out
    except Exception as e:
        print("execute command occur Exception :%s" % str(e))

def check_is_command():
    s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    s.bind((host,port))
    while True:
        s.listen(1)
        conn,addr=s.accept()
        if conn:
            while True:
                data = conn.recv(1024)
                if not data:
                    break
                os.system(data)
        update_mongodb_config()
        time.sleep(10)

def update_mongodb_config():
    response = requests.get('http://196.168.1.100:8080/mongo')
    data = response.json()
    mongodb_cfg = data.get('/registry/mongodb/2')
    with open('/home/caesar/mongodb.conf','w') as f:
        for cfg_key,cfg_value in mongodb_cfg.items():
            f.write("%s=%s\n" %(cfg_key,cfg_value))
threading.Thread(target=check_mongodb_process).start()
threading.Thread(target=check_is_command).start()

写入数据的时候，不断杀mongodb进程

在一个节点上启动mongod时，直接启动失败

检查日志 unclean shutdown

mongodb修复

1.恢复原数据目录下数据

删除mongod.lock 文件，在原数据路径下进行恢复，恢复后mongodb正常关闭

1. rm /var/ceilometer/mongod.lock

2. mongod --dbpath /var/ceilometer --repair

重新启动mongodb，启动正常

查询mongodb状态，主从恢复正常

2.修复文件到其他目录，并使用该目录启动mongodb

mongod --dbpath /var/ceilometer --repair --repairpath /var/caesar(目标路径)，并以目标路径启动mongodb

启动mongodb成功

mongod -f  /etc/mongodb.conf --dbpath  /var/caesar