Docker mongodb 3.4 分片 一主 一副 一仲 鉴权集群部署.

非docker部署

为了避免过分冗余,并且在主节点挂了,还能顺利自动提升,所以加入仲裁节点

为什么要用docker部署,因为之前直接在虚拟机启动10个mongod 进程.多线程并发测试的时候,mongo的进程经常会挂掉.

所以这次使用docker部署测试

之前用的是7200转的固态硬盘

这次用nvme 的固态硬盘,性能是飞跃提升,使用批量插入,每次插入100个文档 ,每个文档里面有内嵌又内嵌文档1=10个文档, 算上内嵌总数相当于关系型数据库5000W数据量,总耗时130秒,而且非常稳定,未见异常情况!

镜像版本: 3.4.10-jessie, 3.4-jessie, 3-jessie, jessie (3.4/Dockerfile)

环境:一台虚拟机

三个configsvr 副本: 端口为 27020,27021,27022

两个分片:

　　shard1:-> 三个副本,端口为 27017,27018,27019

　　shard2:-> 三个副本,端口为 27024,27025,27026

一个路由:mongos -> 端口为 27023

前置条件:

注意:configsvr 的 data 目录是 /data/configdb 而数据节点的目录是 /data/db

创建数据存储文件的目录

mkdir /usr/local/mongodb/data
cd /usr/local/mongodb/data
mkdir c0 && mkdir c1 && mkdir c2 && mkdir s100 && mkdir s101 &&  mkdir s102 && mkdir s200 && mkdir s201 && mkdir s202

生成鉴权需要的keyfile,keyfile 内容不能太长,否则启动不了,权限不能太大,否则也是启动不了

openssl rand -base64  > /usr/local/mongodb/keyfile
chmod  /usr/local/mongodb/keyfile
chown  /usr/local/mongodb/keyfile

副本模式启动 configsvr

第一次启动 时,不要添加 --keyFile /usr/local/mongodb/keyfile 先初始化用户

docker run --name mongo-cs0 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs0:/data/configdb -v /usr/local/mongodb/data/cs0/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet cs --configsvr --port  --bind_ip=localhost,192.168.1.9

docker run --name mongo-cs1 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs1:/data/configdb -v /usr/local/mongodb/data/cs1/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet cs --configsvr --port  --bind_ip=localhost,192.168.1.9

docker run --name mongo-cs2 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs2:/data/configdb -v /usr/local/mongodb/data/cs2/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet cs --configsvr --port  --bind_ip=localhost,192.168.1.9

登陆任意一个节点 初始化配置,以及创建用户 注意,如果初始化配置以后,use db 多几次,会自动切换到primary,需要在primary节点上才可以有创建动作,或者设置slaveok

use admin;

var cnf={_id:"cs",members:[{_id:,host:"192.168.1.9:27020"},{_id:,host:"192.168.1.9:27021"},{_id:,host:"192.168.1.9:27022"}]}

 rs.initiate(cnf)

db.createUser(
   {
     user: "dba",
     pwd: "dba",
     roles: [ { role: "root", db: "admin" } ]
   }
 )

use testdb
db.createUser(
  {
    user: "testuser",
    pwd: "testuser",
    roles: [ { role: "readWrite", db: "testdb" } ]
  }
)

启动分片1

第一次启动 时,不要添加 --keyFile /usr/local/mongodb/keyfile 先初始化用户

docker run --name mongo-s100 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s100:/data/db -v /usr/local/mongodb/data/s100/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard1 --shardsvr --port  --bind_ip=localhost,192.168.1.9

docker run --name mongo-s101 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s101:/data/db -v /usr/local/mongodb/data/s101/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard1 --shardsvr --port  --bind_ip=localhost,192.168.1.9

docker run --name mongo-s102 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s102:/data/db -v /usr/local/mongodb/data/s102/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard1 --shardsvr --port  --bind_ip=localhost,192.168.1.9

登陆路任意一个分片1 注意,如果初始化配置以后,use db 多几次,会自动切换到primary,需要在primary节点上才可以有创建动作,或者设置slaveok

use admin;

var cnf={_id:"shard1",members:[{_id:,host:"192.168.1.9:27017"},{_id:,host:"192.168.1.9:27018"}]}

rs.initiate(cnf)
rs.addArb("192.168.1.9:27019")#仲裁节点
db.createUser(
   {
     user: "dba",
     pwd: "dba",
     roles: [ { role: "root", db: "admin" } ]
   }
 )

use 目标数据库
db.createUser(
  {
    user: "testuser",
    pwd: "testuser",
    roles: [ { role: "readWrite", db: "testdb" } ]
  }
)

启动分片2

第一次启动 时,不要添加 --keyFile /usr/local/mongodb/keyfile 先初始化用户

docker run --name mongo-s200 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s200:/data/db -v /usr/local/mongodb/data/s200/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard2 --shardsvr --port  --bind_ip=localhost,192.168.1.9

docker run --name mongo-s201 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s201:/data/db -v /usr/local/mongodb/data/s201/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard2 --shardsvr --port  --bind_ip=localhost,192.168.1.9

docker run --name mongo-s202 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s202:/data/db -v /usr/local/mongodb/data/s202/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard2 --shardsvr --port  --bind_ip=localhost,192.168.1.9

登陆任意一个分片2 初始化配置,并创建用户 注意,如果初始化配置以后,use db 多几次,会自动切换到primary,需要在primary节点上才可以有创建动作,或者设置slaveok

use admin;

var cnf={_id:"shard2",members:[{_id:,host:"192.168.1.9:27024"},{_id:,host:"192.168.1.9:27025"}]}
rs.initiate(cnf)
rs.addArb("192.168.1.9:27026") #仲裁节点
db.createUser( { user: "dba", pwd: "dba", roles: [ { role: "root", db: "admin" } ] } ) use 目标数据库 db.createUser(   {     user: "testuser",     pwd: "testuser",     roles: [ { role: "readWrite", db: "testdb" } ]   } )

启动路由

docker run -d --name mongos --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/mongos/log:/var/log/mongodb -d d22888af0ce0 mongos --port  --configdb cs/192.168.1.9:,192.168.1.9:,192.168.1.9: --bind_ip=192.168.1.9

登陆路由,添加分片

mongos> sh.addShard("shard1/192.168.1.9:27017")#只需要添加其中一个副本或者仲裁host即可 可选 192.168.1.9:27018 192.168.1.9:27019
{ "shardAdded" : "shard1", "ok" :  }
mongos> sh.addShard("shard2/192.168.1.9:27024")#同上规则
{ "shardAdded" : "shard2", "ok" :  }

mongos> sh.enableSharding("testdb") 
mongos> sh.shardCollection("testdb.orderInfo",{"_id":"hashed"}) Use testdb; db.createUser( { user: "testuser", pwd: "testuser", roles: [ { role: "readWrite", db: "testdb" } ] } )

关闭所有节点,重启加上鉴权

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

重启configsvr

docker run --restart=always --name mongo-cs0 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs0:/data/configdb -v /usr/local/mongodb/data/cs0/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet cs --configsvr --port  --bind_ip=localhost,192.168.1.9

docker run  --restart=always --name mongo-cs1 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs1:/data/configdb -v /usr/local/mongodb/data/cs1/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet cs --configsvr --port  --bind_ip=localhost,192.168.1.9

docker run  --restart=always --name mongo-cs2 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs2:/data/configdb -v /usr/local/mongodb/data/cs2/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet cs --configsvr --port  --bind_ip=localhost,192.168.1.9

重启分片1

docker run  --restart=always --name mongo-s100 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s100:/data/db -v /usr/local/mongodb/data/s100/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard1 --shardsvr --port  --bind_ip=localhost,192.168.1.9

docker run  --restart=always --name mongo-s101 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s101:/data/db -v /usr/local/mongodb/data/s101/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard1 --shardsvr --port  --bind_ip=localhost,192.168.1.9

docker run  --restart=always --name mongo-s102 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s102:/data/db -v /usr/local/mongodb/data/s102/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard1 --shardsvr --port  --bind_ip=localhost,192.168.1.9

重启分片2

docker run  --restart=always --name mongo-s200 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s200:/data/db -v /usr/local/mongodb/data/s200/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard2 --shardsvr --port  --bind_ip=localhost,192.168.1.9

docker run  --restart=always --name mongo-s201 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s201:/data/db -v /usr/local/mongodb/data/s201/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard2 --shardsvr --port  --bind_ip=localhost,192.168.1.9

docker run  --restart=always --name mongo-s202 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s202:/data/db -v /usr/local/mongodb/data/s202/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize  --replSet shard2 --shardsvr --port  --bind_ip=localhost,192.168.1.9

启动路由

docker run  --restart=always -d --name mongos --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/mongos/log:/var/log/mongodb  -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 mongos --keyFile /usr/local/mongodb/keyfile --port  --configdb cs/192.168.1.9:,192.168.1.9:,192.168.1.9: --bind_ip=192.168.1.9

现在可以登陆路由进行测试了

以下测试,片键为公司编码以及用户名,复合键片,实现多热点

插入500W 单文档数据,不包含内嵌文档,速度非常快.只用了56秒

完成插入查看数据分布情况

分布不太均衡,等一会再看一次

非常均衡,因为我用了9个公司编码,14个随机用户名插入.但是只有两个分片,所以导致数据会分发倒第一次插入的分片中,导致数据需要频繁自动均衡.建议有条件的,初始化的时候,创建多个分片