mongodb 3.4 学习 (四）分片
https://www.linode.com/docs/databases/mongodb/build-database-clusters-with-mongodb
由三部分组成
shard: 每个shard包括一个所有shard数据的子集. 每个shard能够部署成replica set
mongos: 做为查询路由, 提供clientt与sharded cluster的接口
config servers: 存储群集的metadata和配置，必须部署为replica set
# 生产环境部署
kvm-70-102 10.0.70.102 # mongos
kvm-70-103 10.0.70.103 # config
kvm-70-102 10.0.70.104 # shard1
kvm-70-103 10.0.70.105 # shard2
config
# 建立数据目录
mkdir -p /opt/mongo_{27021,27022,27023}
# 配置文件
cat > /etc/mongod_27021.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27021.log
storage:
dbPath: /opt/mongo_27021
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27021.pid
net:
port: 27021
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: config
sharding:
clusterRole: configsvr
EOF
# 配置文件
cat > /etc/mongod_27022.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27022.log
storage:
dbPath: /opt/mongo_27022
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27022.pid
net:
port: 27022
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: config
sharding:
clusterRole: configsvr
EOF
# 配置文件
cat > /etc/mongod_27023.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27023.log
storage:
dbPath: /opt/mongo_27023
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27023.pid
net:
port: 27023
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: config
sharding:
clusterRole: configsvr
EOF
# 启动复制集
mongod --config /etc/mongod_27021.conf
mongod --config /etc/mongod_27022.conf
mongod --config /etc/mongod_27023.conf
# 配置config
mongo --port 27021
rs.initiate(
{
_id: 'config',
configsvr: true,
members: [
{ _id : 0, host : '10.0.70.103:27021' },
{ _id : 1, host : '10.0.70.103:27022' },
{ _id : 2, host : '10.0.70.103:27023' }
]
}
)
{ "ok" : 1 }
shard1
# 建立数据目录
mkdir -p /opt/mongo_{27021,27022,27023}
# 配置文件
cat > /etc/mongod_27021.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27021.log
storage:
dbPath: /opt/mongo_27021
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27021.pid
net:
port: 27021
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: shard1
sharding:
clusterRole: shardsvr
EOF
# 配置文件
cat > /etc/mongod_27022.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27022.log
storage:
dbPath: /opt/mongo_27022
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27022.pid
net:
port: 27022
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: shard1
sharding:
clusterRole: shardsvr
EOF
# 配置文件
cat > /etc/mongod_27023.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27023.log
storage:
dbPath: /opt/mongo_27023
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27023.pid
net:
port: 27023
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: shard1
sharding:
clusterRole: shardsvr
EOF
# 启动复制集
mongod --config /etc/mongod_27021.conf
mongod --config /etc/mongod_27022.conf
mongod --config /etc/mongod_27023.conf
# 配置config
mongo --port 27021
rs.initiate(
{
_id: 'shard1',
members: [
{ _id : 0, host : '10.0.70.104:27021' },
{ _id : 1, host : '10.0.70.104:27022' },
{ _id : 2, host : '10.0.70.104:27023' }
]
}
)
{ "ok" : 1 }
shard2
# 建立数据目录
mkdir -p /opt/mongo_{27021,27022,27023}
# 配置文件
cat > /etc/mongod_27021.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27021.log
storage:
dbPath: /opt/mongo_27021
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27021.pid
net:
port: 27021
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: shard2
sharding:
clusterRole: shardsvr
EOF
# 配置文件
cat > /etc/mongod_27022.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27022.log
storage:
dbPath: /opt/mongo_27022
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27022.pid
net:
port: 27022
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: shard2
sharding:
clusterRole: shardsvr
EOF
# 配置文件
cat > /etc/mongod_27023.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27023.log
storage:
dbPath: /opt/mongo_27023
journal:
enabled: true
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27023.pid
net:
port: 27023
security:
keyFile: /etc/mongodb_product_key
replication:
oplogSizeMB: 2048
replSetName: shard2
sharding:
clusterRole: shardsvr
EOF
# 启动复制集
mongod --config /etc/mongod_27021.conf
mongod --config /etc/mongod_27022.conf
mongod --config /etc/mongod_27023.conf
# 配置config
mongo --port 27021
rs.initiate(
{
_id: 'shard2',
members: [
{ _id : 0, host : '10.0.70.105:27021' },
{ _id : 1, host : '10.0.70.105:27022' },
{ _id : 2, host : '10.0.70.105:27023' }
]
}
)
{ "ok" : 1 }
mongos
# 配置文件
cat > /etc/mongod_27021.conf << EOF
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod_27021.log
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod_27021.pid
net:
port: 27021
security:
keyFile: /etc/mongodb_product_key
sharding:
configDB: config/10.0.70.103:27021,10.0.70.103:27022,10.0.70.103:27023
EOF
# 启动mongos
mongos --config /etc/mongod_27021.conf
# 配置
mongo --port 27021
# 添加分片
sh.addShard('shard1/10.0.70.104:27021')
sh.addShard('shard1/10.0.70.104:27022')
sh.addShard('shard1/10.0.70.104:27023')
sh.addShard('shard2/10.0.70.105:27021')
sh.addShard('shard2/10.0.70.105:27022')
sh.addShard('shard2/10.0.70.105:27023')
# 建立超级用户
use admin
db.createUser(
{
user: 'admin',
pwd: '@admin',
roles: [ { role: 'root', db: 'admin' } ]
}
);
# 认证登录
db.auth('admin', '@admin')
# app数据库demo表插入100000条数据
use app
for (var i = 1; i < 100000; i++) db.demo.insert({id: i, name: 'ken'})
WriteResult({ "nInserted" : 1 })
# 查看分片状态
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("59261bca37ceff575b36ef09")
}
shards:
{ "_id" : "shard1", "host" : "shard1/10.0.70.104:27021,10.0.70.104:27022,10.0.70.104:27023", "state" : 1 }
{ "_id" : "shard2", "host" : "shard2/10.0.70.105:27021,10.0.70.105:27022,10.0.70.105:27023", "state" : 1 }
active mongoses:
"3.4.4" : 1
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Balancer lock taken at Thu May 25 2017 07:48:42 GMT+0800 (CST) by ConfigServer:Balancer
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "app", "primary" : "shard1", "partitioned" : false }
# 结果显示没有分区，记录插在shard1服务器
# 配置分片
# 允许分片的数据库
sh.enableSharding('app')
# 为用做shard key的字段建立索引，实际中可以用uuid字段
use app
db.demo.createIndex({id: 1})
{
"raw" : {
"shard1/10.0.70.104:27021,10.0.70.104:27022,10.0.70.104:27023" : {
"createdCollectionAutomatically" : false,
"numIndexesBefore" : 1,
"numIndexesAfter" : 2,
"ok" : 1,
"$gleStats" : {
"lastOpTime" : {
"ts" : Timestamp(1495675335, 1),
"t" : NumberLong(1)
},
"electionId" : ObjectId("7fffffff0000000000000001")
}
}
},
"ok" : 1
}
# 允许分片的collection，指定shard key的字段
sh.shardCollection('app.demo', { id : 'hashed' })
{ "collectionsharded" : "app.demo", "ok" : 1 }
# 插入数据
for (var i = 1; i < 10000; i++) db.demo.insert({id: i, name: 'ken'})
WriteResult({ "nInserted" : 1 })
# 查看状态，可以看到db与collection的分片状态
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("59261bca37ceff575b36ef09")
}
shards:
{ "_id" : "shard1", "host" : "shard1/10.0.70.104:27021,10.0.70.104:27022,10.0.70.104:27023", "state" : 1 }
{ "_id" : "shard2", "host" : "shard2/10.0.70.105:27021,10.0.70.105:27022,10.0.70.105:27023", "state" : 1 }
active mongoses:
"3.4.4" : 1
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Balancer lock taken at Thu May 25 2017 07:48:42 GMT+0800 (CST) by ConfigServer:Balancer
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
1 : Success
databases:
{ "_id" : "app", "primary" : "shard1", "partitioned" : true }
app.demo
shard key: { "id" : "hashed" }
unique: false
balancing: true
chunks:
shard1 2
shard2 2
{ "id" : { "$minKey" : 1 } } -->> { "id" : NumberLong("-4611686018427387902") } on : shard1 Timestamp(2, 2)
{ "id" : NumberLong("-4611686018427387902") } -->> { "id" : NumberLong(0) } on : shard1 Timestamp(2, 3)
{ "id" : NumberLong(0) } -->> { "id" : NumberLong("4611686018427387902") } on : shard2 Timestamp(2, 4)
{ "id" : NumberLong("4611686018427387902") } -->> { "id" : { "$maxKey" : 1 } } on : shard2 Timestamp(2, 5)
# shard1
db.demo.find().count()
4992
# shard2
db.demo.find().count()
5007
如果在允许colletion前demo有数据，执行以下命令时会报错。
sh.shardCollection('app.demo', { id : 'hashed' })
{
"proposedKey" : {
"id" : "hashed"
},
"curIndexes" : [
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "app.demo"
},
{
"v" : 2,
"key" : {
"id" : 1
},
"name" : "id_1",
"ns" : "app.demo"
}
],
"ok" : 0,
"errmsg" : "please create an index that starts with the shard key before sharding."
}
做法是将collection删除，重建索引。
chunksize默认是64M，如果分片过慢的话，可以适当调大，如512M。mongos的配置文件里。
Hashed Sharding
Ranged Sharding
mongodb 3.4 学习 (四）分片的更多相关文章
- 搭建高可用mongodb集群（四）—— 分片
按照上一节中<搭建高可用mongodb集群(三)—— 深入副本集>搭建后还有两个问题没有解决: 从节点每个上面的数据都是对数据库全量拷贝,从节点压力会不会过大? 数据压力大到机器支撑不了的 ...
- 搭建高可用mongodb集群（四）—— 分片
按照上一节中<搭建高可用mongodb集群(三)-- 深入副本集>搭建后还有两个问题没有解决: 从节点每个上面的数据都是对数据库全量拷贝,从节点压力会不会过大? 数据压力大到机器支撑不了的 ...
- MongoDB的使用学习之（七）MongoDB的聚合查询（两种方式）附项目源码
先来张在路上…… 铛铛铛……项目源码下载地址:http://files.cnblogs.com/ontheroad_lee/MongoDBDemo.rar 此项目是用Maven创建的,没有使用Mave ...
- TweenMax动画库学习(四)
目录 TweenMax动画库学习(一) TweenMax动画库学习(二) TweenMax动画库学习(三) Tw ...
- SVG 学习<四> 基础API
目录 SVG 学习<一>基础图形及线段 SVG 学习<二>进阶 SVG世界,视野,视窗 stroke属性 svg分组 SVG 学习<三>渐变 SVG 学习<四 ...
- Android JNI学习(四)——JNI的常用方法的中文API
本系列文章如下: Android JNI(一)——NDK与JNI基础 Android JNI学习(二)——实战JNI之“hello world” Android JNI学习(三)——Java与Nati ...
- mongodb 搭建集群（分片+副本集）
mongodb 搭建集群(分片+副本集) 一.搭建结构图: 二.搭建步骤:
- SCARA——OpenGL入门学习四（颜色）
OpenGL入门学习[四] 本次学习的是颜色的选择.终于要走出黑白的世界了~~ OpenGL支持两种颜色模式:一种是RGBA,一种是颜色索引模式. 无论哪种颜色模式,计算机都必须为每一个像素保存一些数 ...
- mongodb常用命令学习笔记
mongodb常用命令学习笔记 创建数据库 use DATABASE_NAME eg: use users; 如果数据库不存在,则创建数据库,否则切换到指定数据库.要显示刚刚创建的数据库,需要向数据库 ...
随机推荐
- rest-assured的JsonPath使用方法总结
JsonPath对于解析Json格式的数据来说非常简单,比如有下面这样的Json数据: {"lotto":{ "lottoId":5, "winnin ...
- Modular Inverse （拓展欧几里得求逆元）
The modular modular multiplicative inverse of an integer a modulo m is an integer xsuch that a-1≡x ( ...
- tensorflow基础-placeholder
placeholder: 要给节点输入数据时用 placeholder,在 TensorFlow 中用placeholder 来描述等待输入的节点,只需要指定类型即可,然后在执行节点的时候用一个字典来 ...
- CNN 卷积神经网络结构
cnn每一层会输出多个feature map, 每个Feature Map通过一种卷积滤波器提取输入的一种特征,每个feature map由多个神经元组成,假如某个feature map的shape是 ...
- js数组去重 javascript版
//js数组去重 //思路: // 1.放入第一个元素 // 2.放入第n个元素,和第n个之前的元素就行比较,如果有重复,则跳过.没有重复就加入数组中 // 3.返回新的去重后数组 Array.pro ...
- Q205 同构字符串
给定两个字符串 s 和 t,判断它们是否是同构的. 如果 s 中的字符可以被替换得到 t ,那么这两个字符串是同构的. 所有出现的字符都必须用另一个字符替换,同时保留字符的顺序.两个字符不能映射到同一 ...
- [V1-Team] WEDO创意论坛功能规格说明书
项目功能规格说明书 版本说明 版本 内容 时间 V1.0 描述总体目标,用户使用场景,界面原型.功能设计及验收 2019.3.28 附Github仓库:WEDO 正文 1.目标 规范指导整个项目设计与 ...
- linux拓展之 用linux命令 管理windows一秒完成不可思议的操作--本节实战find 移动！！
花里胡哨的东西太多,有时候觉得简单也好! 你学习了Linux,是不是觉得Linux很强大!命令的多样性结合性有没有把你征服? 在那个烈日炎炎的夏日,我下载了辣末多老男孩的视屏----但是突然我只想看t ...
- Nginx 配置跨域权限
今天设置静态资源服务器时发现 Font from origin 'http://start.fbzl.org' has been blocked from loading by Cross-Origi ...
- Oracle 数据库实例和数据库
本文参考自oracle数据库实例,数据库的理解,纯属读书笔记,用于加深记忆. 先看Tom关于这二者的解释: 1.数据库 物理操作系统文件或磁盘的集合(我觉得可以理解为数据文件等).使用Oracle 1 ...