Reroute Unassigned Shards——遇到主shard 出现的解决方法就是重新路由
Red Cluster!
摘自:http://blog.kiyanpro.com/2016/03/06/elasticsearch/reroute-unassigned-shards/
There are 3 cluster states:
- green: All primary and replica shards are active
- yellow: All primary shards are active, but not all replica shards are active
- red: Not all primary shards are active
When cluster health is red, it means cluster is dead. And that means you can do nothing until it’s recovered, which is very bad indeed. I will share with you how to deal with one common situation: when cluster is red due to unassigned shards.
Steps
The general idea is pretty simple: find those shards which are unassigned, manually assign them to a node with reroute API. Let’s see how we can do that step by step. Then we can combine them into a configurable simple script.
Step 1: Check Unassigned Shards
To get cluster information, we usually use cat APIs. There is a GET /_cat/shards
endpoint to show a detailed view of what nodes contain which shards[1].
Cat shards
1
2
3
4
5
6
7
8
9
|
# cat shards verbose
curl "http://your.elasticsearch.host.com:9200/_cat/shards?v"
# cat shards index
curl "http://your.elasticsearch.host.com:9200/_cat/shards/wiki2"
# example return
# wiki2 0 p STARTED 197 3.2mb 192.168.56.10 Stiletto
# wiki2 1 p STARTED 205 5.9mb 192.168.56.30 Frankie Raye
# wiki2 2 p STARTED 275 7.8mb 192.168.56.20 Commander Kraken
|
By piping cat shards to fgrep, we can get all unassigned shards.
Get unassigned shards
1
2
3
4
5
6
|
# cat shards with fgrep
curl "http://your.elasticsearch.host.com:9200/_cat/shards" | fgrep UNASSIGNED
# example return
# wiki1 0 r UNASSIGNED ALLOCATION_FAILED
# wiki1 1 r UNASSIGNED ALLOCATION_FAILED
# wiki1 2 r UNASSIGNED ALLOCATION_FAILED
|
If you don’t want to deal with shell script, you can also find these unassigned shards using another endpoint POST /_flush/synced
[2]. This endpoint is actually not just some information. It allows an administrator to initiate a synced flush manually. This can be particularly useful for a planned (rolling) cluster restart where you can stop indexing and don’t want to wait the default 5 minutes for idle indices to be sync-flushed automatically. It returns with a json response.
_flush/synced
1
|
curl -XPOST "http://your.elasticsearch.host.com:9200/twitter/_flush/synced"
|
If there are failed shards in the response, we can iterate through a failures array to get all unassigned ones.
Example response with failed shards
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
{
"_shards": {
"total": 4,
"successful": 1,
"failed": 1
},
"twitter": {
"total": 4,
"successful": 3,
"failed": 1,
"failures": [
{
"shard": 1,
"reason": "unexpected error",
"routing": {
"state": "STARTED",
"primary": false,
"node": "SZNr2J_ORxKTLUCydGX4zA",
"relocating_node": null,
"shard": 1,
"index": "twitter"
}
}
]
}
}
|
Step 2: Reroute
The reroute command allows to explicitly execute a cluster reroute allocation command including specific commands[3] . An unassigned shard can be explicitly allocated on a specific node.
Reroute example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
"commands" : [ {
"move" :
{
"index" : "test", "shard" : 0,
"from_node" : "node1", "to_node" : "node2"
}
},
{
"allocate" : {
"index" : "test", "shard" : 1, "node" : "node3"
}
}
]
}'
|
There are 3 kinds of commands you can use:
move: Move a started shard from one node to another node. Accepts index and shard for index name and shard number, from_node for the node to move the shard from, and to_node for the node to move the shard to.
cancel: Cancel allocation of a shard (or recovery). Accepts index and shard for index name and shard number, and node for the node to cancel the shard allocation on. It also accepts allow_primary flag to explicitly specify that it is allowed to cancel allocation for a primary shard. This can be used to force resynchronization of existing replicas from the primary shard by cancelling them and allowing them to be reinitialized through the standard reallocation process.
allocate: Allocate an unassigned shard to a node. Accepts the index and shard for index name and shard number, and node to allocate the shard to. It also accepts allow_primary flag to explicitly specify that it is allowed to explicitly allocate a primary shard (might result in data loss).
Combining step 2 with the unassigned shards from Step 1, we can reroute all unassigned shards 1 by 1, thus getting faster cluster recovery from red state.
Example Solutions
Python
Below is a python script I wrote using POST /_flush/synced
and POST /reroute
Shell Script
Below is a shell script I found elsewhere in a blog post[4]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
for shard in $(curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
"commands" : [ {
"allocate" : {
"index" : "t37", # index name
"shard" : $shard,
"node" : "datanode15", # node name
"allow_primary" : true
}
}
]
}'
sleep 5
done
|
EDIT: Based on Vincent’s comment I updated the shell script:
Possible Unassigned Shard Reasons
FYI, these are the possible reasons for a shard be in a unassigned state[1]:
Name | Comment |
---|---|
INDEX_CREATED | Unassigned as a result of an API creation of an index |
CLUSTER_RECOVERED | Unassigned as a result of a full cluster recovery |
INDEX_REOPENED | Unassigned as a result of opening a closed index |
DANGLING_INDEX_IMPORTED | Unassigned as a result of importing a dangling index |
NEW_INDEX_RESTORED | Unassigned as a result of restoring into a new index |
EXISTING_INDEX_RESTORED | Unassigned as a result of restoring into a closed index |
REPLICA_ADDED | Unassigned as a result of explicit addition of a replica |
ALLOCATION_FAILED | Unassigned as a result of a failed allocation of the shard |
NODE_LEFT | Unassigned as a result of the node hosting it leaving the cluster |
REROUTE_CANCELLED | Unassigned as a result of explicit cancel reroute command |
REINITIALIZED | When a shard moves from started back to initializing, for example, with shadow replicas |
REALLOCATED_REPLICA | A better replica location is identified and causes the existing replica allocation to be cancelled |
References
- ElasticSearch Document Cat Shards
- ElasticSearch Document Synced Flush
- ElasticSearch Document Cluster Reroute
- How to fix your elasticsearch cluster stuck in initializing shards mode?
Reroute Unassigned Shards——遇到主shard 出现的解决方法就是重新路由的更多相关文章
- 运行jar提示“没有主清单属性”的解决方法
以下记录的是我export jar包后运行遇到问题的解决方法,如有错误,欢迎批评指正. 1.运行导出jar包,提示"没有主清单属性" 2.回想自己导出jar的操作是否有误,重新ex ...
- mysql插入报主键冲突,解决方法主键索引重新排序
1.备份表结构 create table table_bak like table_name; 2.备份表数据 insert into table_bak select * from table_na ...
- JAVA之中出现无法加载主类的情况解决方法
j今天打代码的时候出现了无法加载主类的情况,我就收集了一些,java无法加载主类的方法 ava无法加载主类解决办法 今天启动项目,又遇到找不到或无法加载主类的情况,清除项目后无法编译,class文件下 ...
- eclipse 导出jar 没有主清单属性的解决方法
eclipse编写导出的jar文件,运行出现了没有主清单属性,问题在哪里呢?有下面几种方法: 1. 导出jar文件的时候选择[可运行的jar文件]而不是[Jar文件]即可,如下图: 2. 在jar文件 ...
- KETTLE 主键不唯一解决方法
SELECT 某一列, COUNT( 某一列 ) FROM 表 GROUP BY 某一列 HAVING
- How to resolve unassigned shards in Elasticsearch——写得非常好
How to resolve unassigned shards in Elasticsearch 转自:https://www.datadoghq.com/blog/elasticsearch-un ...
- springboot打包成jar包后找不到xml,找不到主类的解决方法
springboot打包成jar包后找不到xml,找不到主类的解决方法 请首先保证你的项目能正常运行(即不打包的时候运行无误),我们在打包时经常遇到如下问题: springboot打包成jar包后找不 ...
- Recovering unassigned shards on elasticsearch 2.x——副本shard可以设置replica为0在设置回来
Recovering unassigned shards on elasticsearch 2.x 摘自:https://z0z0.me/recovering-unassigned-shards-on ...
- 如何在Elasticsearch中解析未分配的分片(unassigned shards)
一.精确定位到有问题的shards 1.查看哪些分片未被分配 curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unas ...
随机推荐
- redis主从复制,读写分离
主从复制,读写分离 Master/Slave 是什么 master写入 slave读取 能干嘛 读写分离,更加安全,性能提升 怎么玩 一主二仆.薪火相传.反客为主 周明老师,能够把长篇大论总结的很精辟 ...
- SpringCloud微服务Docker部署
前两写了两篇,都是为SpringCloud+Docker部署做准备,在部署的时候,不同服务器,不同的Docker容器之间的通信,还好没有掉到坑里去,在公司里用了新技术,还是很开心的,小有成就感,之前一 ...
- POJ 2665 模拟,,
It is confirmed that these sections do not overlap with each other. 一句话 就变成水题了,,, // by SiriusRen #i ...
- java代码实现python2中aes加密经历
背景: 因项目需要,需要将一个python2编写的aes加密方式改为java实现. 1.源python2实现 from Crypto.Cipher import AES from binascii i ...
- [转]C#事件-使用事件需要的步骤
事件是C#中另一高级概念,使用方法和委托相关.奥运会参加百米的田径运动员听到枪声,比赛立即进行.其中枪声是事件,而运动员比赛就是这个事件发生后的动作.不参加该项比赛的人对枪声没有反应. 从程序员的角度 ...
- vscode 插件推荐 - 献给所有前端工程师
VScode现在已经越来越完善.性能远超Atom和webstorm,你有什么理由不用它?在这里,我会给你们推荐很多实用的插件,让你对 vscode 有更深刻的体会,渐渐地你就会知道它有多好用. 走马观 ...
- E5中遍历数组的方法
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...
- 发现个很变态的css问题,记录下。
注意看Ul:nth-child(){} 本来是一一对应的,因为后来改版在div中加了<h1></h1>标记后,竟然自动后移了.... 不可思议,莫明奇妙
- USB接口
总结: 1.电脑的usb接口是usb母接口,u盘接口是usb公接口 2.usb otg指的是不需要电脑作为中转站接口,例如如果买一个micro 转otg接口即可将手机直接接u盘 3.方形usb口是u ...
- shell-6.环境变量配置文件
1. 2. 3. 4. 5. 6.