Eleastisearch6.0.0 read_only_allow_delete: false

Eleastisearch6.0.0由单节点升级到多节点集群cluster时候出现的分片同步错误问题解决

原创 2018年01月18日 16:33:21 5

启动多个节点的ES后，ES开始推举master节点并同步分片shard数据到新ES节点上，此时观察Logstash日志抛出以下错误：

logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

这是由于ES新节点的数据目录data存储空间不足，导致从master主节点接收同步数据的时候失败，此时ES集群为了保护数据，会自动把索引分片index置为只读read-only

解决步骤：

1.提供足够的存储空间供数据写入，如需在配置文件中更改ES数据存储目录，注意重启ES

2.放开索引只读设置，在Kibana的开发工具Dev Tools中执行（或在服务器上通过curl工具发起PUT请求，下文同）

PUT _settings

    {

    "index": {

    "blocks": {

    "read_only_allow_delete": "false"

    }

    }

    }

此时观察ES集群状态：curl http://10.0.7.220:9200/_cluster/health?pretty

注意到"active_shards_percent_as_number" : 12.0 该项的值产生变化;

[FORBIDDEN/12/index read-only / allow delete (api)] - read
only elasticsearch indices

If your elasticsearch is responding with 403 and this message:

{
  "type": "cluster_block_exception",
  "reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
}

Then you probably recovered from a full hard drive. The thing is, elasticsearch is switching to read-only if it cannot index more documents because your hard drive is full. With this it ensures availability for read-only queries. Elasticsearch will not switch
back automatically but you can disable it by sending

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'