【原创】大数据基础之ElasticSearch（5）重要配置及调优

Index Settings 重要索引配置

Index level settings can be set per-index. Settings may be:

1 static 静态索引配置

They can only be set at index creation time or on a closed index.

只能在创建索引时设置或者在closed状态的索引上设置；

index.number_of_shards

The number of primary shards that an index should have. Defaults to 5. This setting can only be set at index creation time. It cannot be changed on a closed index. Note: the number of shards are limited to 1024 per index.

2 dynamic 动态索引配置

They can be changed on a live index using the update-index-settings API.

可以在索引存在时通过api修改；

index.number_of_replicas

The number of replicas each primary shard has. Defaults to 1.

index.refresh_interval

How often to perform a refresh operation, which makes recent changes to the index visible to search. Defaults to 1s. Can be set to -1 to disable refresh.

index.blocks.read_only

Set to true to make the index and index metadata read only, false to allow writes and metadata changes.

index.blocks.read

Set to true to disable read operations against the index.

index.blocks.write

Set to true to disable data write operations against the index. Unlike read_only, this setting does not affect metadata. For instance, you can close an index with a write block, but not an index with a read_only block.

index.merge.scheduler.max_thread_count

The maximum number of threads on a single shard that may be merging at once. Defaults to Math.max(1, Math.min(4, Runtime.getRuntime().availableProcessors() / 2)) which works well for a good solid-state-disk (SSD). If your index is on spinning platter drives instead, decrease this to 1.

index.translog.durability

Whether or not to fsync and commit the translog after every index, delete, update, or bulk request. This setting accepts the following parameters:

request: (default) fsync and commit after every request. In the event of hardware failure, all acknowledged writes will already have been committed to disk.
async: fsync and commit in the background every sync_interval. In the event of hardware failure, all acknowledged writes since the last automatic commit will be discarded.

写索引调优

1 Use bulk requests

批量请求

Bulk requests will yield much better performance than single-document index requests.

2 Use multiple workers/threads to send data to Elasticsearch

多线程，但要注意并发量不能太大以至于es无法处理而报错

Make sure to watch for TOO_MANY_REQUESTS (429) response codes (EsRejectedExecutionException with the Java client), which is the way that Elasticsearch tells you that it cannot keep up with the current indexing rate. When it happens, you should pause indexing a bit before trying again, ideally with randomized exponential backoff.

3 Increase the refresh interval

增加刷新间隔

The default index.refresh_interval is 1s, which forces Elasticsearch to create a new segment every second. Increasing this value (to say, 30s) will allow larger segments to flush and decreases future merge pressure.

4 Disable refresh and replicas for initial loads

在第一次大量写索引时禁用刷新和副本

If you need to load a large amount of data at once, you should disable refresh by setting index.refresh_interval to -1 and set index.number_of_replicas to 0. This will temporarily put your index at risk since the loss of any shard will cause data loss, but at the same time indexing will be faster since documents will be indexed only once. Once the initial loading is finished, you can set index.refresh_interval and index.number_of_replicas back to their original values.

5 Disable swapping

禁用swap

You should make sure that the operating system is not swapping out the java process by disabling swapping.

# swapoff -a

6 Give memory to the filesystem cache

The filesystem cache will be used in order to buffer I/O operations. You should make sure to give at least half the memory of the machine running Elasticsearch to the filesystem cache.

7 Use auto-generated ids

尽量使用自动生成id，可以节省查找id是否存在的开销；

When indexing a document that has an explicit id, Elasticsearch needs to check whether a document with the same id already exists within the same shard, which is a costly operation and gets even more costly as the index grows. By using auto-generated ids, Elasticsearch can skip this check, which makes indexing faster.

8 Use faster hardware

使用更快的硬件，比如更多的内存缓存或者ssd

If indexing is I/O bound, you should investigate giving more memory to the filesystem cache (see above) or buying faster drives. In particular SSD drives are known to perform better than spinning disks.

9 Indexing buffer size

增加indices.memory.index_buffer_size，通常每个shard最多需要512M

If your node is doing only heavy indexing, be sure indices.memory.index_buffer_size is large enough to give at most 512 MB indexing buffer per shard doing heavy indexing (beyond that indexing performance does not typically improve).

indices.memory.index_buffer_size

Accepts either a percentage or a byte size value. It defaults to 10%, meaning that 10% of the total heap allocated to a node will be used as the indexing buffer size shared across all shards.

修改配置

1 索引动态配置

$ curl -XPUT -H 'Content-Type: application/json' 'http://localhost:9200/testdoc/_settings' -d '{
    "index": {
        "refresh_interval":"-1",
        "number_of_replicas":0,
        "index.translog.durability":"async"
    }
}'

可反复修改，设置为null即可恢复默认

2 集群配置

$ vi elasticsearch.yml
indices.memory.index_buffer_size: 40%
thread_pool.write.queue_size: 1024

修改后同步到所有节点并重启

注意以下配置已经deprecated

The bulk thread pool has been renamed to the write thread pool. This change was made to reflect the fact that this thread pool is used to execute all write operations: single-document index/delete/update requests, as well as bulk requests.

thread_pool.index.type
thread_pool.index.size
thread_pool.index.queue_size
thread_pool.bulk.type
thread_pool.bulk.size
thread_pool.bulk.queue_size

另外以上配置也不能通过api修改（即http://localhost:9200/_cluster/settings）

The prefix on all thread pool settings has been changed from threadpool to thread_pool.
Thread pool settings are now node-level settings. As such, it is not possible to update thread pool settings via the cluster settings API.

参考：
https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html
https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html
https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-disk-usage.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html
https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules.html
https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules-translog.html
https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules-merge.html