Elasticsearch日志收集

Install pip if necessary

curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"

python get-pip.py

Install Curator for Elasticsearch

Elasticsearch Curator helps you curate, or manage, your Elasticsearch indices and snapshots by:

• Obtaining the full list of indices (or snapshots) from the cluster, as the actionable list
• Iterate through a list of user-defined filters to progressively remove indices (or snapshots) from this actionable list as needed.
• Perform various actions on the items which remain in the actionable list.

pip install elasticsearch-curator

pip install click==6.7

Configure curator

mkdir -p /var/log/elastic

touch /var/log/elastic/curator.log

mkdir ~/.curator

vi ~/.curator/curator.yml

curator.yml

# Remember, leave a key empty if there is no value. None will be a string,

## not a Python "NoneType"

client:

hosts: [Elasticsearch Server IP]

port: 9200

url_prefix:

use_ssl: False

certificate:

client_cert:

client_key:

ssl_no_validate: False

http_auth:

timeout: 30

master_only: False

 

logging:

loglevel: INFO

logfile: /var/log/elastic/curator.log

logformat: default

blacklist: ['elasticsearch', 'urllib3']

Have a test, now you can get the indices list
curator_cli show_indices

Create repository

Configure elasticseach.yml default in /etc/elasticsearch/elasticsearch.yml

elasticsearch.yml

path.repo:  /u01/elasticsearch/backup

http.max_header_size: 16kb

Restart elasticsearch service (service elasticsearch restart) to make the configurations work.

Create repository elasticsearch. Ensure location points to a valid path which is configured in path.repo, accesable from all nodes.

curl -XPUT http://localhost:9200/_snapshot/es_backup -H "Content-Type: application/json" -d @repository.json

repository.json

{

   "type": "fs",

   "settings": {

      "compress": true,

      "location": "/u01/elasticsearch/backup"

   }

}

Have a test

curl -XGET 'localhost:9200/_snapshot/_all?pretty=true'

Create curator yaml action files

daily_backup.yml

Customize the snapshot name in name option
action 1: backup all indices before today to repository elasticsearch with specified snapshot name
action 2: delete indices older than 185 days

daily_backup.yml

---

actions:

  1:

    action: snapshot

    description: >-

      Snapshot selected all indices to repository 'elasticsearch' with the snapshot name

    options:

      repository: es_backup

      name: '<c4cert-{now/d-1d}>'

      wait_for_completion: True

      max_wait: 4800

      wait_interval: 30

    filters:

    - filtertype: age

      source: name

      direction: older

      unit: days

      unit_count: 1

      timestring: "%Y.%m.%d"

 

 

  2:

    action: delete_indices

    description: >-

      Delete indices which is older than 185 days

    filters:

    - filtertype: age

      source: name

      direction: older

      unit: days

      unit_count: 185

      timestring: "%Y.%m.%d"

del_snapshot.yml
action 1: Delete snapshots from repository elasticsearch which is older than 185 days

del_snapshot.yml

---

 

actions:

  1:

    action: delete_snapshots

    description: >-

      Delete snapshots from repository which is older than 185 days

    options:

      repository: es_backup

      retry_interval: 120

      retry_count: 3

    filters:

    - filtertype: age

      source: creation_date

      direction: older

      unit: days

      unit_count: 185

restore.yml
action 1: Restore all indices in the most recent snapshot with state SUCCESS.

restore.yml

---

 

actions:

  1:

    action: restore

    description: >-

      Restore all indices in the most recent snapshot with state SUCCESS.  Wait

      for the restore to complete before continuing.  Do not skip the repository

      filesystem access check.  Use the other options to define the index/shard

      settings for the restore.

    options:

      repository: es_backup

      # If name is blank, the most recent snapshot by age will be selected

      name:

      # If indices is blank, all indices in the snapshot will be restored

      indices:

      wait_for_completion: True

      max_wait: 3600

      wait_interval: 10

    filters:

    - filtertype: state

      state: SUCCESS

Note: use --dry-run option to verify your action without any change. Find the dry run results in log path.
Curator --dry-run daily_backup.yml

Shell script and crontab

run.sh

#!/bin/sh

curator /u01/curator/del_snapshot.yml

curator /u01/curator/daily_backup.yml

crontab -e

Here configured the job run on every 3 AM

crontab

0 3 * * * /bin/sh /u01/curator/run.sh

Restore

Curator restore.yml

Tested OK in CERT env.

Some useful API 

# get all repositories

curl -XGET 'localhost:9200/_snapshot/_all?pretty=true'

 

# delete repository

curl -XDELETE 'localhost:9200/_snapshot/es-snapshot?pretty=true'

 

# show snapshots

curator_cli show_snapshots --repository es_backup

 

# show indices

curator_cli show_indices