ElasticSearch日常使用脚本
1.启动服务要切换到非root账户 （例子：su - elk --command="/usr/local/elk/kibana/bin/kibana serve &"）
2.常用操作命令
vim 进入文件
cd 进入指定目录
w 智能提示
q! 不保存
wq 保存
esc 停止编辑
: 输入命令
i 进入编辑
& 加在最后可以使后台运行
ps -ef | grep elastic 查找ES进程
kill -9 2382（进程号） 杀掉ES进程
sh elasticsearch -d 重启ES
安装Wget：
yum install wget (基于 YUM 的系统)
apt--get install wget (基于 APT 的系统)
du -sh ./* 查看目录属性（大小等）
yum -y install java 安装java
wget url 下载并安装指定的url包
tar -zcvf v5.5.0.tar.gz /usr/local/elasticsearch/plugins/ik 解压
chown -R elk.elk /路径 授权 （例子：chown -R elk.elk /usr/local/elk/elasticsearch/logs/elasticsearch.log）
ifconfig 查看ip
poweroff 关闭虚拟机
su - elastic --command="/usr/local/elk/elasticsearch/bin/elasticsearch -d" 启动es （/usr/local/kibana/bin/kibana &）
虚拟机乔连模式可以生成内网IP
ES:
来获取所有接口：curl '192.168.1.135:9200/_cat?help’
检查集群健康：curl '192.168.1.135:9200/_cat/health?v'
集群中的节点列表：curl '192.168.1.135:9200/_cat/nodes?v'
列出所有的索引：curl '192.168.78.129:9200/_cat/indices?v'
搜索：curl '192.168.78.128:9200/${index}/${type}/_search'
查集群状态：curl '192.168.1.135:9200/_cat/nodes?v’
查看分片状态：curl '192.168.1.135:9200/_cat/shards?v’
查看映射 curl -XGET "http://192.168.1.135:9200/yqtrackv5_cache06/_mapping?pretty"
删除索引 curl -XDELETE 'http://192.168.1.135:9200/shakespeare'
----修改ip
vi /etc/sysconfig/network-scripts/ifcfg-eth0
修改前：
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=172.16.1.186
GATEWAY=172.16.1.1
修改后：
DEVICE=eth0
TYPE=Ethernet
HWADDR=00:0C:29:E6:8E:B6
UUID=7fc9a981-f305-45fc-803c-fdaf471aa9d0
NM_CONTROLLED=yes
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.1.135
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
----搜索区别
Filter 会根据情况缓存bitset，只根据提交筛选，没有其他排序功能
query 不会缓存，可以根据条件搜索，可排序
//设置副本数量
curl -XPUT "http://172.16.1.141:9200/gather.v5api-apitrack-2018.11.20/_settings?pretty=1" -d '{
"index" :{
"number_of_replicas" : 0
}
}'
//删除索引
curl -XDELETE http://172.16.1.141:9200/gather.restapi-countverify-2018.11.03
//获取索引设置
curl -XGET http://172.16.1.141:9200/indexName/_settings?pretty
//迁移分片
curl -XPOST 'http://172.16.1.141:9200/_cluster/reroute' -d '{
"commands":[{
"move":{
"index":"indexName",
"shard":0,
"from_node":"elk05",
"to_node":"elk03"
}}]}'
//重新分配副本分片到节点
curl -XPOST "http://172.16.1.141:9200/_cluster/reroute?retry_failed=true" -d'
{
"commands" : [
{
"allocate_replica" : {
"index" : "seller.thirdpartsitemsg-sellersitemsgnotsupport-2018.11.23", "shard" : 0,
"node" : "elk04"
}
}
]
}'
//手动分配主节点
curl -XPOST 'http://172.16.1.141:9200/_cluster/reroute' -d '{
"commands" : [ {
"allocate_stale_primary" : {
"index" : "indexName",
"shard" : 0,
"node" : "11EbIt7",
"accept_data_loss": true
}
}
]
}'
//关闭集群分片操作
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'
//开启集群分片操作
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'
//开启集群平衡(近针对副本)
curl -XPUT http://72.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.rebalance.enable": "replicas"}}'
//关闭集群平衡
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.rebalance.enable": "none"}}'
//查看集群设置
curl -XGET http://172.16.1.41:9200/_cluster/settings?pretty=1
//设置索引刷新时间
curl -XPUT "http://172.16.1.41:9200/gather.storage.track/_settings?pretty=1" -d ' {
"index" : {
"refresh_interval" : "30s"
}
}'
//添加索引别名
curl -XPOST 'http://172.16.1.41:9200/_aliases?pretty' -d'
{
"actions" : [
{ "add" : { "index" : "seller.online.track.002", "alias" : "seller.online.track.20" } }
]
}'
//添加别名并同时删除索引
curl -XPOST 'http://172.16.1.41:9200/_aliases?pretty' -d'
{
"actions" : [
{ "add" : { "index" : "message.apppush.2018", "alias" : "message.apppush" } },
{ "remove_index" : { "index" : "message.apppush" } }
]
}'
//添加索引并同时设置分片及刷新时间
curl -XPUT "http://172.16.1.141:9200/indexName?pretty" -d '
{
"settings" : {
"index" : {
"refresh_interval" : "5s",
"number_of_shards" : "1",
"number_of_replicas" : "1"
}
}
}'
//设置集群线程池队列长度
curl -XPUT http://172.16.1.14:9200/_cluster/settings?pretty=1 -d '{"persistent":{"thread_pool.bulk.queue_size": 200}}'
//排除集群节点
curl -XPUT http://172.16.1.14:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.exclude._ip": "172.16.1.1"}}'
//清理排除集群节点设置
curl -XPUT http://172.16.1.14:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.exclude._ip": ""}}'
一、数据备份
1、备份源数据库配置文件添加存储仓库(可设置多个)并重启(如果启动已经配置了则不需重启)
path.repo: ["/log/backup"]
2、添加快照位置到存储仓库
curl -XPUT http://127.0.0.1:9200/_snapshot/backup -d '{"type": "fs","settings": {"location": "/log/backup"}}'
3、执行备份可针对指定索引
curl -XPUT http://127.0.0.1:9200/_snapshot/backup/snapshot_1?wait_for_completion=true -d '{"indices": ".kibana"}'
二、数据恢复
1、恢复目标数据库配置文件添加存储仓库(可设置多个)并重启(如果启动已经配置了则不需重启)
path.repo: ["/data/backup"]
2、添加快照位置到存储仓库
curl -XPUT http://127.0.0.1:9200/_snapshot/backup -d '{"type": "fs","settings": {"location": "/data/backup"}}'
3、恢复
curl -XPOST http://127.0.0.1:9200/_snapshot/backup/snapshot_1/_restore?wait_for_completion=true -d '{"indices": ".kibana"}'
三、删除备份
curl -XDELETE http://127.0.0.1:9200/_snapshot/backup/snapshot_1
注意事项:
1、特别注意恢复和备份的位置对应的Elasticsearch进程是否有读写权限
node.master: true
node.data: true
node.ingest: false
/etc/security/limits.conf
elastic soft memlock unlimited
elastic hard memlock unlimited
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'
关闭分片操作
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.rebalance.enable": "all"}}'
curl -XPUT http://72.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.rebalance.enable": "none"}}'
curl -XGET http://172.16.1.41:9200/_cluster/settings?pretty
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{ "commands": [ { "allocate_replica": { "index": "seller.online.track.3", "shard": 4, "node": "yqtrack-es-06" } } ] }'
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'
curl -XPUT http://172.16.1.41:9200/_settings?pretty=1 -d ' {
"index" : {
"number_of_replicas" : 0
}
}'
curl -XPUT http://172.16.1.41:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'
curl -XPUT 'http://node3:9200/xxindex/_settings'
curl -XGET http://172.16.1.41:9200/buyer.online.track.7/trackinfo
重新分配分片?retry_failed=true
curl -XPOST "172.16.1.141:9200/_cluster/reroute?retry_failed=true" -H 'Content-Type: application/json' -d'
{
"commands" : [
{
"allocate_replica" : {
"index" : "buyer.orderapi-operatelog-2018.09.20", "shard" : 0,
"node" : "elk04"
}
}
]
}
'
curl -XPOST 'http://172.16.1.141:9200/_cluster/reroute' -d '{
"commands":[{
"move":{
"index":"nginx-access-2018.11.15",
"shard":0,
"from_node":"elk03",
"to_node":"elk05"
}}]}'
设置副本
curl -XPUT "http://172.16.1.141:9200/gather.restapi-requestcost-2018.11.15/_settings?pretty=1" -d ' {
"index" : {
"number_of_replicas" : 0
}
}'
curl -XPUT "http://172.16.1.41:9200/gather.storage.track/_settings?pretty=1" -d ' {
"index" : {
"refresh_interval" : "30s"
}
}'
设置副本
curl -XPUT "http://172.16.1.141:9200/logstash-gather.schedule-2018.05.18/_settings?pretty=1" -d ' {
"index" : {
"number_of_replicas" : 1
}
}'
开启分片操作
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'
curl -XGET http://127.0.0.1:9200/_cluster/settings?pretty=1
查看集群设置
curl -XGET http://172.16.1.41:9200/_cluster/settings?pretty=1
关闭再平衡
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.rebalance.enable": "none"}}'
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.rebalance.enable": "replicas"}}'
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"transient":{"cluster.routing.allocation.exclude._ip": ""}}'
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"transient":{"cluster.routing.allocation.exclude._ip": ""}}'
四、ES别名
curl -XPOST 'http://172.16.1.41:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
{
"actions" : [
{ "add" : { "index" : "seller.online.track.002", "alias" : "seller.online.track.2" } },
{ "add" : { "index" : "seller.online.track.003", "alias" : "seller.online.track.3" } },
{ "add" : { "index" : "seller.archived.track.002", "alias" : "seller.archived.track.2" } },
{ "add" : { "index" : "seller.archived.track.003", "alias" : "seller.archived.track.3" } }
]
}
'
curl -XPOST 'http://172.16.1.41:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
{
"actions" : [
{ "add" : { "index" : "seller.online.track.002", "alias" : "seller.online.track.20" } }
]
}
'
curl -XPOST 'http://172.16.1.41:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
{
"actions" : [
{ "add" : { "index" : "message.apppush.2018", "alias" : "message.apppush" } },
{ "remove_index" : { "index" : "message.apppush" } }
]
}
'
迁移分片
curl -XPOST 'http://172.16.1.41:9200/_cluster/reroute' -d '{
"commands":[{
"move":{
"index":"gather.storage.track",
"shard":3,
"from_node":"yqtrack-es-06",
"to_node":"yqtrack-es-05"
}}]}'
"actions" : [
{ "remove" : { "index" : "test1", "alias" : "alias1" } }
]
"actions" : [
{ "remove" : { "index" : "test1", "alias" : "alias1" } },
{ "add" : { "index" : "test2", "alias" : "alias1" } }
]
"actions" : [
{ "add": { "index": "test_2", "alias": "test" } },
{ "remove_index": { "index": "test" } }
]
curl -XPOST 'http://172.16.1.41:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
{
"actions" : [
{ "add" : { "index" : "seller.archived.track.10", "alias" : "seller.archived.track.10 } }
]
}
'
Shard Rebalancing Settingsedit
The following dynamic settings may be used to control the rebalancing of shards across the cluster:
cluster.routing.rebalance.enable
Enable or disable rebalancing for specific kinds of shards:
all- (default) Allows shard balancing for all kinds of shards.primaries- Allows shard balancing only for primary shards.replicas- Allows shard balancing only for replica shards.none- No shard balancing of any kind are allowed for any indices.
cluster.routing.allocation.allow_rebalance
Specify when shard rebalancing is allowed:
always- Always allow rebalancing.indices_primaries_active- Only when all primaries in the cluster are allocated.indices_all_active- (default) Only when all shards (primaries and replicas) in the cluster are allocated.
cluster.routing.allocation.cluster_concurrent_rebalance
Allow to control how many concurrent shard rebalances are allowed cluster wide. Defaults to 2. Note that this setting only controls the number of concurrent shard relocations due to imbalances in the cluster. This setting does not limit shard relocations due to allocation filtering or forced awareness.
curl -XPUT "http://172.16.1.141:9200/user.email-amazonsesstat-2018.01.27?pretty" -d '
{
"settings" : {
"index" : {
"refresh_interval" : "5s",
"number_of_shards" : "1",
"number_of_replicas" : "1"
}
}
}
'
curl -XGET http://127.0.0.1:9200/user.email-amazonsesstat-2018.01.27/_settings?pretty
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.cluster_concurrent_rebalance": "1"}}'
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.rebalance.enable": "none"}}'
curl -XGET http://172.16.1.141:9200/_cluster/settings?pretty=1
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'
[2018-01-28T12:50:06,115][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@263307f7 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52f1a6c1[Running, pool size = 12, active threads = 12, queued tasks = 200, completed tasks = 939958]]"})
ifcfg-eth0
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"thread_pool.bulk.queue_size": 1000}}'
curl -XPUT 'http://172.16.1.141:9200/_settings';; -d '{
"number_of_shards" : 1
}'
curl -XGET http://127.0.0.1:9200/_cat/health?v
curl -XPUT http://172.16.1.141:9200/_cluster/settings?pretty=1 -d '{"persistent":{"cluster.routing.allocation.exclude._ip": "172.16.1.144"}}'
curl -XPUT P.P.P.P:9200/_cluster/settings -d '{
"transient" :{
"cluster.routing.allocation.exclude._ip" : "X.X.X.X"
}
}'
.通过Id删除一条
curl -XDELETE 'http://192.168.1.216:9200/deals.log.search/info/6486873743665922049'
.通过查询删除数据，SearchText是keyword类型，es5.x以上版本默认可以使用_delete_by_query，2.x版本需要装插件，scroll_size是每页删除个数，默认是1000，可以自己调整
curl -XPOST "http://192.168.1.216:9200/deals.log.search/_delete_by_query?scroll_size=5" -d '
{
"query": {
"term": {
"SearchText": "all-battery.com"
}
}
}
'
ElasticSearch日常使用脚本的更多相关文章
- Linux/hp unix/AIX日常巡检脚本（转）
以下为Linux/hp unix/AIX日常巡检脚本,大家可以参考着进行改写,用于自己的服务器. #!/usr/bin/ksh syserrdate=`date +"%m/%d"` ...
- linux日常巡检脚本
######################以下是脚本内容开始部分###################################### #!/bin/bash #set -x2012-02-2 ...
- elasticsearch设置执行脚本并添加开机启动 (转）
elasticsearch设置执行脚本并添加开机启动 在/etc/init.d目录下新建文件elasticsearch #!/bin/sh #chkconfig: 2345 80 05 #descri ...
- Elasticsearch集群状态脚本及grafana监控面板导出的json文件
脚本文件: #!/usr/bin/env python import datetime import time import urllib import json import urllib2 imp ...
- elasticsearch开机启动脚本
最近搭建了一个elasticsearch服务,其中机器重启而ES服务没有重启是问题,就有下面的脚本 #!/bin/sh #chkconfig: #description: es export JAVA ...
- elasticsearch 索引清理脚本及常用命令
elastic索引日志清理不及时,很容易产生磁盘紧张,官网给出curl -k -XDELETE可以清理不需要的索引日志. 清理脚本 #!/bin/bash #Author: 648403020@qq. ...
- sqlserver日常维护脚本
SQL code --备份declare @sql varchar(8000) set @sql='backup database mis to disk=''d:\databack\mis\mis' ...
- elasticsearch的监控脚本
监控elasticsearch cluster 集群 通过主动模式将数据发送给zabbix server import json import struct import socket import ...
- Oracle日常维护脚本
1.正常停库流程 ps -ef|grep LOCAL=NO|cut -c 9-15|xargs kill -9 shutdown immediate; 2.备份数据库 bac ...
随机推荐
- 一个极其简易版的vue.js实现
前言 之前项目中一直在用vue,也边做边学摸滚打爬了近一年.对一些基础原理性的东西有过了解,但是不深入,例如面试经常问的vue的响应式原理,可能大多数人都能答出来Object.defineProper ...
- NanoPC-T4/RK3399开发板Ubuntu FriendlyCore系统开机自动运行客户程序
RK3399开机自动运行客户程序 比如hellohello.c 交叉编译:aarch64-linux-gcc hello.c -o hello使用SecureCRT软件通过串口下载到开发板rz修改文件 ...
- Excel技巧--提取中文字串
类似的,如果要提取上图第1列的商品,不要型号,如第2列. 可以考虑使用SEARCHB函数. searchb与search的区别,在于searchb函数以字节为单位搜索,search函数以字符为单位搜索 ...
- React Native学习(一) 环境搭建
需安装工具 RN环境: [必须] Node [必须] react-native-cli [可选] Node Package Manager(npm):node包管理工具,一般安装Node会带上npm ...
- 作业---修改haproxy配置文件
#查询 f=open("C:\\aaaaaaaaaaaaa\\haproxy.txt", "r", encoding="utf-8") ha ...
- Vue开发环境搭建
Nodejs与包管理工具安装: Windows安装: https://nodejs.org/en/ 下载安装包安装就可以. Ubuntu安装: //安装nodejs sudo apt-get inst ...
- C++ 重定义、重载、覆盖
想要用好C++继承和类自身函数实现就必须了解C++得三个概念重定义(redefine).重载(overload).重写(override). 一 基本感念 1 重定义(redefine) 派生类对基类 ...
- goflyway简单使用
前言 一个朋友最近新买的搬瓦工ip突然被强了,要等10周左右才能免费更换ip.而恰巧在网上看到了Goflyway 进阶教程:KCP 协议复活被墙IP 决定试一试.在vultr上临时搭建了测试环境,可能 ...
- VS2017报错 未能加载文件或程序集”Oracle.DataAccess, Version=2.122.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342"或它的某一个依赖项。试图加载格式不正确的程序
1.VS2017调用Oracle数据库报错:(采取Oracle.DataAccess.Client.dll方法调用) 2.解决方案:(将架构由anycpu转成x86或者x64) 3.具体步骤:anyc ...
- ES6系列之let/const及块级作用域
本系列是在平时阅读.学习.实际项目中有关于es6中的新特性.用发的简单总结,目的是记录以备日后温习:本系列预计包含let/const.箭头函数.解构.常用新增方法.Symbol.Set&Map ...