ELK安装和配置及常用插件安装

环境

CentOS 7.3
root 用户
JDK 版本：1.8（最低要求），主推：JDK 1.8.0_121 以上
关闭 firewall
systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动

关闭selinux

　

安装 Elasticsearch

elasticsearch运行需要使用普通用户

修改 /etc/security/limits.conf

*    soft              nofile     600000
*    hard              nofile     600000
*    soft              nproc      60000
*    hard              nproc      60000
jt_app soft memlock unlimited
jt_app hard memlock unlimited

修改/etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
vm.swappiness = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.core.somaxconn = 16384
vm.max_map_count = 262144

修改配置文件：

　生产环境主要配置：

#grep -v '^#' elasticsearch.yml|grep -v '^$'
cluster.name: prod_es_cluster
node.name: elk-log-srv01
node.master: true
node.data: true
path.data: /opt/es_data/data
path.logs: /opt/elasticsearch/logs
bootstrap.memory_lock: false
network.host: elk-log-srv01
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping_timeout: 3s
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_interval: 10s
discovery.zen.ping.unicast.hosts: ["elk-log-srv01", "elk-log-srv02","elk-log-srv03"]
discovery.zen.minimum_master_nodes: 2
gateway.recover_after_nodes: 3
gateway.expected_nodes: 3
gateway.recover_after_time: 5m
indices.query.bool.max_clause_count: 10240
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-credentials: true
search.max_buckets: 1000000

　　

　启动：

./bin/elasticsearch -d

　启动脚本：

[root@elk-log-srv01 elasticsearch]# cat /usr/lib/systemd/system/elasticsearch.service
[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=elasticsearch
Environment=ES_HOME=/opt/elasticsearch
Environment=ES_PATH_CONF=/opt/elasticsearch/config
Environment=PID_DIR=/opt/elasticsearch
#EnvironmentFile=-/etc/sysconfig/elasticsearch
#Environment=JAVA_HOME=/opt/jdk

WorkingDirectory=/opt/elasticsearch

User=jt_app
Group=jt_app

ExecStart=/opt/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

#
LimitMEMLOCK=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

# Built for packages-6.3.2 (packages)
[root@elk-log-srv01 elasticsearch]#

　

安装 Kibana

选择一台节点安装即可

进入安装目录修改配置文件：
    config/kibana.yml

server.port: 5601                                  #端口
server.host: "elk-log-srv01"                       #访问ip地址
elasticsearch.url: "http://elk-log-srv01:9200"      #连接elastic
kibana.index: ".kibana"                            #在elastic中添加.kibana索引
pid.file: /opt/kibana/kibana.pid
logging.dest: /opt/kibana/kibana.log

　启动：

nohup ./bin/kibana &

logstash安装

elasticsearch 常用插件安装

只是版本不一样，方法是一样的，替换成自己的版本即可

采用离线安装插件的方法

1、sql插件

### 项目地址
https://github.com/NLPchina/elasticsearch-sql
历史版本：
https://github.com/NLPchina/elasticsearch-sql/releases
### 下载sql插件
下载
wget https://github.com/NLPchina/elasticsearch-sql/releases/download/5.5.1.0/elasticsearch-sql-5.5.1.0.zip
安装
./bin/elasticsearch-plugin install file:///opt/elasticsearch-sql-5.5.1.0.zip

安装web访问
wget https://github.com/NLPchina/elasticsearch-sql/releases/download/5.4.1.0/es-sql-site-standalone.zip
unzip ./es-sql-site-standalone.zip
cd site-server
npm install express --save
node node-server.js & #后台启动
默认端口：8080
cd _site
vim controllers.js
修改链接es地址
url = "http://localhost:9200"

　2 分词器

项目地址：
https://github.com/medcl/elasticsearch-analysis-ik/
下载地址：
wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.5.1/elasticsearch-analysis-ik-5.5.1.zip
安装插件
./bin/elasticsearch-plugin install file:///opt/elasticsearch-analysis-ik-5.5.1.zip

　　

elasticsearch5.x安装head插件

5.0以上版本中不支持直接安装head插件，需要启动一个服务。
由于head插件本质上还是一个nodejs的工程，因此需要安装node，使用npm来安装依赖的包。（npm可以理解为maven）
#安装git
yum -y install git
#下载源码
git clone git://github.com/mobz/elasticsearch-head.git
安装 nodejs，修改环境变量
node -v
2、安装npm

3、使用npm安装grunt
由于 npm 是国外的源，下载速度比较慢，推荐使用国内淘宝镜像
npm install -g cnpm --registry=https://registry.npm.taobao.org

下面开始修改 head 插件的配置

地址：
https://github.com/mobz/elasticsearch-head

　　

cerebro插件安装

以单独进程启动
下载
wget https://github.com/lmenezes/cerebro/releases/download/v0.6.6/cerebro-0.6.6.zip
解压
unzip cerebro-0.6.6.zip
启动：
bin/cerebro -Dhttp.port=1234 -Dhttp.address=0.0.0.0 &
------------------------------------------------
其他配置
-Dconfig.file=/some/other/dir/alternate.conf

##项目地址
https://github.com/lmenezes/cerebro

　

kibana安装x-pack插件

先下载x-pack-5.5.1.zip
https://artifacts.elastic.co/downloads/packs
在线安装
bin/kibana-plugin install x-pack
离线安装
./bin/kibana-plugin install file:///opt/x-pack-5.5.1.zip

elasticsearch安装此插件一样