ELK-elasticsearch-6.3.2部署

参考博客：linux下ElasticSearch.6.2.2集群安装与head、Kibana、X-Pack..插件的配置安装

参考博客：ELK5.5.1 插件安装实践纪要（head/bigdesk/kopf/cerebo/中文分词插件）

参考博客：ELK构建MySQL慢日志收集平台详解

参考博客：针对Logstash吞吐量一次优化

参考博客：ElasticStack系列之十八 & ElasticSearch5.x XPack 过期新 License 更新

1. 主机规划

主机名称	IP信息	内网IP	操作系统	安装软件	备注：运行程序
mini01	10.0.0.11	172.16.1.11	CentOS 7.4 2G内存	jdk、elasticsearch、kibana	插件head、bigdesk、cerebro
mini02	10.0.0.12	172.16.1.12	CentOS 7.4 4G内存	jdk、elasticsearch、logstash
mini03	10.0.0.13	172.16.1.13	CentOS 7.4 4G内存	jdk、elasticsearch、logstash
mini04	10.0.0.14	172.16.1.14	CentOS 7.4	jdk、logstash

添加hosts信息，保证每台都可以相互ping通

 [root@mini01 ~]# cat /etc/hosts
 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
 ::         localhost localhost.localdomain localhost6 localhost6.localdomain6

 10.0.0.11    mini01
 10.0.0.12    mini02
 10.0.0.13    mini03
 10.0.0.14    mini04
 10.0.0.15    mini05

windows的hosts文件也追加如下信息

 c:\windows\system32\drivers\etc
 ########################################## 追加信息如下：
 10.0.0.11    mini01
 10.0.0.12    mini02
 10.0.0.13    mini03
 10.0.0.14    mini04
 10.0.0.15    mini05

2. 添加用户账号

 # 使用一个专门的用户，避免直接使用root用户
 # 添加用户、指定家目录并指定用户密码
 useradd -d /app yun && echo '' | /usr/bin/passwd --stdin yun
 # sudo提权
 echo "yun  ALL=(ALL)       NOPASSWD: ALL" >>  /etc/sudoers
 # 让其它普通用户可以进入该目录查看信息
 chmod  /app/

3. Jdk【java8】

3.1. 软件安装

 [yun@mini01 software]# pwd
 /app/software
 [yun@mini01 software]# tar xf jdk1..0_112.tar.gz
 [yun@mini01 software]# ll
 total
 drwxr-xr-x             Dec  : jdk1..0_112
 -rw-r--r--  root root  Mar  : jdk1..0_112.tar.gz
 [yun@mini01 software]# mv jdk1..0_112/ /app/
 [yun@mini01 software]# cd /app/
 [yun@mini01 app]# ll
 total
 drwxr-xr-x          Dec  : jdk1..0_112
 [yun@mini01 app]# ln -s jdk1..0_112/ jdk
 [yun@mini01 app]# ll
 total
 lrwxrwxrwx   root root     May  : jdk -> jdk1..0_112/
 drwxr-xr-x          Dec  : jdk1..0_112

3.2. 环境变量

 [root@mini01 ~]$ pwd
 /app
 [root@mini01 ~]$ ll -d jdk*  # 选择jdk1.
 lrwxrwxrwx  yun yun    Mar  : jdk -> jdk1..0_112
 drwxr-xr-x  yun yun  Dec  : jdk1..0_112
 [root@mini01 profile.d]$ pwd
 /etc/profile.d
 [root@mini01 profile.d]$ cat jdk.sh # java环境变量
 export JAVA_HOME=/app/jdk
 export JRE_HOME=/app/jdk/jre
 export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
 export PATH=$JAVA_HOME/bin:$PATH

 [root@mini01 profile.d]# source /etc/profile
 [root@mini01 profile.d]$ java -version
 java version "1.8.0_112"
 Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
 Java HotSpot(TM) -Bit Server VM (build 25.112-b15, mixed mode)

4. elasticsearch部署

4.1. 软件部署

 [yun@mini01 software]$ pwd
 /app/software
 [yun@mini01 software]$ tar xf elasticsearch-6.3..tar.gz
 [yun@mini01 software]$ mv elasticsearch-6.3. /app/
 [yun@mini01 software]$ cd /app/
 [yun@mini01 ~]$ ln -s elasticsearch-6.3./ elasticsearch

4.2. 环境变量

所有安装es的机器都要有该配置

 [root@mini01 profile.d]# pwd
 /etc/profile.d
 [root@mini01 profile.d]# cat es.sh
 export ES_HOME="/app/elasticsearch"
 export PATH=$ES_HOME/bin:$PATH

 [root@mini01 profile.d]# logout
 [yun@mini01 es-data]$ source /etc/profile  # 重加在环境变量

4.3. 配置修改

由于node.name使用了变量，所以所有集群的该配置都可以一样

 [yun@mini01 config]$ pwd
 /app/elasticsearch/config
 [yun@mini01 config]$ vim elasticsearch.yml
 ………………
 # ---------------------------------- Cluster -----------------------------------
 #
 # Use a descriptive name for your cluster:
 #
 #cluster.name: my-application
 # 集群名称
 cluster.name: zhang-es
 #
 # ------------------------------------ Node ------------------------------------
 #
 # Use a descriptive name for the node:
 #
 #node.name: node-
 # 节点名称
 node.name: ${HOSTNAME}

 #
 # Add custom attributes to the node:
 ………………
 # Path to directory where to store the data (separate multiple locations by comma):
 #
 #path.data: /path/to/data
 # 该目录需要创建
 path.data: /app/es-data

 #
 # Path to log files:
 #
 #path.logs: /path/to/logs
 # 该目录需要创建
 path.logs: /app/es-data/logs
 ………………
 # Lock the memory on startup:
 #
 # 锁定内存
 bootstrap.memory_lock: true
 #
 ………………
 # Set the bind address to a specific IP (IPv4 or IPv6):
 #
 #network.host: 192.168.0.1
 # 绑定
 network.host: 0.0.0.0

 #
 # Set a custom port for HTTP:
 #
 http.port:
 #
 ………………
 # Elasticsearch performs poorly when the system is swapping the memory.
 #
 #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
 # 集群节点ip或者主机
 discovery.zen.ping.unicast.hosts: ["mini01", "mini02", "mini03"]

 #
 # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes /  + ):
 #
 # 防止脑裂配置，注意在多master时，这个值应该等于 Math.floor(master候选节点数/)+
 # 意思是master候选节点的数目最少达到多少个，才去选举master
 # 没有这个配置,在多master时容易造成脑裂,出现多个集群
 # 这里只有一个master就设置成1即可  【默认为1】
 #discovery.zen.minimum_master_nodes:
 ………………
 #action.destructive_requires_name: true

 # 下面两行配置为haad和bigdesk插件配置，各服务器一致
 http.cors.enabled: true
 http.cors.allow-origin: "*"

4.4. 启动es程序

 # -d 后台运行   -p 指定pid文件
 [yun@mini01 ~]$ elasticsearch -d -p /app/elasticsearch/es.pid  # 添加了环境变量，所以可以在任何地方启动程序
 [yun@mini01 ~]$ cat elasticsearch/es.pid 

 [yun@mini01 ~]$ netstat -lntup | grep ''
 (Not all processes could be identified, non-owned process info
  will not be shown, you would have to be root to see it all.)
 tcp6              :::                 :::*                    LISTEN      /java
 [yun@mini01 ~]$ kill -    # 停止es
 [yun@mini01 ~]$ ps -ef | grep 'ela'
 yun              : pts/    :: grep --color=auto ela

4.5. 浏览器访问

 http://mini01:9200/
 http://mini02:9200/
 http://mini03:9200/    

4.6. 启动报错与解决

 # 启动报错如下：
 [] bootstrap checks failed
 []: max file descriptors [] for elasticsearch process is too low, increase to at least []
 []: memory locking requested for elasticsearch process but memory is not locked
 []: max virtual memory areas vm.max_map_count [] is too low, increase to at least []

4.6.1. 解决1和2

 # 将 soft nofile和hard nofile从  改为 ，如下：
 # 添加soft memlock和hard memlock
 [yun@mini01 ~]$ tail /etc/security/limits.conf   # 需要退出重新登录才生效
 #*               hard    rss
 #@student        hard    nproc
 #@faculty        soft    nproc
 #@faculty        hard    nproc
 #ftp             hard    nproc
 #@student        -       maxlogins       

 # End of file
 * soft nofile
 * hard nofile
 * soft memlock unlimited
 * hard memlock unlimited

4.6.2. 解决3

 [root@mini01 ~]# vim /etc/sysctl.conf  # 追加如下信息
 ………………

 vm.max_map_count=
 [root@mini01 ~]# sysctl -p  # 生效

5. ES的XPack 过期新 License 更新

当我们直接访问ES的索引，出现如下信息时，表示license已经过期，需要重新更新

其中涉及ES的账号信息------elastic：是可以 build 这个项目的超级用户，对应默认的密码为：changeme

命令行访问

 [yun@mini02 ~]$ curl -XGET -u elastic:changeme 'http://mini01:9200/_xpack/license'
 {
   "license" : {
     "status" : "expired",
     "uid" : "59bc0e32-685b-48a9-bfdb-ddd373f672ab",
     "type" : "trial",
     "issue_date" : "2018-06-03T08:56:33.376Z",
     "issue_date_in_millis" : ,
     "expiry_date" : "2018-07-03T08:56:33.376Z",
     "expiry_date_in_millis" : ,
     "max_nodes" : ,
     "issued_to" : "zhang-es",
     "issuer" : "elasticsearch",
     "start_date_in_millis" : -
   }
 }
 [yun@mini02 ~]$ curl -XGET -u elastic:changeme 'http://mini01:9200/_cat/indices'
 {"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":}

浏览器访问

 http://mini01:9200/_cat/indices    

获取License 

 https://register.elastic.co/marvel_register    

需要你输入你对应的邮箱等基本信息，随便填写，但是邮箱和国家必须是真实的，将对应新下载的 license 上传到对应集群的某一台服务器上即可，我将我下载的一堆很长名字的 license 修改为简单的名称即：

mv xxxxx-license.json license.json

更新 license

我们更新最新的 license 不需要重启相应的 ElasticSearch 节点，只需要通过一个命令就可以动态的生效

 [yun@mini02 ~]$ curl -XPUT -u elastic:changeme 'http://mini01:9200/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @license.json
 {"acknowledged":true,"license_status":"valid"}
 [yun@mini02 ~]$ curl -XGET -u elastic:changeme 'http://mini01:9200/_xpack/license'
 {
   "license" : {
     "status" : "active",
     "uid" : "aad141e1-c24b-453c-92d1-0fdf5ac63540",
     "type" : "basic",
     "issue_date" : "2018-09-07T00:00:00.000Z",
     "issue_date_in_millis" : ,
     "expiry_date" : "2019-09-07T23:59:59.999Z",
     "expiry_date_in_millis" : ,
     "max_nodes" : ,
     "issued_to" : "zhang lia (myself)",
     "issuer" : "Web Form",
     "start_date_in_millis" :
   }
 }

之后就可以正常访问了

 [yun@mini02 ~]$ curl http://mini01:9200/_cat/indices
 green open logstash-2018.08.         MoHGSrCBQgyYrA5PLcHePg          .5kb  .2kb
 green open nginx-access-log-2018.08. TJRUOCELRPaNBLj_t943Ww       .8kb .3kb
 green open httpd-access-log-2018.08. 21NENLdBTNu49oIg9bIlnw       .6kb .6kb
 green open index-demo                  cTz0lN39SmSQfOLAM89GRg          .9kb   .9kb
 green open system-rsyslog-2018.08      zHmPivsQS72dtkQzVhIFBQ       .9kb .9kb
 ………………