日志分析第六章安装elasticsearch

在这里，以两台es集群为例。

es集群健康状况有三种状态，这里我们搭建的es集群，只要两台不同时挂掉，数据不会丢失。

green	所有主要分片和复制分片都可用
yellow	所有主要分片可用，但不是所有复制分片都可用
red	不是所有的主要分片都可用

举个例子：

比如说现在集群节点es1位主节点，es2位复制分片节点，默认情况下，两台es都接收logstash传过来的日志，是负载均衡的。
如果es1宕掉，es2会被提升为主节点，只有es2接收logstash传来得日志数据，同时整个集群状态由green转为red。
修复es1，es1会作为分片节点加入集群中，es1加入到集群中后，会进行修复，es2中新增的数据同步到es1分片节点，同步过程整个集群健康状态转为yellow。
当同步完成，集群状态转为green。

安装elasticsearch

官网：https://www.elastic.co/products/elasticsearch

# tar xf elasticsearch-2.3..tar.gz -C /usr/local/app/

# ln -sv /usr/local/app/elasticsearch-2.3. /usr/local/elasticsearch

# chown –R nobody.nobody /usr/local/app/elasticsearch-2.3./

# cd /usr/local/elasticsearch

编辑配置文件

es1

cluster.name: myES

node.name: es1

node.master: true

path.data: /data/elasticsearch/data

path.logs: /data/elasticsearch/log

bootstrap.mlockall: True

network.host: 10.80.2.83

http.port:

http.enabled: true

transport.tcp.port:

transport.tcp.compress: true

discovery.zen.ping.unicast.hosts: ["10.80.2.83:9300","10.80.2.84:9300"]

discovery.zen.minimum_master_nodes:

discovery.zen.ping.timeout: 3s

discovery.zen.ping.multicast.enabled: false

gateway.recover_after_nodes:

gateway.recover_after_time: 5m

gateway.expected_nodes:

cluster.routing.allocation.node_initial_primaries_recoveries:

indices.recovery.max_size_per_sec: 50mb

index :

  analysis :

    analyzer :

      default :

        tokenizer : keyword

config/elasticsearch.yml

#!/bin/sh

# check in case a user was using this mechanism

if [ "x$ES_CLASSPATH" != "x" ]; then

    cat >& << EOF

Error: Don't modify the classpath with ES_CLASSPATH. Best is to add

additional elements via the plugin mechanism, or if code must really be

added to the main classpath, add jars to lib/ (unsupported).

EOF

    exit

fi

ES_CLASSPATH="$ES_HOME/lib/elasticsearch-2.3.4.jar:$ES_HOME/lib/*"

if [ "x$ES_MIN_MEM" = "x" ]; then

    ES_MIN_MEM=8g

fi

if [ "x$ES_MAX_MEM" = "x" ]; then

    ES_MAX_MEM=8g

fi

if [ "x$ES_HEAP_SIZE" != "x" ]; then

    ES_MIN_MEM=$ES_HEAP_SIZE

    ES_MAX_MEM=$ES_HEAP_SIZE

fi

# min and max heap sizes should be set to the same value to avoid

# stop-the-world GC pauses during resize, and so that we can lock the

# heap in memory on startup to prevent any of it from being swapped

# out.

JAVA_OPTS="$JAVA_OPTS -Xms${ES_MIN_MEM}"

JAVA_OPTS="$JAVA_OPTS -Xmx${ES_MAX_MEM}"

# new generation

if [ "x$ES_HEAP_NEWSIZE" != "x" ]; then

    JAVA_OPTS="$JAVA_OPTS -Xmn${ES_HEAP_NEWSIZE}"

fi

# max direct memory

if [ "x$ES_DIRECT_SIZE" != "x" ]; then

    JAVA_OPTS="$JAVA_OPTS -XX:MaxDirectMemorySize=${ES_DIRECT_SIZE}"

fi

# set to headless, just in case

JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

# Force the JVM to use IPv4 stack

if [ "x$ES_USE_IPV4" != "x" ]; then

  JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

fi

# Add gc options. ES_GC_OPTS is unsupported, for internal testing

if [ "x$ES_GC_OPTS" = "x" ]; then

  ES_GC_OPTS="$ES_GC_OPTS -XX:+UseParNewGC"

  ES_GC_OPTS="$ES_GC_OPTS -XX:+UseConcMarkSweepGC"

  ES_GC_OPTS="$ES_GC_OPTS -XX:CMSInitiatingOccupancyFraction=75"

  ES_GC_OPTS="$ES_GC_OPTS -XX:+UseCMSInitiatingOccupancyOnly"

fi

JAVA_OPTS="$JAVA_OPTS $ES_GC_OPTS"

# GC logging options

if [ -n "$ES_GC_LOG_FILE" ]; then

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDateStamps"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"

  JAVA_OPTS="$JAVA_OPTS -Xloggc:$ES_GC_LOG_FILE"

  # Ensure that the directory for the log file exists: the JVM will not create it.

  mkdir -p "`dirname \"$ES_GC_LOG_FILE\"`"

fi

# Causes the JVM to dump its heap on OutOfMemory.

JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"

# The path to the heap dump location, note directory must exists and have enough

# space for a full heap dump.

#JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=$ES_HOME/logs/heapdump.hprof"

# Disables explicit GC

JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC"

# Ensure UTF- encoding by default (e.g. filenames)

JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF-8"

# Use our provided JNA always versus the system one

JAVA_OPTS="$JAVA_OPTS -Djna.nosys=true"

bin/elasticsearch.in.sh

es2

#集群名字

cluster.name: myES

#节点名字

node.name: es2

#是否有资格成为主节点

node.master: true

#数据目录

path.data: /data/elasticsearch/data

#日志目录

path.logs: /data/elasticsearch/log

#锁住内存

bootstrap.mlockall: True

#绑定的IP地址，可为0.0.0.

#network.bind_host: 10.80.2.84

#该节点与其他节点交互的ip地址，

#network.publish_host: 10.80.2.84

#上面两个参数集合

network.host: 10.80.2.84

#对外服务端口

http.port:

#是否使用http协议对外提供服务

http.enabled: true

#节点间交互tcp端口

transport.tcp.port:

#传输数据时压缩

transport.tcp.compress: true

#集群中主节点初始化列表，通过这些节点自动发现加入其他节点

discovery.zen.ping.unicast.hosts: ["10.80.2.83:9300","10.80.2.84:9300"]

#保证集群中节点可以知道其他N各有主节点资格，默认1，es数超过2个，可设置大一些

discovery.zen.minimum_master_nodes:

#自动发现其他节点时ping超时时间，网络环境差，提高此参数

discovery.zen.ping.timeout: 3s

#是否打开多播发现节点，默认true

discovery.zen.ping.multicast.enabled: false

#集群中几个节点启动时进行数据恢复

gateway.recover_after_nodes:

#初始化数据恢复进程超时时间

gateway.recover_after_time: 5m

#这个es集群中节点数量，一旦这些数目节点启动，进行数据恢复

gateway.expected_nodes:

#初始化数据恢复时，并发恢复的线程个数

cluster.routing.allocation.node_initial_primaries_recoveries:

#数据恢复时最大带宽

indices.recovery.max_size_per_sec: 50mb

# 禁止分词，

index :

  analysis :

    analyzer :

      default :

        tokenizer : keyword

config/elasticsearch.yml

#!/bin/sh

# check in case a user was using this mechanism

if [ "x$ES_CLASSPATH" != "x" ]; then

    cat >& << EOF

Error: Don't modify the classpath with ES_CLASSPATH. Best is to add

additional elements via the plugin mechanism, or if code must really be

added to the main classpath, add jars to lib/ (unsupported).

EOF

    exit

fi

ES_CLASSPATH="$ES_HOME/lib/elasticsearch-2.3.4.jar:$ES_HOME/lib/*"

if [ "x$ES_MIN_MEM" = "x" ]; then

    ES_MIN_MEM=8g

fi

if [ "x$ES_MAX_MEM" = "x" ]; then

    ES_MAX_MEM=8g

fi

if [ "x$ES_HEAP_SIZE" != "x" ]; then

    ES_MIN_MEM=$ES_HEAP_SIZE

    ES_MAX_MEM=$ES_HEAP_SIZE

fi

# min and max heap sizes should be set to the same value to avoid

# stop-the-world GC pauses during resize, and so that we can lock the

# heap in memory on startup to prevent any of it from being swapped

# out.

JAVA_OPTS="$JAVA_OPTS -Xms${ES_MIN_MEM}"

JAVA_OPTS="$JAVA_OPTS -Xmx${ES_MAX_MEM}"

# new generation

if [ "x$ES_HEAP_NEWSIZE" != "x" ]; then

    JAVA_OPTS="$JAVA_OPTS -Xmn${ES_HEAP_NEWSIZE}"

fi

# max direct memory

if [ "x$ES_DIRECT_SIZE" != "x" ]; then

    JAVA_OPTS="$JAVA_OPTS -XX:MaxDirectMemorySize=${ES_DIRECT_SIZE}"

fi

# set to headless, just in case

JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

# Force the JVM to use IPv4 stack

if [ "x$ES_USE_IPV4" != "x" ]; then

  JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

fi

# Add gc options. ES_GC_OPTS is unsupported, for internal testing

if [ "x$ES_GC_OPTS" = "x" ]; then

  ES_GC_OPTS="$ES_GC_OPTS -XX:+UseParNewGC"

  ES_GC_OPTS="$ES_GC_OPTS -XX:+UseConcMarkSweepGC"

  ES_GC_OPTS="$ES_GC_OPTS -XX:CMSInitiatingOccupancyFraction=75"

  ES_GC_OPTS="$ES_GC_OPTS -XX:+UseCMSInitiatingOccupancyOnly"

fi

JAVA_OPTS="$JAVA_OPTS $ES_GC_OPTS"

# GC logging options

if [ -n "$ES_GC_LOG_FILE" ]; then

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDateStamps"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"

  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"

  JAVA_OPTS="$JAVA_OPTS -Xloggc:$ES_GC_LOG_FILE"

  # Ensure that the directory for the log file exists: the JVM will not create it.

  mkdir -p "`dirname \"$ES_GC_LOG_FILE\"`"

fi

# Causes the JVM to dump its heap on OutOfMemory.

JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"

# The path to the heap dump location, note directory must exists and have enough

# space for a full heap dump.

#JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=$ES_HOME/logs/heapdump.hprof"

# Disables explicit GC

JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC"

# Ensure UTF- encoding by default (e.g. filenames)

JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF-8"

# Use our provided JNA always versus the system one

JAVA_OPTS="$JAVA_OPTS -Djna.nosys=true"

bin/elasticsearch.in.sh

日志分析第六章安装elasticsearch的更多相关文章

日志分析第五章安装logstash
logstash是java应用,依赖JDK,首先需要安装JDK,在安装jdk过程中,logstash-2.3.4使用JDK-1.7版本有bug,使用JDK-1.8版本正常,因此我们安装JDK-1.8版 ...
日志分析第七章安装grafana
grafana依赖mysql存储数据,首先需要安装mysql 安装mysql 解压 # groupadd mysql # useradd -s /sbin/nologin -g mysql mysql ...
日志分析第四章安装filebeat
在进行前面准备之后可以开始安装了,我们的安装顺序是filebeat--->logstash--->elasticsearch filebeat安装很简单,先下载filebeat,这里我们使 ...
可视化日志分析工具Gltail的安装与使用
可视化日志分析工具Gltail的安装与使用 GlTail.rb 是一款带有浓郁的 Geek 风格的可视化日志分析工具,它采用 Ruby 技术构建,并利用 OpenGL 图形技术进行渲染,呈现 ...
日志分析平台ELK之搜索引擎Elasticsearch集群
一.简介什么是ELK?ELK是Elasticsearch.Logstash.Kibana这三个软件的首字母缩写:其中elasticsearch是用来做数据的存储和搜索的搜索引擎:logstash是数 ...
日志分析工具--GoAccess的安装部署
需求:及时得到线上用户访问日志分析统计结果,以便给开发.测试.运维.运营人员提供决策! 方案:GoAccess,图文并茂,而且速度快,每秒8W 的日志记录解析速度,websocket10秒刷新统计数据 ...
手把手教你搭建 ELK 实时日志分析平台
本篇文章主要是手把手教你搭建 ELK 实时日志分析平台,那么,ELK 到底是什么呢? ELK 是三个开源项目的首字母缩写,这三个项目分别是:Elasticsearch.Logstash 和 Kiban ...
安装logstash+kibana+elasticsearch+redis搭建集中式日志分析平台
安装logstash+kibana+elasticsearch+redis搭建集中式日志分析平台 2014-01-16 19:40:57| 分类: logstash | 标签:logstash ...
ELK 日志分析体系
ELK 日志分析体系 ELK 是指 Elasticsearch.Logstash.Kibana三个开源软件的组合. logstash 负责日志的收集,处 ...

随机推荐

使用webpack搭建vue开发环境
最近几天项目上使用了vue.js作为一个主要的开发框架,并且为了发布的方便搭配了webpack一起使用.CSS框架使用的是vue-strap(vue 对bootstrap控件做了封装)这篇文章主要总结 ...
使用js和jq去掉左右空格方法
<!DOCTYPE html><html> <head> <meta charset="UTF-8"> <script src ...
oracle游标调试结果显示位置
在SQL窗口输入内容,按F8后,可以在下图看到
A+B
Problem Description Calculate A + B. Input Each line will contain two integers A and B. Process to e ...
[转]response.getWriter().write()与out.print()的区别
原文地址:http://blog.csdn.net/javaloveiphone/article/details/8133772 1.首先介绍write()和print()方法的区别: (1).w ...
innerHTML和appendChild的性能
目测innerHTML比appendChild好了3到4倍, 但是界面渲染还是很慢啊: chrome结果 /** *chrome浏览器; * innerHTML appendChild * 1千条的 ...
堆 poj 2010
选n个人从c个中花费不超过f c个人的成绩和花费求分数中位数最大 n是奇数显然中位数是n/2+1 ~c-n/2之间的(假如存在的话) 用大顶堆维护前n/2个小的花费求出以这个人为中位数的花费 ...
Fiddler+Jmeter+断言详细教程
一.Fiddler抓包工具的配置和使用在编写网关自动化脚本之前,得先学会如何抓包,这里以Fiddler为例.会抓包的同学可以跳过这一步,当然看看也是没坏处的-- 局域网络配置将要进行抓包的手机与电 ...
22 java当中的数组
class Test { public static void main(String args[]) { //数组的静态声明法 int arr[]={5,2,7,9,0}; //数组的动态声明法 i ...
iOS中通知中心(NSNotificationCenter)的使用总结
一.了解几个相关的类 1.NSNotification 这个类可以理解为一个消息对象,其中有三个成员变量. 这个成员变量是这个消息对象的唯一标识,用于辨别消息对象. @property (readon ...

日志分析 第六章 安装elasticsearch

安装elasticsearch

日志分析 第六章 安装elasticsearch的更多相关文章

随机推荐

热门专题

日志分析第六章安装elasticsearch

日志分析第六章安装elasticsearch的更多相关文章