ES项目实战

前置

ES: Java
Spark/Flink Stack + Spring Boot + ES
Scala/Java + Java/Scala + Java
==> 用API的方式来掌握ES的用法(API、SpringBoot的使用)
ES: API RESTFul

ElasticSearch + Kibana
存储 展示/分析
ES Plugin: Head SQL Kibana (三个插件,Kibana也算插件)
最终数据要到SQL (易用性)

ES的安装

地址: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.2.tar.gz



• 解压: tar -zxvf elasticsearch-6.6.2.tar.gz -C ~/app/
• 配置到系统环境变量:
• 修改:
  cd elasticsearch-6.6.2
  删除bin目录中以bat结尾的命令(windows中的,无用)

$ES_HOME (重要的文件)
bin
elasticsearch 启动ES
elasticsearch 前台方法启动
elasticsearch -d 后台方法启动 (需要打开日志查看系统输出信息)
elasticsearch-plugin 管理ES插件
elasticsearch-sql-cli sql客户端
config

elasticsearch.yml es的配置信息

#cluster.name: my-application  (集群名称)
#node.name: node-1   (节点名称)
#path.data: /path/to/data
Path to log files:
path.logs: /path/to/logs
#network.host: 0.0.0.0  (全网)
Set a custom port for HTTP:
http.port: 9200

jvm.optiom es的JVM相关的配置信息

## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g
-Xmx1g

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

# explicitly set the stack size
-Xss1m

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log

# ----------------------------------- Memory -----------------------------------
elasticsearch-migrate       x-pack-env
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

***注:ES对硬盘要求极高(一般SSD、内存大) ***

后台启动ES: elasticsearch -d
去Web UI查看: hadoop000:9200 (Web UI port)
hadoop000:9300 服务器端口(server port)
Lucene_version: "7.6.0" 不对应

技巧:用chrome中的json字符串美化插件 (JSON Formatter)

ES核心概念

Cluster
Node
Index       Database
Type        Table
Document    Row
Field       Column   //这四种的REST API在工作中用的最多
shard(分片--分区集叫分片)
replica(副本)

对应关系:
Index -> Type -> Document -> Field
Database -> Table -> Row -> Column

代码开发

需要: IDEA+Maven+Java
pom.xml

      <!--添加elasticsearch依赖-->
    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>transport</artifactId>
      <version>6.6.2</version>
    </dependency>

    <!--添加junit依赖(默认已有)-->
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <!--<scope>test</scope>(去掉这块),需要重新导入依赖-->
    </dependency>

PreBuiltTransportClient.java

     /**
     * Creates a new transport client with pre-installed plugins.
     *
     * @param settings the settings passed to this transport client
     * @param plugins  an optional array of additional plugins to run with this client
     */
    @SafeVarargs
    public PreBuiltTransportClient(Settings settings, Class<? extends Plugin>... plugins) {
        this(settings, Arrays.asList(plugins));
    }
    //Class<? extends Plugin>...
    // ...代表一个/多个可变参数, 可填可不填

TransportClient.java

/**
     * Adds a transport address that will be used to connect to.
     * <p>
     * The Node this transport address represents will be used if its possible to connect to it.
     * If it is unavailable, it will be automatically connected to once it is up.
     * <p>
     * In order to get the list of all the current connected nodes, please see {@link #connectedNodes()}.
     */
    public TransportClient addTransportAddress(TransportAddress transportAddress) {
        nodesService.addTransportAddresses(transportAddress);
        return this;
    }