全文搜索引擎 Elasticsearch 安装

全文搜索引擎 Elasticsearch 安装

学习了：http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html

拼音：https://www.cnblogs.com/wenbronk/p/6564962.html

head：https://github.com/mobz/elasticsearch-head

head 有chrome插件：http://sina.lt/ftSr

head 插件csdn下载：https://download.csdn.net/download/lu1005287365/10468104

head 启动 npm run start

yml中不能有Tab符号；

head for es5 之后必须单独运行了；

=======================================

学习了：http://www.cnblogs.com/techroad4ca/p/7748293.html

=======================================

elastic search 启动：

bin\elasticsearch

bin\elasticsearch  -d 表示后台启动

elastic search head 启动：

npm install 安装  注意这里可以用 cnpm install安装

npm run start 启动

========================================

对于head插件的使用，修改elasticsearch.yml文件，增加如下内容：

　　http.cors.enabled: true

　　http.cors.allow-origin: "*"

 

ik分词安装：

ik: https://github.com/medcl/elasticsearch-analysis-ik

方法1，install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.2.4/elasticsearch-analysis-ik-6.2.4.zip

            查看安装的目录结果是 elasticsearch-6.2.4\plugins\analysis-ik，下面是如下内容：

commons-codec-1.9.jar
commons-logging-1.2.jar
elasticsearch-analysis-ik-6.2.4.jar
httpclient-4.5.2.jar
httpcore-4.4.4.jar
plugin-descriptor.properties

方法2，下载https://github.com/medcl/elasticsearch-analysis-ik/releases中对应的版本，解压到elasticsearch-6.2.4\plugins\analysis-ik目录，

 

pinyin插件安装：

拼音分词器: https://github.com/medcl/elasticsearch-analysis-pinyin

解压到plugins目录下面，把文件夹改个名字；

 

==============================================================================

帮助文档： https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

所谓的cat API：https://www.elastic.co/guide/en/elasticsearch/reference/current/cat.html

所谓的Cluster API： https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-state.html

例如：http://localhost:9200/_cluster/health  查看集群名称；

=============================================================
学习了：https://blog.csdn.net/qq_16164711/article/details/78892904
官方下载：https://www.elastic.co/guide/en/elasticsearch/reference/6.2/installing-xpack-es.html
xpack默认只能install，在install说明界面有offline安装链接；
https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.2.4.zip
下载的文件，包含了x-pack for elasticsearch, kibana, 和logstash
安装命令： ./bin/elasticsearch-plugin install file:///path/to/x-pack-6.2.4.zip
在https://github.com/mobz/elasticsearch-head的官网说明了需要进行的配置：
在elasticsearch中的config/elasticsearch.yml中增加：
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization----这个有问题的
用这两个都可以
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type # from ldb
http.cors.allow-headers: Authorization,Origin, X-Requested-With, Content-Type, Accept # from php
这个也可以：
    http.cors.allow-headers: Authorization, Content-Type

使用elasticsearch/bin/x-pack/setup-passwords interactive进行密码设置，search8**
但是只能设置一次；
设置之后就可以使用该密码访问http://localhost:9200了；
启动head之后，http://localhost:9100/?auth_user=elastic&auth_password=search8**进行测试
就可以正常访问了；

=============================================================
kibana安装
下载了就可以用，如果需要认证elasticsearch，在kibana-6.2.4-darwin-x86_64/config/kibana.yml中增加es的用户名密码

帮助官网：https://www.elastic.co/guide/index.html  这个可以查看，就在官网的下面就有这个链接

kibana-plugin install file:///servers/elk/install/x-pack-6.2.4.zip   这个比较慢，即便是使用file本地安装也是比较慢的；

还是需要通过邮件进行license的下载，而且需要设置xpack.secutiry.enabled: false; 否则无法正常curl破解license；

6.2.4 x-pack破解：https://www.cnblogs.com/chengjiawei/p/8991859.html

破解java文件，覆盖elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.4.jar 传送门： x-pack-core-6.2.4.tar
在elastic.yml中进行xpack.security.enabled: false的设定；

申请license，然后手动修改；

curl -XPUT -u elastic:pass 'http://ip:9200/_xpack/license' -H "Content-Type: application/json" -d @license.json

不输入密码会在执行的时候提示一下输入，但是现在还没有设置密码呢；

使用elasticsearch/bin/x-pack/setup-passwords interactive设置密码之后，就可以进行http://localhost:9100/?auth_user=elastic&auth_password=pass访问了；

设置完密码之后，kibana/config/kibana.yml 中进行elasticsearch user/pass的设置，然后就可以使用kibana访问了；

这时，在elasticsearch启动中可以看到platinum显示，中kibana启动中可以看到platinum和有效期的显示；而且curl -u user:pass localhost:9200可以访问；

而且使用java中设置用户名密码可以访问；但是中elasticsearch启动中会出现warning，提示要xpack.security.ssl.transport.enabled 设置为true；

=============================================================

增加ssl通信证书

使用certgen -in instances.yml命令生成PEM格式证书；https://www.elastic.co/guide/en/elasticsearch/reference/6.0/certgen.html

该方法depecated了，现在还可以用；elasticsearch.yml中的node.name与instances.yml名字不一致没事；

java客户端可以与es使用同一个证书文件；elasticsearch.yml中不指定network.host，没事；

注意进行密码设置：

bin/elasticsearch-keystore add xpack.security.transport.ssl.secure_key_passphrase

在elasticsearch.yml中进行配置如下，可以使用相对路径配置key/crt/ca.crt等

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: node1/node1.key
xpack.security.transport.ssl.certificate: node1/node1.crt
xpack.security.transport.ssl.certificate_authorities: ca/ca.crt

中java客户端使用如下代码，红色部分与官网手册中不一致；

        TransportClient client = new PreBuiltXPackTransportClient(Settings.builder()
                .put("cluster.name", "elasticsearch")
                .put("xpack.security.user", "elastic:elastic")
                .put("xpack.security.transport.ssl.enabled", "true")
                .put("xpack.security.transport.ssl.verification_mode", "certificate") 
                .put("xpack.ssl.key", "/servers/elk/ca/node1.key")
                .put("xpack.ssl.certificate", "/servers/elk/ca/node1.crt")
                .put("xpack.ssl.certificate_authorities", "/servers/elk/ca/ca.crt")
//                .put("client.transport.ping_timeout", "50s")
                .build())
                .addTransportAddress(new TransportAddress(InetAddress.getByName("localhost"), 9300));

注意maven中增加es的repository

    <repositories>
        <!-- add the elasticsearch repo -->
        <repository>
            <id>elasticsearch-releases</id>
            <url>https://artifacts.elastic.co/maven</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

如果用python，人生苦短！

class ElasticObj:
    def __init__(self, index_name="trademark_around_index", index_type="location_type", ip="localhost"):
        self.index_name = index_name  # 索引名称
        self.index_type = index_type  # 索引类型
        # 无用户名密码状态，这样也会可以的
        # self.es = Elasticsearch(
        #     [
        #         'http://elastic:pass@localhost:9200/'
        #     ],
        #     verify_certs=True
        # )
        # 用户名密码状态
        self.es = Elasticsearch([ip], http_auth=('elastic', 'pass'), port=9200)

使用certutil ca / certutil cert --ca elastic-stack-ca.p12命令来生成PKCS#12格式的密钥，https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html

这是官方推荐的方式，但是命令位于bin/x-pack/certutil中，与官方文档不符；

在生成证书过程中需要输入密码，之后需要使用命令进行密码的存储：

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

在elasticsearch.yml中进行配置：

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 

在java客户端使用代码如下，注意红色部分是根据错误提示之后增加的内容，官方帮助：https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html

        TransportClient client = new PreBuiltXPackTransportClient(Settings.builder()
                .put("cluster.name", "elasticsearch")
                .put("xpack.security.user", "elastic:elastic")
                .put("xpack.security.transport.ssl.enabled", "true")
                .put("xpack.security.transport.ssl.verification_mode", "certificate")
                .put("xpack.ssl.keystore.path","/servers/elk/ca/elastic-certificates.p12")
                .put("xpack.ssl.keystore.password","elastic")
//                .put("client.transport.ping_timeout", "50s")
                .build())
                .addTransportAddress(new TransportAddress(InetAddress.getByName("localhost"), 9300));

==================================================================

说点啥好呢；es7.1.0已经默认包含x-pack了，并且默认都安装了；

在config/elasticsearch.yml中配置：

xpack.security.enabled: true

然后启动es，这个时候就已经不能匿名访问了；就可以进行密码交互设置：

➜  elasticsearch-7.1.0 bin/elasticsearch-setup-passwords interactive

在kibana/config/kibana.yml中进行用户名密码的配置，输入的是elasticSearch的用户名密码：

elasticsearch.username: "elastic"
elasticsearch.password: "elastic"

这样就可以进行用户认证了；

=============================================================

在config/elasticsearch.yml中配置：

http.cors.enabled: true
http.cors.allow-origin: "*"

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.keystore.password: elastic
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.truststore.password: elastic

其中的elastic-certificates.p12文件生成方法如下：

bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

这个时候就可以进行密码的设置了，这里有个坑，先不要设置客户端ssl认证，如果设置了客户端ssl认证，无法进行密码的设置：

➜  elasticsearch-7.1.0 bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

====================================

es 7.1.0线上配置：

cluster.name: my-application
node.name: node-1
network.host: 0.0.0.0
discovery.seed_hosts: ["0.0.0.0", "[::1]"]
cluster.initial_master_nodes: ["node-1"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: /.*/

还需要配置：

/etc/security/limits.conf

root soft nofile 65535
root hard nofile 65535
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
vm.max_map_count=655360

/etc/security/limits.d/20-nproc.conf

* soft nproc 4096
root soft nproc unlimited

/etc/sysctl.conf 
vm.max_map_count=655360

执行 sysctl -p

=====================多个节点的设置=============================

/etc/hosts
node-110 192.168.0.110
node-111 192.168.0.111
node-112 192.168.0.112

# 生成p12认证文件
bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

elasticsearch.yml
discovery.seed_hosts: ["192.168.0.110", "192.168.0.111","192.168.0.112"]
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 

# 进行密码存储
bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
# 可以使用 list/remove 命令进行已经存储内容的显示和删除；
# 存储了一个错误的值，导致无法启动；
# 第三个节点没有进行密码的设置，可以正常使用