Elasticsearch之环境搭建

一、安装 elasticsearch

-- 拉取镜像
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.9.1
-- 创建 docker 网络
docker network create elastic
-- 启动容器，-m 设置内存大小
docker run --name es01 --net elastic -p 9200:9200 -p 9300:9300 -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:8.9.1
-- 将证书从容器复制到本地计算机
docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .

首次启动后，控制台会输出用户名密码和注册令牌

 Elasticsearch security features have been automatically configured!
 Authentication is enabled and cluster connections are encrypted.

️  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
  j1QUuCeQbcawuLkvJ*=u

️  HTTP CA certificate SHA-256 fingerprint:
  82e5fd26d657c97027721a4e47f464063e40cb67a104ef847b5cff3887f2f52e

️  Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjkuMSIsImFkciI6WyIxNzIuMTguMC4yOjkyMDAiXSwiZmdyIjoiODJlNWZkMjZkNjU3Yzk3MDI3NzIxYTRlNDdmNDY0MDYzZTQwY2I2N2ExMDRlZjg0N2I1Y2ZmMzg4N2YyZjUyZSIsImtleSI6ImtSSWlYb29CbEdlcWJ4aEFodTVCOnlaTUIxX2NMVFc2enB6RVhKVm9IVkEifQ==

️ Configure other nodes to join this cluster:
• Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjkuMSIsImFkciI6WyIxNzIuMTguMC4yOjkyMDAiXSwiZmdyIjoiODJlNWZkMjZkNjU3Yzk3MDI3NzIxYTRlNDdmNDY0MDYzZTQwY2I2N2ExMDRlZjg0N2I1Y2ZmMzg4N2YyZjUyZSIsImtleSI6Imt4SWlYb29CbEdlcWJ4aEFodTVCOnA5NlFONVdFUkdDdi1rNy1RYkx3ZncifQ==

  If you're running in Docker, copy the enrollment token and run:
  `docker run -e "ENROLLMENT_TOKEN=<token>" docker.elastic.co/elasticsearch/elasticsearch:8.9.1`

打开浏览器访问 https://localhost:9200 输入用户名 elastic 密码 j1QUuCeQbcawuLkvJ*=u ，页面返回：

{
    name: "c0766560ee1a",
    cluster_name: "docker-cluster",
    cluster_uuid: "pQ8iWzM5TQGPY0W8vQOlVQ",
    version: {
    number: "8.9.1",
    build_flavor: "default",
    build_type: "docker",
    build_hash: "a813d015ef1826148d9d389bd1c0d781c6e349f0",
    build_date: "2023-08-10T05:02:32.517455352Z",
    build_snapshot: false,
    lucene_version: "9.7.0",
    minimum_wire_compatibility_version: "7.17.0",
    minimum_index_compatibility_version: "7.0.0"
    },
    tagline: "You Know, for Search"
}

elasticsearch 令牌过期

-- 刷新令牌
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana --url "https://127.0.0.1:9200"

密码错误

docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

输出结果：

xadyo97zTpBfLl56j38C

二、安装 kibana

-- 拉取镜像
docker pull docker.elastic.co/kibana/kibana:8.9.1
-- 运行容器
docker run --name kib-01 --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:8.9.1
-- 创建本地目录
sudo mkdir -p /opt/docker/kibana/config
-- 修改目录权限
sudo chmod -R 777 /opt/docker/kibana
-- 拷贝配置文件到本地
sudo docker cp kib-01:/usr/share/kibana/config /opt/docker/kibana/
-- 删除容器
docker stop kib-01
docker rm kib-01
-- 运行容器,指定名称、端口、挂载外部配置文件
docker run --name kib-01 --net elastic -p 5601:5601 -v /opt/docker/kibana/config:/usr/share/kibana/config docker.elastic.co/kibana/kibana:8.9.1

访问 http://localhost:5601 页面会弹出配置窗口，输入 elasticsearch 初次启动输出的令牌进行关联

eyJ2ZXIiOiI4LjkuMSIsImFkciI6WyIxNzIuMTguMC4yOjkyMDAiXSwiZmdyIjoiODJlNWZkMjZkNjU3Yzk3MDI3NzIxYTRlNDdmNDY0MDYzZTQwY2I2N2ExMDRlZjg0N2I1Y2ZmMzg4N2YyZjUyZSIsImtleSI6Imt4SWlYb29CbEdlcWJ4aEFodTVCOnA5NlFONVdFUkdDdi1rNy1RYkx3ZncifQ==

输入后 在 kibana 控制台获取到验证码输入即可

配置完成后会弹出 elastic 登录页面，输入 elasticsearch 的用户名和密码即可

如果令牌过期，可执行如下命令：

-- 刷新令牌
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana --url "https://127.0.0.1:9200"

三、安装Logstash

官网下载

注：Logstash 直接采用本地安装包解压方式执行

导入数据

下载 movies.csv 文件，存放到bin目录下

在bin目录下创建 logstash.conf 文件：

input {
  file {
    path => "/{自己的文件目录}/elasticsearch/logstash-8.9.1/bin/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
     hosts => "https://localhost:9200"
     index => "movies"
     document_id => "%{id}"
     ssl_certificate_verification => "true"
     cacert => "/Users/zhulizhong/Software/elasticsearch/logstash-8.9.1/bin/http_ca.crt"
     user => "elastic"
     password => "xadyo97zTpBfLl56j38C"
   }
  stdout {}
}

然后执行导入命令

./logstash -f /Users/zhulizhong/Software/elasticsearch/logstash-8.9.1/bin/logstash.conf

failed to obtain node locks 错误

删除 /data/.lock 文件

unable to find valid certification path to requested target 错误

配置elasticsearch证书文件
配置elasticsearch用户名密码