1 准备analyzer

内置analyzer

参考:https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html

中文分词

smartcn

参考:https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html

ik

$ bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.6.2/elasticsearch-analysis-ik-6.6.2.zip

参考:https://github.com/medcl/elasticsearch-analysis-ik

其他plugins

参考:https://www.elastic.co/guide/en/elasticsearch/plugins/current/index.html

2 创建索引--准备mapping,确定shards、replication

# curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/testdoc -d '
{
"settings": {
"index.number_of_shards" : 10,
"index.number_of_routing_shards" : 30,
"index.number_of_replicas":1,
"index.translog.durability": "async",
"index.merge.scheduler.max_thread_count": 1,
"index.refresh_interval": "30s"
},
"mappings": {
"_doc": {
"_all": {
"enabled": false
},
"_source": {
"enabled": false
},
"properties": {
"title": { "type": "text", "analyzer": "ik_smart"},
"name": { "type": "keyword", "doc_values": false},
"age": { "type": "integer", "index": false},
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}'

其中:

_source 控制是否存储原始json
_all 控制是否对原始json建倒排
analyzer 用于指定分词
doc_values 用于控制是否列式存储
index 用于控制是否倒排

The _source field stores the original JSON body of the document. If you don’t need access to it you can disable it.
By default Elasticsearch indexes and adds doc values to most fields so that they can be searched and aggregated out of the box.

数据类型

参考:https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html

其中String有两种:text和keyword,区别是text会被分词,keyword不会被分词;

text

参考:https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html

keyword

参考:https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html

3 导入数据

3.1 调用index api

参考:https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html

3.2 准备hive外部表

详见:https://www.cnblogs.com/barneywill/p/10300951.html

4 测试

# curl -XPOST -H 'Content-Type: application/json' 'http://localhost:9200/_xpack/sql?format=txt' -d '{"query":"select * from testdoc limit 10"}'

or

# curl -XGET 'http://localhost:9200/testdoc/_search?q=*'

5 问题

报错:all nodes failed

2019-03-27 03:14:50,091 ERROR [main] org.elasticsearch.hadoop.rest.NetworkClient: Node [192.168.0.1:9200] failed (Read timed out); selected next node [192.168.0.1:9200]
2019-03-27 03:15:50,148 ERROR [main] org.elasticsearch.hadoop.rest.NetworkClient: Node [192.168.0.2:9200] failed (Read timed out); selected next node [192.168.0.2:9200]
2019-03-27 03:16:50,207 ERROR [main] org.elasticsearch.hadoop.rest.NetworkClient: Node [192.168.0.3:9200] failed (Read timed out); no other nodes left - aborting...
2019-03-27 03:16:50,208 ERROR [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing operators - failing tree
2019-03-27 03:16:50,210 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Hive Runtime Error while closing operators
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[192.168.0.1:9200, 192.168.0.2:9200, 192.168.0.3:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:152)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:398)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:362)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:366)
at org.elasticsearch.hadoop.rest.RestClient.refresh(RestClient.java:267)
at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.close(BulkProcessor.java:550)
at org.elasticsearch.hadoop.rest.RestRepository.close(RestRepository.java:219)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.doClose(EsOutputFormat.java:214)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.close(EsHiveOutputFormat.java:74)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:190)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1047)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
... 8 more

解决方法:增加 index.number_of_shards,只能在创建索引时指定,默认为5

报错:es_rejected_execution_exception

Caused by: org.elasticsearch.hadoop.EsHadoopException: Could not write all entries for bulk operation [70/1000]. Error sample (first [5] error messages):
org.elasticsearch.hadoop.rest.EsHadoopRemoteException: es_rejected_execution_exception: rejected execution of processing of [7622922][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[test_indix][18]] containing [38] requests, target allocation id: iLlIBScJTxahse559pTINQ, primary term: 1 on EsThreadPoolExecutor[name = 1hxgYU_/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@ce11763[Running, pool size = 32, active threads = 32, queued tasks = 200, completed tasks = 5686436]]

报错原因:

thread_pool.write.queue_size

For single-document index/delete/update and bulk requests. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.

The queue_size allows to control the size of the queue of pending requests that have no threads to execute them. By default, it is set to -1 which means its unbounded. When a request comes in and the queue is full, it will abort the request.

查看thread_pool统计

# curl 'http://localhost:9200/_nodes/stats?pretty'|grep '"write"' -A 7

通常由于写入速度、并发量或者压力较大超过es处理能力,超出queue的大小就会被reject

解决方法:

1)修改配置调优

index.refresh_interval: -1
index.number_of_replicas: 0
indices.memory.index_buffer_size: 40%
thread_pool.write.queue_size: 1024

详见:https://www.cnblogs.com/barneywill/p/10615249.html

2)减小写入压力

【原创】大数据基础之ElasticSearch(4)es数据导入过程的更多相关文章

  1. 【原创】大数据基础之ElasticSearch(2)常用API整理

    Fortunately, Elasticsearch provides a very comprehensive and powerful REST API that you can use to i ...

  2. 【原创】大数据基础之ElasticSearch(1)简介、安装、使用

    ElasticSearch 6.6.0 官方:https://www.elastic.co/ 一 简介 ElasticSearch简单来说是对lucene的分布式封装,增加了shard(每个shard ...

  3. 【原创】大数据基础之ElasticSearch(5)重要配置及调优

    Index Settings 重要索引配置 Index level settings can be set per-index. Settings may be: 1 static 静态索引配置 Th ...

  4. 【原创】大数据基础之ElasticSearch(3)升级

    elasticsearch版本升级方案 常用的滚动升级过程(Rolling Upgrade)如下: $ curl -XPUT '$es_server:9200/_cluster/settings?pr ...

  5. 你的ES数据备份了吗?

    前言: 无论使用哪种存储软件,定期的备份数据都是重中之重,在使用ElasticSearch的时候,随着数据日益积累,存放es数据的磁盘空间也捉襟见肘, 此时对于业务功能使用不到的索引数据,又不能直接删 ...

  6. 【原创】大数据基础之Zookeeper(2)源代码解析

    核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,f ...

  7. 【原创】大数据基础之Impala(2)实现细节

    一 架构 Impala is a massively-parallel query execution engine, which runs on hundreds of machines in ex ...

  8. 大数据基础知识问答----spark篇,大数据生态圈

    Spark相关知识点 1.Spark基础知识 1.Spark是什么? UCBerkeley AMPlab所开源的类HadoopMapReduce的通用的并行计算框架 dfsSpark基于mapredu ...

  9. 大数据篇:ElasticSearch

    ElasticSearch ElasticSearch是什么 ElasticSearch是一个基于Lucene的搜索服务器.它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口. ...

随机推荐

  1. 记自己在spring中使用redis遇到的两个坑

    本人在spring中使用redis作为缓存时,遇到两个坑,现在记录如下,算是作为自己的备忘吧,文笔不好,望大家见谅: 一.配置文件 <!-- 加载Properties文件 --> < ...

  2. SpringBoot 统一时区的方案

    系统采用多时区设计的时候,往往我们需要统一时区,需要统一的地方如下: 服务器(Tomcat服务) 数据库(JPA + Hibernate) 前端数据(前端采用Vuejs) 思路为:将数据库和服务器的时 ...

  3. Python——Django-模板

    一.模板的种类 1.变量 {{变量名}} 2.语句类{% %} 2.1 {%for i in booklist%} {{i}} {%endfor%} 2.2 {%if 10>5%} {%else ...

  4. springboot jpa 复合主键

    https://blog.csdn.net/wyc_cs/article/details/9031991 创建一个复合主键类 public class LevelPostMultiKeysClass ...

  5. Linux 学习 (五) 压缩与解压缩命令

    Linux达人养成计划 I 学习笔记 常用压缩格式:.zip | .gz | .bz2 | .tar.gz | .tar.bz2 .zip zip 压缩文件名 源文件:压缩文件 zip -r 压缩文件 ...

  6. 三、Java多人博客系统-技术架构

    多人博客系统1.0版本,架构和技术还是很简单和很传统的. 1.技术 前端:jsp.html.css.javascript.jquery.easyui.echarts 后端:spring mvc.Hib ...

  7. Intel 11代全新核显出现了!好强 好乱

    Intel最近为其10亿级用户的核芯显卡真是操醉了心,一方面进一步公布了第11代核显的架构细节,另一方面升级驱动程序带来了全新的控制中心. 而在对新驱动进行了一番挖掘之后,赫然发现Intel已经将第1 ...

  8. MongoDB和pymongo的CURD

    一.mongodb 1.介绍 MongoDB是一个基于分布式文件存储的数据库.由C++语言编写.旨在为WEB应用提供可扩展的高性能数据存储解决方案. MongoDB是一个介于关系数据库和非关系数据库之 ...

  9. mysql主从复制、redis基础、持久化和主从复制

    一.mysql(mariadb)基础 1.基础命令(centos7操作系统下) 1.启动mysql systemctl start mariadb 2.linux客户端连接自己 mysql -uroo ...

  10. 【XSY2990】树 组合数学 容斥

    题目描述 同 Comb Avoiding Trees 不过只用求一项. \(n,k\leq {10}^7\) 题解 不难发现一棵 \(n\) 个叶子的树唯一对应了一个长度为 \(2n-2\) 的括号序 ...