ElasticSearch踩坑记录

一、分词错误
[2018-02-06 14:28:30:098] --- [INFO] --- [CjhArticleSimilarityTask.java:66] --- [【SimilarityConsumeTask】=报错结束，时间:2018-02-06 14:28:30,errorMsg:Failed to deserialize response of type [org.elasticsearch.action.admin.indices.analyze.AnalyzeResponse]] ---
[2018-02-06 14:28:30:099] --- [ERROR] --- [LogUtils.java:59] --- [【SimilarityConsumeTask】=报错结束，时间:2018-02-06 14:28:30] ---
TransportSerializationException[Failed to deserialize response of type [org.elasticsearch.action.admin.indices.analyze.AnalyzeResponse]]; nested: IllegalStateException[unexpected byte [0x43]];
     at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1425)
     at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1397)
     at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
     at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
     at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: unexpected byte [0x43]
     at org.elasticsearch.common.io.stream.StreamInput.readBoolean(StreamInput.java:409)
     at org.elasticsearch.common.io.stream.StreamInput.readBoolean(StreamInput.java:399)
     at org.elasticsearch.common.io.stream.StreamInput.readOptionalString(StreamInput.java:315)
     at org.elasticsearch.action.admin.indices.analyze.AnalyzeResponse$AnalyzeToken.readFrom(AnalyzeResponse.java:128)
     at org.elasticsearch.action.admin.indices.analyze.AnalyzeResponse$AnalyzeToken.readAnalyzeToken(AnalyzeResponse.java:110)
     at org.elasticsearch.action.admin.indices.analyze.AnalyzeResponse.readFrom(AnalyzeResponse.java:201)
     at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1422)
     ... 27 more

解决：
生产环境ElasticSearch版本5.2.2，
期初es使用mvn版本 5.6.3，降低maven版本为5.2.2后搞定

二、es 5.2.2写入es报错：
Limit of total fields [1000] in index [my_index] has been exceeded
调整如下设置可以暂时解决：
PUT my_index/_settings { "index.mapping.total_fields.limit": 2000 }

报错的原因是json串写入es时解析字段类型错误，扩大了mapping，经测试逐个字段赋值是可以的，但是不方便。
同样的程序在es6.0下是不会出现该错误的。

三、es 5.2批量写入es报错：
[RemoteTransportException[[node_***][****:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[node_***.*][10.**.***.**:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$7@5e2672b5 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@151d7492[Running, pool size = 32, active threads = 32, queued tasks = 54, completed tasks = 814852676]]];] ---

put("es.batch.size.bytes", "300000000");
                 put("es.batch.size.entries", "500000");
                 put("es.input.json", "true");
                 put("es.batch.write.refresh", "false");
                 put("es.batch.write.retry.count","-1");
                 put("es.batch.write.retry.wait","300");

提交es频率太高，调整方案增大写入间隔和每次写入条数。

四、es日期字段自动识别为字符串
put test_date/test/1
{
   "d1":"2017/11/04",
   "d2":"2017-11-03 21:42:11",
   "d3":"2017/11/03 21:42:11",
   "d4":"2017-08-20T16:00:00.000Z"
}

五、解决报错 availableProcessors is already set to [16], rejecting [16]

@Configuration
public class ElasticSearchConfig {
    /**
     * 防止netty的bug
     * java.lang.IllegalStateException: availableProcessors is already set to [4], rejecting [4]
     */
    @PostConstruct
    void init() {
        System.setProperty("es.set.netty.runtime.available.processors", "false");
    }
}