1、场景

1.1、依赖版本

  • avatica-core 1.11.0
  • druid 0.12.0

1.2、问题重现:

使用Avatica JDBC查询语句:SELECT score FROM student WHERE name='小明'

到Druid变成:SELECT score FROM student WHERE name='??'

2、解决过程

思路:检查请求发送前request body -> 检查收到请求后解析的文本

2.1、初步怀疑请求编码所致

初步怀疑请求的编码格式设置不正确。为了方便查看返回的结果是否是乱码,我们使用EXPLAIN PLAN FOR来调试。查看请求日志,可以知道avatica的内部使用了HttpClient实现的。

17:45:17.557 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> POST /druid/v2/sql/avatica/ HTTP/1.1
17:45:17.557 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 241
17:45:17.557 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Type: application/octet-stream
17:45:17.557 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: localhost:8082
17:45:17.557 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
17:45:17.557 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_131)
17:45:17.557 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "POST /druid/v2/sql/avatica/ HTTP/1.1[\r][\n]"
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 241[\r][\n]"
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: localhost:8082[\r][\n]"
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_131)[\r][\n]"
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
17:45:17.557 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "{"request":"prepareAndExecute","connectionId":"b4490330-c1c1-493a-b40d-a4303698cafa","statementId":1,"sql":"EXPLAIN PLAN FOR SELECT score FROM student WHERE name='[0xe6][0xb4][0x97][0xe5]'","maxRowsInFirstFrame":-1,"maxRowCount":-1}"

可以发现请求的sqlEXPLAIN PLAN FOR SELECT score FROM student WHERE name='小明'发送时变成了EXPLAIN PLAN FOR SELECT score FROM student WHERE name='[0xe6][0xb4][0x97][0xe5]',这是编码乱了吗?其实不是的,我们可以通过指定avatica的HttpClient,来调试请求发送前的数据。

通过复制avatica-core-1.11.0.jar的AvaticaCommonsHttpClientImpl类,改变public byte[] send(byte[] request)的实现,可以改变请求的方式。

同时jdbc调用的时候指定httpclient_impl,即可。

String url = "jdbc:avatica:remote:url=http://localhost:8082/druid/v2/sql/avatica/";
Properties connectionProperties = new Properties();
connectionProperties.setProperty("httpclient_impl", "com.test.MyAvaticaCommonsHttpClientImpl");
conn = DriverManager.getConnection(url, connectionProperties);

最后无论怎么改变send方法的实现,最后返回的结果始终带有??。

17:45:18.907 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
17:45:18.908 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Date: Tue, 15 May 2018 09:45:30 GMT[\r][\n]"
17:45:18.908 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Content-Type: application/json;charset=utf-8[\r][\n]"
17:45:18.908 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Content-Length: 1641[\r][\n]"
17:45:18.908 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Server: Jetty(9.3.19.v20170502)[\r][\n]"
17:45:18.908 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
17:45:18.909 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "{"response":"executeResults","missingStatement":false,"rpcMetadata":{"response":"rpcMetadata","serverAddress":"master:8082"},"results":[{"response":"resultSet","connectionId":"b4490330-c1c1-493a-b40d-a4303698cafa","statementId":1,"ownStatement":false,"signature":{"columns":[{"ordinal":0,"autoIncrement":false,"caseSensitive":true,"searchable":false,"currency":false,"nullable":0,"signed":true,"displaySize":-1,"label":"PLAN","columnName":"PLAN","schemaName":null,"precision":-1,"scale":-2147483648,"tableName":null,"catalogName":null,"type":{"type":"scalar","id":12,"name":"VARCHAR","rep":"STRING"},"readOnly":true,"writable":false,"definitelyWritable":false,"columnClassName":"java.lang.String"}],"sql":"EXPLAIN PLAN FOR SELECT score FROM student WHERE name='??'","parameters":[],"cursorFactory":{"style":"LIST","clazz":null,"fieldNames":null},"statementType":"SELECT"},"firstFrame":{"offset":0,"done":true,"rows":[["DruidQueryRel(query=[{\"queryType\":\"scan\",\"dataSource\":{\"type\":\"table\",\"name\":\"student\"},\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z\"]},\"virtualColumns\":[],\"resultFormat\":\"compactedList\",\"batchSize\":20480,\"filter\":{\"type\":\"selector\",\"dimension\":\"name\",\"value\":\"??\",\"extractionFn\":null},\"columns\":[\"score\"],\"legacy\":false,\"context\":{},\"descending\":false,\"granularity\":{\"type\":\"all\"}}], signature=[{name:STRING])\n"]]},"updateCount":-1,"rpcMetadata":{"response":"rpcMetadata","serverAddress":"master:8082"}}]}[\n]"
17:45:18.910 [main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 OK
17:45:18.910 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Tue, 15 May 2018 09:45:30 GMT
17:45:18.911 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Type: application/json;charset=utf-8
17:45:18.911 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Length: 1641
17:45:18.911 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Server: Jetty(9.3.19.v20170502)

2.2、检查Druid收到的请求以及返回

既然检查发送前的请求没问题,那么接下来检查请求接收后的情况。

通过全局搜索Druid的源码,搜索/druid/v2/sql/avatica/,可以知道我们刚才的请求被DruidAvaticaHandler类接收并处理,最后实际处理的avatica-server,而这里用的是1.10.0版本,通过版本对比,没版本的问题。

通过检查avatica-server的AvaticaJsonHandler类的方法:

public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response)

看到不和谐的地方:

final String jsonRequest = new String(rawRequest.getBytes("ISO-8859-1"), "UTF-8");,

UTF-8编码传来的请求居然用ISO-8859-1转换?原来问题就出现在这里。

3、解决方法

3.1、修改avatica-server源码

上github克隆avatica-server 1.10.0的代码,找到AvaticaJsonHandler类的方法:

public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response)

修改:

final String jsonRequest = new String(rawRequest.getBytes("ISO-8859-1"), "UTF-8");

=> final String jsonRequest = new String(rawRequest.getBytes("UTF-8"), "UTF-8");

3.2、重新编译avatica-server

运行clean install -Dmaven.test.skip=true -Dcheckstyle.skip=true,重新生成avatica-server的jar

3.3、覆盖Druid的依赖

上Druid所在的服务器,进入lib,把avatica-server-1.10.0.jar备份并覆盖,重启服务。

编译好的jar包下载:https://download.csdn.net/download/yongjian_pan/10417162

Druid.io SQL乱码问题的更多相关文章

  1. Druid.io启用SQL支持

    Druid.io的SQL功能虽然在试验阶段,但是也支持了大部分的功能,而且还可以通过 Avatica JDBC查看请求的json,有助于我们理解Druid.io的语法.Druid.io有个比较坑的是, ...

  2. Druid.io系列(一):简介

    原文链接: https://blog.csdn.net/njpjsoftdev/article/details/52955676 Druid.io(以下简称Druid)是面向海量数据的.用于实时查询与 ...

  3. Druid.io系列(五):查询过程

    原文链接: https://blog.csdn.net/njpjsoftdev/article/details/52956194 Druid使用JSON over HTTP 作为底层的查询语言,不过强 ...

  4. druid.io 海量实时OLAP数据仓库 (翻译+总结) (1)

    介绍 我是NDPmedia公司的大数据OLAP的资深高级工程师, 专注于OLAP领域, 现将一个成熟的可靠的高性能的海量实时OLAP数据仓库介绍给大家: druid.io NDPmedia在2014年 ...

  5. druid.io 海量实时OLAP数据仓库 (翻译+总结) (1)——分析框架如hive或者redshift(MPPDB)、ES等

    介绍 我是NDPmedia公司的大数据OLAP的资深高级工程师, 专注于OLAP领域, 现将一个成熟的可靠的高性能的海量实时OLAP数据仓库介绍给大家: druid.io NDPmedia在2014年 ...

  6. SpringMVC4+MyBatis+SQL Server2014+druid 监控SQL运行情况

    前言 在基于SpringMVC+MyBatis的开发过程中,我们希望能看到自己手写SQL的执行情况,在开发阶段我们可以配置log4j在控制台里基于debug模式查看,那么上线后,在生产声我们想查看SQ ...

  7. druid.io本地集群搭建 / 扩展集群搭建

    druid.io 是一个比较重型的数据库查询系统,分为5种节点 . 在此就不对数据库进行介绍了,如果有疑问请参考白皮书: http://pan.baidu.com/s/1eSFlIJS 单台机器的集群 ...

  8. Druid.io通过NiFi摄取流数据

    NiFi是一个易于使用,功能强大且可靠的系统来处理和分发数据. 本文讲述如何用NiFi将Http的Json数据传到Druid.国外的一篇文章讲到如何用NiFi将推文传到Druid,https://co ...

  9. Druid.io系列(九):数据摄入

    1. 概述 Druid的数据摄入主要包括两大类: 1. 实时输入摄入:包括Pull,Push两种 - Pull:需要启动一个RealtimeNode节点,通过不同的Firehose摄取不同种类的数据源 ...

随机推荐

  1. 增加kms计数

    @echo offset skms=10.15.68.62for %%i in (. . . . . . . . . . . . . . . . . . . . . . . . . .) do cal ...

  2. windows版本 rac 报错信息

    原因 - 安装程序无法在一个或多个节点上执行指定的脚本.这可能是由于在节点上执行脚本时出现异常错误. 操作 - 有关详细信息, 请查看日志文件 'C:\Users\ADMINI~1\AppData\L ...

  3. Python Matplotlib绘图库 安装

    一般我们在做科学计算的时候,首先会想到的是matlab,但是呢,一想到matlab安装包那么大,我就有点不想说什么了. Matplotlib 是python最著名的绘图库,它提供了一整套和matlab ...

  4. HDU 2255 奔小康赚大钱 (KM算法 模板题)

    奔小康赚大钱 Time Limit: 1000/1000 MS (Java/Others)    Memory Limit: 32768/32768 K (Java/Others)Total Subm ...

  5. 优先级:P0

    优先级:P0 Priority https://www.zhihu.com/question/23353333?sort=created 一个团队, 每周需求pk的例会,每个产品经理都要给自己提出的需 ...

  6. Java机器学习框架deeplearing4j入门教程

    1.添加项目maven添加依赖 or 导入jar包 or 使用jvm <project xmlns="http://maven.apache.org/POM/4.0.0" x ...

  7. bzoj1026windy数

    题目链接:http://www.lydsy.com/JudgeOnline/problem.php?id=1026 Description windy定义了一种windy数.不含前导零且相邻两个数字之 ...

  8. Mininet 系列实验(七)

    实验内容 本实验在基于 Mininet 脚本的不同拓扑环境下使用 OpenDaylight 控制交换机行为.任务一:一台交换机两台主机,从1端口进入的数据流转发到 2 端口,从 2 端口进入的数据流转 ...

  9. 【bzoj3730】 震波

    http://www.lydsy.com/JudgeOnline/problem.php?id=3730 (题目链接) 题意 给出一棵树,每个节点又一个权值.两个操作,询问距离节点${x}$不超过${ ...

  10. 解题:SCOI 2008 配对

    题面 如果没有两个数不能相同这个限制就两个数组排序后贪心即可.现在加上这个限制,注意到每个数组中的数是两两不同的,所以每次一定能在前面或后面一个数中找一个换过来,这样每次考虑相邻三个数转移就可以了,注 ...