【原创】大叔问题定位分享(33)beeline连接presto报错
hive2.3.4
presto0.215
使用hive2.3.4的beeline连接presto报错
$ beeline -d com.facebook.presto.jdbc.PrestoDriver -u "jdbc:presto://localhost:8080/hive"
Error: Unrecognized connection property 'url' (state=,code=0)
增加--verbose查看详细报错
$ beeline -d com.facebook.presto.jdbc.PrestoDriver -u "jdbc:presto://localhost:8080/hive" --verbose
Error: Unrecognized connection property 'url' (state=,code=0)
java.sql.SQLException: Unrecognized connection property 'url'
at com.facebook.presto.jdbc.PrestoDriverUri.validateConnectionProperties(PrestoDriverUri.java:316)
at com.facebook.presto.jdbc.PrestoDriverUri.<init>(PrestoDriverUri.java:95)
at com.facebook.presto.jdbc.PrestoDriverUri.<init>(PrestoDriverUri.java:85)
at com.facebook.presto.jdbc.PrestoDriver.connect(PrestoDriver.java:87)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
at org.apache.hive.beeline.Commands.connect(Commands.java:1641)
at org.apache.hive.beeline.Commands.connect(Commands.java:1536)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)
at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1273)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1312)
at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:867)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:776)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1010)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
这里是connect过程报错,connect参数为
Usage: connect <url> <username> <password> [driver]
跟进hive和presto代码
org.apache.hive.beeline.BeeLine
boolean dispatch(String line) {
...
if (isBeeLine) {
if (line.startsWith(COMMAND_PREFIX)) {
// handle SQLLine command in beeline which starts with ! and does not end with ;
return execCommandWithPrefix(line);
} else {
return commands.sql(line, getOpts().getEntireLineAsCommand());
}
} else {
return commands.sql(line, getOpts().getEntireLineAsCommand());
}
beeline中的命令和sql都会经过dispatch,命令执行的是execCommandWithPrefix,然后会反射调用Commands
org.apache.hive.beeline.Commands
public boolean connect(String line) throws Exception {
String example = "Usage: connect <url> <username> <password> [driver]"
+ BeeLine.getSeparator(); String[] parts = beeLine.split(line);
if (parts == null) {
return false;
} if (parts.length < 2) {
return beeLine.error(example);
} String url = parts.length < 2 ? null : parts[1];
String user = parts.length < 3 ? null : parts[2];
String pass = parts.length < 4 ? null : parts[3];
String driver = parts.length < 5 ? null : parts[4]; Properties props = new Properties();
if (url != null) {
String saveUrl = getUrlToUse(url);
props.setProperty(JdbcConnectionParams.PROPERTY_URL, saveUrl);
} String value = null;
if (driver != null) {
props.setProperty(JdbcConnectionParams.PROPERTY_DRIVER, driver);
} else {
value = Utils.parsePropertyFromUrl(url, JdbcConnectionParams.PROPERTY_DRIVER);
if (value != null) {
props.setProperty(JdbcConnectionParams.PROPERTY_DRIVER, value);
}
} if (user != null) {
props.setProperty(JdbcConnectionParams.AUTH_USER, user);
} else {
value = Utils.parsePropertyFromUrl(url, JdbcConnectionParams.AUTH_USER);
if (value != null) {
props.setProperty(JdbcConnectionParams.AUTH_USER, value);
}
} if (pass != null) {
props.setProperty(JdbcConnectionParams.AUTH_PASSWD, pass);
} else {
value = Utils.parsePropertyFromUrl(url, JdbcConnectionParams.AUTH_PASSWD);
if (value != null) {
props.setProperty(JdbcConnectionParams.AUTH_PASSWD, value);
}
} value = Utils.parsePropertyFromUrl(url, JdbcConnectionParams.AUTH_TYPE);
if (value != null) {
props.setProperty(JdbcConnectionParams.AUTH_TYPE, value);
}
return connect(props);
} public boolean connect(Properties props) throws IOException {
String url = getProperty(props, new String[] {
JdbcConnectionParams.PROPERTY_URL,
"javax.jdo.option.ConnectionURL",
"ConnectionURL",
});
String driver = getProperty(props, new String[] {
JdbcConnectionParams.PROPERTY_DRIVER,
"javax.jdo.option.ConnectionDriverName",
"ConnectionDriverName",
});
String username = getProperty(props, new String[] {
JdbcConnectionParams.AUTH_USER,
"javax.jdo.option.ConnectionUserName",
"ConnectionUserName",
});
String password = getProperty(props, new String[] {
JdbcConnectionParams.AUTH_PASSWD,
"javax.jdo.option.ConnectionPassword",
"ConnectionPassword",
}); if (url == null || url.length() == 0) {
return beeLine.error("Property \"url\" is required");
}
if (driver == null || driver.length() == 0) {
if (!beeLine.scanForDriver(url)) {
return beeLine.error(beeLine.loc("no-driver", url));
}
} String auth = getProperty(props, new String[] {JdbcConnectionParams.AUTH_TYPE});
if (auth == null) {
auth = beeLine.getOpts().getAuthType();
if (auth != null) {
props.setProperty(JdbcConnectionParams.AUTH_TYPE, auth);
}
} beeLine.info("Connecting to " + url);
if (Utils.parsePropertyFromUrl(url, JdbcConnectionParams.AUTH_PRINCIPAL) == null) {
String urlForPrompt = url.substring(0, url.contains(";") ? url.indexOf(';') : url.length());
if (username == null) {
username = beeLine.getConsoleReader().readLine("Enter username for " + urlForPrompt + ": ");
}
props.setProperty(JdbcConnectionParams.AUTH_USER, username);
if (password == null) {
password = beeLine.getConsoleReader().readLine("Enter password for " + urlForPrompt + ": ",
new Character('*'));
}
props.setProperty(JdbcConnectionParams.AUTH_PASSWD, password);
} try {
beeLine.getDatabaseConnections().setConnection(
new DatabaseConnection(beeLine, driver, url, props));
beeLine.getDatabaseConnection().getConnection(); if (!beeLine.isBeeLine()) {
beeLine.updateOptsForCli();
}
beeLine.runInit(); beeLine.setCompletions();
beeLine.getOpts().setLastConnectedUrl(url);
return true;
} catch (SQLException sqle) {
beeLine.getDatabaseConnections().remove();
return beeLine.error(sqle);
} catch (IOException ioe) {
return beeLine.error(ioe);
}
}
注意props字段的赋值过程
org.apache.hive.beeline.DatabaseConnection
public DatabaseConnection(BeeLine beeLine, String driver, String url,
Properties info) throws SQLException {
this.beeLine = beeLine;
this.driver = driver;
this.url = url;
this.info = info;
} boolean connect() throws SQLException {
...
Map<String, String> hiveVars = beeLine.getOpts().getHiveVariables();
if (hiveVars != null){
for (Map.Entry<String, String> var : hiveVars.entrySet()) {
info.put(HIVE_VAR_PREFIX + var.getKey(), var.getValue());
}
} Map<String, String> hiveConfVars = beeLine.getOpts().getHiveConfVariables();
if (hiveConfVars != null){
for (Map.Entry<String, String> var : hiveConfVars.entrySet()) {
info.put(HIVE_CONF_PREFIX + var.getKey(), var.getValue());
}
} if (isDriverRegistered) {
// if the driver registered in the driver manager, get the connection via the driver manager
setConnection(DriverManager.getConnection(getUrl(), info));
} else {
Commands中props赋值给DatabaseConnection的info
java.sql.DriverManager
public static Connection getConnection(String url,
java.util.Properties info) throws SQLException { return (getConnection(url, info, Reflection.getCallerClass()));
} private static Connection getConnection(
String url, java.util.Properties info, Class<?> caller) throws SQLException {
...
Connection con = aDriver.driver.connect(url, info);
DatabaseConnection的info传递给DriverManager然后传递给Driver,下面看具体的PrestoDriver
com.facebook.presto.jdbc.PrestoDriver
public Connection connect(String url, Properties info)
throws SQLException
{
if (!acceptsURL(url)) {
return null;
} PrestoDriverUri uri = new PrestoDriverUri(url, info); OkHttpClient.Builder builder = httpClient.newBuilder();
uri.setupClient(builder);
QueryExecutor executor = new QueryExecutor(builder.build()); return new PrestoConnection(uri, executor);
}
info传递给PrestoDriverUri进行初始化
com.facebook.presto.jdbc.PrestoDriverUri
public PrestoDriverUri(String url, Properties driverProperties)
throws SQLException
{
this(parseDriverUrl(url), driverProperties);
} private PrestoDriverUri(URI uri, Properties driverProperties)
throws SQLException
{
this.uri = requireNonNull(uri, "uri is null");
address = HostAndPort.fromParts(uri.getHost(), uri.getPort());
properties = mergeConnectionProperties(uri, driverProperties); validateConnectionProperties(properties); // enable SSL by default for standard port
useSecureConnection = SSL.getValue(properties).orElse(uri.getPort() == 443); initCatalogAndSchema();
} private static void validateConnectionProperties(Properties connectionProperties)
throws SQLException
{
for (String propertyName : connectionProperties.stringPropertyNames()) {
if (ConnectionProperties.forKey(propertyName) == null) {
throw new SQLException(format("Unrecognized connection property '%s'", propertyName));
}
} for (ConnectionProperty<?> property : ConnectionProperties.allProperties()) {
property.validate(connectionProperties);
}
}
所以报错“Unrecognized connection property 'url' ”意味着ConnectionProperties.forKey("url") == null
com.facebook.presto.jdbc.ConnectionProperties
ConnectionProperties中确实没有url,所以presto与hive的beeline不兼容;
尝试修改PrestoDriverUri.java,去掉校验:
com.facebook.presto.jdbc.PrestoDriverUri
private static void validateConnectionProperties(Properties connectionProperties)
throws SQLException
{
// for (String propertyName : connectionProperties.stringPropertyNames()) {
// if (ConnectionProperties.forKey(propertyName) == null) {
// throw new SQLException(format("Unrecognized connection property '%s'", propertyName));
// }
// } for (ConnectionProperty<?> property : ConnectionProperties.allProperties()) {
property.validate(connectionProperties);
}
}
将以上几行注释
简单修改jar包中的class的方法,参考:https://www.cnblogs.com/barneywill/p/10552322.html
编译的时候发现一直报错,
presto-jdbc/com/facebook/presto/jdbc/PrestoDriverUri.java:145: error: method setupSocksProxy in class OkHttpUtil cannot be applied to given types;
setupSocksProxy(builder, SOCKS_PROXY.getValue(properties));
^
required: Builder,Optional<com.google.common.net.HostAndPort>
found: Builder,Optional<com.facebook.presto.jdbc.internal.guava.net.HostAndPort>
reason: argument mismatch; Optional<com.facebook.presto.jdbc.internal.guava.net.HostAndPort> cannot be converted to Optional<com.google.common.net.HostAndPort>
presto-jdbc/com/facebook/presto/jdbc/PrestoDriverUri.java:146: error: method setupHttpProxy in class OkHttpUtil cannot be applied to given types;
setupHttpProxy(builder, HTTP_PROXY.getValue(properties));
^
required: Builder,Optional<com.google.common.net.HostAndPort>
found: Builder,Optional<com.facebook.presto.jdbc.internal.guava.net.HostAndPort>
reason: argument mismatch; Optional<com.facebook.presto.jdbc.internal.guava.net.HostAndPort> cannot be converted to Optional<com.google.common.net.HostAndPort>
2 errors
然后发现原来是presto-jdbc打包的时候设置了relocation,会将一些包做shade并且改package,详见:
https://github.com/prestodb/presto/blob/master/presto-jdbc/pom.xml
<properties>
<air.main.basedir>${project.parent.basedir}</air.main.basedir>
<shadeBase>com.facebook.presto.jdbc.internal</shadeBase>
</properties>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createSourcesJar>true</createSourcesJar>
<shadeSourcesContent>true</shadeSourcesContent>
<dependencyReducedPomLocation>${project.build.directory}/pom.xml</dependencyReducedPomLocation>
<relocations>
<relocation>
<pattern>com.facebook.presto.client</pattern>
<shadedPattern>${shadeBase}.client</shadedPattern>
</relocation>
<relocation>
<pattern>com.facebook.presto.spi</pattern>
<shadedPattern>${shadeBase}.spi</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>${shadeBase}.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>${shadeBase}.guava</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<shadedPattern>${shadeBase}.guava</shadedPattern>
</relocation>
<relocation>
<pattern>io.airlift</pattern>
<shadedPattern>${shadeBase}.airlift</shadedPattern>
</relocation>
<relocation>
<pattern>javax.annotation</pattern>
<shadedPattern>${shadeBase}.javax.annotation</shadedPattern>
</relocation>
<relocation>
<pattern>javax.inject</pattern>
<shadedPattern>${shadeBase}.inject</shadedPattern>
</relocation>
<relocation>
<pattern>org.openjdk.jol</pattern>
<shadedPattern>${shadeBase}.jol</shadedPattern>
</relocation>
<relocation>
<pattern>org.joda.time</pattern>
<shadedPattern>${shadeBase}.joda.time</shadedPattern>
</relocation>
<relocation>
<pattern>okhttp3</pattern>
<shadedPattern>${shadeBase}.okhttp3</shadedPattern>
</relocation>
<relocation>
<pattern>okio</pattern>
<shadedPattern>${shadeBase}.okio</shadedPattern>
</relocation>
</relocations>
所以需要手工调整源代码中的package
okhttp3 -> com.facebook.presto.jdbc.internal.okhttp3
com.google.common -> com.facebook.presto.jdbc.internal.guava
com.facebook.presto.client -> com.facebook.presto.jdbc.internal.client
源代码地址:
https://github.com/prestodb/presto/blob/master/presto-jdbc/src/main/java/com/facebook/presto/jdbc/PrestoDriverUri.java
然后再编译就ok了,编译命令:
# javac -cp presto-jdbc-0.215.jar -d presto-jdbc presto-jdbc/com/facebook/presto/jdbc/PrestoDriverUri.java
然后重新打包,再使用beeline连接就ok了
【原创】大叔问题定位分享(33)beeline连接presto报错的更多相关文章
- 【原创】大叔经验分享(38)beeline连接hiveserver2报错impersonate
beeline连接hiveserver2报错 Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost: ...
- beeline链接hive报错
看问题:beeline连接hiveserver2报错.连接串:hive --service beeline -u jdbc:hive2://localhost:10000/hive 错误:Error ...
- PHP连接MySQL报错:SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket 'MySQL' (2)
如下所示,PHP连接MySQL报错: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket 'MySQL' ...
- redis运用连接池报错解决
redis使用连接池报错解决redis使用十几小时就一直报异常 redis.clients.jedis.exceptions.JedisConnectionException: Could not g ...
- Asp.Net连接Mysql报错Out of sync with server
Asp.Net连接Mysql报错Out of sync with server 原因:程序引用的MySql.Data.dll版本高于服务器版本 解决:下载一个低版本的MySql.Data.dll,项目 ...
- Atom远程连接服务器报错服务器版本和客户端版本不一致
Atom远程连接服务器 报错信息: Server version is different than client version Original error message: Version mi ...
- Navicat连接Mysql报错:Client does not support authentication protocol requested by server;
Navicat连接Mysql报错:Client does not support authentication protocol requested by server: 刚安装Mysql,想用Nav ...
- java项目连接jdbc报错:com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server
java项目连接jdbc报错:com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not creat ...
- 【centOS】【xshell】xshell连接虚拟机上的centOS,操作途中突然断开连接,报错:connect closed by foreign host
如题 xshell连接虚拟机上的centOS,操作途中突然断开连接,报错:connect closed by foreign host 快捷解决方法: 在虚拟机上centOS重新启动网络,即可解决问 ...
随机推荐
- Android adb shell 常用命令
Ø 简介 adb 命令是 Android SDK 中自带的一个调试工具,可以调试电脑中的手机或者模拟器,使用 adb 命令前需要将 sdk 中的 platform-tools 目录添加到环境变量中. ...
- 在Linux上安装Git
Git是一个开源的分布式版本控制系统,可以有效.高速的处理从很小到非常大的项目版本管理.而国外的GitHub和国内的Coding都是项目的托管平台.但是在使用Git工具的时候,第一步要学会如何安装gi ...
- XXE攻防技术
http://bobao.360.cn/learning/detail/3841.html http://www.freebuf.com/articles/web/97833.html http:// ...
- Spring 快速开始 启动Spring
[启动Spring必须配置] [web.xml部署描述符方式] 1.配置Servlet级别上下文 <servlet> <servlet-name>springDispatche ...
- git 出现 fatal: refusing to merge unrelated histories 错误
git pull 失败 ,提示:fatal: refusing to merge unrelated histories 其实这个问题是因为 两个 根本不相干的 git 库, 一个是本地库, 一个是远 ...
- Matconvnet环境配置一些坑
1.先安装VS再安装matlab否则安装失败 2.cuda7.5支持MATLABR2016a/b,支持VS2013但是不支持VS2015 3.cuda8.0支持MABTLABR2017a,对应编译需V ...
- vue-组件命名
vue的组件命名,不能带有大写字母. 正确的写法: components:{ 'myder':av } 错误写法: components:{ 'myDer':av }
- c2d遮罩
ClippingNode 节点 ClippingNode setStencil 设置模版 只有模版的区域显示此节点内容 使用了Opengl的 模板测试 http://www.cnblogs.com ...
- ubuntu完全卸载mysql
可以先用 dpkg --list|grep mysql 查看自己的mysql有哪些依赖 一.先卸载 mysql-common sudo apt-get remove mysql-common 二.然后 ...
- Hbase思维导图之物理模型