spark踩坑——dataframe写入hbase连接异常
最近测试环境基于shc[https://github.com/hortonworks-spark/shc]的hbase-connector总是异常连接不到zookeeper,看下报错日志:
18/06/20 10:45:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5175ab05 connecting to ZooKeeper ensemble=localhost:2181
18/06/20 10:45:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x6399f976 connecting to ZooKeeper ensemble=localhost:2181
18/06/20 10:45:02 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/06/20 10:45:02 INFO ZooKeeper: Client environment:host.name=hnode8
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.version=1.8.0_66
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.home=/usr/local/jdk1.8.0_66/jre
...
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.io.tmpdir=/data5/yarn/nm/usercache/hdfs/appcache/application_1527863132022_18903/container_e35_1527863132022_18903_01_000003/tmp
18/06/20 10:45:02 INFO ZooKeeper: Client environment:java.compiler=<NA>
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.name=Linux
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.arch=amd64
18/06/20 10:45:02 INFO ZooKeeper: Client environment:os.version=2.6.32-696.3.1.el6.x86_64
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.name=yarn
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.home=/var/lib/hadoop-yarn
18/06/20 10:45:02 INFO ZooKeeper: Client environment:user.dir=/data5/yarn/nm/usercache/hdfs/appcache/application_1527863132022_18903/container_e35_1527863132022_18903_01_000003
18/06/20 10:45:02 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5175ab050x0, quorum=localhost:2181, baseZNode=/hbase
18/06/20 10:45:02 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x6399f9760x0, quorum=localhost:2181, baseZNode=/hbase
18/06/20 10:45:02 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:02 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:02 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:02 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:03 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:03 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:03 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:03 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/06/20 10:45:04 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/20 10:45:04 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
可以观察到hbase-connector莫名其妙连接的是localhost:2181,检查所有的配置文件都没有错。同样的代码线上就正常运行,对比之下发现是缺少了hbase-site.xml的配置文件。
查找shc的issue发现已经有人提出这种问题了:
https://github.com/hortonworks-spark/shc/issues/227
大意是说,默认会连接localhost:2181,如果需要连接远程的hbase,只需要拷贝hbase-site.xml到类目录下即可。由于我这边还用了nameservice,因此hdfs-site.xml等配置文件也需要一同打包到Jar中。
spark踩坑——dataframe写入hbase连接异常的更多相关文章
- Spark踩坑记——数据库(Hbase+Mysql)
[TOC] 前言 在使用Spark Streaming的过程中对于计算产生结果的进行持久化时,我们往往需要操作数据库,去统计或者改变一些值.最近一个实时消费者处理任务,在使用spark streami ...
- [转]Spark 踩坑记:数据库(Hbase+Mysql)
https://cloud.tencent.com/developer/article/1004820 Spark 踩坑记:数据库(Hbase+Mysql) 前言 在使用Spark Streaming ...
- Spark踩坑记——数据库(Hbase+Mysql)转
转自:http://www.cnblogs.com/xlturing/p/spark.html 前言 在使用Spark Streaming的过程中对于计算产生结果的进行持久化时,我们往往需要操作数据库 ...
- Spark踩坑填坑-聚合函数-序列化异常
Spark踩坑填坑-聚合函数-序列化异常 一.Spark聚合函数特殊场景 二.spark sql group by 三.Spark Caused by: java.io.NotSerializable ...
- Spark踩坑记——Spark Streaming+Kafka
[TOC] 前言 在WeTest舆情项目中,需要对每天千万级的游戏评论信息进行词频统计,在生产者一端,我们将数据按照每天的拉取时间存入了Kafka当中,而在消费者一端,我们利用了spark strea ...
- Spark踩坑记:Spark Streaming+kafka应用及调优
前言 在WeTest舆情项目中,需要对每天千万级的游戏评论信息进行词频统计,在生产者一端,我们将数据按照每天的拉取时间存入了Kafka当中,而在消费者一端,我们利用了spark streaming从k ...
- Spark踩坑记——共享变量
[TOC] 前言 Spark踩坑记--初试 Spark踩坑记--数据库(Hbase+Mysql) Spark踩坑记--Spark Streaming+kafka应用及调优 在前面总结的几篇spark踩 ...
- Spark踩坑记——从RDD看集群调度
[TOC] 前言 在Spark的使用中,性能的调优配置过程中,查阅了很多资料,之前自己总结过两篇小博文Spark踩坑记--初试和Spark踩坑记--数据库(Hbase+Mysql),第一篇概况的归纳了 ...
- Spark踩坑记:共享变量
收录待用,修改转载已取得腾讯云授权 前言 前面总结的几篇spark踩坑博文中,我总结了自己在使用spark过程当中踩过的一些坑和经验.我们知道Spark是多机器集群部署的,分为Driver/Maste ...
随机推荐
- [vb.net]控制台进度条的示例
Private Sub ConsoleProcessBar() Dim isBreak As Boolean = False Dim colorBack As ConsoleColor = Conso ...
- Linux上安装java+tomcat+mysql运行环境
centos6.5jdk"1.7.0_79"mysql5.6apache-tomcat-7.0.53 安装jdk:1.先到oracle下载rpm包:jdk-7u80-linux-x ...
- 02 Django配置信息
2-1 基本配置信息 import os # 项目路径 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # ...
- telnet 命令使用方法详解,telnet命令怎么用
telnet 命令使用方法详解,telnet命令怎么用? 文章类型:电脑教程 原创:天诺时空 什么是Telnet? 对于Telnet的认识,不同的人持有不同的观点,可以把Telnet当成一种通信协 ...
- mysql学习之路_基础知识
Mysql php阶段将数据库分为三个阶 基础阶段: mysql数据库的基本操作(增删改查),以及一些高级操作(视图,触发器,函数,存储过程等),PHP操作没有sql数 ...
- web程序打包详解
重要更新:鉴于很多小伙伴们说看不到图,我这边换了几个浏览器看了下,都看得到的,估计是网速问题,请耐心等待,另外,为了更好的方便大家学习,特此提供源码以及一个word文档,word文档就是本文内容 ...
- org/apache/maven/cli/MavenCli : Unsupported major.minor version 51.0
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/maven/cli/Ma ...
- 《深入浅出MFC》系列之运行时类型识别(RTTI)
/********************************************************************************** 发布日期:2017-11-13 ...
- XCode中安装cocoapods步骤
Ruby是一种功能强大的面向对象的脚本语言 Gem是一个管理Ruby库和程序的标准包,它通过Ruby Gem(如 http://rubygems.org/ )源来查找.安装.升级和卸载软件包,非常的便 ...
- Magic Powder - 2 (CF 670_D)
http://codeforces.com/problemset/problem/670/D2 The term of this problem is the same as the previous ...