【异常】org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:

1 Phoenix远程无法连接但是本地可以连接，详细异常

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/zhangjin/developSoftware/mavenRepository/org/slf4j/slf4j-log4j12/1.7./slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/zhangjin/mycode/wm/wm-bigdata-etl/zjars/phoenix-4.14.-cdh5.16.1-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=, exceptions:
Fri May  :: CST , null, java.net.SocketTimeoutException: callTimeout=, callDuration=: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=localhost,,, seqNum=
    at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:)
    at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:)
    at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:)
    at org.apache.phoenix.compile.CreateTableCompiler$.execute(CreateTableCompiler.java:)
    at org.apache.phoenix.jdbc.PhoenixStatement$.call(PhoenixStatement.java:)
    at org.apache.phoenix.jdbc.PhoenixStatement$.call(PhoenixStatement.java:)
    at org.apache.phoenix.call.CallRunner.run(CallRunner.java:)
    at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:)
    at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:)
    at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl$.call(ConnectionQueryServicesImpl.java:)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl$.call(ConnectionQueryServicesImpl.java:)
    at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:)
    at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:)
    at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:)
    at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:)
    at java.sql.DriverManager.getConnection(DriverManager.java:)
    at java.sql.DriverManager.getConnection(DriverManager.java:)
    at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:)
    at org.apache.phoenix.mapreduce.util.ConnectionUtil.getInputConnection(ConnectionUtil.java:)
    at org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getSelectColumnMetadataList(PhoenixConfigurationUtil.java:)
    at org.apache.phoenix.spark.PhoenixRDD.toDataFrame(PhoenixRDD.scala:)
    at org.apache.phoenix.spark.PhoenixRelation.schema(PhoenixRelation.scala:)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:)
    at org.apache.spark.sql.SQLContext.load(SQLContext.scala:)
    at com.wm.bigdata.spark.etl.ETLDemo$.main(ETLDemo.scala:)
    at com.wm.bigdata.spark.etl.ETLDemo.main(ETLDemo.scala)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=, exceptions:
Fri May  :: CST , null, java.net.SocketTimeoutException: callTimeout=, callDuration=: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=localhost,,, seqNum=
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:)
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:)
    at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:)
    at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:)
    at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:)
    at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:)
    at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:)
    at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:)
    at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:)
    ...  more
Caused by: java.net.SocketTimeoutException: callTimeout=, callDuration=: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=localhost,,, seqNum=
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:)
    at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
    at java.lang.Thread.run(Thread.java:)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:)
    at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:)
    ...  more
Process finished with exit code 

2 仔细观察异常信息，发现连接是localhost信息，如果是本机访问当然没有问题，但是远程访问肯定就有问题，知道问题所在，开始排查

3 语网友说是要设置dns，感觉太复杂了，

https://blog.51cto.com/yaoyinjie/650902

按照这个思路还需要设置dns，搞了半天也没搞明白，只好放弃

 

4 里面提到 Zookeeper客户端连接后，get /hbase/master

发现抛出的是localhost，定位到就是这里的问题，Phoenix通过Zookeeper的地址连接后，访问域名是根据Zookeeper节点里面存储的/hbase/master返回的值来定位的，那么现在就是想办法修改这个值

 

5 

重新初始化后，设置hbase-site.xml 添加nameserver配置 重新初始化

然后 查询是域名后，搞定，hbase-site.xml配置，我的主机名是hdp,全部改成主机名，然后删除Zookeeper中 /hbase目录，重启hbase，再进入Zookeeper终端

检查，如果是get /hbase/master  能看到主机名hdp 或者对应的，那就是成功了

  <property>
    <name>hbase.master.dns.nameserver</name>
    <value>hdp</value>
    <description>The host name or IP address of the name server (DNS)
      which a master should use to determine the host name used
      for communication and display purposes.
    </description>
  </property> 

  <property>
    <name>hbase.regionserver.dns.nameserver</name>
    <value>hdp</value>
    <description>The host name or IP address of the name server (DNS)
      which a region server should use to determine the host name used by the
      master for communication and display purposes.
    </description>
  </property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hdp:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>hdfs://hdp:60000</value>
</property>