hbase0.92 数据修复
现有集群环境；8台region server、1台master server
早晨重启某一region server后，该server的region全部下线未恢复。
重启前
hbase(main):031:0> status 'simple'
7 live servers
ec2d-cntprocdn-03.mypna.com:60020 1556378718902
requestsPerSecond=0, numberOfOnlineRegions=5, usedHeapMB=737, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556378795513
requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=960, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556378718227
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=225, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556378767123
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1013, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556379153372
requestsPerSecond=0, numberOfOnlineRegions=394, usedHeapMB=1136, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556379080369
requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1751, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556587728098
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=143, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 442
将ec2d-cntprocdn-01节点重启后发现该节点热region server并未如期上线
hbase(main):009:0> status 'simple'
7 live servers
ec2d-cntprocdn-03.mypna.com:60020 1556378718902
requestsPerSecond=9, numberOfOnlineRegions=5, usedHeapMB=762, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556378795513
requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=981, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556378718227
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=244, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556378767123
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1031, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556588642407
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=126, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556379080369
requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1771, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556587728098
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=160, maxHeapMB=4480
0 dead servers
Aggregate load: 9, regions: 48
通过 status 'detailed'查看详细region信息
hbase(main):032:0> status 'detailed'
version 0.92.1
3 regionsInTransition
.META.,,1.1028785192 state=OPENING, ts=1556588320882, server=null
vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. state=PENDING_OPEN, ts=1556588026138, server=null
vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OPENING, ts=1556587702216, server=null
master coprocessors: []
发现元数据表'.META.' region无法正常上线
于是常识重启集群后再次查看状态，发现所有server的region下线
hbase(main):025:0> status 'simple'
7 live servers
ec2d-cntprocdn-07.mypna.com:60020 1556589413648
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556589532168
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556589462880
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556589822614
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-03.mypna.com:60020 1556589365893
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=131, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556589914551
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556589364969
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=138, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 0
查看表内容直接报“-ROOT-”表下线
base(main):034:0> scan 'stats_validation'
ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) hbase(main):035:0> scan '.META.'
ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
执行hbase hbck，发现也是-ROOT-,,0 offline
$ hbase hbck Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=, exceptions:
Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner.access$(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner$.connect(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner$.connect(MetaScanner.java:)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.getMetaEntries(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.doWork(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:)
其中'.META.'和'-ROOT-'表的原理和关系可以参考
https://songlee24.github.io/2015/08/15/root-and-meta-table-structure/
修复过程
停止hbase集群
$ stop-hbase.sh
执行hbase元数据修复，报错可以先不用管
$ hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair RROR: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
// :: ERROR util.HBaseFsck: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
java.io.FileNotFoundException: File does not exist: /hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea/.regioninfo
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:)
// :: ERROR util.HBaseFsck: Bailed out due to:
org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:)
启动集群，查看master的日志输出
Mon Apr :: EDT Starting master on ec2d-cntprocnn-
core file size (blocks, -c)
data seg size (kbytes, -d) unlimited
scheduling priority (-e)
file size (blocks, -f) unlimited
pending signals (-i)
max locked memory (kbytes, -l)
max memory size (kbytes, -m) unlimited
open files (-n)
pipe size ( bytes, -p)
POSIX message queues (bytes, -q)
real-time priority (-r)
stack size (kbytes, -s)
cpu time (seconds, -t) unlimited
max user processes (-u)
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
-- ::, WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
on.
-- ::, ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
-- ::, WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
on.
-- ::, ERROR org.mortbay.log: Error starting handlers
java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.format(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:)
at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:)
at org.mortbay.log.Log.warn(Log.java:)
at org.mortbay.component.AbstractLifeCycle.setFailed(AbstractLifeCycle.java:)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:)
at org.mortbay.jetty.Server.doStart(Server.java:)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:)
at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting on ec2d-cntprocdn-.mypna.com,, with ec2d-cntprocdn-.mypna.com,,
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 7eeecbe911ce6add0487d89ba53b7dcc has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 3fb0ccc6a9adc3f56e1f526f0d111672 has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3be02d44bcba07121d9f29494d66d959 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_poi_10066,,.3be02d44bcba07121d9f29494d66d959. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region e29ae55c52ac2d6644a117fdb0e9af19 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.e29ae55c52ac2d6644a117fdb0e9af19. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region c0dbd03ee61162038fc46f7c9f964585 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.c0dbd03ee61162038fc46f7c9f964585. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 96f2f63ed9de87542a8f643704bb0934 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.96f2f63ed9de87542a8f643704bb0934. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region aa2892d5c917027afcc68c7261879765 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.aa2892d5c917027afcc68c7261879765. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1e32343c54316020cb96f24bf6ce5cbe from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.1e32343c54316020cb96f24bf6ce5cbe. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 5f0e9542c55245ef429557447a03883e from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.5f0e9542c55245ef429557447a03883e. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3105e4d146648aff31262166669cd170 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.3105e4d146648aff31262166669cd170. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region b65d05bdbcf0938e281a0df1667d2ed5 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.b65d05bdbcf0938e281a0df1667d2ed5. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 230fef99d67d51aa66a0e7234e7b0766 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.230fef99d67d51aa66a0e7234e7b0766. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but re
gion was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 00c4a8cd6ed22995dadc8d6b7e901175 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.00c4a8cd6ed22995dadc8d6b7e901175. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region bd7401a936fb21c910f662b02e254190 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.bd7401a936fb21c910f662b02e254190. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 30711b0a58809200c20aa18d545aa94d from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.30711b0a58809200c20aa18d545aa94d. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1d4459d84c02de3b0f194f194a183569 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.1d4459d84c02de3b0f194f194a183569. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 46768cc418862ae474f2190bdc7bada5 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.46768cc418862ae474f2190bdc7bada5. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 9ad69996a4668c6cf93592dddc2d2752 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.9ad69996a4668c6cf93592dddc2d2752. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 2f1752fa24c76301bca2d63478b65381 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.2f1752fa24c76301bca2d63478b65381. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region dfd32ba7ea53cc76abf277e9eda8547e from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.dfd32ba7ea53cc76abf277e9eda8547e. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 965b2b3bb59e0b23cfebabd59c03da12 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.965b2b3bb59e0b23cfebabd59c03da12. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-.mypna.com,, due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:)
at $Proxy11.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
... more
-- ::, ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-.mypna.com,, due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:)
at $Proxy11.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
再次查看region信息
hbase(main):023:0* status 'simple'
7 live servers
ec2d-cntprocdn-01.mypna.com:60020 1556591551521
requestsPerSecond=0, numberOfOnlineRegions=393, usedHeapMB=1828, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556591001078
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=1688, maxHeapMB=4480
ec2d-cntprocdn-03.mypna.com:60020 1556591001666
requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=341, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556591062296
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=387, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556591098649
requestsPerSecond=0, numberOfOnlineRegions=10, usedHeapMB=363, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556591459397
requestsPerSecond=0, numberOfOnlineRegions=12, usedHeapMB=425, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556591049767
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=315, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 442
hbase0.92 数据修复的更多相关文章
- nutch-2.2.1 hadoop-1.2.1 hbase-0.92.1 集群部署
国内私募机构九鼎控股打造APP,来就送 20元现金领取地址:http://jdb.jiudingcapital.com/phone.html 内部邀请码:C8E245J (不写邀请码,没有现金送) 国 ...
- hbase-0.92.1表备份还原
原表结构和数据 hbase(main):021:0* describe 'test' DESCRIPTION ENABLED {NAME => ', TTL = true > ', COM ...
- nutch-2.2.1 hadoop-1.2.1 hbase-0.92.1 集群部署(实用)
原文地址: http://www.cnblogs.com/i80386/p/3540389.html 参考网站:http://blog.csdn.net/weijonathan/article/det ...
- hbase-0.92.1集群部署
环境 主机名 IP地址 角色 安装目录 sht-sgmhadoopnn-01 172.16.101.55 NameNode.SecondaryNameNode. JobTracker.HMaster ...
- 详解SQL Server数据修复命令DBCC的使用
严重级别为 21 表示可能存在数据损坏. 可能的原因包括损坏的页链.损坏的 IAM 或该对象的 sys.objects目录视图中存在无效条目. 这些错误通常由硬件或磁盘设备驱动程序故障而引起. MS ...
- 某企业用友U8+中勒索病毒后数据修复及重新实施过程记录
近期某客户中了勒索病毒,虽然前期多次提醒客户注意异地备份,但始终未执行,导致悲剧. 经过几天的努力,该客户信息系统已基本恢复正常运行,现将相关过程记录如下,作为警示. 方案抉择 交赎金解密:风险过高, ...
- 通用高效的数据修复方法：Row level repair
导读:随着大数据的进一步发展,NoSQL 数据库系统迅速发展并得到了广泛的应用.其中,Apache Cassandra 是最广泛使用的数据库之一.对于 Cassandra 的优化是大家研究的热点,而 ...
- Python 爬取 热词并进行分类数据分析-[数据修复]
日期:2020.02.01 博客期:140 星期六 [本博客的代码如若要使用,请在下方评论区留言,之后再用(就是跟我说一声)] 所有相关跳转: a.[简单准备] b.[云图制作+数据导入] c.[拓扑 ...
- mongodb数据修复宝典
mongodb数据修复宝典 1. 版本信息 mongodb版本: 3.6.14 2.mongodb日志内容: 2019-10-04T10:41:38.558+0000 E STORAGE [inita ...
随机推荐
- linux lcd设备驱动剖析一
s3c2440 lcd驱动源码文件是:drivers/video/s3c2410fb.c 看驱动源码首先当然是先看入口函数,这里是s3c2410fb_init函数 [cpp] view plain? ...
- 优化深度神经网络（二）优化算法 SGD Momentum RMSprop Adam
Coursera吴恩达<优化深度神经网络>课程笔记(2)-- 优化算法 深度机器学习中的batch的大小 深度机器学习中的batch的大小对学习效果有何影响? 1. Mini-batch ...
- 分布式爬虫搭建系列 之二-----神器PyCharm的安装
这里我们使用PyCharm作为开发工具,以下过程摘抄于:http://blog.csdn.net/qq_29883591/article/details/52664478 作者:陌上行走 Pytho ...
- Tuple、list的区别以及dict和set
元组(Tuple): 定义方法:使用小括号() 使用方法: count:可以统计某个元组段在整个元组中出现的次数 index:可以查询某个元组段在整个元组中的元组号 name_tuple = ('xi ...
- Java面向对象-Java类的继承及super关键字
Java面向对象-Java类的继承 1,继承定义以及基本使用 定义:子类能够继承父类的属性和方法: 注意点:Java中只支持单继承: 私有方法不能继承: 上代码: package com.java12 ...
- Bootstrap日期插件中文实现
Bootstrap的相关JS和CSS直接跳过. <script type="text/javascript" src="static/js/jquery-1.9.1 ...
- 斯坦福CS229机器学习课程笔记 part2：分类和逻辑回归 Classificatiion and logistic regression
Logistic Regression 逻辑回归 1.模型 逻辑回归解决的是分类问题,并且是二元分类问题(binary classification),y只有0,1两个取值.对于分类问题使用线性回归不 ...
- 管理授权&管理决策&管理组织&管理目标
[管理授权] 1.如果一个管理者不明白为公司培养人的责任,就很可能成为公司的瓶颈.这个问题的解决在于让管理者学会“授权”,把大部分自己紧抓不放的事情下放给部门内其他人做. 2.为一件事负责和亲自做是两 ...
- tomcat报503 或者无法启动应用
一般都是配置文件有问题,或者路径问题,或者jvm的参数路径问题.... 总之,报错实在是不清楚!这点比resin差远了!!
- Web项目开发性能优化解决方案
web开发性能优化---安全篇 1.ip验证 2.操作日志.安全日志.登录日志 3.SQL注入校验 4.权限管理 5.验证规范(前端.后端.数据库约束) 2014-10-29 08:04 2773 ...