hbase0.92 数据修复
现有集群环境；8台region server、1台master server
早晨重启某一region server后，该server的region全部下线未恢复。
重启前
hbase(main):031:0> status 'simple'
7 live servers
ec2d-cntprocdn-03.mypna.com:60020 1556378718902
requestsPerSecond=0, numberOfOnlineRegions=5, usedHeapMB=737, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556378795513
requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=960, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556378718227
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=225, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556378767123
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1013, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556379153372
requestsPerSecond=0, numberOfOnlineRegions=394, usedHeapMB=1136, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556379080369
requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1751, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556587728098
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=143, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 442
将ec2d-cntprocdn-01节点重启后发现该节点热region server并未如期上线
hbase(main):009:0> status 'simple'
7 live servers
ec2d-cntprocdn-03.mypna.com:60020 1556378718902
requestsPerSecond=9, numberOfOnlineRegions=5, usedHeapMB=762, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556378795513
requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=981, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556378718227
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=244, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556378767123
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1031, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556588642407
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=126, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556379080369
requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1771, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556587728098
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=160, maxHeapMB=4480
0 dead servers
Aggregate load: 9, regions: 48
通过 status 'detailed'查看详细region信息
hbase(main):032:0> status 'detailed'
version 0.92.1
3 regionsInTransition
.META.,,1.1028785192 state=OPENING, ts=1556588320882, server=null
vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. state=PENDING_OPEN, ts=1556588026138, server=null
vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OPENING, ts=1556587702216, server=null
master coprocessors: []
发现元数据表'.META.' region无法正常上线
于是常识重启集群后再次查看状态，发现所有server的region下线
hbase(main):025:0> status 'simple'
7 live servers
ec2d-cntprocdn-07.mypna.com:60020 1556589413648
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556589532168
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556589462880
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556589822614
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-03.mypna.com:60020 1556589365893
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=131, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556589914551
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556589364969
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=138, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 0
查看表内容直接报“-ROOT-”表下线
base(main):034:0> scan 'stats_validation'
ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) hbase(main):035:0> scan '.META.'
ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
执行hbase hbck，发现也是-ROOT-,,0 offline
$ hbase hbck Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=, exceptions:
Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner.access$(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner$.connect(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner$.connect(MetaScanner.java:)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.getMetaEntries(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.doWork(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:)
其中'.META.'和'-ROOT-'表的原理和关系可以参考
https://songlee24.github.io/2015/08/15/root-and-meta-table-structure/
修复过程
停止hbase集群
$ stop-hbase.sh
执行hbase元数据修复，报错可以先不用管
$ hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair RROR: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
// :: ERROR util.HBaseFsck: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
java.io.FileNotFoundException: File does not exist: /hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea/.regioninfo
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:)
// :: ERROR util.HBaseFsck: Bailed out due to:
org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:)
启动集群，查看master的日志输出
Mon Apr :: EDT Starting master on ec2d-cntprocnn-
core file size (blocks, -c)
data seg size (kbytes, -d) unlimited
scheduling priority (-e)
file size (blocks, -f) unlimited
pending signals (-i)
max locked memory (kbytes, -l)
max memory size (kbytes, -m) unlimited
open files (-n)
pipe size ( bytes, -p)
POSIX message queues (bytes, -q)
real-time priority (-r)
stack size (kbytes, -s)
cpu time (seconds, -t) unlimited
max user processes (-u)
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
-- ::, WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
on.
-- ::, ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
-- ::, WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
on.
-- ::, ERROR org.mortbay.log: Error starting handlers
java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.format(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:)
at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:)
at org.mortbay.log.Log.warn(Log.java:)
at org.mortbay.component.AbstractLifeCycle.setFailed(AbstractLifeCycle.java:)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:)
at org.mortbay.jetty.Server.doStart(Server.java:)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:)
at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting on ec2d-cntprocdn-.mypna.com,, with ec2d-cntprocdn-.mypna.com,,
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 7eeecbe911ce6add0487d89ba53b7dcc has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 3fb0ccc6a9adc3f56e1f526f0d111672 has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3be02d44bcba07121d9f29494d66d959 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_poi_10066,,.3be02d44bcba07121d9f29494d66d959. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region e29ae55c52ac2d6644a117fdb0e9af19 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.e29ae55c52ac2d6644a117fdb0e9af19. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region c0dbd03ee61162038fc46f7c9f964585 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.c0dbd03ee61162038fc46f7c9f964585. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 96f2f63ed9de87542a8f643704bb0934 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.96f2f63ed9de87542a8f643704bb0934. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region aa2892d5c917027afcc68c7261879765 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.aa2892d5c917027afcc68c7261879765. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1e32343c54316020cb96f24bf6ce5cbe from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.1e32343c54316020cb96f24bf6ce5cbe. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 5f0e9542c55245ef429557447a03883e from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.5f0e9542c55245ef429557447a03883e. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3105e4d146648aff31262166669cd170 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.3105e4d146648aff31262166669cd170. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region b65d05bdbcf0938e281a0df1667d2ed5 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.b65d05bdbcf0938e281a0df1667d2ed5. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 230fef99d67d51aa66a0e7234e7b0766 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.230fef99d67d51aa66a0e7234e7b0766. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but re
gion was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 00c4a8cd6ed22995dadc8d6b7e901175 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.00c4a8cd6ed22995dadc8d6b7e901175. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region bd7401a936fb21c910f662b02e254190 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.bd7401a936fb21c910f662b02e254190. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 30711b0a58809200c20aa18d545aa94d from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.30711b0a58809200c20aa18d545aa94d. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1d4459d84c02de3b0f194f194a183569 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.1d4459d84c02de3b0f194f194a183569. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 46768cc418862ae474f2190bdc7bada5 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.46768cc418862ae474f2190bdc7bada5. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 9ad69996a4668c6cf93592dddc2d2752 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.9ad69996a4668c6cf93592dddc2d2752. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 2f1752fa24c76301bca2d63478b65381 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.2f1752fa24c76301bca2d63478b65381. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region dfd32ba7ea53cc76abf277e9eda8547e from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.dfd32ba7ea53cc76abf277e9eda8547e. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 965b2b3bb59e0b23cfebabd59c03da12 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.965b2b3bb59e0b23cfebabd59c03da12. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-.mypna.com,, due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:)
at $Proxy11.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
... more
-- ::, ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-.mypna.com,, due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:)
at $Proxy11.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
再次查看region信息
hbase(main):023:0* status 'simple'
7 live servers
ec2d-cntprocdn-01.mypna.com:60020 1556591551521
requestsPerSecond=0, numberOfOnlineRegions=393, usedHeapMB=1828, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556591001078
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=1688, maxHeapMB=4480
ec2d-cntprocdn-03.mypna.com:60020 1556591001666
requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=341, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556591062296
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=387, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556591098649
requestsPerSecond=0, numberOfOnlineRegions=10, usedHeapMB=363, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556591459397
requestsPerSecond=0, numberOfOnlineRegions=12, usedHeapMB=425, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556591049767
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=315, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 442
hbase0.92 数据修复的更多相关文章
- nutch-2.2.1 hadoop-1.2.1 hbase-0.92.1 集群部署
国内私募机构九鼎控股打造APP,来就送 20元现金领取地址:http://jdb.jiudingcapital.com/phone.html 内部邀请码:C8E245J (不写邀请码,没有现金送) 国 ...
- hbase-0.92.1表备份还原
原表结构和数据 hbase(main):021:0* describe 'test' DESCRIPTION ENABLED {NAME => ', TTL = true > ', COM ...
- nutch-2.2.1 hadoop-1.2.1 hbase-0.92.1 集群部署(实用)
原文地址: http://www.cnblogs.com/i80386/p/3540389.html 参考网站:http://blog.csdn.net/weijonathan/article/det ...
- hbase-0.92.1集群部署
环境 主机名 IP地址 角色 安装目录 sht-sgmhadoopnn-01 172.16.101.55 NameNode.SecondaryNameNode. JobTracker.HMaster ...
- 详解SQL Server数据修复命令DBCC的使用
严重级别为 21 表示可能存在数据损坏. 可能的原因包括损坏的页链.损坏的 IAM 或该对象的 sys.objects目录视图中存在无效条目. 这些错误通常由硬件或磁盘设备驱动程序故障而引起. MS ...
- 某企业用友U8+中勒索病毒后数据修复及重新实施过程记录
近期某客户中了勒索病毒,虽然前期多次提醒客户注意异地备份,但始终未执行,导致悲剧. 经过几天的努力,该客户信息系统已基本恢复正常运行,现将相关过程记录如下,作为警示. 方案抉择 交赎金解密:风险过高, ...
- 通用高效的数据修复方法：Row level repair
导读:随着大数据的进一步发展,NoSQL 数据库系统迅速发展并得到了广泛的应用.其中,Apache Cassandra 是最广泛使用的数据库之一.对于 Cassandra 的优化是大家研究的热点,而 ...
- Python 爬取 热词并进行分类数据分析-[数据修复]
日期:2020.02.01 博客期:140 星期六 [本博客的代码如若要使用,请在下方评论区留言,之后再用(就是跟我说一声)] 所有相关跳转: a.[简单准备] b.[云图制作+数据导入] c.[拓扑 ...
- mongodb数据修复宝典
mongodb数据修复宝典 1. 版本信息 mongodb版本: 3.6.14 2.mongodb日志内容: 2019-10-04T10:41:38.558+0000 E STORAGE [inita ...
随机推荐
- 异步fifo的读写
这里不讨论异步fifo是如何实现的,而是在实现fifo的前提下,对fifo的读写. 现在遇到的问题是:总线的数据不能写入fifo中,但是地址能加一. 代码如下: if( !fifo_tx_full & ...
- Deep Learning 学习笔记（3）：Linear Regression 数据的预处理
为了获得良好的收敛,在进行梯度下降前,我们可以对数据进行预处理. 目标是使得数据大小在同一个数据数量级上,均值为零. 一般将数据放缩到(-1,1)区间, 我们可以对数据进行如下操作: 其中u1是数据的 ...
- 聚类 高维聚类 聚类评估标准 EM模型聚类
高维数据的聚类分析 高维聚类研究方向 高维数据聚类的难点在于: 1.适用于普通集合的聚类算法,在高维数据集合中效率极低 2.由于高维空间的稀疏性以及最近邻特性,高维的空间中基本不存在数据簇. 在高维聚 ...
- 二进制(signed or unsigned)补码
在计算机系统中,数值一律用补码来表示(存储). 主要原因:使用补码,可以将符号位和其它位统一处理:同时,减法也可按加法来处理.另外,两个用补 码表示的数相加时,如果最高位(符号位)有进位,则进位被舍弃 ...
- Ubantu下安装adobe flash player插件
用火狐看视频,要打开Adobe官网下载xxxx,太麻烦. 可以在Terminal下输入: apt-get install flashplugin-nonfree 好了.
- ubuntu14.10下Qt5.4无法输入中文
最近学习Qt,于是在ubuntu下安装了开发环境.我是从官网上下载安装的Qt5.4版本.安装后发现在开发过程中无法输入中文.于是在网上搜了搜.解决办法如下: 1.安装fcitx-frontend-qt ...
- JBPM具体应用之decision节点的使用
JBPM工作流引擎为我们提供了许多的节点应用,每一个节点都有其不同的作用,其中有四个比较常用的节点,他们分别decision,fork,state和task.在本文中我们先介绍decision节点,余 ...
- 【原】Coursera—Andrew Ng机器学习—课程笔记 Lecture 1_Introduction and Basic Concepts 介绍和基本概念
目录 1.1 欢迎1.2 机器学习是什么 1.2.1 机器学习定义 1.2.2 机器学习算法 - Supervised learning 监督学习 - Unsupervised learning 无 ...
- 【bzoj1444】[Jsoi2009]有趣的游戏
1444: [Jsoi2009]有趣的游戏 Time Limit: 10 Sec Memory Limit: 64 MBSubmit: 1007 Solved: 334[Submit][Statu ...
- laravel中间键组
` php artisan make:middleware Lend这边我定义一个登陆的中间件这边要注意的就是,当条件成立的时候一定要 return $next($request);不写这个larav ...