hadoop的一个节点unused了。然后重启启动hadoop的服务，仍有有一个hadoop的节点起不来。多次重启hadoop和杀进程之后，发现hadoop的master和slave节点上的状态在切换，没有达到同步起停;当master起来的时候，slave节点上的hadoop就unused,  当slave节点上的hadoop状态为running的时候，master节点上的hadoop节点的状态就是unused.主从没法同步起停。

在数据库入库的时候，日志里报如下错误：

这个时候突然想到了hadoop的主从站点之后的关系是不是没有同步好，文件进入了安全模式。

进入到hadoop的bin目录下，执行退出安全模式的命令，第一次是在hadoop服务停了 之后，执行退出安全模式的命令显示没有连接成功。

然后启动hadoop，这个时候hadoop下面的都运行正常了，执行退出安全模式。

退出安全模式命令： hadoop dfsadmin -safemode leave

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.library.path=/opt/oracle/app/client/oracle/product/11.2.0/inoc/lib:/lib:/usr/lib:/usr/java/packages/lib/a
md64:/usr/lib64:/lib64:/lib:/usr/lib]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.io.tmpdir=/tmp]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.compiler=<NA>]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.name=Linux]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.arch=amd64]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.version=2.6.32.59-0.9-default]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.name=acrosspm]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.home=/home/acrosspm]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.dir=/opt/netwatcher/pm4h2/work/conf/pmpadmin]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Initiating client connection, connectString=10.215.133.36:15248 sessionTimeout=180000 watcher=hconnection]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [The identifier of this process is 35646@pmapp]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Opening socket connection to server pmweb.site/10.215.133.36:15248. Will not attempt to authenticate
using SASL (unknown error)]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Socket connection established to pmweb.site/10.215.133.36:15248, initiating session]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Session establishment complete on server pmweb.site/10.215.133.36:15248, sessionid = 0x35cdf7a17c400
28, negotiated timeout = 180000]
[2017-06-25 18:04:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:05:31] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:06:31] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:07:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:08:31] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:09:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:10:34] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:10:54] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:11:56] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:12:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:13:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:14:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:16:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:17:02] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:18:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:19:05] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:20:05] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:20:25] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:21:30] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:22:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:23:33] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:24:36] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:24:56] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:26:01] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:27:10] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:28:13] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:28:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:29:38] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:30:41] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:31:41] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:32:44] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:33:47] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:34:50] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:35:50] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:36:50] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:37:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:38:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:39:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:40:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:41:59] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:42:59] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:44:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:44:04] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 0 of 10 failed; retrying after sleep of 1000]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:05] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 1 of 10 failed; retrying after sleep of 1006]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:06] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 2 of 10 failed; retrying after sleep of 1005]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:07] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 3 of 10 failed; retrying after sleep of 2014]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:09] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 4 of 10 failed; retrying after sleep of 2015]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:11] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 5 of 10 failed; retrying after sleep of 4034]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:15] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 6 of 10 failed; retrying after sleep of 4007]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:19] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 7 of 10 failed; retrying after sleep of 8075]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:27] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 8 of 10 failed; retrying after sleep of 16070]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:43] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 9 of 10 failed; no more retrying.]

hadoop节点挂死的一次分析报表。的更多相关文章

  1. I2C 挂死，SDA一直为低问题分析【转】

    转自:https://blog.csdn.net/winitz/article/details/72460775 版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csd ...

  2. 记一次 .NET WPF布草管理系统 挂死分析

    一:背景 1. 讲故事 这几天看的 dump 有点多,有点伤神伤脑,晚上做梦都是dump,今天早上头晕晕的到公司就听到背后同事抱怨他负责的WPF程序挂死了,然后测试的小姑娘也跟着抱怨...嗨,也不知道 ...

  3. 记一次 .NET 某上市工业智造 CPU+内存+挂死 三高分析

    一:背景 1. 讲故事 上个月有位朋友加wx告知他的程序有挂死现象,询问如何进一步分析,截图如下: 看这位朋友还是有一定的分析基础,可能玩的少,缺乏一定的分析经验,当我简单分析之后,我发现这个dump ...

  4. 记一次 .NET 某纺织工厂 MES系统 API 挂死分析

    一:背景 1. 讲故事 这个月中旬,有位朋友加我wx求助他的程序线程占有率很高,寻求如何解决,截图如下: 说实话,和不同行业的程序员聊天还是蛮有意思的,广交朋友,也能扩大自己的圈子,朋友说他因为这个b ...

  5. MySQL 连接为什么挂死了？

    摘要:本次分享的是一次关于 MySQL 高可用问题的定位过程,其中曲折颇多但问题本身却比较有些代表性,遂将其记录以供参考. 一.背景 近期由测试反馈的问题有点多,其中关于系统可靠性测试提出的问题令人感 ...

  6. MySQL 连接为什么挂死了

    声明:本文为博主原创文章,由于已授权部分平台发表该文章(知乎.云社区),可能造成发布时间方面的困扰. 一.背景 近期由测试反馈的问题有点多,其中关于系统可靠性测试提出的问题令人感到头疼,一来这类问题有 ...

  7. 应用程序出现挂死，.NET Runtime at IP 791F7E06 (79140000) with exit code 80131506.

    工具出现挂死问题 1.问题描述 工具出现挂死问题,巡检IIS发现以下异常日志 现网系统日志: 事件类型:    错误 事件来源:    .NET Runtime 描述: Application: Di ...

  8. 关于用strace工具定位vrrpd进程有时会挂死的bug

    只做工作总结备忘之用. 正在烧镜像,稍总结一下进来改bug遇到的问题. 一个项目里要用到L3 switch的nat,vrrp功能,但实地测试中偶然出现write file挂死的情况,但不是必现.交付在 ...

  9. IIC挂死问题解决过程

    0.环境:arm CPU 带有IIC控制器作为slave端,带有调试串口. 1.bug表现:IIC slave 在系统启动后概率挂死,导致master无法detect到slave. 猜测1:认为IIC ...

随机推荐

  1. AQTime教程

    1 简介 AQTime和MemProof都是AutomatedQA旗下的产品,AQTime比MemProof提供了更丰富强大的功能.该产品含有完整的性能和调试工具集,能够收集程序运行时关键的性能信息和 ...

  2. http://my.oschina.net/u/1177710/blog/284608

    http://my.oschina.net/u/1177710/blog/284608 http://chuhanzhi.com/?p=45 http://www.2cto.com/kf/201501 ...

  3. js怎么获取图片的相对地址

    <!DOCTYPE html> <html> <head> <meta http-equiv="content-type" content ...

  4. linux解压分卷压缩的zip文件

    zip -s 0 records.zip --out 1.zip unzip 1.zip

  5. java实现快速排序算法

    1.算法概念. 快速排序(Quicksort)是对冒泡排序的一种改进.由C. A. R. Hoare在1962年提出.2.算法思想. 通过一趟排序将要排序的数据分割成独立的两部分,其中一部分的所有数据 ...

  6. chrome浏览器 提示Adobe Flash Player未安装的解决方法

    最近遇到了个flash player设置的一个问题,记录一下,可能不同浏览器版本和设置不一样 浏览器版本:版本 61.0.3163.100(正式版本) (64 位) 打开需要flash player的 ...

  7. Office 如何双面打印Word文档

    打印之前勾选手动双面打印,然后开始打印(不管当前文档有几页,要打印几份,会只打印奇数页面)   只要开始打印奇数页面,就会有一个弹出窗口,当完成之后把打印的东西拿出来,整个翻面再放回打印机,点击确定会 ...

  8. Unity Mono foreach BUG性能测试

    # 环境 - Unity 4.6.4 / Windows # 测试代码 # 结果数据 # 结论 foreach存在bug,会导致GC,并且效率低下: 使用GetEnumerator代替,没有GC,并且 ...

  9. POJ1125 Stockbroker Grapevine 多源最短路

    题目大意 给定一个图,问从某一个顶点出发,到其它顶点的最短路的最大距离最短的情况下,是从哪个顶点出发?须要多久? (假设有人一直没有联络,输出disjoint) 解题思路 Floyd不解释 代码 #i ...

  10. 使用wamp访问localhost时查看项目地址不对

    使用wamp访问localhost时查看项目地址不对 直接点击访问不到,http://路径少了一个localhost. 怎么办呢? 找到wamp 的www 目录下的index.php 文件打开后 找到 ...