hadoop的一个节点unused了。然后重启启动hadoop的服务，仍有有一个hadoop的节点起不来。多次重启hadoop和杀进程之后，发现hadoop的master和slave节点上的状态在切换，没有达到同步起停;当master起来的时候，slave节点上的hadoop就unused,  当slave节点上的hadoop状态为running的时候，master节点上的hadoop节点的状态就是unused.主从没法同步起停。

在数据库入库的时候，日志里报如下错误：

这个时候突然想到了hadoop的主从站点之后的关系是不是没有同步好，文件进入了安全模式。

进入到hadoop的bin目录下，执行退出安全模式的命令，第一次是在hadoop服务停了 之后，执行退出安全模式的命令显示没有连接成功。

然后启动hadoop，这个时候hadoop下面的都运行正常了，执行退出安全模式。

退出安全模式命令： hadoop dfsadmin -safemode leave

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.library.path=/opt/oracle/app/client/oracle/product/11.2.0/inoc/lib:/lib:/usr/lib:/usr/java/packages/lib/a
md64:/usr/lib64:/lib64:/lib:/usr/lib]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.io.tmpdir=/tmp]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.compiler=<NA>]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.name=Linux]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.arch=amd64]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.version=2.6.32.59-0.9-default]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.name=acrosspm]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.home=/home/acrosspm]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.dir=/opt/netwatcher/pm4h2/work/conf/pmpadmin]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Initiating client connection, connectString=10.215.133.36:15248 sessionTimeout=180000 watcher=hconnection]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [The identifier of this process is 35646@pmapp]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Opening socket connection to server pmweb.site/10.215.133.36:15248. Will not attempt to authenticate
using SASL (unknown error)]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Socket connection established to pmweb.site/10.215.133.36:15248, initiating session]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Session establishment complete on server pmweb.site/10.215.133.36:15248, sessionid = 0x35cdf7a17c400
28, negotiated timeout = 180000]
[2017-06-25 18:04:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:05:31] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:06:31] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:07:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:08:31] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:09:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:10:34] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:10:54] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:11:56] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:12:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:13:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:14:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:16:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:17:02] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:18:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:19:05] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:20:05] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:20:25] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:21:30] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:22:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:23:33] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:24:36] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:24:56] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:26:01] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:27:10] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:28:13] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:28:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:29:38] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:30:41] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:31:41] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:32:44] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:33:47] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:34:50] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:35:50] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:36:50] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:37:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:38:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:39:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:40:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:41:59] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:42:59] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:44:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:44:04] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 0 of 10 failed; retrying after sleep of 1000]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:05] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 1 of 10 failed; retrying after sleep of 1006]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:06] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 2 of 10 failed; retrying after sleep of 1005]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:07] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 3 of 10 failed; retrying after sleep of 2014]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:09] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 4 of 10 failed; retrying after sleep of 2015]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:11] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 5 of 10 failed; retrying after sleep of 4034]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:15] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 6 of 10 failed; retrying after sleep of 4007]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:19] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 7 of 10 failed; retrying after sleep of 8075]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:27] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 8 of 10 failed; retrying after sleep of 16070]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:43] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 9 of 10 failed; no more retrying.]

hadoop节点挂死的一次分析报表。的更多相关文章

  1. I2C 挂死，SDA一直为低问题分析【转】

    转自:https://blog.csdn.net/winitz/article/details/72460775 版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csd ...

  2. 记一次 .NET WPF布草管理系统 挂死分析

    一:背景 1. 讲故事 这几天看的 dump 有点多,有点伤神伤脑,晚上做梦都是dump,今天早上头晕晕的到公司就听到背后同事抱怨他负责的WPF程序挂死了,然后测试的小姑娘也跟着抱怨...嗨,也不知道 ...

  3. 记一次 .NET 某上市工业智造 CPU+内存+挂死 三高分析

    一:背景 1. 讲故事 上个月有位朋友加wx告知他的程序有挂死现象,询问如何进一步分析,截图如下: 看这位朋友还是有一定的分析基础,可能玩的少,缺乏一定的分析经验,当我简单分析之后,我发现这个dump ...

  4. 记一次 .NET 某纺织工厂 MES系统 API 挂死分析

    一:背景 1. 讲故事 这个月中旬,有位朋友加我wx求助他的程序线程占有率很高,寻求如何解决,截图如下: 说实话,和不同行业的程序员聊天还是蛮有意思的,广交朋友,也能扩大自己的圈子,朋友说他因为这个b ...

  5. MySQL 连接为什么挂死了？

    摘要:本次分享的是一次关于 MySQL 高可用问题的定位过程,其中曲折颇多但问题本身却比较有些代表性,遂将其记录以供参考. 一.背景 近期由测试反馈的问题有点多,其中关于系统可靠性测试提出的问题令人感 ...

  6. MySQL 连接为什么挂死了

    声明:本文为博主原创文章,由于已授权部分平台发表该文章(知乎.云社区),可能造成发布时间方面的困扰. 一.背景 近期由测试反馈的问题有点多,其中关于系统可靠性测试提出的问题令人感到头疼,一来这类问题有 ...

  7. 应用程序出现挂死，.NET Runtime at IP 791F7E06 (79140000) with exit code 80131506.

    工具出现挂死问题 1.问题描述 工具出现挂死问题,巡检IIS发现以下异常日志 现网系统日志: 事件类型:    错误 事件来源:    .NET Runtime 描述: Application: Di ...

  8. 关于用strace工具定位vrrpd进程有时会挂死的bug

    只做工作总结备忘之用. 正在烧镜像,稍总结一下进来改bug遇到的问题. 一个项目里要用到L3 switch的nat,vrrp功能,但实地测试中偶然出现write file挂死的情况,但不是必现.交付在 ...

  9. IIC挂死问题解决过程

    0.环境:arm CPU 带有IIC控制器作为slave端,带有调试串口. 1.bug表现:IIC slave 在系统启动后概率挂死,导致master无法detect到slave. 猜测1:认为IIC ...

随机推荐

  1. [转]SSIS中OLE DB Source中如何执行Store Procedure 以得到源数据

    本文转自:http://www.cnblogs.com/michaelxu/archive/2009/10/16/1584284.html 有很多人喜欢在OLE DB Source中执行Store P ...

  2. Delphi 对象模型学习笔记（转）

    摘要     Borland Object Pascal 对象模型(现在已经正是命名为 Delphi 语言)与其他 OOP 语言一样,都提供了一些基础服务: 如对象创建服务.对象释放服务.对象识别服务 ...

  3. 利用Nodejs & Cheerio & Request抓取Lofter美女图片

    还是参考了这篇文章: http://cnodejs.org/topic/54bdaac4514ea9146862abee 另外有上面文章 nodejs抓取网易公开课的一些经验. 代码如下,注意其中用到 ...

  4. Vuex内容解析和vue cli项目中使用状态管理模式Vuex

    中文文档:vuex官方中文网站 一.vuex里面都有些什么内容? const store = new Vuex.Store({ state: { name: 'weish', age: }, gett ...

  5. scrapy-splash抓取动态数据例子八

    一.介绍 本例子用scrapy-splash抓取界面网站给定关键字抓取咨询信息. 给定关键字:个性化:融合:电视 抓取信息内如下: 1.资讯标题 2.资讯链接 3.资讯时间 4.资讯来源 二.网站信息 ...

  6. 读书笔记--MapReduce 适用场景 及 常见应用

    原文:http://blog.csdn.net/smile0198/article/details/20578389 1.MR解决的问题的特点   MR是应大数据的背景产生,其解决的问题的共性为:大问 ...

  7. Animation学习笔记

    关于动画的实现,Android提供了Animation,在Android SDK介绍了2种Animation模式: 1. Tween Animation:通过对场景里的对象不断做图像变换(平移.缩放. ...

  8. Vi 编辑器

    Vi的用法 1.简介 Vi,是一个文本编辑器,可以执行输出.删除.查找.替换.块操作等众多文本操作,vi不是一个排版程序.不像word或WPS那样可以对字体.格式.段落等属性进行编排,他只是一个文本编 ...

  9. 网页制作，网站制作中put和get的区别

    Http定义了与服务器交互的不同方法,最基本的方法有4种,分别是GET,POST,PUT,DELETE.URL全称是资源描述符,我们可以这样认为:一个URL地址,它用于描述一个网络上的资源,而HTTP ...

  10. Redis源代码分析（五）--- sparkline微线图

    sparkline这个单词,我第一次看的时候.也不知道这什么意思啊,曾经根本没听过啊,可是这真真实实的出如今了redis的代码中了,刚刚開始以为这也是属于普通的队列嘛.就把他分在了struct包里了. ...