ambari metrics collector内置hbase目录位于

/usr/lib/ams-hbase

配置位于

/etc/ams-hbase/conf

通过ruby启动

/usr/lib/ams-hbase/bin/hirb.rb

实际的启动命令为

/usr/lib/ams-hbase/bin/hbase-daemon.sh --config /etc/ams-hbase/conf foreground_start master

但是启动一段时间报错:

java.lang.RuntimeException: Master not initialized after 200000ms
at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197)
at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3100)

查看日志发现第一个错:

2019-01-16 16:20:29,295 WARN [StoreOpener-1588230740-1] util.CommonFSUtils: FileSystem doesn't support setStoragePolicy; HDFS-6584, HDFS-9345 not available. This is normal and expected on earlier Hadoop versions.
java.lang.NoSuchMethodException: org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path, java.lang.String)
at java.lang.Class.getDeclaredMethod(Class.java:2130)
at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:533)
at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:514)
at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:482)
at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:193)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:258)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5571)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1025)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1022)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

这个错可以忽略,然后是第二个错:

2019-01-17 15:49:08,730 ERROR [Thread-24] master.HMaster: Failed to become active master
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1082)
at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:423)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:714)
at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1398)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:857)
at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568)
at java.lang.Thread.run(Thread.java:748)

这个可以通过设置hbase.unsafe.stream.capability.enforce=false解决,详见 https://www.cnblogs.com/barneywill/p/10283076.html

最后发现很多warn:

2019-01-17 18:22:12,657 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:13,660 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:15,660 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:19,661 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:27,661 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:43,662 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.

清空hbase.rootdir,hbase.tmp.dir,hbase.zookeeper.property.dataDir这3个目录,同时设置好用户权限,然后重启hbase即可;

【原创】大叔问题定位分享(25)ambari metrics collector内置standalone hbase启动失败的更多相关文章

  1. 【原创】大叔问题定位分享(24)hbase standalone方式启动报错

    hbase 2.0.2 hbase standalone方式启动报错: 2019-01-17 15:49:08,730 ERROR [Thread-24] master.HMaster: Failed ...

  2. 【原创】大叔问题定位分享(23)Ambari安装向导点击下一步卡住

    ambari安装第一步是输入集群name,点击next时页面卡住不动,如下图: 注意到其中一个接口请求结果异常,http://ambari.server:8080/api/v1/version_def ...

  3. 【原创】大叔问题定位分享(21)spark执行insert overwrite非常慢,比hive还要慢

    最近把一些sql执行从hive改到spark,发现执行更慢,sql主要是一些insert overwrite操作,从执行计划看到,用到InsertIntoHiveTable spark-sql> ...

  4. 【原创】大叔问题定位分享(20)hdfs文件create写入正常,append写入报错

    最近在hdfs写文件的时候发现一个问题,create写入正常,append写入报错,每次都能重现,代码示例如下: FileSystem fs = FileSystem.get(conf); Outpu ...

  5. 【原创】大叔问题定位分享(13)HBase Region频繁下线

    问题现象:hive执行sql报错 select count(*) from test_hive_table; 报错 Error: java.io.IOException: org.apache.had ...

  6. 【原创】大叔问题定位分享(11)Spark中对大表子查询加limit为什么会报Broadcast超时错误

    当两个表需要join时,如果一个是大表,一个是小表,正常的map-reduce流程需要shuffle,这会导致大表数据在节点间网络传输,常见的优化方式是将小表读到内存中并广播到大表处理,避免shuff ...

  7. 【原创】大叔问题定位分享(6)Dubbo monitor服务iowait高,负载高

    一 问题 Dubbo monitor所在服务器状态异常,iowait一直很高,load也一直很高,监控如下: iowait如图: load如图: 二 分析 通过iotop命令可以查看当前系统中磁盘io ...

  8. 【原创】大叔问题定位分享(3)Kafka集群broker进程逐个报错退出

    kafka0.8.1 一 问题现象 生产环境kafka服务器134.135.136分别在10月11号.10月13号挂掉: 134日志 [2014-10-13 16:45:41,902] FATAL [ ...

  9. 【原创】大叔问题定位分享(2)spark任务一定几率报错java.lang.NoSuchFieldError: HIVE_MOVE_FILES_THREAD_COUNT

    最近用yarn cluster方式提交spark任务时,有时会报错,报错几率是40%,报错如下: 18/03/15 21:50:36 116 ERROR ApplicationMaster91: Us ...

随机推荐

  1. 多项目管理中PMO的作用

    随着现代企业规模的不断扩大,多项目同时运行成为了现代企业的常态,以及企业项目化进程的不断深入,大部分企业不再仅仅只运行一个项目,而且数量之大已经超出了人们的想象,如惠普公司每年有3000个左右的项目, ...

  2. CentOS7.5脱机安装SQL Server 2017(NEW)

    发现搜到的都是在线下载安装的,都是只安装了mssql-server服务,没有mssql-server-agent服务.还以为linux下mssql没有agent服务呢.一番测试发现可以脱机安装,但是能 ...

  3. 【Swift】iOS开发笔记(二)

    前言 这个系列主要是一些开发中遇到的坑记录分享,有助于初学者跨过这些坑,攒够 7 条发一篇. 声明  欢迎转载,但请保留文章原始出处:)  博客园:http://www.cnblogs.com 农民伯 ...

  4. 软工+C(1): 题目设计、点评和评分

    // 下一篇:分数和checklist 如何设计题目 教学中的一个问题是老师出题太简单了,题目设计一开始上来就不紧凑,我认为一个好的课程应该上来就给你紧凑感,而不是先上来"轻松2-3周&qu ...

  5. win10设置操作备忘

    添加密码, 更改密码: Win键-->左侧用户图标-->更改帐户设置-->登陆选项-->添加密码 | 更改密码

  6. Magento2 Service contracts 服务合同

    服务合同 Magento是一个模块化系统,它使第三方开发人员能够定制和覆盖其框架的核心部分.然而,这种灵活性是有代价的. 业务逻辑倾向于泄漏Magento系统的各个层,这表现为重复且不一致的代码. 商 ...

  7. mysql shell 定时备份

    #!/bin/sh if [ ! -d "/data/backup" ]; then         mkdir -p /data/backup fi db_user=" ...

  8. pycharm 运行错误信息显示乱码

    pycharm版本 解决方法:

  9. Gym - 101350E Competitive Seagulls (博弈)

    There are two seagulls playing a very peculiar game. First they line up N unit squares in a line, al ...

  10. Java入门:基础算法之二进制转换为十进制

    Java有两种方法可以将二进制数转换为十进制数: 1)使用Integer类的Integer.parseInt()方法. 2)自己编写转换逻辑. 方法1:使用Integer.parseInt()实现二进 ...