Giraph执行报错，Error: Exceeded limits on number of counters - Counters=120 Limit=120, exiting...

HamaWhite 原创，转载请注明出处。欢迎大家增加Giraph
技术交流群： 228591158

1. 近日用Giraph跑大数据的SSSP时，遇到例如以下错误：

org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120 at org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:315) at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:449)
at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:559) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:506) at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88) at org.apache.giraph.counters.HadoopCountersBase.getCounter(HadoopCountersBase.java:60)
at org.apache.giraph.counters.GiraphTimers.getSuperstepMs(GiraphTimers.java:117) at org.apache.giraph.master.MasterThread.run(MasterThread.java:127) FATAL graph.GraphMapper (GraphMapper.java:uncaughtException(113)) - uncaughtException: OverrideExceptionHandler
on thread org.apache.giraph.master.MasterThread, msg = org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120, exiting... java.lang.IllegalStateException: org.apache.hadoop.mapred.Counters$CountersExceededException:
Error: Exceeded limits on number of counters - Counters=120 Limit=120 at org.apache.giraph.master.MasterThread.run(MasterThread.java:181) Caused by: org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters -
Counters=120 Limit=120 at org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:315) at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:449) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:559) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:506)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88) at org.apache.giraph.counters.HadoopCountersBase.getCounter(HadoopCountersBase.java:60) at org.apache.giraph.counters.GiraphTimers.getSuperstepMs(GiraphTimers.java:117)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:127) INFO zk.ZooKeeperManager (ZooKeeperManager.java:run(655)) - run: Shutdown hook started. WARN zk.ZooKeeperManager (ZooKeeperManager.java:run(658)) - onlineZooKeeperServers: Forced a shutdown
hook kill of the ZooKeeper process. INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1159)) - Unable to read additional data from server sessionid 0x145750f19970000, likely server has closed socket, closing socket connection and attempting reconnect INFO zk.ZooKeeperManager
(ZooKeeperManager.java:run(668)) - onlineZooKeeperServers: ZooKeeper process exited with 143 (note that 143 typically means killed).

2. 错误原因分析：该Job的Counters数量限制超过默认的120 。mapred-default.xml配置文件的默认例如以下：

<property>

          <name>mapreduce.job.counters.limit</name>

          <value>120</value>

          <description>Limit on the number of counters allowed per job. </description>

     </property>

3. 改动配置文件mapred-site.xml 。加入例如以下内容：

<property>

          <name>mapreduce.job.counters.limit</name>

          <value>1200</value>

          <description>Limit on the number of counters allowed per job. </description>

     </property>

4. 停止Hadoop服务 。

5. 把改动后的配置文件发送到集群中的每一个节点上。

6. 启动Hadoop服务。

完成！