继承关系:

  1. java.util

Interface Map.Entry<K,V>

description:

public static interface Map.Entry<K,V>

methods:

Modifier and Type Method and Description
boolean equals(Object o)

Compares the specified object with this entry for equality.
K getKey()

Returns the key corresponding to this entry.
V getValue()

Returns the value corresponding to this entry.
int hashCode()

Returns the hash code value for this map entry.
V setValue(V value)

Replaces the value corresponding to this entry with the specified value (optional operation).

2.java.lang.Object

|__ org.apache.hadoop.conf.Configuration

constructor: public class Configuration extends Objectimplements Iterable<Map.Entry<String,String>>, Writable 

3.org.apache.hadoop.util Class ToolRunner java.lang.Object   |__ org.apache.hadoop.util.ToolRunner

description:
public class ToolRunner

extends Object

  ToolRunner can be used to run classes implementing Tool interface. It works in conjunction with GenericOptionsParser to parse the generic hadoop command line arguments and modifies the Configuration of the Tool. The application-specific options are passed along without being modified.

methods:
static int run(Configuration conf, Tool tool, String[] args)
Runs the given Tool by Tool.run(String[]),
after parsing with the given generic arguments.
static int run(Tool tool, String[] args)

Runs the Tool with its
Configuration.
4.org.apache.hadoop.util 

Interface Tooldescription:
public interface Tool

extends Configurablemethods:
int run(String[] args)
Execute the command with the given arguments.
 5.org.apache.hadoop.conf 

Interface Configurable

constructor:

public interface Configurable

methods:
Configuration getConf()
Return the configuration used by this object.
void setConf(Configuration conf)

Set the configuration to be used by this object.
 6.
java.lang.Object
  |__ org.apache.hadoop.conf.Configureddescription:
public class Configured

extends Objectimplements Configurable
constructor:
Configured()
Construct a Configured.
Configured(Configuration conf)

Construct a Configured
 
methods:
Configuration getConf()
Return the configuration used by this object.
void setConf(Configuration conf)

Set the configuration to be used by this object.
Code1 (Configuration里添加的resource是String类型):
 import java.util.Map.Entry;

 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.conf.Configured;
 import org.apache.hadoop.util.ToolRunner;
 import org.apache.hadoop.util.Tool;
 import org.apache.hadoop.fs.Path;

 public class ConfigurationPrinter extends Configured implements Tool {
   static {
     Configuration.addDefaultResource("config.xml");
   }

   @Override
   public int run(String[] args) throws Exception {
     Configuration conf = getConf();
     for (Entry<String, String> hash: conf) {
       System.out.printf("%s=%s\n", hash.getKey(), hash.getValue());
     }
     return 0;
   }

   public static void main(String[] args) throws Exception {
     int exitCode = ToolRunner.run(new ConfigurationPrinter(), args);
     System.exit(exitCode);
   }
 }

注:Configuration class提供只一种静态方法:addDefaultresource(String name), 如上述代码, 添加Resource "config.xml"为String类型时,hadoop将从classpath里查找此文件;若Resource 为Path()类型时,hadoop将从local filesystem里查找此文件: Configuration conf = new Configuration(); conf.addResource(new Path("config.xml"));

code1的执行步骤:

#将自定义的config文件config.xml放在hadoop的$HADOOP_CONF_DIR里
mv config.xml $HADOOP_HOME/etc/hadoop/

#假如我们添加的resource如下:

 <!--cat $HADOOP_HOME/etc/hadoop/config.xml-->
 <configuration>
   <property>
     <name>color</name>
     <value>yellow</value>
   </property>

   <property>
     <name>size</name>
     <value>10</value>
   </property>

   <property>
     <name>weight</name>
     <value>heavy</value>
     <final>true</final>
   </property>
 </configuration>

执行代码:

mkdir class
source $HADOOP_HOME/libexec/hadoop-config.sh
javac  -d class ConfigurationPrinter.java
jar -cvf ConfigurationPrinter.jar -C class ./
export HADOOP_CLASSPATH=ConfigurationPrinter.jar:$CLASSPATH
#下面查找刚才添加的resource是否被读入
#我们在config.xml里添加了一项 <name>color</name>,执行
yarn ConfigurationPrinter|grep "color"
color=yellow
#可见代码是正确的

或者在commandline里指定HADOOP_CONF_DIR, 比如执行:

yarn ConfigurationPrinter --conf config.xml | grep color

color=yellow

也是可以的!

Code2 (Configuration里添加的resource是Path类型):

 import java.util.Map.Entry;

 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.conf.Configured;
 import org.apache.hadoop.util.ToolRunner;
 import org.apache.hadoop.util.Tool;
 import org.apache.hadoop.fs.Path;

 public class ConfigurationPrinter extends Configured implements Tool {
   @Override
   public int run(String[] args) throws Exception {
     Configuration conf = new Configuration();
     conf.addResource(new Path("config.xml"));
     for (Entry<String, String> hash: conf) {
       System.out.printf("%s=%s\n", hash.getKey(), hash.getValue());
     }
     return 0;
   }

   public static void main(String[] args) throws Exception {
     int exitCode = ToolRunner.run(new ConfigurationPrinter(), args);
     System.exit(exitCode);
   }
 }

此时添加的resource类型是Path()类型,故hadoop将从local filesystem里查找config.xml, 不需要将config.xml放在conf/下面,只要在代码中指定config.xml在本地文件系统中的路径即可(new Path("../others/config.xml"))

运行步骤:

mkdir class
source $HADOOP_HOME/libexec/hadoop-config.sh
javac  -d class ConfigurationPrinter.java
jar -cvf ConfigurationPrinter.jar -C class ./
export HADOOP_CLASSPATH=ConfigurationPrinter.jar:$CLASSPATH
#下面查找刚才添加的resource是否被读入
#我们在config.xml里添加了一项 <name>color</name>,执行
yarn ConfigurationPrinter|grep "color"
color=yellow
#可见代码是正确的

 

备注:ConfigurationParser支持set individual properties:

Generic Options
The supported generic options are:

-conf <configuration file>     specify a configuration file
     -D <property=value>            use value for given property
     -fs <local|namenode:port>      specify a namenode
     -jt <local|jobtracker:port>    specify a job tracker
     -files <comma separated list of files>    specify comma separated
                            files to be copied to the map reduce cluster
     -libjars <comma separated list of jars>   specify comma separated
                            jar files to include in the classpath.
     -archives <comma separated list of archives>    specify comma
             separated archives to be unarchived on the compute machines.

可以尝试:

yarn ConfigurationPrinter -d fuck=Japan | grep fuck
#输出为:
fuck=Japan

再次提醒:

ToolRunner can be used to run classes implementing Tool interface. It works in conjunction with GenericOptionsParser to parse the generic hadoop command line arguments and modifies the Configuration of the Tool. The application-specific options are passed along without being modified.

ToolRunnerGenericOptionsParser共同来(解析|修改) generic hadoop command line arguments  (什么是generic hadoop command line arguments? 比如:yarn  command [genericOptions] [commandOptions]

hadoop2.2编程:Tool, ToolRunner, GenericOptionsParser, Configuration的更多相关文章

  1. hadoop2.2编程:从default mapreduce program 来理解mapreduce

    下面写一个default mapreduce 的程序: import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapr ...

  2. hadoop2.2编程:使用MapReduce编程实例(转)

    原文链接:http://www.cnblogs.com/xia520pi/archive/2012/06/04/2534533.html 从网上搜到的一篇hadoop的编程实例,对于初学者真是帮助太大 ...

  3. hadoop2.2编程:MRUnit测试

    引用地址:http://www.cnblogs.com/lucius/p/3442381.html examples: Overview This document explains how to w ...

  4. hadoop2.2编程:矩阵相乘简单实现

    /* matrix-matrix multiplication on Hadoop A x B = C constraint: A, B, C must be of the same size I u ...

  5. hadoop2.2编程:MRUnit

    examples: Overview This document explains how to write unit tests for your map reduce code, and test ...

  6. hadoop2.2编程:DFS API 操作

    1. Reading data from a hadoop URL 说明:想要让java从hadoop的dfs里读取数据,则java 必须能够识别hadoop hdfs URL schema, 因此我 ...

  7. LPCScrypt, DFUSec : USB FLASH download, programming, and security tool, LPC-Link 2 Configuration tool, Firmware Programming

    What does this tool do? The LPC18xx/43xx DFUSec utility is a Windows PC tool that provides support f ...

  8. hadoop2.2编程: SequenceFileWritDemo

    import java.io.IOException; import java.net.URI; import org.apache.hadoop.fs.FileSystem; import org. ...

  9. Hadoop2.2编程:新旧API的区别

    Hadoop最新版本的MapReduce Release 0.20.0的API包括了一个全新的Mapreduce JAVA API,有时候也称为上下文对象. 新的API类型上不兼容以前的API,所以, ...

随机推荐

  1. java集合 collection-list-LinkedList 模拟一个堆栈或者队列数据结构。

    /* 使用LinkedList模拟一个堆栈或者队列数据结构. 堆栈:先进后出 如同一个杯子. 队列:先进先出 First in First out FIFO 如同一个水管. */ import jav ...

  2. OpenJudge/Poj 1004 Financial Management

    1.链接地址: http://poj.org/problem?id=1004 http://bailian.openjudge.cn/practice/1004 2.题目: 总时间限制: 1000ms ...

  3. 【转】MSSQL获取指定表的列名信息,描述,数据类型,长度

    /* --作用:根据特定的表名查询出字段,以及描述,数据类型,长度,精度,是否自增,是否为空等信息 --作者:wonder QQ:37036846 QQ群:.NET顶级精英群 ID:124766907 ...

  4. jquery垂直公告滚动实现代码

    公告滚动想必大家都有见到过吧,实现方法也有很多,下面为大家介绍使用jquery实现垂直公告滚动,感兴趣的朋友不要错过 复制代码代码如下: <!DOCTYPE html PUBLIC " ...

  5. 使用 桌面的 chrome 远程调试 Android 的页面

    手机浏览器是没有开发者工具的,所以调试手机网页是非常麻烦.使用 chrome 的远程调试功能可以像调试桌面端那样调试手机页面. 准备 手机端:chrome for Android, 安装谷歌浏览器 桌 ...

  6. node.js&mongodb&express 搭建个人博客系统

    源码参见于 https://github.com/njaulj/iliujun

  7. about Q&A in installing linux[centos6,7]

    keywords:grub1,grub2,gnome,kde, question describe:install centos7 by U disk,出现问题, 解决办法: install cent ...

  8. 中国.net域名网站的“前世今生”,那些年的光辉

    1987年9月的一天,中国的第一封电子邮件成功发出,邮件的内容大致是“跨越长城,走向世界”,在当时,没有人会想到十年后中国的互联网开始萌芽,并发展成今天的繁荣.1994年,“巴黎统筹委员会”的解散消除 ...

  9. 记录android学习、开发过程温故知新

    记录android学习.开发过程温故知新

  10. 关于c++字符串的while(*temp++)

    首先,上一段代码 static bool reverse_str(const char *str) { const char *temp=str; while(*temp++); temp-=; // ...