CDH- 测试mr

cdh的mr样例算法的jar包在

[zc.lee@ip---- hadoop-0.20-mapreduce]$ pwd
/opt/cloudera/parcels/CDH-5.10.-.cdh5.10.0.p0./lib/hadoop-0.20-mapreduce

查看该目录下的文件

[zc.lee@ip---- hadoop-0.20-mapreduce]$ ll
total
drwxr-xr-x  root root    Jan    bin
-rw-r--r--  root root  Jan    CHANGES.txt
drwxr-xr-x  root root    Jan    cloudera
lrwxrwxrwx  root root      Jul    conf -> /etc/hadoop/conf
drwxr-xr-x  root root    Jan    contrib
drwxr-xr-x  root root    Jan    example-confs
lrwxrwxrwx  root root      Jul    hadoop-ant-2.6.-mr1-cdh5.10.0.jar -> ../../jars/hadoop-ant-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-ant-mr1.jar -> hadoop-ant-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-core-2.6.-mr1-cdh5.10.0.jar -> ../../jars/hadoop-core-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-core-mr1.jar -> hadoop-core-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-examples-2.6.-mr1-cdh5.10.0.jar -> ../../jars/hadoop-examples-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-examples.jar -> hadoop-examples-mr1.jar
lrwxrwxrwx  root root      Jul    hadoop-examples-mr1.jar -> hadoop-examples-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-test-2.6.-mr1-cdh5.10.0.jar -> ../../jars/hadoop-test-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-test-mr1.jar -> hadoop-test-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-tools-2.6.-mr1-cdh5.10.0.jar -> ../../jars/hadoop-tools-2.6.-mr1-cdh5.10.0.jar
lrwxrwxrwx  root root      Jul    hadoop-tools-mr1.jar -> hadoop-tools-2.6.-mr1-cdh5.10.0.jar
drwxr-xr-x  root root    Jan    include
drwxr-xr-x  root root    Jan    lib
-rw-r--r--  root root   Jan    LICENSE.txt
-rw-r--r--  root root     Jan    NOTICE.txt
-rw-r--r--  root root    Jan    README.txt
drwxr-xr-x  root root    Jan    sbin
drwxr-xr-x  root root    Jan    webapps

可以用hadoop-examples.jar里面的wordcount做测试

#hadoop jar hadoop-examples.jar

可以看到里面都有些上面可以使用的类

An example program must be given as the first argument.
 Valid program names are:
 aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
 aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
 bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
 dbcount: An example job that count the pageview counts from a database.
 distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
 grep: A map/reduce program that counts the matches of a regex in the input.
 join: A job that effects a join over sorted, equally partitioned datasets
 multifilewc: A job that counts words from several files.
 pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
 pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
 randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
 randomwriter: A map/reduce program that writes 10GB of random data per node.
 secondarysort: An example defining a secondary sort to the reduce.
 sort: A map/reduce program that sorts the data written by the random writer.
 sudoku: A sudoku solver.
 teragen: Generate data for the terasort
 terasort: Run the terasort
 teravalidate: Checking results of terasort
 wordcount: A map/reduce program that counts the words in the input files.
 wordmean: A map/reduce program that counts the average length of the words in the input files.
 wordmedian: A map/reduce program that counts the median length of the words in the input files.
 wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

这里我直接取wordcount类来做测试，首先上传文件到hdfs准备好计算

hdfs dfs -mkdir /user/zc.lee/input/
hdfs dfs -put /user/PG/conf/type.txt /user/zc.lee/input/

开始计算

hadoop jar hadoop-examples.jar wordcount /user/zc.lee/input/type.txt /user/zc.lee/ouputtest

检查结果

hdfs dfs -text /user/zc.lee/ouputtest/*