hadoop运行wordcount实例，hdfs简单操作

1.查看hadoop版本

[hadoop@ltt1 sbin]$ hadoop version
Hadoop 2.6.-cdh5.12.0
Subversion http://github.com/cloudera/hadoop -r dba647c5a8bc5e09b572d76a8d29481c78d1a0dd
Compiled by jenkins on --29T11:33Z
Compiled with protoc 2.5.
From source with checksum 7c45ae7a4592ce5af86bc4598c5b4
This command was run using /home/hadoop/hadoop260/share/hadoop/common/hadoop-common-2.6.-cdh5.12.0.jar

2.通过hadoop自带的jar文件，可以简单测试一些功能。

提君博客原创

查看hadoop-mapreduce-examples-2.6.0-cdh5.12.0.jar文件所支持的MapReduce功能列表

[hadoop@ltt1 sbin]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.-cdh5.12.0.jar
An example program must be given as the first argument.
Valid program names are:
  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
  dbcount: An example job that count the pageview counts from a database.
  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
  grep: A map/reduce program that counts the matches of a regex in the input.
  join: A job that effects a join over sorted, equally partitioned datasets
  multifilewc: A job that counts words from several files.
  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
  pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
  randomwriter: A map/reduce program that writes 10GB of random data per node.
  secondarysort: An example defining a secondary sort to the reduce.
  sort: A map/reduce program that sorts the data written by the random writer.
  sudoku: A sudoku solver.
  teragen: Generate data for the terasort
  terasort: Run the terasort
  teravalidate: Checking results of terasort
  wordcount: A map/reduce program that counts the words in the input files.
  wordmean: A map/reduce program that counts the average length of the words in the input files.
  wordmedian: A map/reduce program that counts the median length of the words in the input files.
  wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

3.在hdfs上创建文件夹

hadoop fs -mkdir /input

4.查看hdfs的更目录列表

[hadoop@ltt1 ~]$ hadoop fs -ls /
Found 2 items
drwxr-xr-x - hadoop supergroup 0 2017-09-17 08:11 /input
drwx------ - hadoop supergroup 0 2017-09-17 08:07 /tmp

5.上传本地文件到hdfs

hadoop fs -put $HADOOP_HOME/*.txt /input

6.查看hdfs上input目录下文件

[hadoop@ltt1 ~]$ hadoop fs -ls /input
Found  items
-rw-r--r--    hadoop supergroup       -- : /input/LICENSE.txt
-rw-r--r--    hadoop supergroup       -- : /input/NOTICE.txt
-rw-r--r--    hadoop supergroup        -- : /input/README.txt

7.wordcount简单测试。

提君博客原创

[hadoop@ltt1 ~]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.-cdh5.12.0.jar wordcount /input /output
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505605169997_0002
// :: INFO impl.YarnClientImpl: Submitted application application_1505605169997_0002
// :: INFO mapreduce.Job: The url to track the job: http://ltt1.bg.cn:9180/proxy/application_1505605169997_0002/
// :: INFO mapreduce.Job: Running job: job_1505605169997_0002
// :: INFO mapreduce.Job: Job job_1505605169997_0002 running in uber mode : false
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job: Job job_1505605169997_0002 completed successfully
// :: INFO mapreduce.Job: Counters: 50
>>提君博客原创  http://www.cnblogs.com/tijun/  <<
    File System Counters
        FILE: Number of bytes read=
        FILE: Number of bytes written=
        FILE: Number of read operations=
        FILE: Number of large read operations=
        FILE: Number of write operations=
        HDFS: Number of bytes read=
        HDFS: Number of bytes written=
        HDFS: Number of read operations=
        HDFS: Number of large read operations=
        HDFS: Number of write operations=
    Job Counters
        Launched map tasks=
        Launched reduce tasks=
        Data-local map tasks=
        Rack-local map tasks=
        Total time spent by all maps in occupied slots (ms)=
        Total time spent by all reduces in occupied slots (ms)=
        Total time spent by all map tasks (ms)=
        Total time spent by all reduce tasks (ms)=
        Total vcore-milliseconds taken by all map tasks=
        Total vcore-milliseconds taken by all reduce tasks=
        Total megabyte-milliseconds taken by all map tasks=
        Total megabyte-milliseconds taken by all reduce tasks=
    Map-Reduce Framework
        Map input records=
        Map output records=
        Map output bytes=
        Map output materialized bytes=
        Input split bytes=
        Combine input records=
        Combine output records=
        Reduce input groups=
        Reduce shuffle bytes=
        Reduce input records=
        Reduce output records=
        Spilled Records=
        Shuffled Maps =
        Failed Shuffles=
        Merged Map outputs=
        GC time elapsed (ms)=
        CPU time spent (ms)=
        Physical memory (bytes) snapshot=
        Virtual memory (bytes) snapshot=
        Total committed heap usage (bytes)=
    Shuffle Errors
        BAD_ID=
        CONNECTION=
        IO_ERROR=
        WRONG_LENGTH=
        WRONG_MAP=
        WRONG_REDUCE=
    File Input Format Counters
        Bytes Read=
    File Output Format Counters
        Bytes Written=

8.查看wordcount运行结果（由于结果太长，只举出了部分结果）

[hadoop@ltt1 ~]$ hadoop fs -cat /output/*
worldwide,    4
would    1
writing    2
writing,    4
written    19
xmlenc    1
year    1
you    12
your    5
zlib    1
 252.227-7014(a)(1))    1
§    1
“AS    1
“Contributor    1
“Contributor”    1
“Covered    1
“Executable”    1
“Initial    1
“Larger    1
“Licensable”    1
“License”    1
“Modifications”    1
“Original    1
“Participant”)    1
“Patent    1
“Source    1
“Your”)    1
“You”    2
“commercial    3
“control”    1

>>提君博客原创  http://www.cnblogs.com/tijun/  <<

至此，通过一个wordcount的一个小栗子，简介实践了一下hdfs的创建文件夹，上传文件，查看目录，运行wordcount实例。

提君博客原创

>>提君博客原创  http://www.cnblogs.com/tijun/  <<