Hadoop 2.7.3 完全分布式维护-简单测试篇

1. 测试MapReduce Job

1.1 上传文件到hdfs文件系统

$ jps
 Jps
 SecondaryNameNode
 JobHistoryServer
 NameNode
 ResourceManager
$ jps > infile
$ hadoop fs -mkdir /inputdir
$ hadoop fs -put infile /inputdir
$ hadoop fs -ls /inputdir
Found  items
-rw-r--r--    hduser supergroup          -- : /inputdir/infile

1.2 进行word count计算

$ hadoop jar /usr/local/hadoop-2.7./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7..jar wordcount /inputdir /outputdir
// :: INFO client.RMProxy: Connecting to ResourceManager at /172.16.101.55:
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504106569900_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1504106569900_0001
// :: INFO mapreduce.Job: The url to track the job: http://sht-sgmhadoopnn-01:8088/proxy/application_1504106569900_0001/
// :: INFO mapreduce.Job: Running job: job_1504106569900_0001
// :: INFO mapreduce.Job: Job job_1504106569900_0001 running in uber mode : false
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job: Job job_1504106569900_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
    File System Counters
        FILE: Number of bytes read=
        FILE: Number of bytes written=
        FILE: Number of read operations=
        FILE: Number of large read operations=
        FILE: Number of write operations=
        HDFS: Number of bytes read=
        HDFS: Number of bytes written=
        HDFS: Number of read operations=
        HDFS: Number of large read operations=
        HDFS: Number of write operations=
    Job Counters
        Launched map tasks=
        Launched reduce tasks=
        Data-local map tasks=
        Total time spent by all maps in occupied slots (ms)=
        Total time spent by all reduces in occupied slots (ms)=
        Total time spent by all map tasks (ms)=
        Total time spent by all reduce tasks (ms)=
        Total vcore-milliseconds taken by all map tasks=
        Total vcore-milliseconds taken by all reduce tasks=
        Total megabyte-milliseconds taken by all map tasks=
        Total megabyte-milliseconds taken by all reduce tasks=
    Map-Reduce Framework
        Map input records=
        Map output records=
        Map output bytes=
        Map output materialized bytes=
        Input split bytes=
        Combine input records=
        Combine output records=
        Reduce input groups=
        Reduce shuffle bytes=
        Reduce input records=
        Reduce output records=
        Spilled Records=
        Shuffled Maps =
        Failed Shuffles=
        Merged Map outputs=
        GC time elapsed (ms)=
        CPU time spent (ms)=
        Physical memory (bytes) snapshot=
        Virtual memory (bytes) snapshot=
        Total committed heap usage (bytes)=
    Shuffle Errors
        BAD_ID=
        CONNECTION=
        IO_ERROR=
        WRONG_LENGTH=
        WRONG_MAP=
        WRONG_REDUCE=
    File Input Format Counters
        Bytes Read=
    File Output Format Counters
        Bytes Written=

1.3 查看wordcount结果

$ hadoop fs -ls /outputdir
Found  items
-rw-r--r--    hduser supergroup           -- : /outputdir/_SUCCESS
-rw-r--r--    hduser supergroup         -- : /outputdir/part-r-
$ hadoop fs -cat /outputdir/part-r-

JobHistoryServer
Jps
NameNode
ResourceManager
SecondaryNameNode    

2. 测试hdfs分布式存储

2.1 上传测试文件

$ ls -lh hadoop-2.7..tar.gz
-rw-r--r--  root root 205M May   : hadoop-2.7..tar.gz
$ hadoop fs -put hadoop-2.7..tar.gz /inputdir
$ hadoop fs -ls -h /inputdir
Found  items
-rw-r--r--    hduser supergroup    204.2 M -- : /inputdir/hadoop-2.7..tar.gz
-rw-r--r--    hduser supergroup          -- : /inputdir/infile

2.2 查看datanode副本信息