hadoop基准測试

_写測试hadoop jarhadoop-0.20.2-test.jar TestDFSIO -write -nrFiles 10 -fileSize 1000

----- TestDFSIO ----- : write

Date & time: Fri Jul 24 14:24:36 CST 2015

Number of files: 10

Total MBytes processed: 10000

Throughput mb/sec: 24.302163378583963

Average IO rate mb/sec: 24.46268653869629

IOrate std deviation: 2.0279575539782315

Test exec time sec: 72.853

_读測试hadoop jarhadoop-0.20.2-test.jar TestDFSIO -read -nrFiles 10 -fileSize 1000

----- TestDFSIO ----- : read

Date & time: Fri Jul 24 14:27:11 CST 2015

Number of files: 10

Total MBytes processed: 10000

Throughput mb/sec: 80.73566336457803

Average IO rate mb/sec: 106.7965087890625

IOrate std deviation: 70.18198197030486

Test exec time sec: 46.772

清除数据

[root@ZhejiangYiwuF1210 hadoop]# hadoop jarhadoop-0.20.2-test.jar TestDFSIO -clean

_排序測试

1.在各节点上分别执行2个map任务。每一个节点产生1GB大小的随机二进制数据，并输出到文件夹/examples/terasort-input

hadoop jar hadoop-0.20.2-examples.jar teragen 10000000/examples/terasort-input

Launched map tasks=2

FileSystemCounters

HDFS_BYTES_WRITTEN=1000000000

Map-Reduce Framework

Map input records=10000000

Spilled Records=0

Map input bytes=10000000

Map output records=10000000

MapReduce（map:2个,reduce:0个）

2.排序。并将结果输出到文件夹/examples/terasort-output

#hadoop jar hadoop-0.20.2-examples.jar terasort /examples/terasort-input/examples/terasort-output

MapReduce（map:16个,reduce:1个）

3.检查是否已经排好序

# hadoop jar hadoop-0.20.2-examples.jarteravalidate /examples/terasort-output /examples/terasort-validate

MapReduce（map:1个,reduce:1个）

_{Namenode的负载測试}

# hadoop jar hadoop-0.20.2-test.jarnnbench -operation create_write -maps 12 -reduces 6 -blockSize 1 -bytesToWrite0 -numberOfFiles 1000 -replicationFactorPerFile 3 -readFileAfterOpen true-baseDir /benchmarks/NNBench-`hostname
-s`

-------------- NNBench -------------- :

Version:NameNode Benchmark 0.4

Date& time: 2015-07-24 14:52:04,989

TestOperation: create_write

Starttime: 2015-07-24 14:48:46,912

Maps torun: 12

Reducesto run: 6

BlockSize (bytes): 1

Bytes towrite: 0

Bytesper checksum: 1

Numberof files: 1000

Replicationfactor: 3

Successfulfile operations: 0

MapReduce（map:12个,reduce:6个）

_{MapReduce连续性測试}

mrbench会多次反复执行一个小作业，用于检查在机群上小作业的执行是否可反复以及执行是否高效

执行一个小作业10次

hadoop jar hadoop-0.20.2-test.jar mrbench -numRuns 10

DataLines       Maps   Reduces AvgTime (milliseconds)

1               2       1      29637

以上结果表示平均作业完毕时间是29637(milliseconds)

_{文件系统一致性的分布式检查}

hadoop jarhadoop-0.20.2-test.jar DistributedFSCheck

----- DistributedFSCheck ----- :

Date & time: Fri Jul 2415:04:18 CST 2015

Total number of blocks: 58

Total number of  files: 3171

Number of corrupted blocks: 0

Number of corrupted files: 0

MapReduce（map:2个,reduce:1个）