hadoop安装与WordCount例子

1、JDK安装 
下载网址： 
http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u29-download-513648.html 
如果本地有安装包，则用SecureCRT连接Linux机器，然后用rz指令进行上传文件；

下载后获得jdk-6u29-linux-i586-rpm.bin文件，使用sh jdk-6u29-linux-i586-rpm.bin进行安装， 
等待安装完成即可；java默认会安装在/usr/java下；

在命令行输入:vi /etc/profile在里面添加如下内容export JAVA_HOME=/usr/java/jdk1.6.0_29export JAVA_BIN=/usr/java/jdk1.6.0_29/binexport PATH=$PATH:$JAVA_HOME/binexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport JAVA_HOME JAVA_BIN PATH CLASSPATH

进入 /usr/bin/目录cd /usr/binln -s -f /usr/java/jdk1.6.0_29/jre/bin/javaln -s -f /usr/java/jdk1.6.0_29/bin/javac 
在命令行输入java -version屏幕输出:java version "jdk1.6.0_02"Java(TM) 2 Runtime Environment, Standard Edition (build jdk1.6.0_02)Java HotSpot(TM) Client VM (build jdk1.6.0_02, mixed mode)则表示安装JDK1.6完毕.

2、Hadoop安装 
下载网址：http://www.apache.org/dyn/closer.cgi/hadoop/common/ 
如果本地有安装包，则用SecureCRT连接Linux机器，然后用rz指令进行上传文件；

下载后获得hadoop-0.21.0.tar.gz文件

解压 tar zxvf hadoop-0.21.0.tar.gz 
压缩：tar zcvf hadoop-0.21.0.tar.gz 目录名

在命令行输入:vi /etc/profile在里面添加如下内容 
export hadoop_home = /usr/george/dev/install/hadoop-0.21.0 
export JAVA_HOME=/usr/java/jdk1.6.0_29export JAVA_BIN=/usr/java/jdk1.6.0_29/binexport PATH=$PATH:$JAVA_HOME/bin:$hadoop_home/binexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport JAVA_HOME JAVA_BIN PATH CLASSPATH

需要注销用户或重启vm，就可以直接输入hadoop指令了； 
WordCount例子代码 
3.1 Java代码： 
package demo;

import java.io.IOException; 
import java.util.Iterator; 
import java.util.StringTokenizer;

import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.io.IntWritable; 
import org.apache.hadoop.io.LongWritable; 
import org.apache.hadoop.io.Text; 
import org.apache.hadoop.mapred.FileInputFormat; 
import org.apache.hadoop.mapred.FileOutputFormat; 
import org.apache.hadoop.mapred.JobClient; 
import org.apache.hadoop.mapred.JobConf; 
import org.apache.hadoop.mapred.MapReduceBase; 
import org.apache.hadoop.mapred.Mapper; 
import org.apache.hadoop.mapred.OutputCollector; 
import org.apache.hadoop.mapred.Reducer; 
import org.apache.hadoop.mapred.Reporter; 
import org.apache.hadoop.mapred.TextInputFormat; 
import org.apache.hadoop.mapred.TextOutputFormat;

public class WordCount { 
public static class Map extends MapReduceBase implements 
Mapper<LongWritable, Text, Text, IntWritable> { 
private final static IntWritable one = new IntWritable(1); 
private Text word = new Text();

public void map(LongWritable key, Text value, 
OutputCollector<Text, IntWritable> output, Reporter reporter) 
throws IOException { 
String line = value.toString(); 
StringTokenizer tokenizer = new StringTokenizer(line); 
while (tokenizer.hasMoreTokens()) { 
word.set(tokenizer.nextToken()); 
output.collect(word, one); 
} 
} 
}

public static class Reduce extends MapReduceBase implements 
Reducer<Text, IntWritable, Text, IntWritable> { 
public void reduce(Text key, Iterator<IntWritable> values, 
OutputCollector<Text, IntWritable> output, Reporter reporter) 
throws IOException { 
int sum = 0; 
while (values.hasNext()) { 
sum += values.next().get(); 
} 
output.collect(key, new IntWritable(sum)); 
} 
}

public static void main(String[] args) throws Exception { 
JobConf conf = new JobConf(WordCount.class); 
conf.setJobName("wordcount");

conf.setOutputKeyClass(Text.class); 
conf.setOutputValueClass(IntWritable.class);

conf.setMapperClass(Map.class); 
conf.setCombinerClass(Reduce.class); 
conf.setReducerClass(Reduce.class);

conf.setInputFormat(TextInputFormat.class); 
conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf, new Path(args[0])); 
FileOutputFormat.setOutputPath(conf, new Path(args[1]));

JobClient.runJob(conf); 
} 
}

3.2 编译： 
javac -classpath /usr/george/dev/install/hadoop-0.21.0/hadoop-hdfs-0.21.0.jar:/usr/george/dev/install/hadoop-0.21.0/hadoop-mapred-0.21.0.jar:/usr/george/dev/install/hadoop-0.21.0/hadoop-common-0.21.0.jar WordCount.java -d /usr/george/dev/wkspace/hadoop/wordcount/classes 
在windows中，多个classpath参数值用;分割；在linux中用:分割；

编译后，会在/usr/george/dev/wkspace/hadoop/wordcount/classes目录下生成三个class文件： 
WordCount.class  WordCount$Map.class  WordCount$Reduce.class

3.3将class文件打成jar包 
到/usr/george/dev/wkspace/hadoop/wordcount/classes目录，运行jar cvf WordCount.jar *.class就会生成： 
WordCount.class  WordCount.jar  WordCount$Map.class  WordCount$Reduce.class

3.4 创建输入数据： 
创建/usr/george/dev/wkspace/hadoop/wordcount/datas目录，在其下创建input1.txt和input2.txt文件： 
Touch input1.txt 
Vi input1.txt

文件内容如下： 
i love chinaare you ok?

按照同样的方法创建input2.txt，内容如下： 
hello, i love word 
You are ok

创建成功后可以通过cat input1.txt 和 cat input2.txt查看内容；

3.5 创建hadoop输入与输出目录： 
hadoop fs -mkdir wordcount/inputhadoop fs -mkdir wordcount/outputhadoop fs -put input1.txt wordcount/input/hadoop fs -put input2.txt wordcount/input/

Ps : 可以不创建out目录，要不运行WordCount程序时会报output文件已经存在，所以下面的命令行中使用了output1为输出目录； 
3.6运行 
到/usr/george/dev/wkspace/hadoop/wordcount/classes目录，运行 
[root@localhost classes]# hadoop jar WordCount.jar WordCount wordcount/input wordcount/output1 
11/12/02 05:53:59 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000 
11/12/02 05:53:59 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 
11/12/02 05:53:59 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 
11/12/02 05:53:59 INFO mapred.FileInputFormat: Total input paths to process : 2 
11/12/02 05:54:00 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 
11/12/02 05:54:00 INFO mapreduce.JobSubmitter: number of splits:2 
11/12/02 05:54:00 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null 
11/12/02 05:54:00 INFO mapreduce.Job: Running job: job_201112020429_0003 
11/12/02 05:54:01 INFO mapreduce.Job:  map 0% reduce 0% 
11/12/02 05:54:20 INFO mapreduce.Job:  map 50% reduce 0% 
11/12/02 05:54:23 INFO mapreduce.Job:  map 100% reduce 0% 
11/12/02 05:54:29 INFO mapreduce.Job:  map 100% reduce 100% 
11/12/02 05:54:32 INFO mapreduce.Job: Job complete: job_201112020429_0003 
11/12/02 05:54:32 INFO mapreduce.Job: Counters: 33 
        FileInputFormatCounters 
                BYTES_READ=54 
        FileSystemCounters 
                FILE_BYTES_READ=132 
                FILE_BYTES_WRITTEN=334 
                HDFS_BYTES_READ=274 
                HDFS_BYTES_WRITTEN=65 
        Shuffle Errors 
                BAD_ID=0 
                CONNECTION=0 
                IO_ERROR=0 
                WRONG_LENGTH=0 
                WRONG_MAP=0 
                WRONG_REDUCE=0 
        Job Counters 
                Data-local map tasks=2 
                Total time spent by all maps waiting after reserving slots (ms)=0 
                Total time spent by all reduces waiting after reserving slots (ms)=0 
                SLOTS_MILLIS_MAPS=24824 
                SLOTS_MILLIS_REDUCES=6870 
                Launched map tasks=2 
                Launched reduce tasks=1 
        Map-Reduce Framework 
                Combine input records=12 
                Combine output records=12 
                Failed Shuffles=0 
                GC time elapsed (ms)=291 
                Map input records=4 
                Map output bytes=102 
                Map output records=12 
                Merged Map outputs=2 
                Reduce input groups=10 
                Reduce input records=12 
                Reduce output records=10 
                Reduce shuffle bytes=138 
                Shuffled Maps =2 
                Spilled Records=24 
                SPLIT_RAW_BYTES=220

3.7 查看输出目录 
[root@localhost classes]# hadoop fs -ls wordcount/output1 
11/12/02 05:54:59 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000 
11/12/02 05:55:00 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 
Found 2 items 
-rw-r--r--   1 root supergroup          0 2011-12-02 05:54 /user/root/wordcount/output1/_SUCCESS 
-rw-r--r--   1 root supergroup         65 2011-12-02 05:54 /user/root/wordcount/output1/part-00000

[root@localhost classes]# hadoop fs -cat /user/root/wordcount/output1/part-00000 
11/12/02 05:56:05 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000 
11/12/02 05:56:05 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 
You     1 
are     2 
china   1 
hello,i 1 
i       1 
love    2 
ok      1 
ok?     1 
word    1 
you     1