一、规划

(一)硬件资源

10.171.29.191 master

10.173.54.84  slave1

10.171.114.223 slave2



(二)基本资料

用户:  jediael

目录:/opt/jediael/



二、环境配置

(一)统一用户名密码,并为jediael赋予执行所有命令的权限

#passwd
# useradd jediael
# passwd jediael
# vi /etc/sudoers

增加以下一行:

jediael ALL=(ALL) ALL

(二)创建目录/opt/jediael

$sudo chown jediael:jediael /opt
$ cd /opt
$ sudo mkdir jediael

注意:/opt必须是jediael的,否则会在format namenode时出错。



(三)修改用户名及/etc/hosts文件

1、修改/etc/sysconfig/network

NETWORKING=yes
HOSTNAME=*******

2、修改/etc/hosts

10.171.29.191 master
10.173.54.84 slave1
10.171.114.223 slave2

注 意hosts文件不能有127.0.0.1  *****配置,否则会导致出现异常。org.apache.hadoop.ipc.Client: Retrying connect to server: master/10.171.29.191:9000. Already trie

3、hostname命令

hostname ****

(四)配置免密码登录

以上命令在master上使用jediael用户执行:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

然后,将authorized_keys复制到slave1,slave2

scp ~/.ssh/authorized_keys slave1:~/.ssh/
scp ~/.ssh/authorized_keys slave2:~/.ssh/

注意

(1)若提示.ssh目录不存在,则表示此机器从未运行过ssh,因此运行一次即可创建.ssh目录。

(2).ssh/的权限为600,authorized_keys的权限为700,权限大了小了都不行。





(五)在3台机器上分别安装java,并设置相关环境变量

参考http://blog.csdn.net/jediael_lu/article/details/38925871



(六)下载hadoop-1.2.1.tar.gz,并将其解压到/opt/jediael



三、修改配置文件

【3台机器上均要执行】

(一)修改conf/hadoop_env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_51

(二)修改core-site.xml

<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property> <property>
<name>hadoop.tmp.dir</name>
<value>/opt/tmphadoop</value>
</property> 

(三)修改hdfs-site.xml

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

(四)修改mapred-site.xml

<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>

(五)修改master及slaves

master:
master slaves:
slave1
slave2

可以在master中完成上述配置,然后使用scp命令复制到slave1与slave2上。

 如:

$scp core-site.xml slave2:/opt/jediael/hadoop-1.2.1/conf

四、启动并验证





1、格式 化namenode【此步骤在3台机器上均要运行】

[jediael@master hadoop-1.2.1]$  bin/hadoop namenode -format

15/01/21 15:13:40 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = master/10.171.29.191

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.2.1

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG:   java = 1.7.0_51

************************************************************/

Re-format filesystem in /opt/tmphadoop/dfs/name ? (Y or N) Y

15/01/21 15:13:43 INFO util.GSet: Computing capacity for map BlocksMap

15/01/21 15:13:43 INFO util.GSet: VM type       = 64-bit

15/01/21 15:13:43 INFO util.GSet: 2.0% max memory = 1013645312

15/01/21 15:13:43 INFO util.GSet: capacity      = 2^21 = 2097152 entries

15/01/21 15:13:43 INFO util.GSet: recommended=2097152, actual=2097152

15/01/21 15:13:43 INFO namenode.FSNamesystem: fsOwner=jediael

15/01/21 15:13:43 INFO namenode.FSNamesystem: supergroup=supergroup

15/01/21 15:13:43 INFO namenode.FSNamesystem: isPermissionEnabled=true

15/01/21 15:13:43 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

15/01/21 15:13:43 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

15/01/21 15:13:43 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0

15/01/21 15:13:43 INFO namenode.NameNode: Caching file names occuring more than 10 times


15/01/21 15:13:44 INFO common.Storage: Image file /opt/tmphadoop/dfs/name/current/fsimage of size 113 bytes saved in 0 seconds.

15/01/21 15:13:44 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/opt/tmphadoop/dfs/name/current/edits

15/01/21 15:13:44 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/opt/tmphadoop/dfs/name/current/edits

15/01/21 15:13:44 INFO common.Storage: Storage directory /opt/tmphadoop/dfs/name has been successfully formatted.

15/01/21 15:13:44 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at master/10.171.29.191

************************************************************/





2、启动hadoop【此步骤只需要在master上执行】

[jediael@master hadoop-1.2.1]$ bin/start-all.sh 

starting namenode, logging to /opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-namenode-master.out

slave1: starting datanode, logging to /opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-datanode-slave1.out

slave2: starting datanode, logging to /opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-datanode-slave2.out

master: starting secondarynamenode, logging to /opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-secondarynamenode-master.out

starting jobtracker, logging to /opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-jobtracker-master.out

slave1: starting tasktracker, logging to /opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-tasktracker-slave1.out

slave2: starting tasktracker, logging to /opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-tasktracker-slave2.out



3、登录页面验证

NameNode    http://ip:50070  

JobTracker     http://ip50030







4、查看各个主机的java进程

(1)master:

$ jps

17963 NameNode

18280 JobTracker

18446 Jps

18171 SecondaryNameNode

(2)slave1:

$ jps

16019 Jps

15858 DataNode

15954 TaskTracker

(3)slave2:

$ jps

15625 Jps

15465 DataNode

15561 TaskTracker



五、运行一个完整的mapreduce程序。



以下内容均只是master上执行

1、将wordcount.jar包复制至服务器上

程序见http://blog.csdn.net/jediael_lu/article/details/37596469



2、创建输入目录,并将相关文件复制至目录

[jediael@master166 ~]$ hadoop fs -mkdir /wcin
[jediael@master166 projects]$ hadoop fs -copyFromLocal /opt/jediael/hadoop-1.2.1/conf/hdfs-site.xml /wcin 

3、运行程序

[jediael@master166 projects]$ hadoop jar wordcount.jar org.jediael.hadoopdemo.wordcount.WordCount /wcin /wcout


14/08/31 20:04:26 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.


14/08/31 20:04:26 INFO input.FileInputFormat: Total input paths to process : 1

14/08/31 20:04:26 INFO util.NativeCodeLoader: Loaded the native-hadoop library

14/08/31 20:04:26 WARN snappy.LoadSnappy: Snappy native library not loaded

14/08/31 20:04:26 INFO mapred.JobClient: Running job: job_201408311554_0003

14/08/31 20:04:27 INFO mapred.JobClient: map 0% reduce 0%

14/08/31 20:04:31 INFO mapred.JobClient: map 100% reduce 0%

14/08/31 20:04:40 INFO mapred.JobClient: map 100% reduce 100%

14/08/31 20:04:40 INFO mapred.JobClient: Job complete: job_201408311554_0003

14/08/31 20:04:40 INFO mapred.JobClient: Counters: 29

14/08/31 20:04:40 INFO mapred.JobClient: Job Counters

14/08/31 20:04:40 INFO mapred.JobClient: Launched reduce tasks=1

14/08/31 20:04:40 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=4230

14/08/31 20:04:40 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0


14/08/31 20:04:40 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0


14/08/31 20:04:40 INFO mapred.JobClient: Launched map tasks=1

14/08/31 20:04:40 INFO mapred.JobClient: Data-local map tasks=1

14/08/31 20:04:40 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=8531

14/08/31 20:04:40 INFO mapred.JobClient: File Output Format Counters

14/08/31 20:04:40 INFO mapred.JobClient: Bytes Written=284

14/08/31 20:04:40 INFO mapred.JobClient: FileSystemCounters

14/08/31 20:04:40 INFO mapred.JobClient: FILE_BYTES_READ=370

14/08/31 20:04:40 INFO mapred.JobClient: HDFS_BYTES_READ=357

14/08/31 20:04:40 INFO mapred.JobClient: FILE_BYTES_WRITTEN=104958

14/08/31 20:04:40 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=284

14/08/31 20:04:40 INFO mapred.JobClient: File Input Format Counters

14/08/31 20:04:40 INFO mapred.JobClient: Bytes Read=252

14/08/31 20:04:40 INFO mapred.JobClient: Map-Reduce Framework

14/08/31 20:04:40 INFO mapred.JobClient: Map output materialized bytes=370

14/08/31 20:04:40 INFO mapred.JobClient: Map input records=11

14/08/31 20:04:40 INFO mapred.JobClient: Reduce shuffle bytes=370

14/08/31 20:04:40 INFO mapred.JobClient: Spilled Records=40

14/08/31 20:04:40 INFO mapred.JobClient: Map output bytes=324

14/08/31 20:04:40 INFO mapred.JobClient: Total committed heap usage (bytes)=238026752


14/08/31 20:04:40 INFO mapred.JobClient: CPU time spent (ms)=1130

14/08/31 20:04:40 INFO mapred.JobClient: Combine input records=0

14/08/31 20:04:40 INFO mapred.JobClient: SPLIT_RAW_BYTES=105

14/08/31 20:04:40 INFO mapred.JobClient: Reduce input records=20

14/08/31 20:04:40 INFO mapred.JobClient: Reduce input groups=20

14/08/31 20:04:40 INFO mapred.JobClient: Combine output records=0

14/08/31 20:04:40 INFO mapred.JobClient: Physical memory (bytes) snapshot=289288192


14/08/31 20:04:40 INFO mapred.JobClient: Reduce output records=20

14/08/31 20:04:40 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1533636608


14/08/31 20:04:40 INFO mapred.JobClient: Map output records=20



4、查看结果

[jediael@master166 projects]$ hadoop fs -cat /wcout/* 

--> 1

<!-- 1

</configuration> 1

</property> 1

<?xml 1

<?xml-stylesheet 1

<configuration> 1

<name>dfs.replication</name> 1

<property> 1

<value>2</value> 1

Put 1

file. 1

href="configuration.xsl"?> 1

in 1

overrides 1

property 1

site-specific 1

this 1

type="text/xsl" 1

version="1.0"?> 1

cat: File does not exist: /wcout/_logs

版权声明:本文为博主原创文章,未经博主允许不得转载。

安装hadoop1.2.1集群环境 分类: A1_HADOOP 2014-08-29 15:49 1444人阅读 评论(0) 收藏的更多相关文章

  1. 服务器证书安装配置指南(IIS7.5) 分类: ASP.NET 2014-11-05 12:39 105人阅读 评论(0) 收藏

    1.启动IIS管理器,点击开始菜单->所有程序->管理工具->Internet信息服务(IIS)管理器: 2.选择"服务器证书": 3.在右边窗口,选择" ...

  2. Linux上安装JDK 分类: B1_JAVA B3_LINUX 2014-08-29 15:12 449人阅读 评论(0) 收藏

    1.下载rpm文件并安装 rpm -ivh jdk-7u51-linux-x64.rpm 2.修改/etc/profile文件,增加以下配置 export JAVA_HOME=/usr/java/jd ...

  3. 网站通用登录模块代码 分类: ASP.NET 2014-12-06 10:49 615人阅读 评论(0) 收藏

    1.HTML部分:     <form id="form1" runat="server">     <script src=".. ...

  4. Find The Multiple 分类: 搜索 POJ 2015-08-09 15:19 3人阅读 评论(0) 收藏

    Find The Multiple Time Limit: 1000MS Memory Limit: 10000K Total Submissions: 21851 Accepted: 8984 Sp ...

  5. 周赛-DZY Loves Chessboard 分类: 比赛 搜索 2015-08-08 15:48 4人阅读 评论(0) 收藏

    DZY Loves Chessboard time limit per test 1 second memory limit per test 256 megabytes input standard ...

  6. Basic 分类: POJ 2015-08-03 15:49 3人阅读 评论(0) 收藏

    Basic Time Limit: 1000MS Memory Limit: 65536K Total Submissions: 905 Accepted: 228 Description The p ...

  7. Ultra-QuickSort 分类: POJ 排序 2015-08-03 15:39 2人阅读 评论(0) 收藏

    Ultra-QuickSort Time Limit: 7000MS   Memory Limit: 65536K Total Submissions: 48111   Accepted: 17549 ...

  8. Drainage Ditches 分类: POJ 图论 2015-07-29 15:01 7人阅读 评论(0) 收藏

    Drainage Ditches Time Limit: 1000MS Memory Limit: 10000K Total Submissions: 62016 Accepted: 23808 De ...

  9. cf 61E. Enemy is weak 树状数组求逆序数(WA) 分类: Brush Mode 2014-10-19 15:16 104人阅读 评论(0) 收藏

    #include <iostream> #include <algorithm> #include <cstdio> #include <cstring> ...

随机推荐

  1. 对 hiren bootcd 15.2 中的 XP 系统作了汉化, 同时支持中文输入法。提供下载

    对 hiren bootcd 15.2 中的 XP 系统作了汉化, 同时支持中文输入法.提供下载 对该PE 中的 XP 系统作了汉化, 由于一个 中文字库 就要 10M 多:加之原系统过于精简,对中文 ...

  2. Base64是网络上最常见的用于传输8Bit字节代码的编码方式之一

    Base64是网络上最常见的用于传输8Bit字节代码的编码方式之一,大家可以查看RFC2045-RFC2049,上面有MIME的详细规范.Base64编码可用于在HTTP环境下传递较长的标识信息.例如 ...

  3. Encoding encoding = Encoding.GetEncoding("gb2312"); 与byte[] ping = Encoding.UTF8.GetBytes(inputString);区别

    Encoding encoding = Encoding.GetEncoding("gb2312"); 与byte[] ping = Encoding.UTF8.GetBytes( ...

  4. 模板 Fail树

    fail树就是将Trie图的Fail指针反指,从而生成一棵树,这个树的性质是:子节点对应字符串为以当前串为后缀,而子节点为原串的前缀,前缀的后缀就是嵌套在原串中的子串. 模板:BZOJ3172 Des ...

  5. CISP/CISA 每日一题 16

    CISA 每日一题(答) 作业调度软件的优点: 1.作业信息仅需建立一次,减少错误发生概率: 2.可定义作业间的依赖关系,当某一项作业失败时,依赖于该作业的后续作业就不会被执行: 3.所有成功或失败的 ...

  6. C 字符/字符串经常使用函数

    string.h中经常使用函数 char * strchr(char * str ,char ch); 从字符串str中查找首次出现字符ch的位置,若存在返回查找后的地址.若不存在则返回NULL vo ...

  7. ASP.NET MVC案例教程(基于ASP.NET MVC beta)——第四篇:传递表单数据

    摘要      本文将完成我们“MVC公告发布系统”的公告发布功能,以此展示在ASP.NET MVC中如何传递处理表单的数据. 前言      通过前几篇文章,我们已经能比较自如的使用ASP.NET ...

  8. VUE错误记录 - 品牌后台练习 search(); 数组 splice forEach push 箭头函数

    methods:{ add(){ var car = { id: this.id, name: this.name, ctime: new Date()}; this.list.push(car); ...

  9. python之经典猜数字

    题目:猜数字1.让用户输入1-20,猜数字,可以猜5次.2.每次有提示,大了,或者小了!3.如果超过5次,提示game over. # !/usr/bin/env python # -*- codin ...

  10. hibernate hbm.xml配置映射

    <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.sp ...