hadoop伪分布模式安装
软件环境
操作系统 : OracleLinux-R6-U6
主机名: hadoop
java: jdk1.7.0_75
hadoop: hadoop-2.4.1
环境搭建
1、软件安装
由于所需的软件均为绿色包,所以将java和hadoop分别解压到操作系统根目录即可。
[root@hadoop training]# ls -l /
总用量 110
dr-xr-xr-x. 2 root root 4096 5月 17 19:13 bin
dr-xr-xr-x. 5 root root 1024 5月 17 17:45 boot
drwxr-xr-x. 2 root root 4096 10月 15 2014 cgroup
drwxr-xr-x. 19 root root 3780 5月 18 01:36 dev
drwxr-xr-x. 131 root root 12288 5月 18 17:59 etc
drwxr-xr-x. 11 67974 users 4096 5月 18 18:22 hadoop-2.4.1
drwxr-xr-x. 2 root root 4096 11月 1 2011 home
drwxr-xr-x. 8 uucp 143 4096 12月 19 2014 jdk1.7.0_75
2、配置环境变量
修改profile文件
[root@hadoop training]# cat ~/.bash_profile
# .bash_profile # Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH # set python environment
PYTHON_PATH=/python2.7
export PATH=$PYTHON_PATH/bin:$PATH # set java environment 分为JAVA JDK CLASSPATH三类
export JAVA_HOME=/jdk1.7.0_75
export JRE_HOME=/jdk1.7.0_75/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
export HADOOP_HOME=/hadoop-2.4.1
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
修改hosts文件
[root@hadoop ~]# cat /etc/hosts
127.0.0.1 localhost
172.10.236.21 hadoop
3、hadoop分布式文件配置
hadoop的所有配置文件均在/hadoop-2.4.1/etc/hadoop/目录下
配置hadoop-env.sh,修改java_home
# The java implementation to use.
export JAVA_HOME=/jdk1.7.0_75
配置hdfs-site.xml,伪分布只需要设置一个复制节点即可。
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
配置core-site.xml,设置namenode格式化数据的存储目录,操作系统每次重启/tmp目录下的数据被清除,所以需要为namenode数据设置一个别的目录。
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-2.4.1/tmp</value>
</property>
</configuration>
配置mapred-site.xml,由于默认只有mapred-site.xml.template文件,所以拷贝一份mapred-site.xml.template为mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
配置yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration> <!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
最后一步:
格式化namenode
至此hadoop所有配置文件已全部配置完成,现在格式化namenode,以记录处理hadoop分布式信息了。
# hdfs namenode -format
……
17/05/18 18:28:19 INFO util.GSet: VM type = 32-bit
17/05/18 18:28:19 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
17/05/18 18:28:19 INFO util.GSet: capacity = 2^19 = 524288 entries
17/05/18 18:28:19 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/05/18 18:28:19 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/05/18 18:28:19 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
17/05/18 18:28:19 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/05/18 18:28:19 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/05/18 18:28:19 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/05/18 18:28:19 INFO util.GSet: VM type = 32-bit
17/05/18 18:28:19 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
17/05/18 18:28:19 INFO util.GSet: capacity = 2^16 = 65536 entries
17/05/18 18:28:19 INFO namenode.AclConfigFlag: ACLs enabled? false
17/05/18 18:28:20 INFO namenode.FSImage: Allocated new BlockPoolId: BP-39137453-172.10.236.21-1495103299866
17/05/18 18:28:20 INFO common.Storage: Storage directory /hadoop-2.4.1/tmp/dfs/name has been successfully formatted.
17/05/18 18:28:20 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/05/18 18:28:20 INFO util.ExitUtil: Exiting with status 0
17/05/18 18:28:20 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop/172.10.236.21
************************************************************/
查看格式化信息
[root@hadoop hadoop]# ls /hadoop-2.4.1/tmp/
dfs nm-local-dir
至此,hadoop环境已经可以使用了,可以通过start-all.sh来启动hadoop所有服务
检查进程
[root@hadoop ~]# jps
16570 Jps
15893 DataNode
16461 NodeManager
16179 ResourceManager
16041 SecondaryNameNode
15774 NameNode
hdfs环境可用性测试
创建dfs目录
# hdfs dfs -mkdir /logs
查看创建的目录
# hdfs dfs -ls /
Found 1 items
drwxr-xr-x - root supergroup 0 2017-05-18 18:32 /logs
向新建目录发送数据文件
# hdfs dfs -put install.log /logs
查看文件发送结果
[root@hadoop ~]# hdfs dfs -ls /logs
Found 1 items
-rw-r--r-- 1 root supergroup 57162 2017-05-18 18:32 /logs/install.log
hadoop伪分布模式配置成功并可以使用了。
无密码验证配置
由于每次启动hadoop服务都需要输入密码,对于hadoop集群节点太多的情况下显然不合适,所以需要设置启动hadoop服务无密码的方法。
生成密码文件,并将公钥文件拷贝到hadoop服务器(这里是自己)
# ssh-keygen -t rsa
# 将id_rsa.pub内容拷贝到authorized_keys中
# ssh-copy-id id_rsa.pub hadoop
[root@hadoop ~]# ls ~/.ssh/
authorized_keys id_rsa id_rsa.pub known_hosts
启动hadoop
[root@hadoop ~]# start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [hadoop]
The authenticity of host 'hadoop (172.10.236.21)' can't be established.
RSA key fingerprint is b9:d5:64:bb:f9:34:77:22:d7:a7:09:a6:1e:ab:ba:83.
Are you sure you want to continue connecting (yes/no)? yes
hadoop: Warning: Permanently added 'hadoop' (RSA) to the list of known hosts.
hadoop: starting namenode, logging to /hadoop-2.4.1/logs/hadoop-root-namenode-hadoop.out
localhost: starting datanode, logging to /hadoop-2.4.1/logs/hadoop-root-datanode-hadoop.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /hadoop-2.4.1/logs/hadoop-root-secondarynamenode-hadoop.out
starting yarn daemons
starting resourcemanager, logging to /hadoop-2.4.1/logs/yarn-root-resourcemanager-hadoop.out
localhost: starting nodemanager, logging to /hadoop-2.4.1/logs/yarn-root-nodemanager-hadoop.out
hadoop伪分布模式安装的更多相关文章
- Hadoop 伪分布模式安装
( 温馨提示:图片中有id有姓名,不要盗用哦,可参考流程,有问题评论区留言哦 ) 一.任务目标 1.了解Hadoop的3种运行模式 2.熟练掌握Hadoop伪分布模式安装流程 3.培养独立完成Hado ...
- Hadoop学习笔记(3)hadoop伪分布模式安装
为了学习这部分的功能,我们这里的linux都是使用root用户登录的.所以每个命令的前面都有一个#符号. 伪分布模式安装步骤: 关闭防火墙 修改ip地址 修改hostname 设置ssh自动登录 安装 ...
- hadoop伪分布模式的配置和一些常用命令
大数据的发展历史 3V:volume.velocity.variety(结构化和非结构化数据).value(价值密度低) 大数据带来的技术挑战 存储容量不断增加 获取有价值的信息的难度:搜索.广告.推 ...
- 【原】Hadoop伪分布模式的安装
Hadoop伪分布模式的安装 [环境参数] (1)Host OS:Win7 64bit (2)IDE:Eclipse Version: Luna Service Release 2 (4.4.2) ( ...
- Spark新手入门——2.Hadoop集群(伪分布模式)安装
主要包括以下三部分,本文为第二部分: 一. Scala环境准备 查看 二. Hadoop集群(伪分布模式)安装 三. Spark集群(standalone模式)安装 查看 Hadoop集群(伪分布模式 ...
- Hadoop单点伪分布模式安装
Hadoop单点伪分布模式安装 概述 单点 single-node,单节点,即一台计算机. 伪分布式模式 pseudo-distributed mode 所谓集群,表面上看是多台计算机联合完成任务:但 ...
- 伪分布模式安装hadoop
准备工具: 虚拟机:VMware Linux系统:CentOS hadoop-1.1.2.tar.gz jdk-7u75-linux-x64.gz CentOS的网络配置 1.设置主机中VMware ...
- Hadoop伪分布模式配置
本作品由Man_华创作,采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可.基于http://www.cnblogs.com/manhua/上的作品创作. 请先按照上一篇文章H ...
- Linux环境搭建Hadoop伪分布模式
Hadoop有三种分布模式:单机模式.伪分布.全分布模式,相比于其他两种,伪分布是最适合初学者开发学习使用的,可以了解Hadoop的运行原理,是最好的选择.接下来,就开始部署环境. 首先要安装好Lin ...
随机推荐
- Android Library项目发布到JCenter最简单的配置方法
前沿 网上的步骤看起来实在太麻烦,gituhb上偶然间看到的一个项目,经过实际验证确实可行.github连接:https://github.com/xiaopansky/android-library ...
- 如何虚拟机里安装Win8操作系统
不多说,直接上干货! Windows Server 2003.2008.2012系统的安装 推荐网址:打开MSDN网站(http://msdn.itellyou.cn ) 关于给电脑换系统,很多人会花 ...
- Python -- 网络编程 -- Socket发送文件
客户端如果直接send两次,一次发文件名,一次发文件内容 服务端接受的时候会一起接收,不知怎么分开发送,或者分开接收, 或者全部接收再解析内容 今天发现传送mp3文件的时候没问题,传送文本文件的话,以 ...
- spring整合mongodb
使用spring整合mongodb maven 依赖 <dependency> <groupId>org.mongodb</groupId> <artifac ...
- 移动设备的HTML页面中图片实现滚动加载
如今移动互联网风靡全球,移动页面的元素也是丰富多彩,一个移动页面的图片超过10张已经是再正常不过的事情了.但是相对,很多移动用户还停留在2G,3G这样的网络中.那么这样带宽的用户,在浏览这样的页面时, ...
- can/socket can
1. 概念 参考:Linux-CAN编程详解 can引脚: cn2: 15:CAN1_H 19 CAN1_L 根据每组报文开头的 11 位标识符(扩展帧为29位标识符.CAN 2.0A 规范)解释数据 ...
- Window环境配置Mongodb
Mongodb这几天也了解了一下,今天配置了下环境,从今天开始学下Mongodb数据库. 一.下载 在这个网址中选择要下载的开发环境https://www.mongodb.com/download-c ...
- Linux errno详解
1. 错误码 / errno Linux中系统调用的错误都存储于 errno中,errno由操作系统维护,存储就近发生的错误,即下一次的错误码会覆盖掉上一次的错误. PS: 只有当系统调用或者调用li ...
- java课件运行实践
两数相加 源文件:Addition.java 源代码: // An addition program import javax.swing.JOptionPane; // import class ...
- Bootstrap table 跨页全选
此代码是针对于bootstrap table中分页的跨页全选. 以下是整个bootstrap的定义 <script type="text/javascript" src=&q ...