install Hadoop
Installing Java
Hadoop runs on both Unix and Windows operating systems, and requires Java to be
installed. For a production installation, you should select a combination of operating
system, Java, and Hadoop that has been certified by the vendor of the Hadoop distribution
you are using. There is also a page on the Hadoop wiki that lists combinations
that community members have run with success.
#Master 192.168.1.201 jdk、hadoop NameNode、DFSZKFailoverController(zkfc)、ResourceManager、JournalNode
#Slave1 192.168.1.202 jdk、hadoop NameNode、DFSZKFailoverController(zkfc)、ResourceManager
、JournalNode
#Slave2 192.168.1.203 jdk、hadoop、zookeeper DataNode、NodeManager、QuorumPeerMain
#Slave3 192.168.1.203 jdk、hadoop、zookeeper DataNode、NodeManager、QuorumPeerMain
#Slave4 192.168.1.204 jdk、hadoop、zookeeper DataNode、NodeManager、QuorumPeerMain
##############虚拟机集中配置本机环境######################
sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config
chkconfig iptables off
service iptables stop
rm /etc/udev/rules.d/70-persistent-net.rules
#del MAC form /etc/sysconfig/network-scripts/ifcfg-eth0
tar -zxvf jdk-8u74-linux-x64.tar.gz -C /usr/java
rm /usr/java/latest -rf
ln -s /usr/java/jdk1.8.0_74 latest
echo "export JAVA_HOME=/usr/java/latest" >> /etc/profile
echo "export HADOOP_HOME=/usr/local/hadoop" >> /etc/profile
echo "export HBASE_HOME=/usr/local/hbase" >> /etc/profile
echo "export ZOOKEEPER_HOME=/usr/local/zookeeper" >> /etc/profile
echo "export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:" >> /etc/profile
source /etc/profile
cd && tar -zxvf hadoop-2.6.4.tar.gz -C /usr/local/
mv /usr/local/hadoop-2.6.4 /usr/local/hadoop
echo "slave1" >> /usr/local/hadoop/etc/hadoop/slaves
echo "slave2" >> /usr/local/hadoop/etc/hadoop/slaves
echo "slave3" >> /usr/local/hadoop/etc/hadoop/slaves
##############虚拟机集中配置本机环境######################
##############虚拟机分散配置克隆环境########################
sed -i '/^HOSTNAME=/cHOSTNAME=master' /etc/sysconfig/network
sed -i '/^HOSTNAME=/cHOSTNAME=slave1' /etc/sysconfig/network
sed -i '/^HOSTNAME=/cHOSTNAME=slave2' /etc/sysconfig/network
sed -i '/^HOSTNAME=/cHOSTNAME=slave3' /etc/sysconfig/network
echo "192.168.26.130 master" >> /etc/hosts
echo "192.168.26.163 slave1" >> /etc/hosts
echo "192.168.26.164 slave2" >> /etc/hosts
echo "192.168.26.165 slave3" >> /etc/hosts
cd && mkdir .ssh
ssh-keygen -N '' -t rsa -q -f id_rsa
ssh-copy-id master
ssh-copy-id slave1
ssh-copy-id slave2
ssh-copy-id slave3
##############虚拟机分散配置克隆环境########################
#######################配置Zookeeper####################################################
tar -zxvf zookeeper-3.4.8.tar.gz -C /usr/local/
mv /usr/local/zookeeper-3.4.8/ /usr/local/zookeeper
cd /usr/local/zookeeper/conf/
cp zoo_sample.cfg zoo.cfg
mkdir /usr/local/zookeeper/data -p
sed -i '/^dataDir/cdataDir=/usr/local/zookeeper/data' /usr/local/zookeeper/conf/zoo.cfg
echo "server.0=slave1:2288:3388" >> /usr/local/zookeeper/conf/zoo.cfg
echo "server.1=slave2:2288:3388" >> /usr/local/zookeeper/conf/zoo.cfg
echo "server.2=slave3:2288:3388" >> /usr/local/zookeeper/conf/zoo.cfg
echo 0 > /usr/local/zookeeper/data/myid
#echo 1 > /usr/local/zookeeper/data/myid
#echo 2 > /usr/local/zookeeper/data/myid
#######################配置Zookeeper####################################################
#######################配置HADOOP#######################################################
#hadoo-env.sh
cd /usr/local/hadoop/etc/hadoop
sed -i '/^export JAVA_HOME/cexport JAVA_HOME=/usr/java/latest' hadoop-env.sh
#core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- 指定hdfs的nameservice为masters -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://masters</value>
</property>
<!-- 指定hadoop临时目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<!-- 指定zookeeper地址 -->
<property>
<name>ha.zookeeper.quorum</name>
<value>slave1:2181,slave2:2181,slave3:2181</value>
</property>
</configuration>
#hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!--指定hdfs的nameservice为masters，需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>masters</value>
</property>
<!-- Master下面有两个NameNode，分别是Master，Slave1 -->
<property>
<name>dfs.ha.namenodes.masters</name>
<value>master,slave1</value>
</property>
<!-- Master的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.masters.master</name>
<value>master:9000</value>
</property>
<!-- Master的http通信地址 -->
<property>
<name>dfs.namenode.http-address.masters.master</name>
<value>master:50070</value>
</property>
<!-- Slave1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.masters.slave1</name>
<value>slave1:9000</value>
</property>
<!-- Slave1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.masters.slave1</name>
<value>slave1:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://slave1:8485;slave2:8485;slave3:8485/masters</value>
</property>
<!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/local/hadoop/journal</value>
</property>
<!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.masters</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行-->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
</value>
</property>
<!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!-- 配置sshfence隔离机制超时时间 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<!-- 指定mr框架为yarn方式 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
<configuration>
<!-- 开启RM高可靠 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>RM_HA_ID</value>
</property>
<!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>slave1</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<!-- 指定zk集群地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>slave1:2181,slave2:2181,slave3:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
scp -r /usr/local/hadoop/etc/hadoop slave1:/usr/local/hadoop/etc/hadoop
scp -r /usr/local/hadoop/etc/hadoop slave2:/usr/local/hadoop/etc/hadoop
scp -r /usr/local/hadoop/etc/hadoop slave3:/usr/local/hadoop/etc/hadoop
#######################配置HADOOP#######################################################
##############配置hbase#########################hbase-**
tar -zxvf hbase-** /usr/local/hbase
##############配置hbase#########################hbase-**
##############COMMAND##############################
cd /usr/local/hadoop/sbin
hadoop-daemon.sh start journalnode
cd /usr/local/zookeeper/bin/
zkServer.sh start
zkServer.sh status
hdfs namenode -format
scp -r /usr/local/hadoop/tmp slave1:/usr/local/hadoop/
hdfs zkfc -formatZK
start-dfs.sh
start-yarn.sh
yarn-daemon.sh start resourcemanager
hadoop fs -put /etc/profile /profile
hadoop fs -ls /
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount /profile /out
start-hbase.sh
##############COMMAND#########################
install Hadoop的更多相关文章
- Hadoop学习日志- install hadoop
资料来源 : http://www.tutorialspoint.com/hadoop/hadoop_enviornment_setup.htm Hadoop 安装 创建新用户 $ su passwo ...
- install hadoop on xubuntu
0. install xubuntu we recommend to set username as "hadoop" after installation, set user & ...
- mac osx 系统 brew install hadoop 安装指南
mac osx 系统 brew install hadoop 安装指南 brew install hadoop 配置 core-site.xml:配置hdfs文件地址(记得chmod 对应文件夹 ...
- How to install Hadoop
1.How to install Hadoop 3.0.0 http://blog.sina.com.cn/s/blog_4a1f59bf01010kx3.html 2.How to install ...
- [Spark] 00 - Install Hadoop & Spark
Hadoop安装 Java环境配置 安装课程:安装配置 配置手册:Hadoop安装教程_单机/伪分布式配置_Hadoop2.6.0/Ubuntu14.04[依照步骤完成配置] jsk安装使用的链接中第 ...
- Steps to Install Hadoop on CentOS/RHEL 6---reference
http://tecadmin.net/steps-to-install-hadoop-on-centosrhel-6/# The Apache Hadoop software library is ...
- yum install hadoop related client
yum list avaliable hadoop\* yum list installed yum repolist repo is in /etc/yum.repos.d yum install ...
- How to install Hadoop Cluster
https://dwbi.org/etl/bigdata/183-setup-hadoop-cluster https://www.linode.com/docs/databases/hadoop/h ...
- Install hadoop on windows(non-virtual machine, such cygwin)
DownloadBefore starting make sure you have this two softwares Hadoop 2.7.1 Java – Jdk 1.7+ Extract d ...
随机推荐
- 用Python获取沪深两市上市公司股票信息，提取创近10天股价新高的、停牌的、复牌不超过一天或者新发行的股票，并存入mysql数据库
#该脚本可以提取沪深两市上市公司股票信息,并按以下信息分类:(1)当天股价创近10个交易日新高的股票:(2)停牌的股票:(3)复牌不超过一个交易日或者新发行的股票 #将分类后的股票及其信息(股价新高. ...
- SQL.集合转字符串
场景:有个表,里面有NUM1,NUM2,NUM3,NUM4,我们打算把所有数据拼接成一个值. 在.net中使用的方法应该是两层for循环拼接起来,下面来介绍下sql中的方式 第一种方法,动态sql d ...
- c语言中gets ，getschar 和fgets 的用法及三者之间的差别，还有scanf
①gets [1]函数:gets(字符指针) [2]头文件:stdio.h(c中),c++不需包含此头文件 [3]原型:char*gets(char*buffer); [4]功能:从stdin流中读取 ...
- oracle xmltype导入并解析Excel数据 (一)创建表与序
表说明: T_EXCEL_IMPORT_DATASRC: Excel数据存储表,(使用了xmltype存储Excel数据) 部分字段说明: BUSINESSTYPE: Excel模板类型,一个Exce ...
- Spring的Bean的基本概念
Spring其实就是一个大型的工厂,而Spring容器中的Bean就是该工厂的产品.Spring容器能够生产哪些产品,取决于配置文件的配置. 对于我们而言,使用Spring框架做两件事:开发Bean. ...
- Android 简易崩溃日志保存
仅仅做了简单的保存到了本地而已: 根据需要可以继续增加功能: 下一次启动上传到服务器: 增加应用版本,机型系统版本信息等: public class CrashSaver { public stati ...
- B样条基函数（cubic spline basis）
B样条基函数用作权重 reference http://blog.csdn.net/tuqu
- 以策略为导向的VI设计
从美学角度评价一个标志(LOGO)不是最佳方法,标志设计不是为了参加选美.标识要求除了审美功能外还应包括很多元素. 要说标志,先从CIS(企业形象系统)开始入手,里面分成MI.VI和BI,分别指企业的 ...
- Mac OS 后台服务注册
/Library/LaunchAgents/ /Library/LaunchDaemons
- web框架思考
以前一直不明白web框架是怎样实现路由.orm.接受请求的.今天看了下廖雪峰的python 实现web框架博客才明白. 简单总结并记录: http请求->wsgi->处理请求->返回 ...