1.linux环境配置

首先说一下,这里是虚拟机环境.

1.用vbox安装centos6.8-mini

注意不要使用复制的方式安装,复制的虚拟机网络不通

安装如下:

主机	ip	角色	内存
hadoop1	192.168.0.10	namenode,HMASTER	2G
hadoop2	192.168.0.11	namenode,HiverServer2,HiveMetaStore	1G
hadoop3	192.168.0.12	ZooKeeper,DateNode,HRegionServer,JournalNode	2G
hadoop4	192.168.0.13	ZooKeeper,DateNode,HRegionServer,JournalNode	2G
hadoop5	192.168.0.14	ZooKeeper,DateNode,HRegionServer,JournalNode	2G

2.修改/etc/hosts

所有机器上执行

echo "127.0.0.1     localhost localhost">/etc/hosts
echo "192.168.0.10  hadoop1">>/etc/hosts
echo "192.168.0.11  hadoop2">>/etc/hosts
echo "192.168.0.12  hadoop3">>/etc/hosts
echo "192.168.0.13  hadoop4">>/etc/hosts
echo "192.168.0.14  hadoop5">>/etc/hosts

3.配置SSH互信

最好把root hdfs yarn都配上ssh互信

1.生成公钥

所有节点执行

ssh-keygen -t rsa -P ''

2.所有节点执行

ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.10
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.11
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.12
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.13
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.14

这样所有节点之间都有了互信

3.创建用户和组

在所有节点创建hdfs yarn hive zookeeper hbase用户,后续添加其它用户

groupadd hadoop
useradd -g hadoop hdfs
passwd hdfs <<EOF
hdfs
hdfs
EOF
useradd -g hadoop yarn
passwd yarn <<EOF
yarn
yarn
EOF
groupadd zookeeper
useradd -g zookeeper zookeeper
passwd zookeeper <<EOF
zookeeper
zookeeper
EOF
groupadd hive8
useradd -g hive hive
passwd hive <<EOF
hive
hive
EOF
groupadd hbase
useradd -g hbase hbase
passwd hbase <<EOF
hbase
hbase
EOF

4.修改系统参数

echo "session          required       /lib64/security/pam_limits.so ">> /etc/pam.d/login
echo "@hdfs            -              nofile         4096 ">>/etc/security/limits.conf
echo "@hive            -              nofile         4096 ">>/etc/security/limits.conf
echo "@yarn            -              nofile         4096 ">>/etc/security/limits.conf
echo "@zookeeper       -              nofile         4096 ">>/etc/security/limits.conf
echo "@hbase           -              nofile         4096 ">>/etc/security/limits.conf
echo "@hdfs            -              nproc          1024 ">>/etc/security/limits.conf
echo "@hive            -              nproc          1024 ">>/etc/security/limits.conf
echo "@yarn            -              nproc          1024 ">>/etc/security/limits.conf
echo "@zookeeper       -              nproc          1024 ">>/etc/security/limits.conf
echo "@hbase           -              nproc          1024 ">>/etc/security/limits.conf
echo "@hdfs            -              memlock        32768" >>/etc/security/limits.conf

设置完后,应用程序需要重启生效.

5.关闭防火墙及SELINUX

#关闭防火墙
service iptables stop
chkconfig iptables off
setenforce 0 #关闭selinux

永久关闭SELINUX:

vi /etc/selinux/config
SELINUX=disabled

6.安装ntp

yum -y install ntp

在ntp服务器节点:

vi /etc/ntp.conf
#本子网内主机都可以同步
restrict 192.168.0.0 mask 255.255.0.0 nomodify
#优先时间服务器
server  192.168.0.10 prefer
#当ntp server不可用,把本地时间做为ntp时间给端用
fudge 127.127.1.0 stratum 8
#日志文件位置
logfile /var/log/ntp.log

启动ntpd服务

service ntpd start

这里将使用192.168.0.10作为时间服务器,建议使用网上的时间服务器比较准确.

客户端节点:

vi /etc/ntp.conf
#优先时间服务器
server  192.168.0.10 prefer
#日志文件位置
logfile /var/log/ntp.log

客户端先手动同步一次时间:

nptdate hadoop1

然后启动nptd服务:

service ntpd start

查看npt状态:

watch ntpq -p

7.禁用IPV6

IPV6会引起某些问题,有些程序默认搜索IPV6.

参考:https://linux.cn/article-5417-1.html

vi /etc/sysctl.conf
# 禁用整个系统所有接口的IPv6
net.ipv6.conf.all.disable_ipv6 = 1
# 禁用某一个指定接口的IPv6(例如：eth0, lo)
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1

#使用修改生效:
$ sudo sysctl -p /etc/sysctl.conf

8.配置环境变量

下载java1.8,解压到/opt/下. vi /etc/profile

export JAVA_HOME=/opt/jdk1.8.0_131
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME
export PATH

测试java:

java -version

来自为知笔记(Wiz)