HDP集群部署

一、环境准备

1. centos7 安装和系统环境处理
   
   1）运行初始化脚本

!/bin/bash

-------------------------------------

系统环境初始化脚本

本脚本仅支持centos， centos7.7验证成功

使用root用户运行

------------------------------------

1 依赖安装

yum clean all

yum makecache

yum update -y

yum install -y openssh-server vim gcc gcc-c++ glibc-headers bzip2-devel lzo-devel curl wget openssh-clients zlib-devel autoconf automake cmake libtool openssl-devel fuse-devel snappy-devel telnet unzip zip net-tools.x86_64 firewalld systemd svn ncurses-devel bzip2 psmisc libxslt-devel libxml2-devel libxslt.so.1 cyrus-sasl-plain yum groupinstall chinese-support mod_ssl ntp

2 关闭防火墙

echo "close firewall ----------------------------------------------------------"

firewall-cmd --state

systemctl status firewalld.service

systemctl stop firewalld.service

systemctl disable firewalld.service

systemctl status firewalld.service

3 关闭SELINUX

echo "close SELINUX ----------------------------------------------------------"

setenforce 0

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' '/etc/selinux/config'

4 修改UMASK

echo "set umask 0022 ----------------------------------------------------------"

umask 0022

echo umask 0022 >> /etc/profile

5 关闭swappiness

swap_dev=swapon -s | awk '{if (NR>1){print $1}}'

echo "swap device is ${swap_dev} ----------------------------------------------------------"

echo "close swap device ${swap_dev} ----------------------------------------------------------"

swapoff ${swap_dev}

if [ "$?" != 0 ] ; then

echo "close swap device ${swap_dev} failed ----------------------------------------------------------"

else

echo "close swap device ${swap_dev} success ----------------------------------------------------------"

fi

6 资源限制

echo "set ulimit 102400 ----------------------------------------------------------"

ulimit -Hn 102400

ulimit -Sn 102400

echo "* - nproc 102400" >> /etc/security/limits.conf

echo "* - nofile 102400" >> /etc/security/limits.conf

7 时钟同步

echo "start ntp ----------------------------------------------------------"

systemctl start ntpd

systemctl enable ntpd

systemctl status ntpd

8 禁止huge_page

echo "close huge_page ----------------------------------------------------------"

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local

echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

9 生成公钥私钥

echo "ssh-keygen ----------------------------------------------------------"

mkdir ~/.ssh

ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ''

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 700 ~/.ssh

chmod 600 ~/.ssh/authorized_keys

2）如果时区不合适，需要修改时区

[root@centos ~]# timedatectl

3）hostname 修改

[root@centos ~]# hostnamectl set-hostname hdp-master01

4）添加hosts

vim /etc/hosts

5）免密登陆

ambari-server root用户ssh免密登陆所有其他节点

[root@centos ~]# ssh-copy-id root@node03

2. java8

在 /etc/profile.d/目录下，增加java.sh

!/bin/bash

java.sh

JAVA_HOME=/usr/local/services/jdk1.8.0_191

JRE_HOME=/usr/local/services/jdk1.8.0_191/jre

PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

export JAVA_HOME JRE_HOME PATH CLASSPATH

3. python2.7.5

yum -y install python-pip

pip install --upgrade pip

4. mysql

Node03:3306

root/12345678

yum install mysql-connector-java*

5. 目录规划

mkdir /hadoop

将所有磁盘挂载或者链接到此目录下

/hadoop/disk01

/hadoop/disk02

二、建设本地yum repo

有网络可以直接通过网络，无网络需要先拷贝所有需要下载的文件

依赖安装

yum install yum-utils createrepo httpd

创建目录

mkdir -p /var/www/html

mkdir -p /var/www/html/ambari

mkdir -p /var/www/html/hdp

下载repo url文件

cd /etc/yum.repos.d/

wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari.repo

wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/hdp.repo

查看repo

yum repolist

下载repo文件到本地

cd /var/www/html/ambari

reposync -r ambari-2.7.4.0

cd /var/www/html/hdp

reposync -r HDP-3.1.4.0

reposync -r HDP-UTILS-1.1.0.22

新建本地repo

createrepo /var/www/html/ambari/ambari-2.7.4.0

createrepo /var/www/html/hdp/HDP-3.1.4.0

createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.22

启动http服务

systemctl start httpd.service

systemctl enable httpd.service

修改repo web地址

[root@node03 ~]#vim /etc/yum.repos.d/ambari.repo

[ambari-2.7.4.0]

baseurl=http://node03/ambari/ambari-2.7.4.0

...

[root@node03 ~]#vim /etc/yum.repos.d/hdp.repo

[HDP-3.1.4.0]

name=HDP Version - HDP-3.1.4.0

baseurl=http://node03/hdp/HDP-3.1.4.0

...

[HDP-UTILS-1.1.0.22]

name=HDP-UTILS Version - HDP-UTILS-1.1.0.22

baseurl=http://node03/hdp/HDP-UTILS-1.1.0.22

...

/etc/yum.repos.d/ambari.repo和hdp.repo需要拷贝到ambari-server将要部署的服务器

三、部署ambari-server

1.准备数据库

CREATE DATABASE ambari default character set utf8mb4 collate utf8mb4_unicode_ci;

GRANT ALL PRIVILEGES ON . TO 'root'@'%' IDENTIFIED BY '12345678' WITH GRANT OPTION;

1.安装ambari-server

yum install ambari-server

2.设置

[root@node01 /etc/yum.repos.d]#ambari-server setup

Customize user account for ambari-server daemon [y/n] (n)? y

Enter user account for ambari-server daemon (root):root

Checking JDK...

[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

[2] Custom JDK

Enter choice (1): 2

Path to JAVA_HOME: /usr/local/services/jdk1.8.0_191

GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html

Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y

Enter advanced database configuration [y/n] (n)? y

Configuring database...

Choose one of the following options:

[3] - MySQL / MariaDB

Enter choice (1): 3

Hostname (localhost): node03

Port (3306): 3306

Database name (ambari): ambari

Username (ambari): root

Enter Database Password (bigdata):

Re-enter password: 12345678

Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y

Configuring remote database connection properties...

WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Proceed with configuring remote database connection properties [y/n] (y)? y

3.建表

use ambari

source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

4.启动

端口 8080

ambari-server start

ambari-server status

ambari-server stop

ambari-server restart

四、添加组件

注册 admin/admin

创建集群

选择版本设置仓库

添加节点

上传ambari-server所在服务器的私钥id_rsa

配置用户和ssh端口

使用的host不是域名，下一步

注册所有节点

添加组件

选择组件实际部署节点

设置grafana 帐号密码以及Activity 密码

设置文件目录（需要提前规划）

设置各个组件的启动用户

HDFS服务设置

部署

五、测试

测试hdfs 上传文件

hadoop fs mkdir /tmp/test/in

hadoop fs -put text.txt /tmp/test/in

测试yarn 运行任务

hadoop jar /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /tmp/test/in /tmp/test/out