CentOS7部署HDP3.1.0.0

Apache Ambari是一个基于Web的支持Apache Hadoop集群的供应、管理和监控的开源工具，Ambari已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI进行可视化的集群管理，简化了大数据平台的安装、使用难度。

主机配置规划

主机名	IP	配置	操作系统	角色
c1.heboan.com	10.1.1.121	2核/8G	CentOS Linux release 7.2.1511	Ambari server, mariadb
c2.heboan.com	10.1.1.120	2核/8G	CentOS Linux release 7.2.1511	agent
c3.heboan.com	10.1.1.120	2核/8G	CentOS Linux release 7.2.1511	agent

准备好软件包放到/root/rools/目录下

设置主机名

绑定主机名与ip的关系

10.1.1.120 c1.heboan.com

10.1.1.121 c2.heboan.com

10.1.1.122 c3.heboan.com

/etc/hosts

配置文件打开数

#vim /etc/security/limits.conf

//末尾加上

* soft nofile

* hard nofile

* soft nproc

* hard nproc unlimited

* hard memlock ulimited

* soft memlock unlimited

/etc/security/limits.conf

关闭防火墙和selinux

ntp服务配置

设置swap

echo vm.swappiness =  >> /etc/sysctl.conf

sysctl -p

/etc/sysctl.conf

设置透明大页面

# echo never > /sys/kernel/mm/transparent_hugepage/defrag

# echo never > /sys/kernel/mm/transparent_hugepage/enabled

将如下脚本添加到/etc/rc.d/rc.local文件中

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi

设置透明大页面

安装jdk

tar xf jdk-8u131-linux-x64.tar.gz -C /usr/local/

cd /usr/local/;ln -s jdk1..0_131 jdk

vim /etc/profile.d/jdk.sh

    export JAVA_HOME=/usr/local/jdk

    export JAVA_BIN=/usr/local/jdk/bin

    export PATH=$PATH:$JAVA_HOME/bin

    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

    export JAVA_HOME JAVA_BIN PATH CLASSPATH

source /etc/profile.d/jdk.sh

/etc/profile.d/jdk.sh

安装jdbc驱动

mkdir /usr/share/java

mv mysql-connector-java-5.1..jar /usr/share/java/mysql-connector-java.jar

chmod +x /usr/share/java/mysql-connector-java.jar

配置ssh免密互信

ssh-keygen

ssh-copy-id -i ~/.ssh/id_rsa.pub c1.heboan.com

ssh-copy-id -i ~/.ssh/id_rsa.pub c2.heboan.com

ssh-copy-id -i ~/.ssh/id_rsa.pub c3.heboan.com

cd .ssh/

scp id_rsa id_rsa.pub c2.heboan.com:/root/.ssh/

scp id_rsa id_rsa.pub c3.heboan.com:/root/.ssh/

ssh配置

搭建本地yum源（c1.heboan.com）

安装httpd服务

yum install -y httpd

systemctl start httpd

配置仓库

yum install -y createrepo

tar xf ambari-2.7.3.0-centos7.tar.gz -C /var/www/html/

tar xf HDP-3.1.0.0-centos7-rpm.tar.gz -C /var/www/html/

tar xf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/

tar xf HDP-GPL-3.1.0.0-centos7-gpl.tar.gz -C /var/www/html/

cd /var/www/html/ambari/;createrepo .

cd /var/www/html/HDP;createrepo .

cd /var/www/html/HDP-UTILS/;createrepo .

cd /var/www/html/HDP-GPL;createrepo .

#/etc/yum.repos.d/am.repo

[amrepo]

name = am_repo

baseurl = http://c1.heboan.com/ambari

enable = true

gpgcheck = false

#/etc/yum.repos.d/hdp.repo

[hdprepo]

name = hdp_repo

baseurl = http://c1.heboan.com/HDP

enable = true

gpgcheck = false

#/etc/yum.repos.d/hdp-utils.repo

[hdp-utils-repo]

name = hdp-utils_repo

baseurl = http://c1.heboan.com/HDP-UTILS

enable = true

gpgcheck = false

#/etc/yum.repos.d/hdp-gpl.repo

[hdp-gpl-repo]

name = hdp-gpl_repo

baseurl = http://c1.heboan.com/HDP-GPL

enable = true

gpgcheck = false

yum repolist

#然后也am.repo, hdp.repo, hdp-utils.repo,hdp-gpl.repo 传到其他机器节点

yum配置

配置数据库MariaDB(c1.heboan.com)

安装数据库

yum install -y mariadb mariadb-server mariadb-devel

修改字符集

# vim /etc/my.cnf.d/server.cnf

...

[mysqld]

lower_case_table_names=

character-set-server = utf8

collation-server = utf8_general_ci

启动数据库并且初始化配置

systemctl start mariadb

systemctl enable mariadb

mysql_secure_installation

创建库并账号授权

> create database ambari;

> grant all privileges on *.* to  'bigdata'@'%' identified by 'Bigdata@123';

> flush privileges;

部署ambari（c1.heoan.com）

安装ambari

yum -y install ambari-server

导入数据库

> use ambari

> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

配置ambari

[root@c1 ~]# ambari-server setup

Using python  /usr/bin/python

Setup ambari-server

Checking SELinux...

SELinux status is 'disabled'

Customize user account for ambari-server daemon [y/n] (n)?

Adjusting ambari-server permissions and ownership...

Checking firewall status...

Checking JDK...

Do you want to change Oracle JDK [y/n] (n)? y

[] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files

[] Custom JDK

==============================================================================

Enter choice ():

WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.

WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.

Path to JAVA_HOME: /usr/local/jdk

Validating JDK on Ambari Server...done.

Check JDK version for Ambari Server...

JDK version found:

Minimum JDK version is  for Ambari. Skipping to setup different JDK for Ambari Server.

Checking GPL software agreement...

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)? y

Configuring database...

==============================================================================

Choose one of the following options:

[] - PostgreSQL (Embedded)

[] - Oracle

[] - MySQL / MariaDB

[] - PostgreSQL

[] - Microsoft SQL Server (Tech Preview)

[] - SQL Anywhere

[] - BDB

==============================================================================

Enter choice ():

Hostname (localhost):

Port ():

Database name (ambari):

Username (bigdata): bigdata

Enter Database Password ():

Configuring ambari database...

Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y

Configuring remote database connection properties...

WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Proceed with configuring remote database connection properties [y/n] (y)? y

Extracting system views...

ambari-admin-2.7.3.0..jar

....

Ambari repo file doesn't contain latest json url, skipping repoinfos modification

Adjusting ambari-server permissions and ownership...

Ambari Server 'setup' completed successfully.

配置

启动ambari-server

[root@c1 ~]# ambari-server start

Using python  /usr/bin/python

Starting ambari-server

Ambari Server running with administrator privileges.

Organizing resource files at /var/lib/ambari-server/resources...

Ambari database consistency check started...

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start............................................

Server started listening on 

DB configs consistency check: no errors and warnings were found.

Ambari Server 'start' completed successfully.

浏览器访问http://10.1.1.120:8080, 账号密码：admin/admin

启动向导创建集群

配置集群名字

配置本地仓库

配置主机和私钥

配置主机

根据实际需求选择安装的组件

服务节点分布

选择安装slaves and clients

配置密码

配置hive 元数据存储数据库，选择新的mysql

[root@c1 ~]# ambari-server setup --jdbc-db=mysql  --jdbc-driver=/usr/share/java/mysql-connector-java.jar