基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库01 —— 准备环境，搭建本地仓库，安装ambari
一、集群软硬件环境准备：
操作系统： centos 7 x86_64.1804
Ambari版本：2.7.0
HDP版本：3.0.0
HAWQ版本：2.3.0
5台PC作为工作站：
| ep-bd01 | ep-bd02 | ep-bd03 | ep-bd04 | ep-bd05 |
其中ep-bd01作为主节点，用于安装ambari-server。
二、配置操作系统，安装必备软件
1，安装CentOS 7操作系统：环境配置，安装必备软件。
2，安装配置NTP服务，保证集群时间保持同步，以防止由于时间不同而造成掉线故障。
详细看随笔：基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库之——安装配置NTP服务，保证集群时间保持同步
见《安装配置NTP服务》
3，安装MariaDB Server用于Ambari server以及Hue和Hive
详细过程，参见：基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库之一 —— MariaDB 安装配置
4，安装yum priorities plugin
yum install yum-plugin-priorities -y
三、搭建本地仓库：
1，下载软件包：
cd /root
mkdir downloads
cd downloads
wget http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.0.0.0/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/hdp.repo
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari.repo
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-centos7-rpm.tar.gz
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari-2.7.0.0-centos7.tar.gz
2、搭建本地仓库：
安装并开启Apache HTTP服务
yum install httpd -y
systemctl enable httpd
systemctl start httpd
安装HTTPD服务
确保/var/www/html目录存在，没有的话创建。
mkdir -p /var/www/html
创建HDP，HDF子目录
cd /var/www/html
mkdir hdp hdf
解开下载的软件包：
cd /var/www/html
tar -zxvf /root/downloads/ambari-2.7.0.0-centos7.tar.gz -C .
tar -zxvf /root/downloads/HDP-3.0.0.0-centos7-rpm.tar.gz -C ./hdp
tar -zxvf /root/downloads/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz -C ./hdp
tar -zxvf /root/downloads/HDP-UTILS-1.1.0.22-centos7.tar.gz -C ./hdp
修改下载的ambari.repo，
vim ambari.repo
安装如下内容修改，[注意版本号，需要根据具体下载的版本不同修改，解压后自己查看一下]：
#VERSION_NUMBER=2.7.0.0-
[ambari-2.7.0.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.0.0
baseurl=http://ep-bd01/ambari/centos7/2.7.0.0-897
gpgcheck=1
gpgkey=http://ep-bd01/ambari/centos7/2.7.0.0-897/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
复制到/etc/yum.repos.d
cp ambari.repo /etc/yum.repos.d/ambari.repo
修改下载的hdp.repo，
vim hdp.repo
安装如下内容修改，[注意版本号，需要根据具体下载的版本不同修改，解压后自己查看一下]：
#VERSION_NUMBER=3.0.0.0-1634
[HDP-3.0]
name=HDP Version - HDP-3.0.0.0
baseurl=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-3.0-GPL]
name=HDP GPL Version - HDP-GPL-3.0.0.0
baseurl=http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
保存退出，复制到/etc/yum.repos.d/
cp hdp.repo /etc/yum.repos.d/hdp.repo
四、主节点安装ambari server
1，使用刚才配置好的本地仓库，直接yum命令安装。
yum install ambari-server -y
2，查看ambari server 状态
systemctl status ambari-server ● ambari-server.service - LSB: ambari-server daemon
Loaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)
Active: inactive (dead)
Docs: man:systemd-sysv-generator()
看到ambari server已成功安装了。
3，配置mariadb，建立用户和数据库供ambari使用
建立数据库用户ambari
mysql -uroot -p
MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [mysql]> grant all privileges on *.* to 'ambari'@'%' identified by 'ambari';
Query OK, 0 rows affected (0.06 sec)
MariaDB [mysql]> flush privileges;
Query OK, 0 rows affected (0.00 sec)
MariaDB [mysql]>
建立数据库ambari，并运行ambari数据库建表sql命令文件。
MariaDB [mysql]> create database ambari;
Query OK, row affected (0.01 sec) MariaDB [mysql]> use ambari;
Database changed
MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
Query OK, rows affected (0.01 sec) Query OK, rows affected (0.00 sec) Query OK, rows affected (0.00 sec) Query OK, rows affected (0.06 sec) Query OK, rows affected (0.00 sec)
Statement prepared
... ...
4，配置ambari server
执行命令
ambari-server setup
回答选择项，其中JDK选择“”Custom“”，给出系统安装目录，数据库一定要选择高级配置，指定mariadb数据库和用户，本人系统中详细过程如下：
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files
[] Custom JDK
==============================================================================
Enter choice ():
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? ^C
Aborting ... Keyboard Interrupt.
[root@ep-bd01 downloads]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files
[] Custom JDK
==============================================================================
Enter choice ():
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: ^C
Aborting ... Keyboard Interrupt.
[root@ep-bd01 downloads]# echo $JAVA_HOME
/usr/java/jdk1..0_181-amd64
[root@ep-bd01 downloads]# $JAVA_HOME/bin/java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) -Bit Server VM (build 25.181-b13, mixed mode)
[root@ep-bd01 downloads]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files
[] Custom JDK
==============================================================================
Enter choice ():
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: ^C
Aborting ... Keyboard Interrupt.
[root@ep-bd01 downloads]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files
[] Custom JDK
==============================================================================
Enter choice ():
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz
ERROR: Exiting with exit code .
REASON: Downloading or installing JDK failed: 'Fatal exception: Failed to download JDK: <urlopen error [Errno -2] Name or service not known>. Please check that the JDK is available at http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz. Also you may specify JDK file location in local filesystem using --jdk-location command line argument., exit code 1'. Exiting.
[root@ep-bd01 downloads]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files
[] Custom JDK
==============================================================================
Enter choice ():
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1..0_181-amd64
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found:
Minimum JDK version is for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[] - PostgreSQL (Embedded)
[] - Oracle
[] - MySQL / MariaDB
[] - PostgreSQL
[] - Microsoft SQL Server (Tech Preview)
[] - SQL Anywhere
[] - BDB
==============================================================================
Enter choice ():
Hostname (localhost): ep-bd01
Port ():
Database name (ambari): ambari
Username (ambari): ambari
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.0.0..jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
5，命令行方式设置mysql数据库连接库，用于oozie和ranger连接mariadb时使用（上面设置后不起作用，必须如下操作）：
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
6，配置ambari-server自启动，启动ambari-server
[root@ep-bd01 downloads]# systemctl enable ambari-server
[root@ep-bd01 downloads]# systemctl start ambari-server
[root@ep-bd01 downloads]# systemctl status ambari-server
● ambari-server.service - LSB: ambari-server daemon
Loaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)
Active: active (running) since Tue -- :: CST; 2min 36s ago
Docs: man:systemd-sysv-generator()
Process: ExecStart=/etc/rc.d/init.d/ambari-server start (code=exited, status=/SUCCESS)
CGroup: /system.slice/ambari-server.service
└─ /usr/java/jdk1..0_181-amd64/bin/java -server -XX:NewRatio= -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction= -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -... Aug :: ep-bd01 ambari-server[]: Organizing resource files at /var/lib/ambari-server/resources...
Aug :: ep-bd01 ambari-server[]: Ambari database consistency check started...
Aug :: ep-bd01 ambari-server[]: Server PID at: /var/run/ambari-server/ambari-server.pid
Aug :: ep-bd01 ambari-server[]: Server out at: /var/log/ambari-server/ambari-server.out
Aug :: ep-bd01 ambari-server[]: Server log at: /var/log/ambari-server/ambari-server.log
Aug :: ep-bd01 ambari-server[]: Waiting for server start.....................
Aug :: ep-bd01 ambari-server[]: Server started listening on
Aug :: ep-bd01 ambari-server[]: DB configs consistency check: no errors and warnings were found.
Aug :: ep-bd01 ambari-server[]: Ambari Server 'start' completed successfully.
Aug :: ep-bd01 systemd[]: Started LSB: ambari-server daemon.systemctl status ambari-server
● ambari-server.service - LSB: ambari-server daemon
Loaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)
Active: active (running) since Tue -- :: CST; 2min 36s ago
Docs: man:systemd-sysv-generator()
Process: ExecStart=/etc/rc.d/init.d/ambari-server start (code=exited, status=/SUCCESS)
CGroup: /system.slice/ambari-server.service
└─ /usr/java/jdk1..0_181-amd64/bin/java -server -XX:NewRatio= -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction= -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -... Aug :: ep-bd01 ambari-server[]: Organizing resource files at /var/lib/ambari-server/resources...
Aug :: ep-bd01 ambari-server[]: Ambari database consistency check started...
Aug :: ep-bd01 ambari-server[]: Server PID at: /var/run/ambari-server/ambari-server.pid
Aug :: ep-bd01 ambari-server[]: Server out at: /var/log/ambari-server/ambari-server.out
Aug :: ep-bd01 ambari-server[]: Server log at: /var/log/ambari-server/ambari-server.log
Aug :: ep-bd01 ambari-server[]: Waiting for server start.....................
Aug :: ep-bd01 ambari-server[]: Server started listening on
Aug :: ep-bd01 ambari-server[]: DB configs consistency check: no errors and warnings were found.
Aug :: ep-bd01 ambari-server[]: Ambari Server 'start' completed successfully.
Aug :: ep-bd01 systemd[]: Started LSB: ambari-server daemon.
ok，Ambari-server安装完成。浏览器访问ambari-server服务：
http://ep-bd01:8080
默认用户名/密码为：admin/admin，登陆后界面如下：
五、所有主机节点，安装ambari-agent，并配置自启动
yum install ambari-agent -y
systemctl enable ambari-agent
systemctl restart ambari-agent && systemctl status ambari-agent
基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库01 —— 准备环境，搭建本地仓库，安装ambari的更多相关文章
- 基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库——操作系统配置，安装必备软件
注意未经说明,所有本文中所有操作都默认需要作为root用户进行操作. 一.安装zmodem,用于远程上传下载文件,安装gedit,方便重定向到远程windows上编辑文件(通过xlanuch) [ro ...
- Solr4.8.0源码分析(4)之Eclipse Solr调试环境搭建
Solr4.8.0源码分析(4)之Eclipse Solr调试环境搭建 由于公司里的Solr调试都是用远程jpda进行的,但是家里只有一台电脑所以不能jpda进行调试,这是因为jpda的端口冲突.所以 ...
- 基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库02 ——使用ambari-server安装HDP
本文记录使用ambari-server安装HDP的过程,对比于使用cloudera-manager安装CDH,不得不说ambari的易用性差的比较多~_~,需要用户介入的过程较多,或者说可定制性更高. ...
- 基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库04 —— 安装HAWQ插件PXF3.3.0.0
一. 安装PXF3.3.0.0,这里所安装的pxf的包文件都包含在apache-hawq-rpm-2.3.0.0-incubating.tar.gz里面下面步骤都是以root身份执行这里注意,pxf插 ...
- HDP2.0.6+hadoop2.2.0+eclipse（windows和linux下）调试环境搭建
花了好几天,搭建好windows和linux下连接HDP集群的调试环境,在此记录一下 hadoop2.2.0的版本比hadoop0.x和hadoop1.x结构变化很大,没有eclipse-hadoop ...
- 基于WDF的PCI/PCIe接口卡Windows驱动程序（1）-WDF概述及开发环境搭建
原文出处:http://www.cnblogs.com/jacklu/p/4619110.html 本科毕业设计是这方面的工作,所以想开几篇博客来介绍使用WDF开发PCI/PCIe接口卡的驱动程序方法 ...
- vs2012+qt5.2.0环境搭建/vs2013 + qt5.3.2 环境搭建
分类: Windows Qt2014-01-17 00:50 15434人阅读 评论(18) 收藏 举报 此文章已作废,请参考我的新文章: vs2013 + qt5.3.2 环境搭建 ( http:/ ...
- Cocos2d-x v3.0正式版尝鲜体验【1】 环境搭建和新建项目
Cocos2d-x v3.0在前天最终公布正式版了,等了大半年最终出来了.一直没去碰之前的3.0各种beta,rc版本号,就想等正式版出来再尝试. 昨天也參加了触控科技在成都举办的沙龙活动.看到作者王 ...
- .net core 2.0小白笔记（一）：开发运行环境搭建
小白一枚,有任何不妥之处敬请指教 这里不讨论什么设计模式,什么架构,什么什么,就是入门,简单的入门,虽然能跨平台,但是这里还是在win的环境下进行,不扯的那么远 其实官网文档写的挺不错的了,就是偶尔有 ...
随机推荐
- 谈谈canvas的性能优化（主要讲缓存问题）
声明:本文为原创文章,如需转载,请注明来源WAxes,谢谢! canvas玩多了后,就会自动的要开始考虑性能问题了.怎么优化canvas的动画呢? [使用缓存] 使用缓存也就是用离屏canvas进行预 ...
- g++ 动态库的编译及使用
#ifndef __HELLO_H_ #define __HELLO_H_ void print(); #endif #include "hello.h" #include < ...
- Cheapest Palindrome [POJ3280] [区间DP] [经典]
一句话题意:每个字母添加和删除都相应代价(可以任意位置 增加/删除),求把原串变成回文串的最小代价 Description 保持对所有奶牛的跟踪是一项棘手的任务,因此农场主约翰已经安装了一个系统来实现 ...
- 安装并运行Hello World
新建虚拟环境并安装Flask pip install Flask 运行HelloWorld from flask import Flask #导入Flask类 app = Flask(__name__ ...
- Linux内容
1.“~”:当前用户主目录“.”:当前工作目录“..”:当前工作目录的父目录使用pwd命令可以确定当前所在目录的绝对路径$ pwd2.shell命令:(which命令判断是否是外部命令)$ which ...
- django session源码剖析
首先要明白,session和cookie,session是保存在服务器端,cookie存储在浏览器上,我们称为客户端,客户端向服务端发送请求时,会将cookie一起发送给服务端.服务端接收到请求后,会 ...
- 法嵌入互操作类型“SHDocVw.ShellWindowsClass”请改用适用的接口-解决方法
点DLL名字,---属性----------嵌入互操作类型(设置为false)
- Idea checkstyle插件的使用
File->Setting 选择Plugins,查询是否已经安装了checkstyle,如果没有安装,可以点击下面的“Browse repositories...”按钮 查询到checkstyl ...
- 【不做标题党，只做纯干货】HashMap在jdk1.7和1.8中的实现
同步首发:http://www.yuanrengu.com/index.php/20181106.html Java集合类的源码是深入学习Java非常好的素材,源码里很多优雅的写法和思路,会让人叹为 ...
- hadoop2.7.3编译，支持snappy、bzip2本地压缩
软件包: apache-ant-1.9.9-bin.tar.gz apache-maven-3.3.9-bin.tar.gz apache-tomcat-6.0.44.tar.gz CentOS-6. ...