A record--Offline deployment of Big Data Platform CDH Cluster

Tags： Cloudera-Manager CDH Hadoop Deploy Cluster

Abstract: Deployment and Management of Hadoop clusters need tools, such as Cloudera Manager. In this article, I compare the tools briefly, and then record the step of deploying CDH cluster offline in detail. Finally, I expound the theory of 'handle delicately'.

A record--Offline deployment of Big Data Platform CDH Cluster

Preface

The emergence of Big Data technology led by Apache Hadoop, makes small and medium-sized enterprises also have the ability to handle the storage and processing of big data. At present, there'r lots of Hadoop distributions, such as HUAWEI Distribution, Intel Distribution, Cloudera’s Distribution Including Apache Hadoop (CDH free), and Hortonworks Data Platform (HDP free), etc. All of these are based on the Hadoop Apache Community Edition.

The deployment and management of a Hadoop cluster which has tens or more nodes needs advanced tools. Apache Ambari from Hortonworks is this kind of tools, it provided an easy-to-use RESTfull web site to manage Hadoop. Cloudera also provided a similar tool, Cloudera Manager(CM) to configure, monitor and manage CDH clusters.

The main content of this paper is a record of building a CDH cluster. Special attention is required to choose Cloudera Manager version, which depends on Operating system, el7 isn't supported by Cloudera Manager at this moment. You should follow [the official document][1], otherwise the installation will run into a stone wall.

This paper is based on CentOS 6.5, 64-bit；Cloudera Manager 5.3.6；JDK 1.7.

Deploy CDH

Configure network (All nodes)

[root@cdh-server ~]# vi /etc/sysconfig/network #修改hostname:

NETWORKING=yes

HOSTNAME=cdh-server

[root@cdh-server ~]# vi /etc/hosts #修改ip与主机名的对应关系:

192.168.180.173 cdh-server

192.168.180.175 node175

[root@cdh-server ~]# service network restart #重启网络服务生效

Install JDK (All nodes)

#卸载OpenJDK

[root@cdh-server user1]# rpm -qa | grep java

[root@cdh-server user1]# rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64

[root@cdh-server user1]# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

[root@cdh-server user1]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

#安装JDK

[root@cdh-server user1]# chmod a+x jdk-7u79-linux-x64.rpm

[root@cdh-server user1]# rpm -ivh jdk-7u79-linux-x64.rpm

[root@cdh-server user1]# echo "JAVA_HOME=/usr/java/jdk1.7.0_79/" >>

Install MySQL (Master)

[user1@cdh-server]$ cd /home/user1

[user1@cdh-server]$ tar -zxvf mysql-5.6.26-linux-glibc2.5-x86_64.tar.gz

[user1@cdh-server]$ mv mysql-5.6.26-linux-glibc2.5-x86_64 mysql-5.6.26

[user1@cdh-server]$ cd mysql-5.6.26/

[user1@cdh-server]$ vi support-files/my.cnf	#新建文件

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[mysqld]

character-set-server=utf8

default-storage-engine=INNODB

# Uncomment the following if you are using InnoDB tables

innodb_data_home_dir = /home/user1/mysql-5.6.26/data

innodb_data_file_path = ibdata1:10M:autoextend

innodb_log_group_home_dir = /home/user1/mysql-5.6.26/data

# You can set .._buffer_pool_size up to 50 - 80 %

# of RAM but beware of setting memory usage too high

innodb_buffer_pool_size = 16M

innodb_additional_mem_pool_size = 2M

# Set .._log_file_size to 25 % of buffer pool size

innodb_log_file_size = 5M

innodb_log_buffer_size = 8M

innodb_flush_log_at_trx_commit = 1

innodb_lock_wait_timeout = 50

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Initialize MySQL (Master)

[user1@cdh-server]$ ./scripts/mysql_install_db --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data --user=user1

[user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &

[user1@cdh-server]$ ./bin/mysqladmin -u root password '123456'

[user1@cdh-server mysql-5.6.26]$ ./bin/mysql -uroot -p'123456'

#hive

mysql> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.00 sec)

#Activity Monitor使用

mysql> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.01 sec)

#Navigator Audit Server使用

mysql> create database audit DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.01 sec)

#Navigator Metadata Server

mysql> create database metadata DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.01 sec)

mysql> grant all privileges on *.* to 'root'@'localhost' identified by '123456' with grant option;

Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on *.* to 'root'@'cdh-server' identified by '123456' with grant option;

Query OK, 0 rows affected (0.00 sec)

#this user scm is for  cloudera manager

mysql> grant all privileges on *.* to 'scm'@'localhost' identified by 'scm' with grant option;

Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on *.* to 'scm'@'cdh-server' identified by 'scm' with grant option;

Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

Deploy & Start CM-Server (Master)

[user1@cdh-server ~]$ tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz

[user1@cdh-server ~]$ cp mysql-connector-java-5.1.33-bin.jar ./cm-5.3.6/share/cmf/lib/

[user1@cdh-server ~]$ su - root

[root@cdh-server ~]# cd /home/user1/

[root@cdh-server user1]# cp -rf cloudera /opt

[root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel

[root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha

[root@cdh-server user1]# mv manifest.json /opt/cloudera/parcel-repo/manifest.json

[root@cdh-server user1]# ./cm-5.3.6/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost:3306 -uroot -p123456 --scm-host localhost scm scm scm

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start

Starting cloudera-scm-server:                              [  OK  ]

[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log

Stop iptables (All nodes)

#停止iptables

[root@cdh-server user1]# service iptables stop

#通过浏览器访问验证

http://192.168.180.173:7180/

Deploy & Start CM-Agent (Slaves)

[root@cdh-server user1]# tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz

[root@cdh-server user1]# vi cm-5.3.6/etc/cloudera-scm-agent/config.ini

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

# Hostname of the CM server.

#server_host=localhost

server_host=cdh-server

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@cdh-server user1]# useradd -G sys --home=/home/user1/cm-5.3.6/run/cloudera-scm-server --no-create-home  --comment "Cloudera SCM User" cloudera-scm

[root@cdh-server user1]# useradd --comment "Cloudera SCM User" cloudera-scm	#若上一步执行正确，则此步省略

[root@cdh-server user1]# echo 0 > /proc/sys/vm/swappiness

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start

Starting cloudera-scm-agent:                               [  OK  ]

[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log

Configure CDH

Load Cloudera Manager http://192.168.180.173:7180/, then create a new Cluster names Cluster_user1，Start and configure Services.

#安装配置hive出错时，在hiveServer上:

[root@hive-server user1]# cp mysql-connector-java-5.1.33-bin.jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/hive/lib/

#同理：use this jar for Navigator Audit Server and Navigator Metadata Server or Activity Server

[root@cdh-server user1]# cp mysql-connector-java-5.1.33-bin.jar /usr/share/java/mysql-connector-java.jar

Others

Stop CDH

Stop Cloudera Management Service & Cluster_user1
Stop Agent (Slaves)

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent stop

Stop Server (Master)

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server stop

Start CDH

Start MySQL (Master)

[user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &

Start Agent (Slaves)

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start

Starting cloudera-scm-agent:                               [  OK  ]

[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log

Start Server (Master)

[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start

Starting cloudera-scm-server:                              [  OK  ]

[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log

The theory of handle delicately

Handle delicately is a kind of feeling, but also a skill. When in seller's market, a company can make a profit while may needn't to handle delicately. However, if it wishes to pursue more, it will handle more delicately, such as pay more attention to detail, User Experience or others. When in buyer's market, the relation between supply and demand makes companies handle delicately to survive.

Handle delicately is not only the driving force pushing social to continually advance, but also the result of this advance. Today, lots of internet firms are in buyer's market, who has more users who will win the fight, delicately handling makes them at the forefront of social evolution and technological innovation.

Handle delicately doesn't only appear in companies, but also individuals, regions and countries, etc. The contry who handles more delicately, who is more developed. The company who pays more attention to details, who is more competitive. But, the one who handles more delicately doesn't mean it will make more profits, for various reasons.

Writer: @Angel Wang

aitanjupt@hotmail.com

2015 - 10 - 18

[1]: http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/pcm_os.html

果然cnblogs不支持目录啊。不过代码格式还算漂亮。

此篇文章是本人另一英文文章的版本，中文版在此：朝花夕拾之--大数据平台CDH集群离线搭建 http://www.cnblogs.com/wgp13x/p/4990484.html ，多谢指教！