大数据软件安装之Hive（查询）

一、安装及配置

官方文档：

https://cwiki.apache.org/confluence/display/Hive/GettingStarted

安装Hive2.3

1）上传apache-hive-2.3.0-bin.tar.gz 到/opt/software目录下，并解压到/opt/module

[test@hadoop102 software]$ tar -zxvf apache-hive-2.3.6-bin.tar.gz -C /opt/module/

2）修改apache-hive-2.3.6-bin名称为hive

[test@hadoop102 module]$ mv apache-hive-2.3.6-bin hive

3）将Mysql的mysql-connector-java-5.1.27-bin.jar拷贝到/opt/module/hive/lib/

[test@hadoop102 module]$ cp /opt/software/mysql-libs/mysql-connector-java-5.1.27/mysql-connector-java-5.1.27-bin.jar /opt/module/hive/lib/

4）在/opt/module/hive/conf路径上，创建hive-site.xml文件

[test@hadoop102 conf]$ vim hive-site.xml

添加如下内容

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://hadoop102:3306/metastore?createDatabaseIfNotExist=true</value>

<description>JDBC connect string for a JDBC metastore</description>

</property>

<property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

<description>Driver class name for a JDBC metastore</description>

</property>

<property>

<name>javax.jdo.option.ConnectionUserName</name>

<value>root</value>

<description>username to use against metastore database</description>

</property>

<property>

<name>javax.jdo.option.ConnectionPassword</name>

<value></value>

<description>password to use against metastore database</description>

</property>

<property>

<name>hive.metastore.warehouse.dir</name>

<value>/user/hive/warehouse</value>

<description>location of default database for the warehouse</description>

</property>

<property>

<name>hive.cli.print.header</name>

<value>true</value>

</property>

<property>

<name>hive.cli.print.current.db</name>

<value>true</value>

</property>

<property>

<name>hive.metastore.schema.verification</name>

<value>false</value>

</property>

<property>

<name>datanucleus.schema.autoCreateAll</name>

<value>true</value>

</property>

<property>

<name>hive.metastore.uris</name>

<value>thrift://hadoop102:9083</value>

</property>

</configuration>

注意：hive安装在哪个服务器节点，thrift://hadoop102:9083中的主机名就更换为相应的主机名。

3）启动服务

[test@hadoop102 hive]$ nohup bin/hive --service metastore &

[test@hadoop102 hive]$ nohup bin/hive --service hiveserver2 &

注意：hive2.x版本需要启动两个服务metastore和hiveserver2，否则会报错Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

4）服务启动完毕后在启动Hive

[test@hadoop102 hive]$ bin/hive

二、Hadoop集群配置

必须先启动hdfs和yarn

[test@hadoop102 hadoop-2.7.2]$ sbin/start-dfs.sh

[test@hadoop102 hadoop-2.7.2]$ sbin/start-yarn.sh

三、安装MySQL

1 、安装包准备

1）查看MySQL是否安装，如果安装了，卸载MySQL

（1）查看

[root@hadoop102 桌面]# rpm -qa|grep mysql

mysql-libs-5.1.73-7.el6.x86_64

（2）卸载

[root@hadoop102 桌面]# rpm -e --nodeps mysql-libs-5.1.73-7.el6.x86_64

2）解压mysql-libs.zip文件到当前目录

[root@hadoop102 software]# unzip mysql-libs.zip

[root@hadoop102 software]# ls

mysql-libs.zip

mysql-libs

3）进入到mysql-libs文件夹下

[root@hadoop102 mysql-libs]# ll

总用量 76048

-rw-r--r--. 1 root root 18509960 3月  26 2015 MySQL-client-5.6.24-1.el6.x86_64.rpm

-rw-r--r--. 1 root root  3575135 12月  1 2013 mysql-connector-java-5.1.27.tar.gz

-rw-r--r--. 1 root root 55782196 3月  26 2015 MySQL-server-5.6.24-1.el6.x86_64.rpm

2 、安装MySql服务器

1）安装mysql服务端

[root@hadoop102 mysql-libs]# rpm -ivh MySQL-server-5.6.24-1.el6.x86_64.rpm

2）查看产生的随机密码

[root@hadoop102 mysql-libs]# cat /root/.mysql_secret

OEXaQuS8IWkG19Xs

3）查看mysql状态

[root@hadoop102 mysql-libs]# service mysql status

4）启动mysql

[root@hadoop102 mysql-libs]# service mysql start

3 、安装MySql客户端

1）安装mysql客户端

[root@hadoop102 mysql-libs]# rpm -ivh MySQL-client-5.6.24-1.el6.x86_64.rpm

2）链接mysql

[root@hadoop102 mysql-libs]# mysql -uroot -pOEXaQuS8IWkG19Xs

3）修改密码

mysql>SET PASSWORD=PASSWORD('000000');

4）退出mysql

mysql>exit

4、 MySql中user表中主机配置

配置只要是root用户+密码，在任何主机上都能登录MySQL数据库。

1）进入mysql

[root@hadoop102 mysql-libs]# mysql -uroot -p000000

2）显示数据库

mysql>show databases;

3）使用mysql数据库

mysql>use mysql;

4）展示mysql数据库中的所有表

mysql>show tables;

5）展示user表的结构

mysql>desc user;

6）查询user表

mysql>select User, Host, Password from user;

7）修改user表，把Host表内容修改为%

mysql>update user set host='%' where host='localhost';

8）删除root用户的其他host

mysql>

delete from user where Host='hadoop102';

delete from user where Host='127.0.0.1';

delete from user where Host='::1';

9）刷新

mysql>flush privileges;

10）退出

mysql>quit;

四、Hive集成引擎Tez

Tez是一个Hive的运行引擎，性能优于MR。

用Hive直接编写MR程序，假设有四个有依赖关系的MR作业，上图中，绿色是Reduce Task，云状表示写屏蔽，需要将中间结果持久化写到HDFS。

Tez可以将多个有依赖的作业转换为一个作业，这样只需写一次HDFS，且中间节点较少，从而大大提升作业的计算性能。

1 、安装包准备

1）下载tez的依赖包：http://tez.apache.org

2）拷贝apache-tez-0.9.1-bin.tar.gz到hadoop102的/opt/software目录

[test@hadoop102 software]$ ls

apache-tez-0.9.1-bin.tar.gz

3）将apache-tez-0.9.1-bin.tar.gz上传到HDFS的/tez目录下。

[test@hadoop102 conf]$ hadoop fs -mkdir /tez

[test@hadoop102 conf]$ hadoop fs -put /opt/software/apache-tez-0.9.1-bin.tar.gz/ /tez

4）解压缩apache-tez-0.9.1-bin.tar.gz

[test@hadoop102 software]$ tar -zxvf apache-tez-0.9.1-bin.tar.gz -C /opt/module

5）修改名称

[test@hadoop102 module]$ mv apache-tez-0.9.1-bin/ tez-0.9.1

2、 集成Tez

1）进入到Hive的配置目录：/opt/module/hive/conf

[test@hadoop102 conf]$ pwd

/opt/module/hive/conf

2）在Hive的/opt/module/hive/conf下面创建一个tez-site.xml文件

[test@hadoop102 conf]$ pwd

/opt/module/hive/conf

[test@hadoop102 conf]$ vim tez-site.xml

添加如下内容

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>tez.lib.uris</name>

<value>${fs.defaultFS}/tez/apache-tez-0.9.1-bin.tar.gz</value>

</property>

<property>

<name>tez.use.cluster.hadoop-libs</name>

<value>true</value>

</property>

<property>

<name>tez.history.logging.service.class</name>

<value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>

</property>

</configuration>

2）在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置

[test@hadoop102 conf]$ mv hive-env.sh.template hive-env.sh

[test@hadoop102 conf]$ vim hive-env.sh

添加如下配置

# Set HADOOP_HOME to point to a specific hadoop install directory

export HADOOP_HOME=/opt/module/hadoop-2.7.2

# Hive Configuration Directory can be controlled by:

export HIVE_CONF_DIR=/opt/module/hive/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:

export TEZ_HOME=/opt/module/tez-0.9.1    #是你的tez的解压目录

export TEZ_JARS=""

for jar in `ls $TEZ_HOME |grep jar`; do

export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar

done

for jar in `ls $TEZ_HOME/lib`; do

export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar

done

export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS

3）在hive-site.xml文件中添加如下配置，更改hive计算引擎

<property>

<name>hive.execution.engine</name>

<value>tez</value>

</property>

3 、测试

1）启动Hive

[test@hadoop102 hive]$ bin/hive

2）创建表

hive (default)> create table student(

id int,

name string);

3）向表中插入数据

hive (default)> insert into student values(1,"zhangsan");

4）如果没有报错就表示成功了

hive (default)> select * from student;

1       zhangsan

4 、注意事项

1）运行Tez时检查到用过多内存而被NodeManager杀死进程问题：

Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1546781144082_0005 failed 2 times due to AM Container for appattempt_1546781144082_0005_000002 exited with  exitCode: -103

For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1546781144082_0005Then, click on links to logs of each attempt.

Diagnostics: Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.

这种问题是从机上运行的Container试图使用过多的内存，而被NodeManager kill掉了。

[摘录] The NodeManager is killing your container. It sounds like you are trying to use hadoop streaming which is running as a child process of the map-reduce task. The NodeManager monitors the entire process tree of the task and if it eats up more memory than the maximum set in mapreduce.map.memory.mb or mapreduce.reduce.memory.mb respectively, we would expect the Nodemanager to kill the task, otherwise your task is stealing memory belonging to other containers, which you don't want.

2）解决方法：

（1）关掉虚拟内存检查，修改yarn-site.xml，

<property>

<name>yarn.nodemanager.vmem-check-enabled</name>

<value>false</value>

</property>

（2）修改后一定要分发，并重新启动hadoop集群。

[test@hadoop102 hadoop]$ xsync yarn-site.xml