Linux 系统基于 Hadoop 安装 Hive

【注意】安装hive前提是要先安装hadoop集群，并且hive只需要在hadoop的namenode节点集群里安装即可(在所有的namenode上安装)，可以不在datanode节点的机器上安装。

一、安装和配置Hive

1. 下载Hive

Hive的官方下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3.7/apache-hive-2.3.7-bin.tar.gz

2. 安装Hive

1. 将hive的二进制包上传到服务器中并解压到/opt/hive目录下

root@hserver1:~# mkdir /opt/hive
root@hserver1:~# tar zxf apache-hive-2.3.7-bin.tar.gz -C /opt/hive/

1. 修改环境变量配置，在/etc/profile文件中添加如下内容

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/opt/hadoop/hadoop-2.9.2
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"
export HIVE_HOME=/opt/hive/apache-hive-2.3.7-bin
export HIVE_CONF_DIR=${HIVE_HOME}/conf
export CLASS_PATH=.:${JAVA_HOME}/lib:${HIVE_HOME}/lib:$CLASS_PATH
export PATH=.:${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${HIVE_HOME}/bin:$PATH 

3. 配置Hive

1. 进入到/opt/hive/apache-hive-2.3.7-bin/conf目录下，将hive-default.xml.template文件复制一份，并且改名为hive-site.xml。在hive-site.xml文件中存在如下一段配置

  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>

1. 所以需要根据如上的配置在hadoop中新建一个HDFS目录

$HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/warehouse

1. 为刚才新建的目录赋予读写权限

$HADOOP_HOME/bin/hadoop fs -chmod 777 /user/hive/warehouse

1. 再新建一个/tmp/hive目录并赋予读写权限

$HADOOP_HOME/bin/hadoop fs -mkdir -p /tmp/hive
$HADOOP_HOME/bin/hadoop fs -chmod 777 /tmp/hive

1. 检查创建的两个hdfs目录是否创建成功

root@hserver1:/opt/hive/apache-hive-2.3.7-bin/conf# $HADOOP_HOME/bin/hadoop fs -ls /user/hive
Found 1 items
drwxrwxrwx   - root supergroup          0 2020-05-07 15:17 /user/hive/warehouse
root@hserver1:/opt/hive/apache-hive-2.3.7-bin/conf# $HADOOP_HOME/bin/hadoop fs -ls /tmp
Found 1 items
drwxrwxrwx   - root supergroup          0 2020-05-07 15:42 /tmp/hive

1. 将hive-site.xml文件中的${system:java.io.tmpdir}替换为hive的临时目录，本文中替换为/opt/hive/tmp，该目录如果不存在则要自己手工创建，并且赋予读写权限。

sed -i 's/${system:java.io.tmpdir}/\/opt\/hive\/tmp/g' hive-site.xml

1. 将hive-site.xml文件中的${system:user.name}都替换为root

sed -i 's/${system:user.name}/root/g' hive-site.xml

1. hive需要搭配MySQL数据库使用，本文不对MySQL的安装进行说明，可参考其他笔记。安装完成后需要注意MySQL的访问权限配置。
2. 安装完MySQL后，需要修改hive-site.xml文件中关于MySQL相关的配置，首先搜索javax.jdo.option.ConnectionURL，将该name对应的value修改为MySQL的地址

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://192.168.92.11:3306/hive?createDatabaseIfNotExist=true</value>
  </property>

1. 搜索javax.jdo.option.ConnectionDriverName，将该name对应的value修改为MySQL驱动类路径

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>

1. 搜索javax.jdo.option.ConnectionUserName，将对应的value修改为MySQL数据库登录名

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    <description>Username to use against metastore database</description>
  </property>

1. 搜索javax.jdo.option.ConnectionPassword，将对应的value修改为MySQL数据库登录密码

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>123456</value>
    <description>password to use against metastore database</description>
  </property>

1. 搜索hive.metastore.schema.verification，将对应的value修改为false

  <property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
  </property>

1. 将hive-env.sh.template文件复制一份，改名为hive-env.sh并且添加如下内容

export HADOOP_HOME=/opt/hadoop/hadoop-2.9.2
export HIVE_CONF_DIR=/opt/hive/apache-hive-2.3.7-bin/conf
export HIVE_AUX_JARS_PATH=/opt/hive/apache-hive-2.3.7-bin/lib

1. 将MySQL驱动包上传到hive的lib目录下，驱动包下载地址：https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-java-5.1.41.tar.gz

root@hserver1:/opt/hive/apache-hive-2.3.7-bin/lib# ll mysql-connector-java-5.1.41.jar
-rw-r--r-- 1 root root 992808 May  7 16:19 mysql-connector-java-5.1.41.jar

二、启动Hive并测试

1. 初始化hive数据库

在启动hive之前，首先要将mysql设置为hive的元数据库。操作步骤如下：

1. 进入hive目录

cd /opt/hive/apache-hive-2.3.7-bin/bin

1. 执行如下命令进行初始化

root@hserver1:/opt/hive/apache-hive-2.3.7-bin/bin# schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/apache-hive-2.3.7-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://192.168.92.11:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 root
Thu May 07 16:31:56 CST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Thu May 07 16:31:58 CST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Initialization script completed
Thu May 07 16:32:03 CST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
schemaTool completed

1. 初始化成功后，可以看到MySQL数据库的hive库中已经创建了一些表

mysql> show tables;
+---------------------------+
| Tables_in_hive            |
+---------------------------+
| AUX_TABLE                 |
| BUCKETING_COLS            |
| CDS                       |
| COLUMNS_V2                |
| COMPACTION_QUEUE          |
| COMPLETED_COMPACTIONS     |
...
...
| WRITE_SET                 |
+---------------------------+
57 rows in set (0.01 sec)

2. 启动hive

2.1 启动hive命令行

1. 启动hive需要进入到hive的bin目录下执行如下命令：

root@hserver1:/opt/hive/apache-hive-2.3.7-bin/bin# ./hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/apache-hive-2.3.7-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/hive/apache-hive-2.3.7-bin/lib/hive-common-2.3.7.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>

1. 启动成功后就进入了hive的命令行模式，首先执行查看函数的命令

hive> show functions;
OK
!
!=
$sum0
%
&
*
+
......
......
xpath_string
year
|
~
Time taken: 7.45 seconds, Fetched: 271 row(s)

1. 查看sum函数的详细信息

hive> desc function sum;
OK
sum(x) - Returns the sum of a set of numbers
Time taken: 0.232 seconds, Fetched: 1 row(s)

2.2 以服务的方式启动

上面的启动方式只适用于本地操作hive，如果希望hive启动后，可以在其他机器上远程连接，那么就需要让hive以服务的方式启动，步骤如下：

1. 进去hive的安装路径的 bin 目录下，执行如下命令

nohup ./hiveserver2 &

1. 命令执行完成后，服务会默认监听在10000端口

root@hserver1:~# netstat -anplut | grep 10000
tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN      18935/java

1. 在其他机器上可以使用beeline命令远程连接hive数据库（hive安装路径 bin 目录下也带有beeline命令）

beeline -u jdbc:hive2://hserver1:10000/irp_sns_hive -n root

3. 测试新建表和导入数据

1. 在hive中新建数据库

hive> create database db_hive_edu;
OK
Time taken: 1.062 seconds

1. 创建数据表

hive> create database db_hive_edu;
OK
Time taken: 1.062 seconds
hive> use db_hive_edu;
OK
Time taken: 0.053 seconds
hive> create table student(id int,name string) row format delimited fields terminated by '\t';
OK
Time taken: 1.719 seconds

1. 将文件数据写入到表中，首先在操作系统上的/home目录下新建一个名为student.txt的文件，并在文件中添加如下内容（注意，id和name之间是tab键，不是空格，因为在上面创建表的语句中用了terminated by '\t'，所以这个文本里id和name的分割必须是用TAB键。行与行之间不能有空格，否则会将NULL导入表中，文件的格式也要使用unix格式）：

001	aaron
002	alex
003	tom
004	peter
005	jackson

1. 编辑完成后开始导入，在hive命令行中执行如下命令导入

hive> load data local inpath '/home/student.txt' into table db_hive_edu.student;
Loading data to table db_hive_edu.student
OK
Time taken: 4.403 seconds

1. 使用select命令查看数据是否写入成功

hive> select * from student;
OK
1	aaron
2	alex
3	tom
4	peter
5	jackson
Time taken: 3.981 seconds, Fetched: 5 row(s)

1. 此时可以在namenode的web界面上查看到刚才导入的信息（注意hive使用的HDFS目录是/user/hive/warehouse）

2. 点击表的名字其实就相当于直接访问http://192.168.92.11:50070/explorer.html#/user/hive/warehouse/db_hive_edu.db/student，进入页面后可以看到student.txt这个文件

3. 点击文件的名字可以查看到文件中的信息

4. 进入到MySQL数据库中，使用如下语句也可以查询到hive创建表的信息

mysql> select * from TBLS \G
*************************** 1. row ***************************
            TBL_ID: 1
       CREATE_TIME: 1588841019
             DB_ID: 6
  LAST_ACCESS_TIME: 0
             OWNER: root
         RETENTION: 0
             SD_ID: 1
          TBL_NAME: student
          TBL_TYPE: MANAGED_TABLE
VIEW_EXPANDED_TEXT: NULL
VIEW_ORIGINAL_TEXT: NULL
IS_REWRITE_ENABLED:
1 row in set (0.00 sec)