Hadoop-Hive组件部署
目录
一、基础环境
Hive 组件需要基于之前已部署完毕的 Hadoop 全分布系统，在 master 节点上实现 Hive 组件安装。
1、Hadoop-全分布式配置（全部配置）
2、Hadoop-集群运行（部分配置）
# 查看之前部署的环境
[root@master ~]# su - hadoop
[hadoop@master ~]$ jps
47967 Jps
[hadoop@master hadoop]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out
192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out
192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out
192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out
192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out
[hadoop@master hadoop]$ jps
44678 NameNode
45030 ResourceManager
45289 Jps
44874 SecondaryNameNode
[hadoop@master hadoop]$ exit
logout
要确保 Hadoop 系统能够正常运行
二、安装MySQL
注意：使用 root 用户登录
1、传输需要的包到/opt目录下
[root@master ~]# cd /opt/software/
[root@master software]# ls
apache-hive-2.0.0-bin.tar.gz jdk-8u152-linux-x64.tar.gz mysql-connector-java-5.1.46.jar
hadoop-2.7.1.tar.gz mysql-5.7.18.zip
2、卸载自带的MariaDB
[root@master software]# rpm -qa | grep mariadb
mariadb-libs-5.5.52-1.el7.x86_64
[root@master software]# rpm -e --nodeps mariadb-libs-5.5.52-1.el7.x86_64
3、解压mysql-5.7.18.zip
[root@master software]# ls
apache-hive-2.0.0-bin.tar.gz jdk-8u152-linux-x64.tar.gz mysql-connector-java-5.1.46.jar
hadoop-2.7.1.tar.gz mysql-5.7.18.zip
[root@master software]# unzip mysql-5.7.18.zip
Archive: mysql-5.7.18.zip
creating: mysql-5.7.18/
inflating: mysql-5.7.18/mysql-community-client-5.7.18-1.el7.x86_64.rpm
inflating: mysql-5.7.18/mysql-community-common-5.7.18-1.el7.x86_64.rpm
inflating: mysql-5.7.18/mysql-community-devel-5.7.18-1.el7.x86_64.rpm
inflating: mysql-5.7.18/mysql-community-libs-5.7.18-1.el7.x86_64.rpm
inflating: mysql-5.7.18/mysql-community-server-5.7.18-1.el7.x86_64.rpm
4、依次安装MySQL数据库
[root@master software]# cd mysql-5.7.18/
[root@master mysql-5.7.18]# ls
mysql-community-client-5.7.18-1.el7.x86_64.rpm mysql-community-libs-5.7.18-1.el7.x86_64.rpm
mysql-community-common-5.7.18-1.el7.x86_64.rpm mysql-community-server-5.7.18-1.el7.x86_64.rpm
mysql-community-devel-5.7.18-1.el7.x86_64.rpm
[root@master mysql-5.7.18]# rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-common-5.7.18-1.e################################# [100%]
[root@master mysql-5.7.18]# rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-5.7.18-1.el7################################# [100%]
[root@master mysql-5.7.18]# rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-client-5.7.18-1.e################################# [100%]
[root@master mysql-5.7.18]# rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-server-5.7.18-1.e################################# [100%]
5、修改配置
添加配置信息到/etc/my.cnf 文件 symbolic-links=0 的下方
[root@master mysql-5.7.18]# vi /etc/my.cnf
default-storage-engine=innodb
innodb_file_per_table
collation-server=utf8_general_ci
init-connect='SET NAMES utf8'
character-set-server=utf8
6、启动MySQL
[root@master mysql-5.7.18]# systemctl start mysqld
[root@master mysql-5.7.18]# systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2022-04-09 10:12:27 CST; 44s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 45577 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
Process: 45499 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 45580 (mysqld)
CGroup: /system.slice/mysqld.service
└─45580 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
Apr 09 10:12:17 master.example.com systemd[1]: Starting MySQL Server...
Apr 09 10:12:27 master.example.com systemd[1]: Started MySQL Server.
7、查询MySQL默认密码
[root@master ~]# cat /var/log/mysqld.log | grep password
2022-04-09T02:12:24.671882Z 1 [Note] A temporary password is generated for root@localhost: /RrHz+J-t1fj
默认密码为/RrHz+J-t1fj
8、初始化MySQL
是否更改 root 用户密码n
是否使用设定的密码继续y
是否删除匿名用户y
是否拒绝 root 用户远程登录n
是否删除测试数据库y
是否重新加载授权表y
[root@master ~]# mysql_secure_installation
Securing the MySQL server deployment.
Enter password for user root:
The existing password for the user account root has expired. Please set a new password.
New password:
Re-enter new password:
The 'validate_password' plugin is installed on the server.
The subsequent steps will run with the existing configuration
of the plugin.
Using existing password for root.
Estimated strength of the password: 100
Change the password for root ? ((Press y|Y for Yes, any other key for No) : n
Estimated strength of the password: 100
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.
Remove anonymous users? (Press y|Y for Yes, any other key for No) : n
... skipping.
Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n
... skipping.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : n
... skipping.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : n
... skipping.
All done!
9、进入数据库，添加权限
添加 root 用户从本地和远程访问 MySQL 数据库表单的授权
[root@master ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 5.7.18 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> grant all privileges on *.* to root@'localhost' identified by 'Wangzhigang123!';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> grant all privileges on *.* to root@'%' identified by 'Wangzhigang123!';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> select user,host from mysql.user where user='root';
+------+-----------+
| user | host |
+------+-----------+
| root | % |
| root | localhost |
+------+-----------+
2 rows in set (0.00 sec)
mysql> exit;
Bye
[root@master ~]#
二、配置Hive组件
1、设置Hive环境变量
# 解压hive文件
[root@master ~]# tar -zxf /opt/software/apache-hive-2.0.0-bin.tar.gz
[root@master ~]# mv apache-hive-2.0.0-bin/ /usr/local/src/hive
# 修改用户和用户组为 hadoop
[root@master ~]# chown -R hadoop:hadoop /usr/local/src/hive
# 添加配置信息
[root@master ~]# vi /etc/profile
# set hive environment
export HIVE_HOME=/usr/local/src/hive
export PATH=$PATH:$HIVE_HOME/bin
[root@master ~]# source /etc/profile
2、修改Hive组件配置文件
[root@master ~]# su - hadoop
[hadoop@master ~]$ cp /usr/local/src/hive/conf/hive-default.xml.template /usr/local/src/hive/conf/hive-site.xml
[hadoop@master ~]$ vi /usr/local/src/hive/conf/hive-site.xml
# 设置 MySQL 数据库连接
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false</value>
<description>JDBC connect string for a JDBC metastore</description>
# 配置 MySQL 数据库 root 的密码
<name>javax.jdo.option.ConnectionPassword</name>
<value>Wangzhigang123!</value>
<description>password to use against metastore database</description>
# 验证元数据存储版本一致性,默认 false
<name>hive.metastore.schema.verification</name>
<value>false</value>
# 配置数据库驱动
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
# 配置数据库用户名
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>Username to use against metastore database</description>
# 配置目录信息（共4处）
<name>hive.querylog.location</name>
<value>/usr/local/src/hive/tmp</value>
<description>Location of Hive run time structured log file</description>
<name>hive.exec.local.scratchdir</name>
<value>/usr/local/src/hive/tmp</value>
<name>hive.downloaded.resources.dir</name>
<value>/usr/local/src/hive/tmp/resources</value>
<name>hive.server2.logging.operation.log.location</name>
<value>/usr/local/src/hive/tmp/operation_logs</value>
[hadoop@master ~]$ mkdir -p /usr/local/src/hive/tmp{resources,operation_logs}
3、重启all.sh
[hadoop@master ~]$ cp /opt/software/mysql-connector-java-5.1.46.jar /usr/local/src/hive/lib/
[hadoop@master ~]$ stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
192.168.100.20: stopping datanode
192.168.100.30: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
192.168.100.20: stopping nodemanager
192.168.100.30: stopping nodemanager
no proxyserver to stop
[hadoop@master ~]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out
192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out
192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out
192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out
192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out
4、初始化数据库
[hadoop@master ~]$ schematool -initSchema -dbType mysql
which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Starting metastore schema initialization to 2.0.0
Initialization script hive-schema-2.0.0.mysql.sql
Initialization script completed
schemaTool completed
5、启动hive
[hadoop@master ~]$ hive
which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
OK
default
Time taken: 0.241 seconds, Fetched: 1 row(s)
hive> exit;
[hadoop@master ~]$
6、验证
[hadoop@master ~]$ mysql -uroot -pWangzhigang123!
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 24
Server version: 5.7.18 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| hive |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
mysql> exit;
Bye
[hadoop@master ~]$ jps
47057 NameNode
47411 ResourceManager
47254 SecondaryNameNode
47967 Jps
[hadoop@master ~]$
slave1节点
[root@slave1 ~]# su - hadoop
[hadoop@slave1 ~]$ jps
43621 Jps
43351 DataNode
43464 NodeManager
slave2节点
[root@slave2 ~]# su - hadoop
[hadoop@slave2 ~]$ jps
41607 Jps
41337 DataNode
41450 NodeManager
声明：未经许可，不得转载
Hadoop-Hive组件部署的更多相关文章
- hadoop hive组件介绍及常用cli命令
Hive架构图 Hive产生原因 1 关系型数据库以产生多年sql成熟 2 简化开发降低成本 3 java成员可编写udf函数 Hive是什么 Hive是基于hadoop的一个数据库工具,使用Hql作 ...
- Hadoop生态组件Hive，Sqoop安装及Sqoop从HDFS/hive抽取数据到关系型数据库Mysql
一般Hive依赖关系型数据库Mysql,故先安装Mysql $: yum install mysql-server mysql-client [yum安装] $: /etc/init.d/mysqld ...
- Docker+Hadoop+Hive+Presto 使用Docker部署Hadoop环境和Presto
Background 一. 什么是Presto Presto通过使用分布式查询,可以快速高效的完成海量数据的查询.如果你需要处理TB或者PB级别的数据,那么你可能更希望借助于Hadoop和HDFS来完 ...
- Hadoop.之.入门部署
一.课程目标 ->大数据是什么?大数据能做什么? ->什么是Hadoop?Hadoop的设计思想? ->Hadoop如何解决大数据的问题?(什么是hdfs与yarn.MapReduc ...
- 单机安装hadoop+hive+presto
系统环境 在个人笔记本上使用virtualbox虚拟机 os:centos -7.x86-64.everything.1611 ,内核 3.10.0-514.el7.x86_64 注:同样可以使用r ...
- Hive 组件安装配置
下载和解压安装文件 基础环境和安装准备 Hive组件的部署规划和软件包路径如下: (1)当前环境中已安装 Hadoop全分布系统 (2)本地安装 MySQL数据库(账号 root,密码 Passwor ...
- 远程调试hadoop各组件
远程调试对应用程序开发十分有用.例如,为不能托管开发平台的低端机器开发程序,或在专用的机器上(比如服务不能中断的 Web 服务器)调试程序.其他情况包括:运行在内存小或 CUP 性能低的设备上的 Ja ...
- Sqoop是一款开源的工具，主要用于在HADOOP(Hive)与传统的数据库(mysql、oracle...)间进行数据的传递
http://niuzhenxin.iteye.com/blog/1706203 Sqoop是一款开源的工具,主要用于在HADOOP(Hive)与传统的数据库(mysql.postgresql.. ...
- Hadoop生态圈-Sqoop部署以及基本使用方法
Hadoop生态圈-Sqoop部署以及基本使用方法 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. Sqoop(发音:skup)是一款开源的工具,主要用于在Hadoop(Hive)与 ...
随机推荐
- ArcMap操作随记（12）
1.[取色器]工具 [自定义]|[自定义模式]|[命令] 2.批量修改符号 [符号系统]→右键,[所有符号的属性] 3.将地图元素转换为图形 转换工具 4.好看的地图边框 [布局视图]→数据框上右键→ ...
- Spring AOP Xml配置过程及解释
目录 Spring AOP(基于xml) 专业术语: 基于xml的声明式AspectJ 具体实践 Spring AOP(基于xml) 目前主流的AOP框架有两个,分别是Spring AOP和Aspec ...
- 分布式 PostgreSQL 集群(Citus)，分布式表中的分布列选择最佳实践
确定应用程序类型 在 Citus 集群上运行高效查询要求数据在机器之间正确分布.这因应用程序类型及其查询模式而异. 大致上有两种应用程序在 Citus 上运行良好.数据建模的第一步是确定哪些应用程序类 ...
- 旅游清单一步搭建，Angular助力你的踏春计划
春天的脚步愈发临近,相信很多小伙伴已经开始规划自己的踏春计划了,无论是欣赏名胜古迹,还是走访风土人文,你都需要提前准备一份旅游清单!有了这款Angular旅游计划应用,从地点到预算,它都能帮助你创建自 ...
- 搭建Loki、Promtail、Grafana轻量级日志系统（centos7）
搭建Loki.Promtail.Grafana轻量级日志系统(centos7)--简称PLG 需求 公司项目采用微服务的架构,服务很多,每个服务都有自己的日志,分别存放在不同的服务器上.当查找日志时需 ...
- python练习册 每天一个小程序 第0008题
1 # -*-coding:utf-8-*- 2 __author__ = 'Deen' 3 ''' 4 题目描述: 5 一个HTML文件,找出里面的正文. 6 7 思路: 8 利用Beautiful ...
- Rabbit MQ 怎么保证可靠性、幂等性、消费顺序？
RabbitMQ如何保证消息的可靠性 RabbitMQ消息丢失的三种情况 生产者弄丢消息时的解决方法 方法一:生产者在发送数据之前开启RabbitMQ的事务(采用该种方法由于事务机制,会导致吞吐量下降 ...
- leedcode算法
1.LeetCode - 合并两个链表 2. 3. 链表是否有环 https://jingyan.baidu.com/article/066074d69afb8d83c31cb068.ht ...
- python模块相互依赖的解决方案
第一种:将相互依赖的文件中的其中一个文件的代码移植到另一个文件中... 第二种:将 import .... 或 from ... import 语句的位置移动到def函数内部,由于import和fro ...
- django中动态新建postgres数据库表
import psycopg2def create_new_table(table_id): conn = psycopg2.connect(database='Test', user='postgr ...