Sqoop是一个用来完成Hadoop和关系型数据库中的数据相互转移的工具,它可以将关系型数据库中的数据导入到Hadoop的HDFS中,也可以将HDFS的数据导入到关系型数据库中。

  Kafka是一个开源的分布式消息订阅系统

  一、Sqoop的安装

  1.http://www-eu.apache.org/dist/sqoop/1.4.7/下载sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz并解压到/home/jun下

[jun@master sqoop-1.4..bin__hadoop-2.6.]$ ls -l
total
drwxr-xr-x. jun jun Dec bin
-rw-rw-r--. jun jun Dec build.xml
-rw-rw-r--. jun jun Dec CHANGELOG.txt
-rw-rw-r--. jun jun Dec COMPILING.txt
drwxr-xr-x. jun jun Dec conf
drwxr-xr-x. jun jun Dec docs
drwxr-xr-x. jun jun Dec ivy
-rw-rw-r--. jun jun Dec ivy.xml
drwxr-xr-x. jun jun Dec lib
-rw-rw-r--. jun jun Dec LICENSE.txt
-rw-rw-r--. jun jun Dec NOTICE.txt
-rw-rw-r--. jun jun Dec pom-old.xml
-rw-rw-r--. jun jun Dec README.txt
-rw-rw-r--. jun jun Dec sqoop-1.4..jar
-rw-rw-r--. jun jun Dec sqoop-patch-review.py
-rw-rw-r--. jun jun Dec sqoop-test-1.4..jar
drwxr-xr-x. jun jun Dec src
drwxr-xr-x. jun jun Dec testdata

  2.配置MySQL连接器

[jun@master sqoop-1.4..bin__hadoop-2.6.]$ cp /home/jun/Resources/mysql-connector-java-5.1./mysql-connector-java-5.1..jar   /home/jun/sqoop-1.4..bin__hadoop-2.6./lib/

  3.配置Sqoop环境变量

  编辑配置文件

[jun@master lib]$ cd /home/jun/sqoop-1.4..bin__hadoop-2.6./conf/
[jun@master conf]$ ls
oraoop-site-template.xml sqoop-env-template.cmd sqoop-env-template.sh sqoop-site-template.xml sqoop-site.xml
[jun@master conf]$ cp sqoop-env-template.sh sqoop-env.sh
[jun@master conf]$ gedit sqoop-env.sh

  增加下面的配置

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/jun/hadoop #Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/jun/hadoop #set the path to where bin/hbase is available
export HBASE_HOME=/home/jun/hbase-1.2.6.1 #Set the path to where bin/hive is available
export HIVE_HOME=/home/jun/apache-hive-2.3.-bin #Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/local/zk

  4.配置linux环境变量

#sqoop
export SQOOP_HOME=/home/jun/sqoop-1.4..bin__hadoop-2.6.
export PATH=$PATH:$SQOOP_HOME/bin

  5.启动Sqoop,如果出现下面的内容就说明安装成功

[jun@master ~]$ sqoop-help
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
usage: sqoop COMMAND [ARGS] Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information See 'sqoop help COMMAND' for information on a specific command.

  6.测试与MySQL的连接

  (1)列出MySQL的所有数据库

[jun@master ~]$ sqoop-list-databases --connect jdbc:mysql://localhost:3306 --username root -P
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul :: CST WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive_db
mysql
performance_schema
sys

  (2)列出数据库下的所有数据表

[jun@master ~]$ sqoop-list-tables --connect jdbc:mysql://localhost:3306/mysql --username root -P
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul :: CST WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
columns_priv
db
engine_cost
event
func
general_log
gtid_executed
help_category
help_keyword
help_relation
help_topic
innodb_index_stats
innodb_table_stats
ndb_binlog_index
plugin
proc
procs_priv
proxies_priv
server_cost
servers
slave_master_info
slave_relay_log_info
slave_worker_info
slow_log
tables_priv
time_zone
time_zone_leap_second
time_zone_name
time_zone_transition
time_zone_transition_type
user

  (3)执行MySQL的查询语句

[jun@master ~]$ sqoop-eval --connect jdbc:mysql://localhost:3306/mysql --username root -P --query "select * from plugin"
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul :: CST WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
-----------------------------------------------
| name | dl |
-----------------------------------------------
| validate_password | validate_password.so |
-----------------------------------------------

Sqoop的安装和验证的更多相关文章

  1. 【sqoop】安装配置测试sqoop1

    3.1.1 下载sqoop1:sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz 3.1.2 解压并查看目录: [hadoop@hadoop01 ~]$ tar -zxvf sq ...

  2. Hadoop 2.6.0-cdh5.4.0集群环境搭建和Apache-Hive、Sqoop的安装

    搭建此环境主要用来hadoop的学习,因此我们的操作直接在root用户下,不涉及HA. Software: Hadoop 2.6.0-cdh5.4.0 Apache-hive-2.1.0-bin Sq ...

  3. sqoop的安装

    Sqoop是一个用来完成Hadoop和关系型数据库中的数据相互转移的工具, 他可以将关系型数据库(MySql,Oracle,Postgres等)中的数据导入Hadoop的HDFS中, 也可以将HDFS ...

  4. Hive/Hbase/Sqoop的安装教程

    Hive/Hbase/Sqoop的安装教程 HIVE INSTALL 1.下载安装包:https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3 ...

  5. Sqoop的安装及简单使用

    SQOOP是用于对数据进行导入导出的. (1)把MySQL.Oracle等数据库中的数据导入到HDFS.Hive.HBase中   (2)把HDFS.Hive.HBase中的数据导出到MySQL.Or ...

  6. Sqoop的安装配置及使用

    一.Sqoop基础:连接关系型数据库与Hadoop的桥梁 1.1 Sqoop的基本概念 Hadoop正成为企业用于大数据分析的最热门选择,但想将你的数据移植过去并不容易.Apache Sqoop正在加 ...

  7. Sqoop环境安装

    环境下载 首先将下载的 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz放到 /usr/hadoop/目录下(该目录可以自定义,一般为Hadoop集群安装目录),然 ...

  8. 大数据之路week07--day06 (Sqoop 的安装及配置)

    Sqoop 的安装配置比较简单. 提供安装需要的安装包和连接mysql的驱动的百度云链接: 链接:https://pan.baidu.com/s/1pdFj0u2lZVFasgoSyhz-yQ 提取码 ...

  9. Python下载安装及验证

      目录: 一.Python介绍 二.python安装及验证 一.Python介绍 Python是著名的“龟叔”Guido van Rossum在1989年圣诞节期间,为了打发无聊的圣诞节而编写的一个 ...

随机推荐

  1. .NET Core使用NPOI导出复杂Word详解

    前言: 最近使用NPOI做了个导出Word文档的功能,关于使用.NET Core 导出Word文档的方式有很多.最终我为什么选择了NPOI来实现了这个功能,首先是NPOI是一个开源,免费且容易上手的第 ...

  2. elasticsearch深度分页问题

    elasticsearch专栏:https://www.cnblogs.com/hello-shf/category/1550315.html 一.深度分页方式from + size es 默认采用的 ...

  3. Hadoop核心组件之YARN

    YARN概述 Yet Another Resource Negotiator:另外资源的协调者 通用的资源管理系统 为上层应用提供统一的资源管理和调度 操作系统级别的调度框架,可以让各种计算框架运行在 ...

  4. httpclient整理

    package com.yjl.util; import net.sf.json.JSONObject; import org.apache.commons.lang3.StringUtils; im ...

  5. Ubuntu 查看操作系统的位数

    查看Ubuntu操作系统的位数是32位还是64位,可以通过以下命令来查看: getconf LONG_BIT 返回32或64 :如图

  6. 并发编程之多线程(Java)

    一.线程与进程区别 每个正在系统上运行的程序都是一个进程.每个进程包含一到多个线程.线程是一组指令的集合,或者是程序的特殊段,它可以在程序里独立执行.也可以把它理解为代码运行的上下文.所以线程基本上是 ...

  7. Vagrant定制个性化CentOS7虚拟机模板

    关于虚拟机模板 想用vagrant搭建hadoop集群,要完成以下准备工作: 三个虚拟机实例操作系统都是CentOS7的server版: 每个实例都要安装同样的应用.关闭防火墙.关闭swap等: 今天 ...

  8. 记录使用echarts的graph类型绘制流程图全过程(一)-x,y位置的计算

    先说下本次案例业务需求,输入2个节点,获取数据后绘制出2个节点间的路径,之前使用的是网状图,但是网状图的效果不佳,需要转换成流程图的模式: 那么如何在不修改数据的情况下,实现类似效果尼? 看了下ech ...

  9. 向net core 3.0进击——April.WebApi从2.2爬到3.0

    目录 前言 升级之路 测试 小结 前言 在之前对Swagger的变化做了调整后,就开始想着要不把之前的工程升级得了,这样就还是个demo工程,来做各种测试(当然还是因为懒),这就有了今天这个比较折腾的 ...

  10. Scala 学习笔记之集合(5)

    import collection.mutable.Buffer object CollectionDemo6 { def main(args: Array[String]): Unit = { // ...