sqoop 安装与使用

Sqoop(发音：skup)是一款开源的工具，主要用于在Hadoop(Hive)与传统的数据库间进行数据的传递，可以将一个关系型数据库（例如 ： MySQL ,Oracle ,Postgres等）中的数据导进到Hadoop的HDFS中，也可以将HDFS的数据导进到关系型数据库中。

官网：http://sqoop.apache.org/

由于官方不建议将sqoop 2用于生产环境，因为仍然使用sqoop 1

1、准备

下载

找到对应的镜像地址安装，复制下载地址

wget http://apache-mirror.8birdsvideo.com/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

解压

tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

2、安装

mv sqoop-1.4.7.bin__hadoop-2.6.0  /usr/local/

添加jdbc到sqoop的lib中:

cp mysql-connector-java-5.1.17.jar /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/lib/

3、配置

配置sqoop，加入path

vim /etc/profile

export SQOOP_HOME=/usr/local/sqoop-1.4..bin__hadoop-2.6.
export PATH=$SQOOP_HOME/bin:$PATH

配置sqoop-evn.sh

在sqoop的conf目录下

cp sqoop-env-template.sh sqoop-env.sh

vim sqoop-env.sh

文件内容增加

export HADOOP_COMMON_HOME=/lib/hadoop
export HADOOP_MAPRED_HOME=/lib/hadoop-mapreduce
export HIVE_HOME=/search/hive
export HIVE_CONF_DIR=/search/hive/conf

4、测试

测试与mysql的连通性

sqoop list-databases --connect jdbc:mysql://${HOST}:3306/test?characterEncoding=UTF-8 --username ${username} --password '$PASSWORD'
Warning: /usr/local/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
// :: WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
test

测试从mysql中导入到hive中

sqoop import \
--connect jdbc:mysql://***:3306/test?characterEncoding=UTF-8 \
--username ***\
--password '***' \
--table testxxx \
-m  \
--warehouse-dir /user/teemo/hive/warehouse/lyj_temp.db \
--hive-database lyj_temp \
--hive-import \
--hive-table testxxx

运行结果:

进入hive，select  一下能看到结果

遇到得问题：

问题1

// :: INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/5c81c7e9c9d08472a6c1a1a272e4eeb3/testxxx.jar
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/InputFormat
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:)
        at java.net.URLClassLoader.access$(URLClassLoader.java:)
        at java.net.URLClassLoader$.run(URLClassLoader.java:)
        at java.net.URLClassLoader$.run(URLClassLoader.java:)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:)
        at java.net.URLClassLoader.access$(URLClassLoader.java:)
        at java.net.URLClassLoader$.run(URLClassLoader.java:)
        at java.net.URLClassLoader$.run(URLClassLoader.java:)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:)
        at java.net.URLClassLoader.access$(URLClassLoader.java:)
        at java.net.URLClassLoader$.run(URLClassLoader.java:)
        at java.net.URLClassLoader$.run(URLClassLoader.java:)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:)
        at java.net.URLClassLoader.access$(URLClassLoader.java:)
        at java.net.URLClassLoader$.run(URLClassLoader.java:)
        at java.net.URLClassLoader$.run(URLClassLoader.java:)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at org.apache.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:)
        at com.cloudera.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.InputFormat
        at java.net.URLClassLoader.findClass(URLClassLoader.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        ...  more

配置的HIVE_HOME这个变量有问题，应该是hadoop-mapreduce这个目录

问题2

// :: ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
        at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:)
        at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:)
        at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:)
        at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:)
        at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
        at java.net.URLClassLoader.findClass(URLClassLoader.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:)
        at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:)
        ...  more

将hive的lib里面的拷贝到sqoop的lib目录下

cp hive-exec-1.2.1.jar /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/lib/