Sqoop入门

1 下载地址

        http://archive.cloudera.com/cdh5/cdh/5/

        版本 sqoop-1.4.6-cdh5.7.0

        安装包  sqoop-1.4.6-cdh5.7.0.tar.gz  上传到hadoop服务器环境

 

2 解压软件

        [root@hdp1 /mnt/software]#tar -zvxf sqoop-1.4.6-cdh5.7.0.tar.gz

 

3 重命名配置文件

        [root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0]#cd conf/

        [root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/conf]#ls

        oraoop-site-template.xml  sqoop-env-template.sh    sqoop-site.xml

        sqoop-env-template.cmd    sqoop-site-template.xml

        [root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/conf]#cp sqoop-env-template.sh sqoop-env.sh

 

4 修改配置文件

       [root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/conf]#vim sqoop-env.sh

        export HADOOP_COMMON_HOME=/mnt/software/hadoop-2.6.0-cdh5.7.0

        export HADOOP_MAPRED_HOME=/mnt/software/hadoop-2.6.0-cdh5.7.0

        export HIVE_HOME=/mnt/software/hive-1.1.0-cdh5.7.0

        export ZOOKEEPER_HOME=/mnt/software/zookeeper-3.4.5-cdh5.7.05

        export ZOOCFGDIR=/mnt/software/zookeeper-3.4.5-cdh5.7.0

 

5 拷贝jdbc驱动 到 Sqoop lib目录

        [root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/conf]#cp /mnt/software/hive-1.1.0-cdh5.7.0/lib/mysql-connector-java-5.1.45.jar /mnt/software/sqoop-1.4.6-cdh5.7.0/lib/

 

6 验证Sqoop

      配置Sqoop环境变量 

         [root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/bin]#vim /etc/profile

         export SQOOP_HOME=/mnt/software/sqoop-1.4.6-cdh5.7.0

         export PATH=.:$PATH:$SQOOP_HOME/bin 

        [root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0/bin]#source /etc/profile

        验证Sqoop

        

   

7 测试Sqoop是否能够成功连接数据库

[root@hdp1 /mnt/software/sqoop-1.4.6-cdh5.7.0]#sqoop list-databases --connect jdbc:mysql://hdp1:3306/ --username root --password yourpasswd

 

8 导入数据

(1)确定Mysql服务开启正常

(2)在Mysql中新建一张表并插入一些数据

[root@hdp1 /mnt/software/hive-1.1.0-cdh5.7.0]#mysql -uroot -p yourpasswd

mysql> create database company;

mysql> create table company.staff(id int(4) primary key not null auto_increment, name varchar(255), sex varchar(255));

mysql>  insert into company.staff(name, sex) values('ajin', 'Male’);

mysql> insert into company.staff(name, sex) values('awen', 'FeMale');

(3)导入数据

    全部导入

        sqoop import \

--connect jdbc:mysql://hdp1:3306/company \

--username root \

--password yourpasswd \

--table staff \

--target-dir /mnt/software/tmp/sqoopdir \

--delete-target-dir \

--num-mappers 1 \

--fields-terminated-by  “\t"

 

遇到异常

Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject

 

解决异常 

    这是因为sqoop缺少java-json.jar包. 下载jar包导入lib目录下  http://www.java2s.com/Code/Jar/j/Downloadjavajsonjar.htm

    mv java-json.jar /mnt/software/sqoop-1.4.6-cdh5.7.0/lib

 

继续导入

     再次遇到异常

        

 

设置accumulo目录

[root@hdp1 /mnt/software/tmp]#mkdir /mnt/software/tmp/accumulo

导入环境变量    

[root@hdp1 /mnt/software/tmp]#export ACCUMULO_HOME=/mnt/software/tmp/accumulo

再次导入 成功