环境:Hadoop 2.3.0

sqoop 1.4.5

1、下载并解压sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz (解压完,名字会很长,可以根据需要自己修改下目录名)

  1. tar -zxvfsqoop-1.4.5.bin__hadoop-2.0.4-alpha.tar.gz

2、配置环境变量

  1. export SQOOP_HOME=/home/grid2/sqoop-1.4.5
  2. export PATH=$PATH:$SQOOP_HOME/bin

3、修改配置文件

  1. cd /home/grid2/sqoop-1.4.5/conf
  2. cp sqoop-env-template.sh sqoop-env.sh
  3. vi sqoop-env.sh
  1. exportHADOOP_COMMON_HOME=/home/grid2/hadoop-2.3.0
  2. exportHIVE_HOME=/home/grid2/apache-hive-0.13.1-bin
  1. cd /home/grid2/sqoop-1.4.5/bin
  2. vi configure-sqoop

将HBASE,ZOOKEEPER,ACCUMULO部分全部注释掉

4、拷贝mysql连接器(注意:对于1.4.5版本的sqoop,要用5.1.31版本的mysql连接器)

  1. cp mysql-connector-java-5.1.31-bin.jar./sqoop-1.4.5/lib/

5、检查sqoop是否配置成功

  1. sqoop help

6、如果想要使用sqoop的job需要配置


  1. <property>
  2. <name>sqoop.metastore.client.autoconnect.url</name>
  3. <value>jdbc:hsqldb:file:/usr/local/sqoop-1.4.6/metastore/meta.db;shutdown=true</value>
  4. <description>The connect string to use when connecting to a
  5. job-management metastore. If unspecified, uses ~/.sqoop/.
  6. You can specify a different path here.
  7. </description>
  8. </property>
  9. <property>
  10. <name>sqoop.metastore.client.autoconnect.username</name>
  11. <value>SA</value>
  12. <description>The username to bind to the metastore.
  13. </description>
  14. </property>
  15. <property>
  16. <name>sqoop.metastore.client.autoconnect.password</name>
  17. <value></value>
  18. <description>The password to bind to the metastore.
  19. </description>
  20. </property>
  21. <property>
  22. <name>sqoop.metastore.client.record.password</name>
  23. <value>true</value>
  24. <description>If true, allow saved passwords in the metastore.
  25. </description>
  26. </property>
  27. <property>
  28. <name>sqoop.metastore.server.location</name>
  29. <value>/usr/local/sqoop-1.4.6/metastore/shared.db</value>
  30. <description>Path to the shared metastore database files.
  31. If this is not set, it will be placed in ~/.sqoop/.
  32. </description>
  33. </property>
  34. <property>
  35. <name>sqoop.metastore.server.port</name>
  36. <value>16000</value>
  37. <description>Port that this metastore should listen on.
  38. </description>
  39. </property>

出错记录

问题1:

  1. ERROR manager.SqlManager: Error reading from database:
  2. java.sql.SQLException: Streaming result
  3. setcom.mysql.jdbc.RowDataDynamic@6c4fc156 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
  4. java.sql.SQLException: Streaming result set
  5. com.mysql.jdbc.RowDataDynamic@6c4fc156 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.

解决:

后经定位,发现是mysql-connect-java jar包的版本不对

改为mysql-connector-java-5.1.31 这个版本后就可以了

问题2:

  1. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)

解决:

hive/lib换成mysql-connector-java-5.1.31-bin.jar

quit退出hive,然后重新进入hive

问题3:

  1. [hadoop@Master bin]$ sqoop export --connect jdbc:mysql://localhost:3306/test --username dyh --password 000000 --table users --export-dir /user/hive/warehouse/users/part-m-00000 --input-fields-terminated-by '\0001'
  2. Warning: /usr/lib/hbase does not exist! HBase imports will fail.
  3. Please set $HBASE_HOME to the root of your HBase installation.
  4. Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
  5. Please set $HCAT_HOME to the root of your HCatalog installation.
  6. Warning: $HADOOP_HOME is deprecated.
  7. 13/12/12 19:50:38 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
  8. 13/12/12 19:50:38 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
  9. 13/12/12 19:50:38 INFO tool.CodeGenTool: Beginning code generation
  10. 13/12/12 19:50:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `users` AS t LIMIT 1
  11. 13/12/12 19:50:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `users` AS t LIMIT 1
  12. 13/12/12 19:50:38 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hadoop
  13. Note: /tmp/sqoop-hadoop/compile/9731783979d46a3414a9f86d700bec33/users.java uses or overrides a deprecated API.
  14. Note: Recompile with -Xlint:deprecation for details.
  15. 13/12/12 19:50:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/9731783979d46a3414a9f86d700bec33/users.jar
  16. 13/12/12 19:50:39 INFO mapreduce.ExportJobBase: Beginning export of users
  17. 13/12/12 19:50:41 INFO input.FileInputFormat: Total input paths to process : 1
  18. 13/12/12 19:50:41 INFO input.FileInputFormat: Total input paths to process : 1
  19. 13/12/12 19:50:41 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  20. 13/12/12 19:50:41 WARN snappy.LoadSnappy: Snappy native library not loaded
  21. 13/12/12 19:50:42 INFO mapred.JobClient: Running job: job_201312051716_0034
  22. 13/12/12 19:50:43 INFO mapred.JobClient: map 0% reduce 0%
  23. 13/12/12 19:50:51 INFO mapred.JobClient: map 25% reduce 0%
  24. 13/12/12 19:50:53 INFO mapred.JobClient: map 50% reduce 0%
  25. 13/12/12 19:50:58 INFO mapred.JobClient: Task Id : attempt_201312051716_0034_m_000002_0, Status : FAILED
  26. java.io.IOException: java.sql.SQLException: Access denied for user 'dyh'@'localhost' (using password: YES)
  27. Caused by: java.sql.SQLException: Access denied for user 'dyh'@'localhost' (using password: YES)
  28. 13/12/12 19:50:58 INFO mapred.JobClient: Task Id : attempt_201312051716_0034_m_000003_0, Status : FAILED
  29. java.io.IOException: java.sql.SQLException: Access denied for user 'dyh'@'localhost' (using password: YES)

解决办法:

将jdbc:mysql://localhost:3306/test 中的localhost 改为ip地址即可。

sqoop安装的更多相关文章

  1. 如何将mysql数据导入Hadoop之Sqoop安装

    Sqoop是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql.postgresql...)间进行数据的传递,可以将一个关系型数据库(例如 : MySQL ,Oracle , ...

  2. Sqoop安装与应用过程

    1.  参考说明 参考文档: http://sqoop.apache.org/ http://sqoop.apache.org/docs/1.99.7/admin/Installation.html ...

  3. sqoop安装部署(笔记)

    sqoop是一个把关系型数据库数据抽向hadoop的工具.同时,也支持将hive.pig等查询的结果导入关系型数据库中存储.由于,笔者部署的hadoop版本是2.2.0,所以sqoop的版本是:sqo ...

  4. sqoop安装与简单实用

    一,sqoop安装 1.解压源码包 2.配置环境变量 3.在bin目录下的 /bin/configsqoop 注释掉check报错信息 4.配置conf目录下 /conf/sqoop-env.sh 配 ...

  5. cdh版本的sqoop安装以及配置

    sqoop安装需要提前安装好sqoop依赖:hadoop .hive.hbase.zookeeper hadoop安装步骤请访问:http://www.cnblogs.com/xningge/arti ...

  6. [Hadoop]&nbsp;Sqoop安装过程详解

    Sqoop是一个用来将Hadoop和关系型数据库中的数据相互转移的工具,可以将一个关系型数据库(例如 : MySQL ,Oracle ,Postgres等)中的数据导进到Hadoop的HDFS中,也可 ...

  7. hadoop(八) - sqoop安装与使用

    一. sqoop安装: 安装在一台节点上就能够了. 1. 使用winscp上传sqoop 2. 安装和配置 加入sqoop到环境变量 将数据库连接驱动mysql-connector-5.1.8.jar ...

  8. sqoop 安装

    Sqoop是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql.postgresql...)间进行数据的传递,可以将一个关系型数据库(例如 : MySQL ,Oracle , ...

  9. Sqoop 安装部署

    1. 上传并解压 Sqoop 安装文件 将 sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz 安装包上传到 node-01 的 /root/ 目录下并将其解压 [root@no ...

  10. Sqoop安装及操作

    一.集群环境: Hostname IP Hadoop版本 Hadoop 功能 系统 node1 192.168.1.151 0.20.0 namenode hive+sqoop rhel5.4X86 ...

随机推荐

  1. SqlServer数据文件增长也很快,到底是哪些表增长造成的呢?

    查询数据库中所有表的大小,哪些表的数据量较大 create table #t (name ), rows ), data ), index_size ), unused )) exec sp_MSfo ...

  2. 初级——程序如何打包成apk文件

    将Eclipse Android项目打包成APK文件是本文要介绍的内容,主要是来了解并学习Eclipse Android打包的内容,具体关于Eclipse Android内容的详解来看本文.Eclip ...

  3. IOS开发-UI基础-视图

    //------------------------------UIWindow--------------------------// 1.UIWindow:是 UIView 的子类,用于管理.协调 ...

  4. 理解javascript中的原型模式

    一.为什么要用原型模式. 早期采用工厂模式或构造函数模式的缺点:  1.工厂模式:函数creatPerson根据接受的参数来构建一个包含所有必要信息的person对象,这个函数可以被无数次的调用,工厂 ...

  5. Android官方提供的支持不同屏幕大小的全部方法

    转载请注明出处:http://blog.csdn.net/guolin_blog/article/details/8830286 原文地址为:http://developer.android.com/ ...

  6. 【转】如何使用PhoneGap打包Web App

    如何使用PhoneGap打包Web App 最近做了一款小游戏,定位是移动端访问,思来想去最后选择了jQuery mobile最为框架,制作差不多以后,是否可以打包成App,恰好以前对PhoneGap ...

  7. SQL Server附加数据库文件出错

    场景: 新装一台数据库服务器,装好后,附加数据库时出错.附加前的数据库架构没有在新服务器上安装.新服务器上只有默认dbo架构. 解决: 以windows身份验证登录,附加正常. 错误码可能为5120. ...

  8. haproxy实现负载均衡

    一.安装tar zxvf haproxy-1.4.22.tar.gzcd haproxy-1.4.22make TARGET=linux26 PREFIX=/usr/local/haproxy ins ...

  9. DG_Oracle DataGuard Switchover主备节点切换(案例)

    2014-06-09 Created By BaoXinjian Thanks and Regards http://wenku.baidu.com/view/dc9f00d349649b6648d7 ...

  10. POJ 3984 迷宫问题(BFS)

    迷宫问题 Description 定义一个二维数组: int maze[5][5] = { 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, ...