sqoop安装
环境:Hadoop 2.3.0
sqoop 1.4.5
1、下载并解压sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz (解压完,名字会很长,可以根据需要自己修改下目录名)
tar -zxvfsqoop-1.4.5.bin__hadoop-2.0.4-alpha.tar.gz
2、配置环境变量
export SQOOP_HOME=/home/grid2/sqoop-1.4.5
export PATH=$PATH:$SQOOP_HOME/bin
3、修改配置文件
cd /home/grid2/sqoop-1.4.5/conf
cp sqoop-env-template.sh sqoop-env.sh
vi sqoop-env.sh
exportHADOOP_COMMON_HOME=/home/grid2/hadoop-2.3.0
exportHIVE_HOME=/home/grid2/apache-hive-0.13.1-bin
cd /home/grid2/sqoop-1.4.5/bin
vi configure-sqoop
将HBASE,ZOOKEEPER,ACCUMULO部分全部注释掉
4、拷贝mysql连接器(注意:对于1.4.5版本的sqoop,要用5.1.31版本的mysql连接器)
cp mysql-connector-java-5.1.31-bin.jar./sqoop-1.4.5/lib/
5、检查sqoop是否配置成功
sqoop help
6、如果想要使用sqoop的job需要配置
<property>
<name>sqoop.metastore.client.autoconnect.url</name>
<value>jdbc:hsqldb:file:/usr/local/sqoop-1.4.6/metastore/meta.db;shutdown=true</value>
<description>The connect string to use when connecting to a
job-management metastore. If unspecified, uses ~/.sqoop/.
You can specify a different path here.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.username</name>
<value>SA</value>
<description>The username to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.password</name>
<value></value>
<description>The password to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.client.record.password</name>
<value>true</value>
<description>If true, allow saved passwords in the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.server.location</name>
<value>/usr/local/sqoop-1.4.6/metastore/shared.db</value>
<description>Path to the shared metastore database files.
If this is not set, it will be placed in ~/.sqoop/.
</description>
</property>
<property>
<name>sqoop.metastore.server.port</name>
<value>16000</value>
<description>Port that this metastore should listen on.
</description>
</property>
出错记录
问题1:
ERROR manager.SqlManager: Error reading from database:
java.sql.SQLException: Streaming result
setcom.mysql.jdbc.RowDataDynamic@6c4fc156 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set
com.mysql.jdbc.RowDataDynamic@6c4fc156 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
解决:
后经定位,发现是mysql-connect-java jar包的版本不对
改为mysql-connector-java-5.1.31 这个版本后就可以了
问题2:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)
解决:
hive/lib换成mysql-connector-java-5.1.31-bin.jar
quit退出hive,然后重新进入hive
问题3:
[hadoop@Master bin]$ sqoop export --connect jdbc:mysql://localhost:3306/test --username dyh --password 000000 --table users --export-dir /user/hive/warehouse/users/part-m-00000 --input-fields-terminated-by '\0001'
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: $HADOOP_HOME is deprecated.
13/12/12 19:50:38 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/12/12 19:50:38 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/12/12 19:50:38 INFO tool.CodeGenTool: Beginning code generation
13/12/12 19:50:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `users` AS t LIMIT 1
13/12/12 19:50:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `users` AS t LIMIT 1
13/12/12 19:50:38 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hadoop
Note: /tmp/sqoop-hadoop/compile/9731783979d46a3414a9f86d700bec33/users.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/12/12 19:50:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/9731783979d46a3414a9f86d700bec33/users.jar
13/12/12 19:50:39 INFO mapreduce.ExportJobBase: Beginning export of users
13/12/12 19:50:41 INFO input.FileInputFormat: Total input paths to process : 1
13/12/12 19:50:41 INFO input.FileInputFormat: Total input paths to process : 1
13/12/12 19:50:41 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/12/12 19:50:41 WARN snappy.LoadSnappy: Snappy native library not loaded
13/12/12 19:50:42 INFO mapred.JobClient: Running job: job_201312051716_0034
13/12/12 19:50:43 INFO mapred.JobClient: map 0% reduce 0%
13/12/12 19:50:51 INFO mapred.JobClient: map 25% reduce 0%
13/12/12 19:50:53 INFO mapred.JobClient: map 50% reduce 0%
13/12/12 19:50:58 INFO mapred.JobClient: Task Id : attempt_201312051716_0034_m_000002_0, Status : FAILED
java.io.IOException: java.sql.SQLException: Access denied for user 'dyh'@'localhost' (using password: YES)
Caused by: java.sql.SQLException: Access denied for user 'dyh'@'localhost' (using password: YES)
13/12/12 19:50:58 INFO mapred.JobClient: Task Id : attempt_201312051716_0034_m_000003_0, Status : FAILED
java.io.IOException: java.sql.SQLException: Access denied for user 'dyh'@'localhost' (using password: YES)
解决办法:
将jdbc:mysql://localhost:3306/test 中的localhost 改为ip地址即可。
sqoop安装的更多相关文章
- 如何将mysql数据导入Hadoop之Sqoop安装
Sqoop是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql.postgresql...)间进行数据的传递,可以将一个关系型数据库(例如 : MySQL ,Oracle , ...
- Sqoop安装与应用过程
1. 参考说明 参考文档: http://sqoop.apache.org/ http://sqoop.apache.org/docs/1.99.7/admin/Installation.html ...
- sqoop安装部署(笔记)
sqoop是一个把关系型数据库数据抽向hadoop的工具.同时,也支持将hive.pig等查询的结果导入关系型数据库中存储.由于,笔者部署的hadoop版本是2.2.0,所以sqoop的版本是:sqo ...
- sqoop安装与简单实用
一,sqoop安装 1.解压源码包 2.配置环境变量 3.在bin目录下的 /bin/configsqoop 注释掉check报错信息 4.配置conf目录下 /conf/sqoop-env.sh 配 ...
- cdh版本的sqoop安装以及配置
sqoop安装需要提前安装好sqoop依赖:hadoop .hive.hbase.zookeeper hadoop安装步骤请访问:http://www.cnblogs.com/xningge/arti ...
- [Hadoop] Sqoop安装过程详解
Sqoop是一个用来将Hadoop和关系型数据库中的数据相互转移的工具,可以将一个关系型数据库(例如 : MySQL ,Oracle ,Postgres等)中的数据导进到Hadoop的HDFS中,也可 ...
- hadoop(八) - sqoop安装与使用
一. sqoop安装: 安装在一台节点上就能够了. 1. 使用winscp上传sqoop 2. 安装和配置 加入sqoop到环境变量 将数据库连接驱动mysql-connector-5.1.8.jar ...
- sqoop 安装
Sqoop是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql.postgresql...)间进行数据的传递,可以将一个关系型数据库(例如 : MySQL ,Oracle , ...
- Sqoop 安装部署
1. 上传并解压 Sqoop 安装文件 将 sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz 安装包上传到 node-01 的 /root/ 目录下并将其解压 [root@no ...
- Sqoop安装及操作
一.集群环境: Hostname IP Hadoop版本 Hadoop 功能 系统 node1 192.168.1.151 0.20.0 namenode hive+sqoop rhel5.4X86 ...
随机推荐
- SqlServer数据文件增长也很快,到底是哪些表增长造成的呢?
查询数据库中所有表的大小,哪些表的数据量较大 create table #t (name ), rows ), data ), index_size ), unused )) exec sp_MSfo ...
- 初级——程序如何打包成apk文件
将Eclipse Android项目打包成APK文件是本文要介绍的内容,主要是来了解并学习Eclipse Android打包的内容,具体关于Eclipse Android内容的详解来看本文.Eclip ...
- IOS开发-UI基础-视图
//------------------------------UIWindow--------------------------// 1.UIWindow:是 UIView 的子类,用于管理.协调 ...
- 理解javascript中的原型模式
一.为什么要用原型模式. 早期采用工厂模式或构造函数模式的缺点: 1.工厂模式:函数creatPerson根据接受的参数来构建一个包含所有必要信息的person对象,这个函数可以被无数次的调用,工厂 ...
- Android官方提供的支持不同屏幕大小的全部方法
转载请注明出处:http://blog.csdn.net/guolin_blog/article/details/8830286 原文地址为:http://developer.android.com/ ...
- 【转】如何使用PhoneGap打包Web App
如何使用PhoneGap打包Web App 最近做了一款小游戏,定位是移动端访问,思来想去最后选择了jQuery mobile最为框架,制作差不多以后,是否可以打包成App,恰好以前对PhoneGap ...
- SQL Server附加数据库文件出错
场景: 新装一台数据库服务器,装好后,附加数据库时出错.附加前的数据库架构没有在新服务器上安装.新服务器上只有默认dbo架构. 解决: 以windows身份验证登录,附加正常. 错误码可能为5120. ...
- haproxy实现负载均衡
一.安装tar zxvf haproxy-1.4.22.tar.gzcd haproxy-1.4.22make TARGET=linux26 PREFIX=/usr/local/haproxy ins ...
- DG_Oracle DataGuard Switchover主备节点切换(案例)
2014-06-09 Created By BaoXinjian Thanks and Regards http://wenku.baidu.com/view/dc9f00d349649b6648d7 ...
- POJ 3984 迷宫问题(BFS)
迷宫问题 Description 定义一个二维数组: int maze[5][5] = { 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, ...