sqoop1.99.4 JAVA API操作

貌似天国还没有介绍1.99.4的java操作代码的，自己吃一次螃蟹吧

如果你是MAVEN项目

 <dependency>

   <groupId>org.apache.sqoop</groupId>

     <artifactId>sqoop-client</artifactId>

     <version>1.99.4</version>

 </dependency>

如果你是java项目

导入sqoop1.99.4中shell目录下的lib里面全部jar包就行(不用server中的）

HDFS->MYSQL

 package org.admln.sqoopOperate;

 import org.apache.sqoop.client.SqoopClient;

 import org.apache.sqoop.model.MFromConfig;

 import org.apache.sqoop.model.MJob;

 import org.apache.sqoop.model.MLink;

 import org.apache.sqoop.model.MLinkConfig;

 import org.apache.sqoop.model.MSubmission;

 import org.apache.sqoop.model.MToConfig;

 import org.apache.sqoop.submission.counter.Counter;

 import org.apache.sqoop.submission.counter.CounterGroup;

 import org.apache.sqoop.submission.counter.Counters;

 import org.apache.sqoop.validation.Status;

 public class HDFSToMysql {

     public static void main(String[] args) {

         sqoopTransfer();

     }

     public static void sqoopTransfer() {

         //初始化

         String url = "http://hadoop:12000/sqoop/";

         SqoopClient client = new SqoopClient(url);

         //创建一个源链接 HDFS

         long fromConnectorId = 1;

         MLink fromLink = client.createLink(fromConnectorId);

         fromLink.setName("HDFS connector");

         fromLink.setCreationUser("admln");

         MLinkConfig fromLinkConfig = fromLink.getConnectorLinkConfig();

         fromLinkConfig.getStringInput("linkConfig.uri").setValue("hdfs://hadoop:8020/");

         Status fromStatus = client.saveLink(fromLink);

         if(fromStatus.canProceed()) {

          System.out.println("创建HDFS Link成功，ID为: " + fromLink.getPersistenceId());

         } else {

          System.out.println("创建HDFS Link失败");

         }

         //创建一个目的地链接 JDBC

         long toConnectorId = 2;

         MLink toLink = client.createLink(toConnectorId);

         toLink.setName("JDBC connector");

         toLink.setCreationUser("admln");

         MLinkConfig toLinkConfig = toLink.getConnectorLinkConfig();

         toLinkConfig.getStringInput("linkConfig.connectionString").setValue("jdbc:mysql://hadoop:3306/hive");

         toLinkConfig.getStringInput("linkConfig.jdbcDriver").setValue("com.mysql.jdbc.Driver");

         toLinkConfig.getStringInput("linkConfig.username").setValue("hive");

         toLinkConfig.getStringInput("linkConfig.password").setValue("hive");

         Status toStatus = client.saveLink(toLink);

         if(toStatus.canProceed()) {

          System.out.println("创建JDBC Link成功，ID为: " + toLink.getPersistenceId());

         } else {

          System.out.println("创建JDBC Link失败");

         }

         //创建一个任务

         long fromLinkId = fromLink.getPersistenceId();

         long toLinkId = toLink.getPersistenceId();

         MJob job = client.createJob(fromLinkId, toLinkId);

         job.setName("HDFS to MySQL job");

         job.setCreationUser("admln");

         //设置源链接任务配置信息

         MFromConfig fromJobConfig = job.getFromJobConfig();

         fromJobConfig.getStringInput("fromJobConfig.inputDirectory").setValue("/out/aboutyunLog/HiveExport/ipstatistical/data");

         //创建目的地链接任务配置信息

         MToConfig toJobConfig = job.getToJobConfig();

         toJobConfig.getStringInput("toJobConfig.schemaName").setValue("aboutyunlog");

         toJobConfig.getStringInput("toJobConfig.tableName").setValue("ipstatistical");

         //toJobConfig.getStringInput("fromJobConfig.partitionColumn").setValue("id");

         // set the driver config values

         //MDriverConfig driverConfig = job.getDriverConfig();

         //driverConfig.getStringInput("throttlingConfig.numExtractors").setValue("3");//这句还没弄明白

         Status status = client.saveJob(job);

         if(status.canProceed()) {

          System.out.println("JOB创建成功，ID为: "+ job.getPersistenceId());

         } else {

          System.out.println("JOB创建失败。");

         }

         //启动任务

         long jobId = job.getPersistenceId();

         MSubmission submission = client.startJob(jobId);

         System.out.println("JOB提交状态为 : " + submission.getStatus());

         while(submission.getStatus().isRunning() && submission.getProgress() != -1) {

           System.out.println("进度 : " + String.format("%.2f %%", submission.getProgress() * 100));

           //三秒报告一次进度

           try {

             Thread.sleep(3000);

           } catch (InterruptedException e) {

             e.printStackTrace();

           }

         }

         System.out.println("JOB执行结束... ...");

         System.out.println("Hadoop任务ID为 :" + submission.getExternalId());

         Counters counters = submission.getCounters();

         if(counters != null) {

           System.out.println("计数器:");

           for(CounterGroup group : counters) {

             System.out.print("\t");

             System.out.println(group.getName());

             for(Counter counter : group) {

               System.out.print("\t\t");

               System.out.print(counter.getName());

               System.out.print(": ");

               System.out.println(counter.getValue());

             }

           }

         }

         if(submission.getExceptionInfo() != null) {

           System.out.println("JOB执行异常，异常信息为 : " +submission.getExceptionInfo());

         }

         System.out.println("HDFS通过sqoop传输数据到MySQL统计执行完毕");

     }

 }

MYSQL->HDFS

 package org.admln.sqoopOperate;

 import org.apache.sqoop.client.SqoopClient;

 import org.apache.sqoop.model.MDriverConfig;

 import org.apache.sqoop.model.MFromConfig;

 import org.apache.sqoop.model.MJob;

 import org.apache.sqoop.model.MLink;

 import org.apache.sqoop.model.MLinkConfig;

 import org.apache.sqoop.model.MSubmission;

 import org.apache.sqoop.model.MToConfig;

 import org.apache.sqoop.submission.counter.Counter;

 import org.apache.sqoop.submission.counter.CounterGroup;

 import org.apache.sqoop.submission.counter.Counters;

 import org.apache.sqoop.validation.Status;

 public class MysqlToHDFS {

     public static void main(String[] args) {

         sqoopTransfer();

     }

     public static void sqoopTransfer() {

         //初始化

         String url = "http://hadoop:12000/sqoop/";

         SqoopClient client = new SqoopClient(url);

         //创建一个源链接 JDBC

         long fromConnectorId = 2;

         MLink fromLink = client.createLink(fromConnectorId);

         fromLink.setName("JDBC connector");

         fromLink.setCreationUser("admln");

         MLinkConfig fromLinkConfig = fromLink.getConnectorLinkConfig();

         fromLinkConfig.getStringInput("linkConfig.connectionString").setValue("jdbc:mysql://hadoop:3306/hive");

         fromLinkConfig.getStringInput("linkConfig.jdbcDriver").setValue("com.mysql.jdbc.Driver");

         fromLinkConfig.getStringInput("linkConfig.username").setValue("hive");

         fromLinkConfig.getStringInput("linkConfig.password").setValue("hive");

         Status fromStatus = client.saveLink(fromLink);

         if(fromStatus.canProceed()) {

          System.out.println("创建JDBC Link成功，ID为: " + fromLink.getPersistenceId());

         } else {

          System.out.println("创建JDBC Link失败");

         }

         //创建一个目的地链接HDFS

         long toConnectorId = 1;

         MLink toLink = client.createLink(toConnectorId);

         toLink.setName("HDFS connector");

         toLink.setCreationUser("admln");

         MLinkConfig toLinkConfig = toLink.getConnectorLinkConfig();

         toLinkConfig.getStringInput("linkConfig.uri").setValue("hdfs://hadoop:8020/");

         Status toStatus = client.saveLink(toLink);

         if(toStatus.canProceed()) {

          System.out.println("创建HDFS Link成功，ID为: " + toLink.getPersistenceId());

         } else {

          System.out.println("创建HDFS Link失败");

         }

         //创建一个任务

         long fromLinkId = fromLink.getPersistenceId();

         long toLinkId = toLink.getPersistenceId();

         MJob job = client.createJob(fromLinkId, toLinkId);

         job.setName("MySQL to HDFS job");

         job.setCreationUser("admln");

         //设置源链接任务配置信息

         MFromConfig fromJobConfig = job.getFromJobConfig();

         fromJobConfig.getStringInput("fromJobConfig.schemaName").setValue("sqoop");

         fromJobConfig.getStringInput("fromJobConfig.tableName").setValue("sqoop");

         fromJobConfig.getStringInput("fromJobConfig.partitionColumn").setValue("id");

         MToConfig toJobConfig = job.getToJobConfig();

         toJobConfig.getStringInput("toJobConfig.outputDirectory").setValue("/usr/tmp");

         MDriverConfig driverConfig = job.getDriverConfig();

         driverConfig.getStringInput("throttlingConfig.numExtractors").setValue("3");

         Status status = client.saveJob(job);

         if(status.canProceed()) {

          System.out.println("JOB创建成功，ID为: "+ job.getPersistenceId());

         } else {

          System.out.println("JOB创建失败。");

         }

         //启动任务

         long jobId = job.getPersistenceId();

         MSubmission submission = client.startJob(jobId);

         System.out.println("JOB提交状态为 : " + submission.getStatus());

         while(submission.getStatus().isRunning() && submission.getProgress() != -1) {

           System.out.println("进度 : " + String.format("%.2f %%", submission.getProgress() * 100));

           //三秒报告一次进度

           try {

             Thread.sleep(3000);

           } catch (InterruptedException e) {

             e.printStackTrace();

           }

         }

         System.out.println("JOB执行结束... ...");

         System.out.println("Hadoop任务ID为 :" + submission.getExternalId());

         Counters counters = submission.getCounters();

         if(counters != null) {

           System.out.println("计数器:");

           for(CounterGroup group : counters) {

             System.out.print("\t");

             System.out.println(group.getName());

             for(Counter counter : group) {

               System.out.print("\t\t");

               System.out.print(counter.getName());

               System.out.print(": ");

               System.out.println(counter.getValue());

             }

           }

         }

         if(submission.getExceptionInfo() != null) {

           System.out.println("JOB执行异常，异常信息为 : " +submission.getExceptionInfo());

         }

         System.out.println("MySQL通过sqoop传输数据到HDFS统计执行完毕");

     }

 }

别问为什么没有MYSQL和HBASE、HIVE互导的代码

20150102

sqoop1.99.4 JAVA API操作的更多相关文章

MongoDB Java API操作很全的整理
MongoDB 是一个基于分布式文件存储的数据库.由 C++ 语言编写,一般生产上建议以共享分片的形式来部署. 但是MongoDB官方也提供了其它语言的客户端操作API.如下图所示: 提供了C.C++ ...
hive-通过Java API操作
通过Java API操作hive,算是测试hive第三种对外接口测试hive 服务启动 package org.admln.hive; import java.sql.SQLException; i ...
hadoop2-HBase的Java API操作
Hbase提供了丰富的Java API,以及线程池操作,下面我用线程池来展示一下使用Java API操作Hbase. 项目结构如下: 我使用的Hbase的版本是 hbase-0.98.9-hadoop ...
使用Java API操作HDFS文件系统
使用Junit封装HFDS import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.*; import org ...
Kafka系列三 java API操作
使用java API操作kafka 1.pom.xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xs ...
Hadoop之HDFS（三）HDFS的JAVA API操作
HDFS的JAVA API操作 HDFS 在生产应用中主要是客户端的开发,其核心步骤是从 HDFS 提供的 api中构造一个 HDFS 的访问客户端对象,然后通过该客户端对象操作(增删改查)HDFS ...
zookeeper的java api操作
zookeeper的java api操作创建会话: Zookeeper(String connectString,int sessionTimeout,Watcher watcher) Zookee ...
java api操作
java api操作导入开发包将hbase安装包中lib下包导入java项目创建表 Configuration conf = HBaseConfiguration.create(); c ...
HDFS 05 - HDFS 常用的 Java API 操作
目录 0 - 配置 Hadoop 环境(Windows系统) 1 - 导入 Maven 依赖 2 - 常用类介绍 3 - 常见 API 操作 3.1 获取文件系统(重要) 3.2 创建目录.写入文件 ...

随机推荐

各种边框样式。。本以为border不是这么用的。
关于文本框样式□ 文本框样式 □ 显示虚线边框的文本框(IE5.5才可看到效果) <INPUT style="border-width: 1px,1px,1px,1px;border ...
第三百五十五天 how can I 坚持
快一年了,三百五十五天了,等写个程序算算时间,看看日期和天数能不能对的上,哈哈. 计划还是未制定,天气预报还是没有写完,立马行动,发完这个博客,立马行动. 计划:设计模式1个月,三大框架3个月,计算机 ...
Flume日志收集
进入 http://blog.csdn.net/zhouleilei/article/details/8568147
（转载）javascript函数作用域和提前声明
http://www.cnblogs.com/ArthurPatten/p/3274080.html 一些语言如C.java都有块级作用域,即花括号内的每一段代码都具有各自的作用域,而且变量在声明它们 ...
ML 徒手系列拉格朗日乘子法
拉格朗日乘子法是解决极值问题的方法. 本方法是计算多元函数在约束条件下的极值问题的方法. 1.多元函数与约束问题如下图所示,f(x,y)为多元函数,g(x,y)=c为约束条件.目的是计算在约束条件下 ...
python sleep
Python Sleep休眠函数 Python 编程中使用 time 模块可以让程序休眠,具体方法是time.sleep(秒数),其中"秒数"以秒为单位,可以是小数,0.1秒则代表 ...
getName()、getCanonicalName()、getSimpleName()异同
package classes; class Box { class Inner { } } public class TestGetName { public static void main(St ...
【Python实战02】共享Python代码到PyPI社区
之前学习了Python的列表,以及编写了一个函数来进行列表的输出,这次我们就继续来学习如何把我们已经编写好的代码共享到PyPI社区,这里以上篇文章中编写的print_lol函数为例. 函数转换为模块 ...
window.print打印指定div
window.print可以打印网页,但有时候我们只希望打印特定控件或内容,怎么办呢? 首先我们可以把要打印的内容放在div中,然后用下面的代码进行打印. <html> <head& ...
opennebula 出错截图与调试

sqoop1.99.4 JAVA API操作

sqoop1.99.4 JAVA API操作的更多相关文章

随机推荐

热门专题