sqoop1.99.4 JAVA API操作

貌似天国还没有介绍1.99.4的java操作代码的，自己吃一次螃蟹吧

如果你是MAVEN项目

 <dependency>

   <groupId>org.apache.sqoop</groupId>

     <artifactId>sqoop-client</artifactId>

     <version>1.99.4</version>

 </dependency>

如果你是java项目

导入sqoop1.99.4中shell目录下的lib里面全部jar包就行(不用server中的）

HDFS->MYSQL

 package org.admln.sqoopOperate;

 import org.apache.sqoop.client.SqoopClient;

 import org.apache.sqoop.model.MFromConfig;

 import org.apache.sqoop.model.MJob;

 import org.apache.sqoop.model.MLink;

 import org.apache.sqoop.model.MLinkConfig;

 import org.apache.sqoop.model.MSubmission;

 import org.apache.sqoop.model.MToConfig;

 import org.apache.sqoop.submission.counter.Counter;

 import org.apache.sqoop.submission.counter.CounterGroup;

 import org.apache.sqoop.submission.counter.Counters;

 import org.apache.sqoop.validation.Status;

 public class HDFSToMysql {

     public static void main(String[] args) {

         sqoopTransfer();

     }

     public static void sqoopTransfer() {

         //初始化

         String url = "http://hadoop:12000/sqoop/";

         SqoopClient client = new SqoopClient(url);

         //创建一个源链接 HDFS

         long fromConnectorId = 1;

         MLink fromLink = client.createLink(fromConnectorId);

         fromLink.setName("HDFS connector");

         fromLink.setCreationUser("admln");

         MLinkConfig fromLinkConfig = fromLink.getConnectorLinkConfig();

         fromLinkConfig.getStringInput("linkConfig.uri").setValue("hdfs://hadoop:8020/");

         Status fromStatus = client.saveLink(fromLink);

         if(fromStatus.canProceed()) {

          System.out.println("创建HDFS Link成功，ID为: " + fromLink.getPersistenceId());

         } else {

          System.out.println("创建HDFS Link失败");

         }

         //创建一个目的地链接 JDBC

         long toConnectorId = 2;

         MLink toLink = client.createLink(toConnectorId);

         toLink.setName("JDBC connector");

         toLink.setCreationUser("admln");

         MLinkConfig toLinkConfig = toLink.getConnectorLinkConfig();

         toLinkConfig.getStringInput("linkConfig.connectionString").setValue("jdbc:mysql://hadoop:3306/hive");

         toLinkConfig.getStringInput("linkConfig.jdbcDriver").setValue("com.mysql.jdbc.Driver");

         toLinkConfig.getStringInput("linkConfig.username").setValue("hive");

         toLinkConfig.getStringInput("linkConfig.password").setValue("hive");

         Status toStatus = client.saveLink(toLink);

         if(toStatus.canProceed()) {

          System.out.println("创建JDBC Link成功，ID为: " + toLink.getPersistenceId());

         } else {

          System.out.println("创建JDBC Link失败");

         }

         //创建一个任务

         long fromLinkId = fromLink.getPersistenceId();

         long toLinkId = toLink.getPersistenceId();

         MJob job = client.createJob(fromLinkId, toLinkId);

         job.setName("HDFS to MySQL job");

         job.setCreationUser("admln");

         //设置源链接任务配置信息

         MFromConfig fromJobConfig = job.getFromJobConfig();

         fromJobConfig.getStringInput("fromJobConfig.inputDirectory").setValue("/out/aboutyunLog/HiveExport/ipstatistical/data");

         //创建目的地链接任务配置信息

         MToConfig toJobConfig = job.getToJobConfig();

         toJobConfig.getStringInput("toJobConfig.schemaName").setValue("aboutyunlog");

         toJobConfig.getStringInput("toJobConfig.tableName").setValue("ipstatistical");

         //toJobConfig.getStringInput("fromJobConfig.partitionColumn").setValue("id");

         // set the driver config values

         //MDriverConfig driverConfig = job.getDriverConfig();

         //driverConfig.getStringInput("throttlingConfig.numExtractors").setValue("3");//这句还没弄明白

         Status status = client.saveJob(job);

         if(status.canProceed()) {

          System.out.println("JOB创建成功，ID为: "+ job.getPersistenceId());

         } else {

          System.out.println("JOB创建失败。");

         }

         //启动任务

         long jobId = job.getPersistenceId();

         MSubmission submission = client.startJob(jobId);

         System.out.println("JOB提交状态为 : " + submission.getStatus());

         while(submission.getStatus().isRunning() && submission.getProgress() != -1) {

           System.out.println("进度 : " + String.format("%.2f %%", submission.getProgress() * 100));

           //三秒报告一次进度

           try {

             Thread.sleep(3000);

           } catch (InterruptedException e) {

             e.printStackTrace();

           }

         }

         System.out.println("JOB执行结束... ...");

         System.out.println("Hadoop任务ID为 :" + submission.getExternalId());

         Counters counters = submission.getCounters();

         if(counters != null) {

           System.out.println("计数器:");

           for(CounterGroup group : counters) {

             System.out.print("\t");

             System.out.println(group.getName());

             for(Counter counter : group) {

               System.out.print("\t\t");

               System.out.print(counter.getName());

               System.out.print(": ");

               System.out.println(counter.getValue());

             }

           }

         }

         if(submission.getExceptionInfo() != null) {

           System.out.println("JOB执行异常，异常信息为 : " +submission.getExceptionInfo());

         }

         System.out.println("HDFS通过sqoop传输数据到MySQL统计执行完毕");

     }

 }

MYSQL->HDFS

 package org.admln.sqoopOperate;

 import org.apache.sqoop.client.SqoopClient;

 import org.apache.sqoop.model.MDriverConfig;

 import org.apache.sqoop.model.MFromConfig;

 import org.apache.sqoop.model.MJob;

 import org.apache.sqoop.model.MLink;

 import org.apache.sqoop.model.MLinkConfig;

 import org.apache.sqoop.model.MSubmission;

 import org.apache.sqoop.model.MToConfig;

 import org.apache.sqoop.submission.counter.Counter;

 import org.apache.sqoop.submission.counter.CounterGroup;

 import org.apache.sqoop.submission.counter.Counters;

 import org.apache.sqoop.validation.Status;

 public class MysqlToHDFS {

     public static void main(String[] args) {

         sqoopTransfer();

     }

     public static void sqoopTransfer() {

         //初始化

         String url = "http://hadoop:12000/sqoop/";

         SqoopClient client = new SqoopClient(url);

         //创建一个源链接 JDBC

         long fromConnectorId = 2;

         MLink fromLink = client.createLink(fromConnectorId);

         fromLink.setName("JDBC connector");

         fromLink.setCreationUser("admln");

         MLinkConfig fromLinkConfig = fromLink.getConnectorLinkConfig();

         fromLinkConfig.getStringInput("linkConfig.connectionString").setValue("jdbc:mysql://hadoop:3306/hive");

         fromLinkConfig.getStringInput("linkConfig.jdbcDriver").setValue("com.mysql.jdbc.Driver");

         fromLinkConfig.getStringInput("linkConfig.username").setValue("hive");

         fromLinkConfig.getStringInput("linkConfig.password").setValue("hive");

         Status fromStatus = client.saveLink(fromLink);

         if(fromStatus.canProceed()) {

          System.out.println("创建JDBC Link成功，ID为: " + fromLink.getPersistenceId());

         } else {

          System.out.println("创建JDBC Link失败");

         }

         //创建一个目的地链接HDFS

         long toConnectorId = 1;

         MLink toLink = client.createLink(toConnectorId);

         toLink.setName("HDFS connector");

         toLink.setCreationUser("admln");

         MLinkConfig toLinkConfig = toLink.getConnectorLinkConfig();

         toLinkConfig.getStringInput("linkConfig.uri").setValue("hdfs://hadoop:8020/");

         Status toStatus = client.saveLink(toLink);

         if(toStatus.canProceed()) {

          System.out.println("创建HDFS Link成功，ID为: " + toLink.getPersistenceId());

         } else {

          System.out.println("创建HDFS Link失败");

         }

         //创建一个任务

         long fromLinkId = fromLink.getPersistenceId();

         long toLinkId = toLink.getPersistenceId();

         MJob job = client.createJob(fromLinkId, toLinkId);

         job.setName("MySQL to HDFS job");

         job.setCreationUser("admln");

         //设置源链接任务配置信息

         MFromConfig fromJobConfig = job.getFromJobConfig();

         fromJobConfig.getStringInput("fromJobConfig.schemaName").setValue("sqoop");

         fromJobConfig.getStringInput("fromJobConfig.tableName").setValue("sqoop");

         fromJobConfig.getStringInput("fromJobConfig.partitionColumn").setValue("id");

         MToConfig toJobConfig = job.getToJobConfig();

         toJobConfig.getStringInput("toJobConfig.outputDirectory").setValue("/usr/tmp");

         MDriverConfig driverConfig = job.getDriverConfig();

         driverConfig.getStringInput("throttlingConfig.numExtractors").setValue("3");

         Status status = client.saveJob(job);

         if(status.canProceed()) {

          System.out.println("JOB创建成功，ID为: "+ job.getPersistenceId());

         } else {

          System.out.println("JOB创建失败。");

         }

         //启动任务

         long jobId = job.getPersistenceId();

         MSubmission submission = client.startJob(jobId);

         System.out.println("JOB提交状态为 : " + submission.getStatus());

         while(submission.getStatus().isRunning() && submission.getProgress() != -1) {

           System.out.println("进度 : " + String.format("%.2f %%", submission.getProgress() * 100));

           //三秒报告一次进度

           try {

             Thread.sleep(3000);

           } catch (InterruptedException e) {

             e.printStackTrace();

           }

         }

         System.out.println("JOB执行结束... ...");

         System.out.println("Hadoop任务ID为 :" + submission.getExternalId());

         Counters counters = submission.getCounters();

         if(counters != null) {

           System.out.println("计数器:");

           for(CounterGroup group : counters) {

             System.out.print("\t");

             System.out.println(group.getName());

             for(Counter counter : group) {

               System.out.print("\t\t");

               System.out.print(counter.getName());

               System.out.print(": ");

               System.out.println(counter.getValue());

             }

           }

         }

         if(submission.getExceptionInfo() != null) {

           System.out.println("JOB执行异常，异常信息为 : " +submission.getExceptionInfo());

         }

         System.out.println("MySQL通过sqoop传输数据到HDFS统计执行完毕");

     }

 }

别问为什么没有MYSQL和HBASE、HIVE互导的代码

20150102

sqoop1.99.4 JAVA API操作的更多相关文章

MongoDB Java API操作很全的整理
MongoDB 是一个基于分布式文件存储的数据库.由 C++ 语言编写,一般生产上建议以共享分片的形式来部署. 但是MongoDB官方也提供了其它语言的客户端操作API.如下图所示: 提供了C.C++ ...
hive-通过Java API操作
通过Java API操作hive,算是测试hive第三种对外接口测试hive 服务启动 package org.admln.hive; import java.sql.SQLException; i ...
hadoop2-HBase的Java API操作
Hbase提供了丰富的Java API,以及线程池操作,下面我用线程池来展示一下使用Java API操作Hbase. 项目结构如下: 我使用的Hbase的版本是 hbase-0.98.9-hadoop ...
使用Java API操作HDFS文件系统
使用Junit封装HFDS import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.*; import org ...
Kafka系列三 java API操作
使用java API操作kafka 1.pom.xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xs ...
Hadoop之HDFS（三）HDFS的JAVA API操作
HDFS的JAVA API操作 HDFS 在生产应用中主要是客户端的开发,其核心步骤是从 HDFS 提供的 api中构造一个 HDFS 的访问客户端对象,然后通过该客户端对象操作(增删改查)HDFS ...
zookeeper的java api操作
zookeeper的java api操作创建会话: Zookeeper(String connectString,int sessionTimeout,Watcher watcher) Zookee ...
java api操作
java api操作导入开发包将hbase安装包中lib下包导入java项目创建表 Configuration conf = HBaseConfiguration.create(); c ...
HDFS 05 - HDFS 常用的 Java API 操作
目录 0 - 配置 Hadoop 环境(Windows系统) 1 - 导入 Maven 依赖 2 - 常用类介绍 3 - 常见 API 操作 3.1 获取文件系统(重要) 3.2 创建目录.写入文件 ...

随机推荐

Git 的核心概念解读
本文不是Git使用教学篇,而是偏向理论方面,旨在更加深刻的理解Git,这样才能更好的使用它,让工具成为我们得力的助手. 版本控制系统 Git 是目前世界上最优秀的分布式版本控制系统.版本控制系统是能够 ...
Django Signals 从实践到源码分析（转）
原文:http://foofish.net/blog/66/django-signals 当某个事件发生的时候,signal(信号)允许senders(发送者)用来通知receivers(接收者),通 ...
ArcObjects10.0MapControl不显示地图内容
添加MapControl控件,右键属性设置MXD 文档之后,运行显示一片空白.拖放另一个控件axLicenseControl之后OK.
GLSL Notes
[GLSL Notes] API of shader: glCreateShader(), glShaderSource(), glCompileShader(), glGetShadrInfoLog ...
c#动态加载dll文件
1.在写一个记录日志到文件中的类库(生成dll文件copy到一个目录中去,然后在主函数的appconfig中去配置. using System; using System.Collections.Ge ...
Compactness问题
今天偶然在论坛讨论在halcon下紧密度的问题,现在总结一下 Calculation: If L is the length of the contour (see contlength) and F ...
OC：基础学习
OC面向对象的编程语言思想类与对象.继承与实例化.属性点语法.内存管理.字符串.可见度. 类是一组具有相同特征和行为的事物的抽象 OC的与C相比所具有的新的特点: 定义新的类.类的实例和方法.方法的 ...
Combox选中项注意事项
一般我们选中某个combox的下拉框会用如下方式: 1.combox.SelectedIndex=下拉框下标,如0是选中第一个.-1是不选中任何项等等: 2.combox.SelectedItem=某 ...
Form动态下拉框
FORM级触发器:WHEN-NEW-FORM-INSTANCE 1.定义: V_LIST_NAME11 VARCHAR2(100) := 'QUERY_FIND.UPDATE_TYPE' ...
3DTouch-ShortcutItem - iOS9 - xcode7
据说苹果某个神秘的团队闭门潜心研发多年的3DTouch终于,应用在iOS9上,且公开了API. 在系统主界面用力按压 APP 图标,如上会出现自定义菜单有两种方法可以实现一.代码(这种方法也是可以动 ...

sqoop1.99.4 JAVA API操作

sqoop1.99.4 JAVA API操作的更多相关文章

随机推荐

热门专题