building confluentinc kafka-connect-hdfs
When I try to compile I get an error about a missing SNAPSHOT dependency.
The error looks something like this:
Failed to execute goal on project kafka-connect-hdfs: Could not resolve dependencies for project
io.confluent:kafka-connect-hdfs:jar:3.1.-SNAPSHOT: Failure to find
io.confluent:kafka-connect-avro-converter:jar:3.1.-SNAPSHOT in http://packages.confluent.io/maven/ was cached in the local
repository, resolution will not be reattempted until the update interval of confluent has elapsed or updates are forced
The HDFS connector uses a few dependencies which we sometimes use SNAPSHOT versions of during development of a new release in order to build and test against new features. If you want to build a development version, you may need to build and install these dependencies to your local Maven repository in order to build the connector:
| Project | Clone | Build with |
|---|---|---|
| Kafka | https://github.com/apache/kafka/ | ./gradlew installAll |
| Confluent Common library | https://github.com/confluentinc/common | mvn clean install |
| Confluent Rest-Utils | https://github.com/confluentinc/rest-utils | mvn clean install |
| Avro Converter | https://github.com/confluentinc/schema-registry | mvn clean install |
| Common library for Storage Connectors | https://github.com/confluentinc/kafka-connect-storage-common | mvn clean install |
building confluentinc kafka-connect-hdfs的更多相关文章
- Kafka Connect HDFS
概述 Kafka 的数据如何传输到HDFS?如果仔细思考,会发现这个问题并不简单. 不妨先想一下这两个问题? 1)为什么要将Kafka的数据传输到HDFS上? 2)为什么不直接写HDFS而要通过Kaf ...
- 使用kafka connect,将数据批量写到hdfs完整过程
版权声明:本文为博主原创文章,未经博主允许不得转载 本文是基于hadoop 2.7.1,以及kafka 0.11.0.0.kafka-connect是以单节点模式运行,即standalone. 首先, ...
- Kafka到Hdfs的数据Pipeline整理
作者:Syn良子 出处:http://www.cnblogs.com/cssdongl 转载请注明出处 找时间总结整理了下数据从Kafka到Hdfs的一些pipeline,如下 1> Kafka ...
- Streaming data from Oracle using Oracle GoldenGate and Kafka Connect
This is a guest blog from Robin Moffatt. Robin Moffatt is Head of R&D (Europe) at Rittman Mead, ...
- Kafka connect in practice(3): distributed mode mysql binlog ->kafka->hive
In the previous post Kafka connect in practice(1): standalone, I have introduced about the basics of ...
- Kafka Connect Architecture
Kafka Connect's goal of copying data between systems has been tackled by a variety of frameworks, ma ...
- Build an ETL Pipeline With Kafka Connect via JDBC Connectors
This article is an in-depth tutorial for using Kafka to move data from PostgreSQL to Hadoop HDFS via ...
- Kafka+Storm+HDFS整合实践
在基于Hadoop平台的很多应用场景中,我们需要对数据进行离线和实时分析,离线分析可以很容易地借助于Hive来实现统计分析,但是对于实时的需求Hive就不合适了.实时应用场景可以使用Storm,它是一 ...
- Kafka connect快速构建数据ETL通道
摘要: 作者:Syn良子 出处:http://www.cnblogs.com/cssdongl 转载请注明出处 业余时间调研了一下Kafka connect的配置和使用,记录一些自己的理解和心得,欢迎 ...
- [转载] Kafka+Storm+HDFS整合实践
转载自http://www.tuicool.com/articles/NzyqAn 在基于Hadoop平台的很多应用场景中,我们需要对数据进行离线和实时分析,离线分析可以很容易地借助于Hive来实现统 ...
随机推荐
- 容器上使用Docker还是Rocket?为什么不一起用呢?
在2014年接近尾声时,CoreOS在Docker发布了Rocket,声称Docker的流程模型有着“根本性的缺陷”.虽然CoreOS的创始人Alex Polvi尽量缓和语气,但是他坚持自己的基本观点 ...
- mysql官方下载安装教程(centos)
Linux下Mysql 5.6.30 tar包安装 (2016-04-27 22:45:39) 转载▼ 环境:centos 6.4 x64 先下载mysql安装包 打开 http://dev.mysq ...
- springcloud服务提供producer and 服务调用consumer
---------------------------------producer------------------------------------------- 1.pom文件中,作为客户端的 ...
- tslint.json的配置项说明
tslint.json的配置项说明 extends: 内设配置项名称 rules: 规则 { //ts专用 adjacent-overload-signatures : true, // Enfo ...
- Makefile:248: /usr/local/otp_src_18.1/make/x86_64-unknown-linux-gnu/otp_ded.mk: No such file
安装erlang的时候,使用make命令一直报这个错 Makefile:248: /usr/local/otp_src_18.1/make/x86_64-unknown-linux-gnu/otp_d ...
- idea项目打包和在linux的部署
1.将项目打包 2.先clean,后package 3.将打好的包上传到linux 使用cd指令到要上传的文件的目录 4.输入rz -y 5.解压tar.gz格式: tar -zxvf filenam ...
- mysql更新数据,条件为实时查询出来的数据
--将更新条件保存到临时表里 CREATE TABLE tmp3 AS (SELECT username FROM oa_user WHERE username NOT IN (SELECT user ...
- 32 | 为什么还有kill不掉的语句?
在 MySQL 中有两个 kill 命令:一个是 kill query + 线程 id,表示终止这个线程中正在执行的语句:一个是 kill connection + 线程 id,这里 connecti ...
- Postgresql pg_dump 与 pg_restore 使用举例
pg_dump备份 备份本地osdb数据库,全备份,不需要密码 pg_dump osdb > osdb.sql 备份远程osdb数据库 pg_dump -h 192.168.122.1 -Uos ...
- luogu 2934
同 bzoj3694 需要先求出最短路树 #include <iostream> #include <cstdio> #include <algorithm> #i ...