building confluentinc kafka-connect-hdfs
When I try to compile I get an error about a missing SNAPSHOT dependency.
The error looks something like this:
Failed to execute goal on project kafka-connect-hdfs: Could not resolve dependencies for project
io.confluent:kafka-connect-hdfs:jar:3.1.-SNAPSHOT: Failure to find
io.confluent:kafka-connect-avro-converter:jar:3.1.-SNAPSHOT in http://packages.confluent.io/maven/ was cached in the local
repository, resolution will not be reattempted until the update interval of confluent has elapsed or updates are forced
The HDFS connector uses a few dependencies which we sometimes use SNAPSHOT versions of during development of a new release in order to build and test against new features. If you want to build a development version, you may need to build and install these dependencies to your local Maven repository in order to build the connector:
| Project | Clone | Build with |
|---|---|---|
| Kafka | https://github.com/apache/kafka/ | ./gradlew installAll |
| Confluent Common library | https://github.com/confluentinc/common | mvn clean install |
| Confluent Rest-Utils | https://github.com/confluentinc/rest-utils | mvn clean install |
| Avro Converter | https://github.com/confluentinc/schema-registry | mvn clean install |
| Common library for Storage Connectors | https://github.com/confluentinc/kafka-connect-storage-common | mvn clean install |
building confluentinc kafka-connect-hdfs的更多相关文章
- Kafka Connect HDFS
概述 Kafka 的数据如何传输到HDFS?如果仔细思考,会发现这个问题并不简单. 不妨先想一下这两个问题? 1)为什么要将Kafka的数据传输到HDFS上? 2)为什么不直接写HDFS而要通过Kaf ...
- 使用kafka connect,将数据批量写到hdfs完整过程
版权声明:本文为博主原创文章,未经博主允许不得转载 本文是基于hadoop 2.7.1,以及kafka 0.11.0.0.kafka-connect是以单节点模式运行,即standalone. 首先, ...
- Kafka到Hdfs的数据Pipeline整理
作者:Syn良子 出处:http://www.cnblogs.com/cssdongl 转载请注明出处 找时间总结整理了下数据从Kafka到Hdfs的一些pipeline,如下 1> Kafka ...
- Streaming data from Oracle using Oracle GoldenGate and Kafka Connect
This is a guest blog from Robin Moffatt. Robin Moffatt is Head of R&D (Europe) at Rittman Mead, ...
- Kafka connect in practice(3): distributed mode mysql binlog ->kafka->hive
In the previous post Kafka connect in practice(1): standalone, I have introduced about the basics of ...
- Kafka Connect Architecture
Kafka Connect's goal of copying data between systems has been tackled by a variety of frameworks, ma ...
- Build an ETL Pipeline With Kafka Connect via JDBC Connectors
This article is an in-depth tutorial for using Kafka to move data from PostgreSQL to Hadoop HDFS via ...
- Kafka+Storm+HDFS整合实践
在基于Hadoop平台的很多应用场景中,我们需要对数据进行离线和实时分析,离线分析可以很容易地借助于Hive来实现统计分析,但是对于实时的需求Hive就不合适了.实时应用场景可以使用Storm,它是一 ...
- Kafka connect快速构建数据ETL通道
摘要: 作者:Syn良子 出处:http://www.cnblogs.com/cssdongl 转载请注明出处 业余时间调研了一下Kafka connect的配置和使用,记录一些自己的理解和心得,欢迎 ...
- [转载] Kafka+Storm+HDFS整合实践
转载自http://www.tuicool.com/articles/NzyqAn 在基于Hadoop平台的很多应用场景中,我们需要对数据进行离线和实时分析,离线分析可以很容易地借助于Hive来实现统 ...
随机推荐
- Mybatis3.1-[tp_34-35]-_映射文件_select_resultMap关联查询_collection定义关联集合封装规则_collection分步查询_延迟加载
笔记要点出错分析与总结工程组织 1.定义接口 interface DepartmentMapper package com.dao; import com.bean.Department; publi ...
- Mybatis的一级缓存机制简介
1.接口 public interface MemberMapperCache { public Members selectMembersById(Integer id); } 2.配置文件xml ...
- Spring第二次案例和AOP
Spring加上AOP com.mapper.entity.UserInfo package com.Spring.entity; public class UserInfo { private In ...
- sql server newid() 的用法
sql newid()随机函数 从A表随机取2条记录,用SELECT TOP 10 * FROM ywle order by newid()order by 一般是根据某一字段排序,newid() ...
- tmux-cheatsheet
http://tmuxcheatsheet.com/ http://louiszhai.github.io/2017/09/30/tmux/ 滚屏 开启 Ctrl-b pageup/pagedown ...
- selenium+chromeDriver配合使用(运行js脚本)
在python中调用selenium,访问百度,并运行js脚本爬取内容 python入口程序 from selenium import webdriver import time with open( ...
- Fidller抓包分析post请求
目的:抓包是为了最近做接口测试做准备,以前没有用过这个工具,最近来学下,但是网上很多文章了,所以不一一记录,有一部分参考即可 1.如何抓取想要的web端或者手机端包,已经有很多文章谢了,推荐的参考文章 ...
- 安装cartographer遇到Unrecognized syntax identifier "proto3". This parser only recognizes "proto2"问题
https://stackoverflow.com/questions/38605734/mac-cannot-find-eigen3 https://blog.csdn.net/qq_4214518 ...
- windows下java环境变量标准配置
配置步骤 1.“此电脑”右键,选择“属性”,点击“高级系统设置”,点击“环境变量”. 2.在“系统变量”这一栏,点击“新建”,变量名:JAVA_HOME,变量值:C:\Program Files\Ja ...
- CF786E ALT
题意 有一棵 \(n\) 个点的树和 \(m\) 个人,第 \(i\) 个人从 \(u_i\) 走到 \(v_i\) 现在要发宠物,要求一个人要么他自己发到宠物,要么他走的路径上的都有宠物. 求最小代 ...