Error- Overloaded method value createDirectStream in error Spark Streaming打包报错
直接上代码
StreamingExamples.setStreamingLogLevels()
val Array(brokers, topics) = args // Create context with 2 second batch interval
// 创建conf,spark streaming至少要启动两个线程,一个负责接受数据,一个负责处理数据
val conf = new SparkConf().setMaster("local[4]").setAppName("NetworkWordCount") // 创建StreamingContext,每隔2秒产生一个批次
val ssc = new StreamingContext(conf, Seconds(2)); val topicsSet = topics.split(",").toSet // 配置Kafka参数
val kafkaParams = Map[String, String]("metadata.broker.list" -> brokers) // 用直连方式读取Kafka数据,在Kafka中读取偏移量
val messages = KafkaUtils.createDirectStream[String, String](
ssc,
LocationStrategies.PreferConsistent,// 位置策略(如果Kafka和spark程序在同一台机器,会从最优位置读取数据【当前位置】)
ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams))// 订阅策略(可以指定用正则的方式读取topic【topic-*】) //====================在下面写业务逻辑============================
val lines = messages.map(_.value())
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x=>(x, 1L)).reduceByKey(_+_)
wordCounts.print()
//====================在上面写业务逻辑============================ ssc.start()
ssc.awaitTermination()
打包报错
Error:(44, 49) overloaded method value createDirectStream with alternatives:
(jssc: org.apache.spark.streaming.api.java.JavaStreamingContext,locationStrategy: org.apache.spark.streaming.kafka010.LocationStrategy,consumerStrategy: org.apache.spark.streaming.kafka010.ConsumerStrategy[String,String],perPartitionConfig: org.apache.spark.streaming.kafka010.PerPartitionConfig)org.apache.spark.streaming.api.java.JavaInputDStream[org.apache.kafka.clients.consumer.ConsumerRecord[String,String]] <and>
(jssc: org.apache.spark.streaming.api.java.JavaStreamingContext,locationStrategy: org.apache.spark.streaming.kafka010.LocationStrategy,consumerStrategy: org.apache.spark.streaming.kafka010.ConsumerStrategy[String,String])org.apache.spark.streaming.api.java.JavaInputDStream[org.apache.kafka.clients.consumer.ConsumerRecord[String,String]] <and>
(ssc: org.apache.spark.streaming.StreamingContext,locationStrategy: org.apache.spark.streaming.kafka010.LocationStrategy,consumerStrategy: org.apache.spark.streaming.kafka010.ConsumerStrategy[String,String],perPartitionConfig: org.apache.spark.streaming.kafka010.PerPartitionConfig)org.apache.spark.streaming.dstream.InputDStream[org.apache.kafka.clients.consumer.ConsumerRecord[String,String]] <and>
(ssc: org.apache.spark.streaming.StreamingContext,locationStrategy: org.apache.spark.streaming.kafka010.LocationStrategy,consumerStrategy: org.apache.spark.streaming.kafka010.ConsumerStrategy[String,String])org.apache.spark.streaming.dstream.InputDStream[org.apache.kafka.clients.consumer.ConsumerRecord[String,String]]
cannot be applied to (org.apache.spark.streaming.StreamingContext, org.apache.spark.streaming.kafka010.LocationStrategy, org.apache.spark.streaming.kafka010.ConsumerStrategy[Nothing,Nothing])
val messages = KafkaUtils.createDirectStream[String, String](
这是一个很长的信息,说主题需要设置[字符串],而不是设置[字符]。
我能看到解决这个问题的最佳方法是:
val topicsSet = topics.toString.split(",").toSet
但是,如果你真的只有一个主题,那么只需按照上面的Set(topics)将字符串拆分成一组单个字符。
Error- Overloaded method value createDirectStream in error Spark Streaming打包报错的更多相关文章
- Spark程序编译报错error: object apache is not a member of package org
Spark程序编译报错: [INFO] Compiling 2 source files to E:\Develop\IDEAWorkspace\spark\target\classes at 156 ...
- Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty 报错
项目中遇到的获取https的数据接口数据时,Unexpected error: java.security.InvalidAlgorithmParameterException: the trustA ...
- Mac Angular打包报错xcode-select: error: tool 'xcodebuild' requires Xcode
Mac Angular打包报错: Error: xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer ...
- iOS开发——打包报错error: linker command failed with exit code 1
真机运行没问题,打包报错: clang: error: linker command failed with exit code 1 (use -v to see invocation) 原因:在Xc ...
- maven打包报错 Fatal error compiling: tools.jar not found: C:\Program Files\Java\jre1.8.0_151\..\lib\tool
maven 打包报错 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:comp ...
- spark提交任务报错: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
spark提交任务报错: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes ...
- error: Failed to start domain lb error: Failed to activate service 'org.freedesktop.machine1': timed out 报错的解决办法
能正常查看kvm虚拟机列表: virsh list 但在执行virsh start lb启动虚拟机时卡顿了好几秒,然后报以下错误: error: Failed to start domain lber ...
- 解决Error: ENOENT: no such file or directory, scandir 安装node-sass报错
新项目开发需要安装依赖,但是安装完之后通过gulp运行项目,产生了一下的报错: 解决方案是执行一些方法: npm rebuild node-sass 可是有时就是网络问题导致上面命令安装失败,查下失败 ...
- 解决"Uncaught (in promise) Error: Navigation cancelled from "/" to "/login" with a new navigation"报错处理
Uncaught (in promise) Error: Navigation cancelled from "/" to "/login" with a ne ...
随机推荐
- jest操作 Elasticsearch
package com.lgmall.search; import com.lgmall.search.esEntity.Article;import com.lgmall.search.esEnti ...
- B2C自营商城的订单设计方案
B2C自营商城的订单设计方案 2018年06月01日 17:19:00 lkx94 阅读数 1640 去年我们的美妆社区APP,上线了自有商城.之后经过多次版本迭代,商城系统的模块已经基本健全,值 ...
- 修改Win7登陆界面墙纸
修改Win7登陆界面墙纸 修改注册表.reg Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\W ...
- pycharm安装第三方库失败module 'pip' has no attribute 'main'
用的pycharm2017.3,新创建一个项目,在安装appium-python-client时报错module 'pip' has no attribute 'main'.通过强大的度娘,知道是pi ...
- element-ui 里面el-checkbox多选框,实现全选单选
data里面定义了 data:[], actionids:[],//选择的那个actionid num1:0,//没选择的计数 num2:0,//选中的计数 ...
- c#代码规则,C#程序中元素的命名规范
俩种命名方法 1.Pascal 命名法,第一个字母大写其它字母小写Userid 2.Camel命名法,所有单第一方写大写,其它小写,骆峰命名法,userId C#程序中元素的命名规范项目名:公司名.项 ...
- [NOIP2016]蚯蚓 题解
题目描述 本题中,我们将用符号[c]表示对c向下取整,例如:[3.0」= [3.1」= [3.9」=3.蛐蛐国最近蚯蚓成灾了!隔壁跳蚤国的跳蚤也拿蚯蚓们没办法,蛐蛐国王只好去请神刀手来帮他们消灭蚯蚓. ...
- php面试专题---11、开发环境及配置考点
php面试专题---11.开发环境及配置考点 一.总结 一句话总结: 了解php运行原理及常见的配置项 1.版本控制软件? 集中式:CVS和SVN 分布式:Git 2.请简述CGI.FastCGI和P ...
- Oracle-数据表对象
表一般指的是一个关系表,也可以生成对象表以及临时表.齐总,对象表是通过用户定义的数据类型生成的,临时表用于存储专用于某个事务或会话的临时数据 字符类型: 定长:char (1-2000字节) 变长:v ...
- 16/7/14-MySQL-关键字
Mysql 关键字-保留字 转载:http://linux.it.net.cn/e/data/mysql/2014/1202/9460.html 时间:2014-12-02 13:09来源:linux ...