apache 是一个流处理框架,官方提供了docker 镜像,同时也提供了基于docker-compose 运行的说明

docker-compose file

version: "2.1"
services:
jobmanager:
image: flink
expose:
- "6123"
ports:
- "8081:8081"
command: jobmanager
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager taskmanager:
image: flink
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager

运行

docker-compose up -d

效果

编写简单job

使用maven

  • 脚手架生成

    按照提示输入group 等信息。。

mvn archetype:generate \
-DarchetypeGroupId=org.apache.flink \
-DarchetypeArtifactId=flink-quickstart-java \
-DarchetypeVersion=1.6.0 - JOB_MANAGER_RPC_ADDRESS=jobmanager
  • 修改代码
默认项目结构
├── pom.xml
└── src
└── main
├── java
│ └── com
│ └── dalong
│ └── app
│ ├── BatchJob.java
│ ├── StreamingJob.java
└── resources
└── log4j.properties
添加一个技术的批处理
添加之后项目结构
├── pom.xml
└── src
└── main
├── java
│ └── com
│ └── dalong
│ └── app
│ ├── BatchJob.java
│ ├── StreamingJob.java
│ └── util
│ └── WordCountData.java
└── resources
└── log4j.properties
代码说明:
BatchJob.java
package com.dalong.app; import org.apache.flink.api.java.ExecutionEnvironment; import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.core.fs.FileSystem.WriteMode;
import com.dalong.app.util.WordCountData;
import org.apache.flink.util.Collector;
public class BatchJob {
public static class Word { // fields
private String word;
private int frequency; // constructors
public Word() {} public Word(String word, int i) {
this.word = word;
this.frequency = i;
} // getters setters
public String getWord() {
return word;
} public void setWord(String word) {
this.word = word;
} public int getFrequency() {
return frequency;
} public void setFrequency(int frequency) {
this.frequency = frequency;
} @Override
public String toString() {
return "Word=" + word + " freq=" + frequency;
}
} public static void main(String[] args) throws Exception {
final ParameterTool params = ParameterTool.fromArgs(args); // set up the execution environment
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); // make parameters available in the web interface
env.getConfig().setGlobalJobParameters(params); // get input data
DataSet<String> text;
if (params.has("input")) {
// read the text file from given input path
text = env.readTextFile(params.get("input"));
} else {
// get default test text data
System.out.println("Executing WordCount example with default input data set.");
System.out.println("Use --input to specify file input.");
text = WordCountData.getDefaultTextLineDataSet(env);
} DataSet<Word> counts =
// split up the lines into Word objects (with frequency = 1)
text.flatMap(new Tokenizer())
// group by the field word and sum up the frequency
.groupBy("word")
.reduce(new ReduceFunction<Word>() {
@Override
public Word reduce(Word value1, Word value2) throws Exception {
return new Word(value1.word, value1.frequency + value2.frequency);
}
}); if (params.has("output")) {
counts.writeAsText(params.get("output"), WriteMode.OVERWRITE);
// execute program
env.execute("WordCount-Pojo Example");
} else {
System.out.println("Printing result to stdout. Use --output to specify output path.");
counts.print();
} } public static final class Tokenizer implements FlatMapFunction<String, Word> { @Override
public void flatMap(String value, Collector<Word> out) {
// normalize and split the line
String[] tokens = value.toLowerCase().split("\\W+"); // emit the pairs
for (String token : tokens) {
if (token.length() > 0) {
out.collect(new Word(token, 1));
}
}
}
} }
util/WordCountData.java
package com.dalong.app.util; import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment; public class WordCountData {
public static final String[] WORDS = new String[] {
"To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer",
"The slings and arrows of outrageous fortune",
"Or to take arms against a sea of troubles,",
"And by opposing end them?--To die,--to sleep,--",
"No more; and by a sleep to say we end",
"The heartache, and the thousand natural shocks",
"That flesh is heir to,--'tis a consummation",
"Devoutly to be wish'd. To die,--to sleep;--",
"To sleep! perchance to dream:--ay, there's the rub;",
"For in that sleep of death what dreams may come,",
"When we have shuffled off this mortal coil,",
"Must give us pause: there's the respect",
"That makes calamity of so long life;",
"For who would bear the whips and scorns of time,",
"The oppressor's wrong, the proud man's contumely,",
"The pangs of despis'd love, the law's delay,",
"The insolence of office, and the spurns",
"That patient merit of the unworthy takes,",
"When he himself might his quietus make",
"With a bare bodkin? who would these fardels bear,",
"To grunt and sweat under a weary life,",
"But that the dread of something after death,--",
"The undiscover'd country, from whose bourn",
"No traveller returns,--puzzles the will,",
"And makes us rather bear those ills we have",
"Than fly to others that we know not of?",
"Thus conscience does make cowards of us all;",
"And thus the native hue of resolution",
"Is sicklied o'er with the pale cast of thought;",
"And enterprises of great pith and moment,",
"With this regard, their currents turn awry,",
"And lose the name of action.--Soft you now!",
"The fair Ophelia!--Nymph, in thy orisons",
"Be all my sins remember'd."
}; public static DataSet<String> getDefaultTextLineDataSet(ExecutionEnvironment env) {
return env.fromElements(WORDS);
}
}
  • 构建
cd  flink-app
mvn clean package
  • 提交job

    生成的代码很简单

  • 运行job
  • 效果

参考资料

https://hub.docker.com/r/library/flink/
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/wordcount/
https://github.com/rongfengliang/flink-docker-compose-demo

 
 
 
 

apache flink docker-compose 运行试用的更多相关文章

  1. dotnet跨平台 - 使用Nginx+Docker Compose运行.NETCore项目

    参考文档: https://docs.docker.com/install/linux/docker-ce/centos/ http://www.dockerinfo.net/document htt ...

  2. .NET遇上Docker - 使用Docker Compose组织Ngnix和.NETCore运行

    本文工具准备: Docker for Windows Visual Studio 2015 与 Visual Studio Tools for Docker 或 Visual Studio 2017 ...

  3. Docker:Docker Compose 详解

    Docker Compose 概述与安装? 前面我们使用 Docker 的时候,定义 Dockerfile 文件,然后使用 docker build.docker run 等命令操作容器.然而微服务架 ...

  4. 使用 pycharm调试docker环境运行的Odoo

    2019日 星期一 安装docker windows系统,参考 docker官方文档 Mac系统,参考 docker官方文档 构建自定义ODOO镜像 标准ODOO镜像可能不包含特别的python模块, ...

  5. apache flink kubernetes 运行试用

    类似docker-compose 运行模式,使用的是deploy 的模式 deploy yaml 文件 deploy-k8s-yaml apiVersion: extensions/v1beta1 k ...

  6. Apache Dolphin Scheduler - Docker Compose 详解

    Apache DolphinScheduler 是一个分布式去中心化,易扩展的可视化 DAG 工作流任务调度系统.简称 DS,包括 Web 及若干服务,它依赖 PostgreSQL 和 Zookeep ...

  7. 官宣 | Apache Flink 1.12.0 正式发布,流批一体真正统一运行!

    官宣 | Apache Flink 1.12.0 正式发布,流批一体真正统一运行! 原创 Apache 博客 [Flink 中文社区](javascript:void(0) 翻译 | 付典 Revie ...

  8. [phvia/dkc] Docker Compose 快速构建(LNMP+Node)运行环境

    快速构建(LNMP+Node)运行环境. dkc 在此作为 docker-compose 的缩写,你可以理解为 alias dkc=docker-compose 准备 安装 docker 选择1) 从 ...

  9. ASP.NET Core 如何在运行Docker容器时指定容器外部端口(docker compose)

    前面我写了一系列关于持续集成的文章,最终构建出来的镜像运行之后,应该会发现每次构建运行之后端口都变了,这对于我们来说是十分不方便的,所以我们可以通过修改docker compose的配置文件来完成我们 ...

随机推荐

  1. 5-11敏捷开发rails的章节: Rspec(使用方法) ,Slim(使用操作简介)

    Rspec: test Slim :可以取代ERB的模版语言.(简单了解了以下,方便写代码,但我觉得不方便读.还是用原生的html) Webpack管理css: 不再使用app/assets/styl ...

  2. Sonya and Ice Cream CodeForces - 1004E 树的直径, 贪心

    题目链接 set维护最小值贪心, 刚开始用树的直径+单调队列没调出来... #include <iostream>#include <cstdio> #include < ...

  3. 让CLOVER默认引导WINDOWS

    解决问题帖子: http://www.insanelymac.com/forum/topic/296000-force-clover-to-always-choose-win-81-efi-as-de ...

  4. C++中的初始化参数列表

    c++中以下几种情况的变量的初始化不可以写在构造函数里,而是要写在初始化参数列表中 1.const常量 class AA { public : const int num; public : AA() ...

  5. 贪心(二)NYOJ14题

    #include <iostream> #include<cmath> #include "algorithm" using namespace std; ...

  6. OAF 小知识

    打开新窗口链接地址 1.link的动态使用 link组件有一个属性叫 Target Frame,设置为_blank就可以新开窗口 link组件默认是打开当前系统中的连接地址,如果要打开外部网页,可以在 ...

  7. CSS3动画和JS动画的比较

    前言 之前有被问到一个问题,css3动画和js动画性能谁更好,为什么.据我的经验,当然觉得css3动画性能更好,至于为什么一时还真答不上来,所以特意查了一下资料总结一波. JS动画 优点: js动画控 ...

  8. 小程序配置单个页面导航栏的属性(微信小程序交流群:604788754)

    配置单个页面导航栏的属性: 就在所要配置页面相对应的json文件中写入以下想要设置的属性: { "navigationBarBackgroundColor": "#fff ...

  9. python *args **kwargs,传入不固定的参数给函数,或者传入很多的内容给函数,常用在构造函数中。

    ''' 例1:展示*args的用法,传入多个参数,不进行预先定义. 本例传入了3个参数.没有预先定义.在函数内自动生成元组() ''' def q1(*args): print('例1') print ...

  10. NetworkManager 命令配置nmcli注意

    1.使用nmcli connection modify对connection进行修改的时候,ipv4.addresses的输入格式为"ip/mask空格gateway",例如172 ...