How To Size Your Apache Flink® Cluster: A Back-of-the-Envelope Calculation
January 11, 2018- Apache Flink




A favorite session from Flink Forward Berlin 2017 was Robert Metzger’s “Keep It Going: How to Reliably and Efficiently Operate Apache Flink”. One of the topics that Robert touches on is how to roughly size an Apache Flink cluster. Flink Forward attendees mentioned that his cluster sizing guidelines were helpful to them, and so we’ve converted that section of his talk into a blog post. Enjoy!
One of the most frequently-asked questions in the Flink community is how to size a cluster when moving from development to production. The definitive answer to this question, is, of course, “it depends,” but that’s not a helpful answer. This post outlines a series of questions to ask to arrive at some numbers you can use as guidance.
Do the Math and Establish a Baseline
The first step is to think through your application’s operational metrics to arrive at a baseline of required resources.
The key metrics to consider are:
- The number of records per second and the size per record
- The number of distinct keys you have and the state size per key
- The number of state updates and the access patterns of your state backend
Finally, a more pragmatic concern is your service-level agreements (SLAs) around downtime, latency, and max throughput with your customers as these directly influence your capacity planning.
Next, look at what resources you have available based on your budget. For example:
- The network capacity, taking into account any external services that also use the network, such as Kafka, HDFS, etc.
- Your disk bandwidth, if you are relying on a disk-based state backend like RocksDB (and considering other disk use like Kafka or HDFS)
- The number of machines and the CPU and memory they have available
Based on all these factors, you can now build a baseline for normal operation, plus a buffer of resources used for recovery catch-up or to handle load spikes. I recommend you also consider the resources used during checkpointing when establishing the baseline.
Example: Let’s run some numbers
I will now plan a job deployment on a hypothetical cluster to visualize the process of establishing a resource usage baseline. These numbers are rough “back-of-the-envelope” values, and they’re not comprehensive–at the end of the post, I’ll also identify some of the aspects that I ignored while making this calculation.
Example Flink Streaming Job and Hardware
Example Flink Streaming job topology
For this example, I am going to deploy a typical Flink streaming job that reads data from a Kafka topic using Flink’s Kafka consumer. The stream is then transformed using a keyed, aggregating window operator. The window operator performs aggregations on time windows of 5 minutes. As there is always fresh data, I’ll configure the window to be a sliding window with a 1-minute slide.
This means I’ll get the aggregates for the past 5 minutes updated every minute. The streaming job creates an aggregate per userId
. The messages consumed from the Kafka topic have a size (on average) of 2 KB.
The throughput is 1 million messages per second. To understand the state size of the window operator, you need to know the number of distinct keys. In this case, it’s the number of userId
s, which is 500,000,000 unique users. For each user, you are computing four numbers, stored as longs (8 bytes).
Let’s summarize the job’s key metrics:
- Message size: 2KB
- Throughput: 1,000,000 msg/sec
- Distinct keys: 500,000,000 (aggregation in window: 4 longs per key)
- Checkpointing: Once every minute.
Hypothetical Hardware Setup
There are five machines running the job, each running a Flink TaskManager (Flink’s worker nodes). Disks are network-attached (common in cloud setups), and there is a 10 Gigabit Ethernet connection from the main switch to each machine running a TaskManager. The Kafka brokers are running on separate machines.
Each machine has 16 CPU cores. For simplicity, I won’t consider CPU and memory requirements. In the real world, depending on your application logic and the state backend in use, you would need to pay attention to memory. This example uses a RocksDB-based state backend, which is robust and has low memory requirements.
A Single Machine’s Perspective
To understand the resource requirements of the whole job deployment, it’s easiest to focus on the operations in one machine and one TaskManager first. You can then use the numbers derived from one machine to calculate the overall resource requirements.
By default (if all operators have the same parallelism and there are no special scheduling restrictions), all operators of a streaming job are running on each machine.
In this case, the Kafka source (or consumer), window operator, and Kafka sink (or producer) are all running on each of the five machines.
A machine perspective – TaskManager nkeyBy
is a separate operator in the figure above so that calculating the resource requirements is easier. In reality, keyBy
is an API construct and translates into a configuration attribute for the connection between the Kafka source and window operator.
I will now go through each of these operators from top to bottom to understand their network resource requirements.
The Kafka source
To calculate the amount of data received by an individual Kafka source, first, compute the aggregate Kafka input. The sources receive 1,000,000 messages per second that are 2KB each.
Dividing 2GB/s by the number of machines (5) leads to the following result:
Each of the 5 Kafka sources running in the cluster receives data with an average throughput of 400 MB/s.
The Kafka source calculation
The Shuffle / keyBy
Next, you need to ensure that all events with the same key (in this case the userId) end up on the same machine. The data in the Kafka topic you are reading from might be partitioned according to a different partitioning scheme.
The shuffling process sends all data with the same key to one machine, so you are splitting the 400MB/s stream of data coming from Kafka into a userId-partitioned stream:
On average, you have to send 80 MB/s of data to each of the machines. This analysis is from the perspective of a single machine which means that some of the data is already on the designated target machine, so subtract 80MB/s to account for that:
Each machine receives and sends user data at a rate of 320MB/s.
The shuffle calculation
Window Emit and Kafka Sink
The next question to ask is how much data the window operator emits and sends through to the Kafka sink. It’s 67MB/s, and let’s explain how we arrived at this number.
The window operator keeps an aggregate of 4 numbers (represented as longs) for each key. Once every minute, the operator emits the current aggregate values. Each key emits 2 int
s (user_id
, window_ts
) and 4 longs from the aggregation:
Then factor in the keys (500,000,000 divided by the number of machines):
…from each machine.
Then calculate the per-second size:
…emitted by each TaskManager.
This means that each TaskManager emits on average 67 MB/s of user data from the window operators. Since there is a Kafka sink running on each TaskManager (next to the window operator), and there’s no further repartitioning, this is the amount of data emitted from Flink to Kafka.
User data: From Kafka, shuffled to the window operators and back to Kafka
The emission of data from the window operators is expected to be “bursty,” because they are emitting the data once every minute. In practice, the operator will not send data at a constant rate of 67 MB/s, but rather max out the available bandwidth for a few seconds every minute.
This all totals to:
- Data in: 720MB/s (400 + 320) per machine
- Data out: 387MB/s (320 + 67) per machine
State Access and Checkpointing
That’s not everything. So far, I’ve only looked at the user data that Flink is processing. You need to include the overhead from disk access to RocksDB for storing state and checkpointing. To understand the disk access costs, you look at how the window operator accesses state. The Kafka source also keeps some state, but it is negligible compared to the window operator.
To understand the state size of the window operator, look at it from a different angle. Flink is computing five-minute windows with a 1-minute slide. Flink implements sliding windows by maintaining five windows, one for each “slide.” As mentioned earlier, you maintain 40 bytes of state for each window and each key for the aggregations when using a window implementation which is performing an eager aggregation. For every incoming event, you first need to retrieve the current aggregation values from disk (read 40 bytes), update the aggregates, and then write the new value back (write 40 bytes).
Window State
This means:
…of read or write disk access per machine. As said in the beginning, the disks are network attached, so I need to add these numbers to the overall throughput calculations.
The totals are now:
- Data in: 760MB/s (400 MB/s data in + 320 MB/s shuffle + 40 MB/s state)
- Data out: 427MB/s (320 MB/s shuffle + 67 MB/s data out + 40 MB/s state)
The above considerations are for the state access, which happens consistently as new events arrive at the window operator. You also have checkpointing enabled for fault-tolerance. If a machine or anything else fails, you want to restore your window contents and continue processing.
Checkpointing is set to an interval of one checkpoint per minute, and each checkpoint copies the entire state of the job into a network-attached file system.
Let’s quickly see how big the entire state on each machine is:
And, to get the per-second value:
Similar to the window operator, checkpointing has a bursty pattern, and once every minute, it tries to send its data at full speed to external storage. Checkpointing causes additional state access to RocksDB (which in this example is located on network attached disks). Since Flink 1.3, the RocksDB state backend supports incremental checkpointing, reducing the required network transfers on each checkpoint, by conceptually only sending the “diff” since the last checkpoint, but this feature is not used in this example.
This updates the totals to:
- Data in: 760MB/s (400 + 320 + 40)
- Data out: 760MB/s (320 + 67 + 40 + 333)
This means that the overall network traffic is:
The 400 is total of the 80MB state access (read and write) process across the 5 machines, and 2335 is the total of the Kafka in and out processes across the cluster.
Or just over half the available network capacity in the hardware setup above.
Networking requirements
There’s a disclaimer I’d like to add. None of these calculations include protocol overhead such as TCP, Ethernet, and RPC calls from Flink, Kafka, or the file system. This is still good starting point to understand what sort of hardware you will need for a job and to have an indication of performance.
Scale Your Way
Based on my analysis, this example, with a 5-node cluster, and in typical operation, each machine would need to handle 760 MB/s of data, both in and out, from a total capacity of 1250 MB/s. That reserves about 40% of the network capacity for the complexities I glossed over, such as network protocol overheads, heavy load during event replay when recovering from a checkpoint, and uneven load balancing across the cluster caused by data skew.
There’s no one-size-fits-all answer to whether 40% is an appropriate amount of headroom, but this arithmetic should give you a good starting point. Try the calculations above, swapping out the number of machines, the number of keys, or the messages per second to get a selection of values to consider and then balance that with your budget and operational factors. Happy scaling!




Tags: capacity planning, cluster sizing, resource planning, throughput
How To Size Your Apache Flink® Cluster: A Back-of-the-Envelope Calculation的更多相关文章
- Flink监控:Monitoring Apache Flink Applications
This post originally appeared on the Apache Flink blog. It was reproduced here under the Apache Lice ...
- Managing Large State in Apache Flink®: An Intro to Incremental Checkpointing
January 23, 2018- Apache Flink, Flink Features Stefan Richter and Chris Ward Apache Flink was purpos ...
- Apache Flink 开发环境搭建和应用的配置、部署及运行
https://mp.weixin.qq.com/s/noD2Jv6m-somEMtjWTJh3w 本文是根据 Apache Flink 系列直播课程整理而成,由阿里巴巴高级开发工程师沙晟阳分享,主要 ...
- apache flink源码挖坑 (未完待续)
Apache Flink 源码解读(一) By yyz940922原创 项目模块 (除去.git, .github, .idea, docs等): flink-annotations: flink ...
- Peeking into Apache Flink's Engine Room
http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html Join Processin ...
- Apache Flink
Flink 剖析 1.概述 在如今数据爆炸的时代,企业的数据量与日俱增,大数据产品层出不穷.今天给大家分享一款产品—— Apache Flink,目前,已是 Apache 顶级项目之一.那么,接下来, ...
- Apache Flink Quickstart
Apache Flink 是新一代的基于 Kappa 架构的流处理框架,近期底层部署结构基于 FLIP-6 做了大规模的调整,我们来看一下在新的版本(1.6-SNAPSHOT)下怎样从源码快速编译执行 ...
- 新一代大数据处理引擎 Apache Flink
https://www.ibm.com/developerworks/cn/opensource/os-cn-apache-flink/index.html 大数据计算引擎的发展 这几年大数据的飞速发 ...
- 腾讯大数据平台Oceanus: A one-stop platform for real time stream processing powered by Apache Flink
January 25, 2019Use Cases, Apache Flink The Big Data Team at Tencent In recent years, the increa ...
随机推荐
- Solr 05 - Solr Web管理界面的基本使用
目录 1 Dashboard - 仪表盘 2 Logging - 日志信息 3 CoreAdmin - Solr核心 4 Java Properties - Java参数 5 Thread Dump ...
- 【Java基础】【01初识Java】
01.01_计算机基础知识(计算机概述)(了解) A:什么是计算机?计算机在生活中的应用举例 计算机(Computer)全称:电子计算机,俗称电脑.是一种能够按照程序运行,自动.高速处理海量数据的现代 ...
- ASP.NET Core 2.1 : 十三.httpClient.GetAsync 报SSL错误的问题
不知什么时候 ,出现了这样的一个奇怪问题,简单的httpClient.GetAsync("xxxx")居然报错了.(ASP.NET Core 系列目录) 一.问题描述 把原来的程序 ...
- Docker 镜像之存储管理
笔者在<Docker 镜像之进阶篇>中介绍了镜像分层.写时复制以及内容寻址存储(content-addressable storage)等技术特性,为了支持这些特性,docker 设计了一 ...
- Jenkins+Maven+Gitlab+Tomcat 自动化构建打包、部署
一.环境需求 本帖针对的是Linux环境,Windows或其他系统也可借鉴.具体只讲述Jenkins配置以及整个流程的实现. 1.JDK(或JRE)及Java环境变量配置,我用的是JDK1.8.0_1 ...
- eclipse创建的maven项目,pom.xml文件报错解决方法
[错误一:]maven 编译级别过低 [解决办法:] 使用 maven-compiler-plugin 将 maven 编译级别改为 jdk1.6 以上: <!-- java编译插件 --> ...
- sql server去掉某个字段前后空格问题
数据通过页面表单保存到数据库,由于有个选项是一个树形的下拉框,导致保存的这个字段的数据前面有空格,在sql server中可以使用 SELECT LTRIM(RTRIM(BelongPartyCode ...
- O(n*logn)级别的算法之二(快速排序)的三种实现方法详解及其与归并排序的对比
一,单路快排1.测试用例: #ifndef INC_06_QUICK_SORT_DEAL_WITH_NEARLY_ORDERED_ARRAY_SORTTESTHELPER_H #define INC_ ...
- Spring(二)继承jdbcDaoSupport的方式实现(增删改查)
一 首先创建数据库表和相应的字段,并创建约束 二 建立项目,导入jar包(ioc,aop,dao,数据库驱动,连接池)并且将applicationContext.xml文件放在src下 三 开启组件扫 ...
- 20190321-HTML基本结构
目录 1.HTML概念 超文本标记语言 2.HTML版本 HTML HTML5 3.HTML基本结构 基本结构 元素.标签.属性 4.HTML常用标签 内容 1.HTML概念 HTML(HyperTe ...