[Storm] 并发度的理解
Tasks & executors relation
Q1. However I'm a bit confused by the concept of "task". Is a task an running instance of the component(spout or bolt) ? An executor having multiple tasks actually is saying the same component is executed for multiple times by the executor, am I correct ?
A1: Yes, and yes
task只是某个component(spout或者bolt)的实例。Executor线程在执行期间会调用该task的nextTuple或execute方法
Q2. Moreover in a general parallelism sense, Storm will spawn a dedicated thread(executor) for a spout or bolt, but what is contributed to the parallelism by an executor(thread) having multiple tasks ?
A2: Running more than one task per executor does not increase the level of parallelism -- an executor always has one thread that it uses for all of its tasks, which means that tasks run serially on an executor.
运行多个task并不会增加并行度,因为一个executor只是一个线程,这意味着它会顺序执行所有的task
- The number of executor threads can be changed after the topology has been started (see
storm rebalance
command). - The number of tasks of a topology is static.
And by definition, there is the invariant of #executors <= #tasks
.
一个topology的task个数是固定的,但是executor数(线程数)是可以动态改变的。默认的,executor数 <= tasks数
So one reason for having 2+ tasks per executor thread is to give you the flexibility to expand/scale up the topology through the storm rebalance
command in the future without taking the topology offline. For instance, imagine you start out with a Storm cluster of 15 machines but already know that next week another 10 boxes will be added. Here you could opt for running the topology at the anticipated parallelism level of 25 machines already on the 15 initial boxes (which is, of course, slower than 25 boxes). Once the additional 10 boxes are integrated you can then storm rebalance
the topology to make full use of all 25 boxes without any downtime.
Another reason to run 2+ tasks per executor is for (primarily functional) testing. For instance, if your dev machine or CI server is only powerful enough to run, say, 2 executors alongside all the other stuff running on the machine, you can still run 30 tasks (here: 15 per executor) to see whether code such as your custom Storm grouping is working as expected.
一个executor运行2+task数的情况通常有:
- 为了给topology运行提供多大的灵活度,在运行中可以扩展并发度
- 为了功能测试
In practice we normally we run 1 task per executor.
PS: Note that Storm will actually spawn a few more threads behind the scenes. For instance, each executor has its own "send thread" that is responsible for handling outgoing tuples. There are also "system-level" background threads for e.g. acking tuples that run alongside "your" threads. IIRC the Storm UI counts those acking threads in addition to "your" threads.
实际上我们通常是 executors数 = task数
Storm如何partition
我们知道 A stream grouping defines how that stream should be partitioned among the bolt's tasks. 简单理解,partition就是定义数据如何分给多个每个task。Storm给出了spout/bolt之间的数据partition算法
- Shuffle grouping: Tuples are randomly distributed across the bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples.
- Fields grouping: The stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the "user-id" field, tuples with the same "user-id" will always go to the same task, but tuples with different "user-id"'s may go to different tasks.
- All grouping: The stream is replicated across all the bolt's tasks. Use this grouping with care.
- Global grouping: The entire stream goes to a single one of the bolt's tasks. Specifically, it goes to the task with the lowest id.
- None grouping: This grouping specifies that you don't care how the stream is grouped. Currently, none groupings are equivalent to shuffle groupings. Eventually though, Storm will push down bolts with none groupings to execute in the same thread as the bolt or spout they subscribe from (when possible).
- Direct grouping: This is a special kind of grouping. A stream grouped this way means that the producer of the tuple decides which task of the consumer will receive this tuple. Direct groupings can only be declared on streams that have been declared as direct streams. Tuples emitted to a direct stream must be emitted using one of the [emitDirect](javadocs/backtype/storm/task/OutputCollector.html#emitDirect(int, int, java.util.List) methods. A bolt can get the task ids of its consumers by either using the provided TopologyContext or by keeping track of the output of the
emit
method inOutputCollector (which returns the task ids that the tuple was sent to). - Local or shuffle grouping: If the target bolt has one or more tasks in the same worker process, tuples will be shuffled to just those in-process tasks. Otherwise, this acts like a normal shuffle grouping.
然而,如何对进入Spout的数据做partition呢?这个问题storm并没有回答。有几种思路
1. 用户自定义partition的方式,比如基于某个字段进行shuffle
2. 干脆不考虑partition,spout的并发设置为1,task个数也为1
3. 还有一种是spout有并发度,多线程抢占式的消费数据,这时候可能需要有一个保持一致性状态的锁。
Reference
http://stackoverflow.com/questions/17257448/what-is-the-task-in-storm-parallelism
http://www.cnblogs.com/yufengof/p/storm-worker-executor-task.html
http://storm.apache.org/releases/0.9.6/Understanding-the-parallelism-of-a-Storm-topology.html
[Storm] 并发度的理解的更多相关文章
- Storm并发度和Grouping方式
Storm并发度和Grouping方式 .note-content {font-family: "Helvetica Neue",Arial,"Hiragino Sans ...
- 关于Storm 中Topology的并发度的理解
来自:https://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html htt ...
- storm并发度理解
1. 核心原理 一个运行中的拓扑是由什么组成的:worker进程,executors和tasks.Storm是按照下面3种主要的部分来区分Storm集群中一个实际运行的拓扑的:Worker进程.Exe ...
- storm基础系列之一----storm并发度概念剖析
前言: 学了几天storm的基础,发现如果有hadoop基础,再理解起概念来,容易的多.不过,涉及到一些独有的东西,如调度,如并发度,还是很麻烦.那么,从这一篇开始,力争清晰的梳理这些知识. 在正式学 ...
- storm源码之理解Storm中Worker、Executor、Task关系 + 并发度详解
本文导读: 1 Worker.Executor.task详解 2 配置拓扑的并发度 3 拓扑示例 4 动态配置拓扑并发度 Worker.Executor.Task详解: Storm在集群上运行一个To ...
- 用实例的方式去理解storm的并发度
什么是storm的并发度 一个topology(拓扑)在storm集群上最总是以executor和task的形式运行在suppervisor管理的worker节点上.而worker进程都是运行在jvm ...
- 理解Storm并发
作者:Jack47 PS:如果喜欢我写的文章,欢迎关注我的微信公众账号程序员杰克,两边的文章会同步,也可以添加我的RSS订阅源. 注:本文主要内容翻译自understanding-the-parall ...
- Storm基本概念以及Topology的并发度
Spouts,流的源头 Spout是Storm里面特有的名词,Stream的源头,通常是从外部数据源读取tuples,并emit到topology Spout可以同时emit多个tupic strea ...
- Storm并发机制详解
本文可作为 <<Storm-分布式实时计算模式>>一书1.4节的读书笔记 在Storm中,一个task就可以理解为在集群中某个节点上运行的一个spout或者bolt实例. 记住 ...
随机推荐
- nginx整理
一.为什么选择Nginx搭建Web服务器 Apache和Nginx是目前使用最火的两种Web服务器,Apache出现比Nginx早.Apache HTTP Server(简称Apache)是世界使用排 ...
- 【C++】继承(虚基类)
类的继承与派生 面向对象技术强调软件的可重用性,这种重用性通过继承机制来实现.而在类的继承过程中,被重用的原有类称为基类,新创建的类称为派生类.派生类定义语法格式如下: class <派生类名& ...
- SCNU 2015ACM新生赛初赛【1006. 3D打印】解题报告
题目链接详见SCNU 2015新生网络赛 1006. 3D打印 .出题思路来自codevs 3288. 积木大赛,属于模拟题. 首先我们把“选择从第L部分到第R部分”理 ...
- ACCELEROMETER
顾名思义,是加速感应器.有2种应用吧:1,电脑保护,例如当笔记本掉落时,可以被自动检测到,此时会自动关闭硬盘操作以保护数据不在强烈冲击时丢失.
- 一个简单移动页面ionic打包成app
先贴JS代码好了,缓动和调整透明度的功能,最后用ionic打包成应用就可以 window.onload=function(){ search(); move(); calc();}function s ...
- [LeetCode] Scramble String 爬行字符串
Given a string s1, we may represent it as a binary tree by partitioning it to two non-empty substrin ...
- Ubuntu安装Hadoop与Spark
更新apt 用 hadoop 用户登录后,我们先更新一下 apt,后续我们使用 apt 安装软件,如果没更新可能有一些软件安装不了.按 ctrl+alt+t 打开终端窗口,执行如下命令: sudo a ...
- UDP通信
package com.slp; import java.io.IOException; import java.net.DatagramPacket; import java.net.Datagra ...
- c# Entity DbArithmeticExpression arguments must have a numeric common type
在Entity Lambda表达式中计算DateTime类型的时候 不能直接用DateTime类型进行相减计算差值 需要使用DbFunctions提供的一些方法. 例如: DbFunctions.Di ...
- Code Conventions for the Java
Oracel官方JAVA编码规范指引 http://www.oracle.com/technetwork/java/codeconvtoc-136057.html