Storm 配置图文解析

Storm 配置图文解析

參考阅读：http://www.xiaofateng.com/?

p=959

     ==============================
     |     sample-topology        |
     |  ------------------------  |   Task 1           Task 2          Task 3
     |  |   Worker Process 1   |  |     T1               T2              T3
     |  |           +--------+ |  |    Spout     =>     Bolt      =>    Bolt
     |  | +------+  | +----+ | |  | parallelism      parallelism     parallelism
     |  | |  T3  |  | | T2 | | |  |    hint=2          hint=2          hint=6
     |  | +------+  | +----+ | |  |
     |  |           | +----+ | |  |    combined parallelism = 2 + 2 + 6 = 10
     |  | +------+  | | T2 | | |  |
     |  | |  T3  |  | +----+ | |  |  Each of the 2 worker processes will spawn 10/2=5 threads
     |  | +------+  +--------+ |  |
     |  |                      |  |    T1: parallelism hint = initial executors
     |  | +------+  +--------+ |  |
     |  | |  T3  |  |   T1   | |  |    T2: the T2 bolt was configured to use 2 executors and four tasks.
     |  | +------+  +--------+ |  |        For this reason each executor runs two tasks for this bolt.
     |  ------------------------  |
     |                            |
     |  ------------------------  |    Config conf = new Config();
     |  |   Worker Process 2   |  |
     |  |           +--------+ |  |    // run as 2 workers on 2 supervisors
     |  | +------+  | +----+ | |  |    conf.setNumWorkers(2);
     |  | |  T3  |  | | T2 | | |  |
     |  | +------+  | +----+ | |  |    // T1: 2 executors for spout
     |  |           | +----+ | |  |    topologyBuilder.setSpout("T1-spout", new T1Spout(), 2);
     |  | +------+  | | T2 | | |  |
     |  | |  T3  |  | +----+ | |  |    // T2: 2 executors for bolt with total 4 tasks
     |  | +------+  +--------+ |  |    topologyBuilder.setBolt("T2-bolt", new T2Bolt(), 2)
     |  |                      |  |        .setNumTasks(4).shuffleGrouping("T1-spout");
     |  | +------+  +--------+ |  |    // T3: 6 executors for bolt (default 1 task for 1 executor)
     |  | |  T3  |  |   T1   | |  |    topologyBuilder.setBolt("T3-bolt", new T3Bolt(), 6).shuffleGrouping("T2-bolt");
     |  | +------+  +--------+ |  |
     |  ------------------------  |    StormSubmitter.submitTopology("sample-topology", conf, topologyBuilder.createTopology());
     ==============================

说明：

一个worker进程(process)会产生N个线程(executor)，那么并行度(parallelism)就是全部线程的数目。setNumWorkers

任务(task)是线程运行的工作队列。线程的任务数说明线程的吞吐能力。一个线程的各个任务之间并非并发的。

setNumTasks

线程(executor)是运行任务的上下文环境。

參照上图理解各个配置的含义。