What's New In Zeebe: Scaling Zeebe, New Client APIs, Faster Requests, Timestamps, NodeJS Client, and Default Topic is Back!

Written by Daniel Meyer on May 16 2018 in the What's New In Zeebe category.

Welcome to the first-ever edition of “What’s New In Zeebe”, where we share our progress on the journey of building Zeebe, the world’s first high-throughput, resilient, and horizontally-scalable workflow engine.

We might not publish a “What’s New In Zeebe” post every non-release week, but when there are exciting developments to share in between releases, we want to be sure we’re getting the word out.

If you have questions for the Zeebe team, or you’d like to get involved with the project, take a look at the Zeebe Community page. There, you’ll find our user support forum, public Slack channel, the Zeebe Github project, and our issue tracker. We gladly welcome new users and contributors to the Zeebe community.

Scaling Zeebe

Felix has been running tests on AWS, particularly to test the new clustering layer released in 0.9.0. Apparently, Zeebe scales :)

This is a great milestone for Zeebe because prior to introducing the replication factor, each broker would replicate each partition, so adding more brokers would not allow you to scale throughput–it would only make the system more resilient by having more copies of the data.

New Client APIs

Thorben and Philipp pushed the first increment of a new Java API that’s a lot easier to use and makes it clearer to users how to formulate fluent API requests like this:

final ZeebeFuture<WorkflowInstanceEvent> workflowInstanceEvt = zeebe.workflowClient()
    .newCreateInstanceCommand()
    .bpmnProcessId("foo")
    .latestVersion()
    .send();

Note how it always returns a future that you can decide to block on or not.

There is a lot more to the new API, and I am sure that the team will provide us with a more detailed update after the next alpha release.

Sam also adjusted the Golang client to the new protocol, and we can expect full support for it in the next release.

Faster Requests

We have also been working on making the transport layer faster and scale better when communicating with multiple brokers from one client or broker. The results are great:

You can find more details in this Github issue.

To achieve this, we rewrote the buffer management in the transport layer, and now it is also possible to send requests asynchronously in a loop without having to wait for the response.

Felix is particularly happy that he can write his performance tests like this:

for (int i = 0; i < 1_000_000; i++)
{
    zeebe.workflowClient()
      .newCreateInstanceCommand()
      .bpmnProcessId("foo")
      .latestVersion()
      .send(); // async sending of the request
}

Timestamps

Thanks to Nicolas, each record in Zeebe now has a timestamp. Timestamps have millisecond precision and reflect when the record (event or command) was appended to the partition / log.

This work prepares us for upcoming support for Zeebe in Camunda Optimize and another webapp we are going to build but that is still “on the QT and very hush hush”…

NodeJS Client

Sebastian, Sam, and Felix have made great progress on exposing the Golang client over FFI so that it can be used as a shared library in an upcoming NodeJS client.

As far as I hear, there was a major breakthrough and both Request-Response but also Subscription Style interactions work.

Work is being conducted here: https://github.com/zeebe-io/zbc-js

Default Topic Is Back!

It is now possible to define one or multiple topics along with partition count and replication factor in the configuration file:

[[topics]]
name = "development"
partitions = 3
replicationFactor = 1

[[topics]]
name = "production"
partitions = 16
replicationFactor = 3

Also, the broker will ship with a default topic configured out of the box, so no more creating a topic in each demo or unit test that you write. I know a certain guy who will be very happy about that!