My topology supervision features of streaming frameworks (or lack thereof) post introduced the three streaming frameworks that were bundled in HDP, but now after the merger of Hortonworks and Cloudera we have swapped out Apache Storm with Apache Flink as our top-of-the-line streaming framework.

This post is to show how Flink stacks up with the other frameworks with respect to what services are available to help with lifecyle events, scalability, management, and monitoring.

Apache Flink

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been design to run in all common cluster environments, perform computations at in-memory speed and at any scale. –http://flink.apache.org/flink-architecture.html

Apache Flink’s running components consist of a master, the Job Manager, and as many Task Manager worker processes required for the desired parallelism. These can run in multiple cluster technologies, including Hadoop YARN.

The actual operations are executed within Task Slots.

Updated Feature Analysis

Summary & Recommendations

The original post’s thoughts still stand and this updated grid above basically shows that Flink is a first-class streaming framework and absolutely worth considering in regards to supervision features.

That said, and this is coming from a big Storm fan, I’m really liking Flink. If looking at a holistic question of which streaming framework to pick from the Cloudera stack, I think this small decision tree/table below should give you an initial start on you eventual selection.