By migrating to https://beam.apache.org/, social networking service LinkedIn unified its streaming and batch source code files, and reduced data processing time by 94%. The lambda architecture was replaced with the Beam API which required only one source code file for batch and streaming.

That LinkedIn recently reduced its data processing time by 94% by unifying its streaming and batch pipelines with Apache Beam makes a big win for the simplification argument.

Apache Beam is an open source, unified model for defining batch and streaming data-parallel processing pipelines.

Batch and stream processing jobs accept different inputs and return different outputs even in the instance of Beam when the source code is the same.

Related Articles