1 year ago

Organizations that require both real-time insights and in-depth analytics often encounter the conundrum of using streaming tools for real-time data processing with basic analytics or a data warehouse with advanced analytics but at high latency. Historically, builders and users of data platforms were forced to choose between a basic analytic insight in real time or a comprehensive analytic insight well after an event occurred.

In a modern data architecture, speed layers combine batch and real-time processing methods to handle large and fast-moving data sets. The speed layer fills the gap between traditional data warehouses or lakes and streaming tools.

A “speed layer” is an architectural pattern that combines real-time processing with the contextual and historical data of a data warehouse or lake.