2 years ago

Category: Database, Data, Cloud, automation

The modern data stack emerged a decade ago, a direct response to the shortcomings of big data. Companies that undertook big data projects ran headlong into the high cost, rigidity and complexity of managing complex on-premises data stacks.

The modern data stack introduced a set of cloud native data solutions such as Fivetran for data ingestion, Snowflake, Amazon Web Services‘ Redshift or Google BigQuery for data warehousing, and Looker or Mode for data visualization.

I’ve briefly explained how we’ve arrived at this moment for the modern real-time data stack, as well as some of the use cases that make real-time data so powerful.

Here are five characteristics of real-time data that the batch-oriented modern data stack has fundamental problems coping with: The real-time wave extends some of the core concepts of the modern data stack in natural ways: I talk to both customers and vendors in this space every day, and here’s my view of the must-have technologies for a modern real-time data stack: Some companies have parts of the modern real-time data stack today such as a Kafka stream.