The Architect’s Guide to Data and File Formats

2 years ago thenewstack.io

Summary: This is a summary of an article originally published by The New Stack. Read the full original article here →

There is a revolution occurring in the data world. Driven by technological advancements, the current wave of open source data formats is changing the game for the entire ecosystem — from vendor to enterprise.

While this data is easy to store in a data lake, querying it will require scanning a significant amount of data if stored in row-based formats.

It is a standardized format used to represent and manipulate data in a variety of systems, including data storage systems, data processing frameworks and machine learning libraries.

The Arrow format is often used in big data and analytics applications, where it can be used to efficiently transfer data between systems and process data in-memory.

DevOps Articles