2 years ago

Category: Software, Database, Data, Infrastructure, Architecture, automation

Data mesh organizes its software teams by domain: Each team handles its own pipelines and provides its data via APIs as “products,” moving from a single to multiple points of consumption. The architecture’s technology-agnostic principles also address two hurdles arising in decentralized systems: duplication of efforts to maintain pipelines and infrastructure (through a central self-service platform), and non-interoperable data products (via federated governance policies such as standardizing data formats, metadata representation and global identity).

For the data to be fully understood (be joinable to other data products) and relied upon, much agreement needs to happen on common data elements beyond what data mesh prescribes in the platform and federated governance layers (model descriptions and formats).

We turn now to analysis on multiple data products, which is how the original quest for data integration is formulated on a data mesh.

Authoring, tracking changes and distributing reference data values to data products must be governed: This capability, akin to, and supported by, MDM systems, is a candidate to include in a data mesh platform.