What is Data Mesh?

Data EngineeringData Science

What is Data Mesh?

A data mesh is a new approach that evolves beyond traditional methods of data accessibility. It is a way of thinking about the data based on the decentralized architecture.

It improves the organizational ability to empower data producers and consumers with accessibility and management without expert team intervention. It mainly focuses on distributing the data ownership among the teams who can individually manage the data as a separate product.

It also reduces the bottlenecks without sacrificing the data governance aspects. It also addresses the few limitations of data warehousing and data lake. It also targets to elevate the business outcomes w.r.t data-centric solutions of modern data architecture that improves organizational capability on the decision making. 

Four principles of Data Mesh

four principles of data mesh

Data mesh is driven and governed by four major principles. 

  1. Domain ownership:
    It defines data ownership with the domain teams instead of the central data team.

    It eases the way of starting the transformational journey from a domain-driven approach to data ownership which tackles some of the hard problems around domain boundaries.

  2. Data as Product:
    It describes how a change in perspective will have a deep impact on how we collect, curate, manage, and share the data to answer cross-domain queries with the highest quality.


    It is one of the significant pillars for growing an innovation culture where data is readily available for experimental purposes. 

  3. Self-Serve Data Infrastructure:
    It majorly removes the friction to delivering quality data to producers and enables consumers to discover, understand and use the data at their fingertips with utmost accessibility.


    It also ensures compliance and security aspects while delivering data insights.

  4. Federal Computational Governance:
    It ensures data delivery to all business users by adhering to the highest data safety, compliance, organizational rules, and industry standards.

Benefits of the Data Mesh: 

  1. By making the data more discoverable, it helps to reduce the data silos and operational bottlenecks. 
  2. It also enables entrepreneurs to make faster decision making on the business. 
  3. It moves away the batch processing, and instead, it promotes the cloud data platform and streaming pipelines to collect the data in near real-time. 
  4. It also removes the technical debt as the centralized data infrastructure causes complexity and requires collaboration in maintaining the systems. Using this distributed approach, the domain owners own data pipelines, and teams have better control and reduce the technical strains. 
  5. It also ensures the organization follows the government regulations such as HIPPA etc., and supports compliance through enabling audits. Logging and tracing data also allows the auditors to understand the accessibility of data and the frequency of their usage. 
  6. Localize changes to the domains to move the changes faster than traditional. 

Building a data mesh in your organization:

Organizations are currently in an experimental phase with different technologies as they attempt to build a data mesh for specific use cases.

However, it is rare that the entire organization has adopted the data mesh as the whole and sole solution for maintaining their data.

There is no clear way to define the implementation approach to the data mesh, but here are a few recommendations while implementing the data mesh solution  

  • Initially, analyze your existing data thoroughly for the business use case definition. 
  • Implement the global data governance policies and compliance policies to it. 
  • Build your own self-serve data platform to serve the use case of answering cross-domain questionnaires. 
  • Choose the best suitable technologies for collecting, curating, manage & sharing the data. 
  • Start with part of the cultural shift towards the new approach data management solution. 

Summary:

Data Mesh is an effective way to implement an enterprise data platform. Data mesh is best for large and complex organizations with a set up of independent business units. Not all organizations can implement data mesh.

Author

  • Venkat A.

    Venkata S is a Solution Architect working on Cloud projects. He has profound skills on Azure and Snowflake and has a 12+ years of experience in leading the projects.

    View all posts

Leave a Reply

CALL NOW
× WhatsApp