I have been working as a Data Engineer for many years and have helped migrate, transform and modernize data team for many companies but end of the day #DataTeam ends up becoming the blocker and slow down our stakeholders. Read this book: Data Mesh by O’Reilly , to identify how to implement Data Mesh and modernize data architecture.
Data Engineering Team often does not understand the business enough to implement the BI layer or perform analytics and end up being dependent on Analyst and Analyst being dependent on Engineers to bring data. Data Analytics Team wants to perform Analytics quickly instead of waiting on DE to do the engineering which ends up causing dependency and delays.
So, How do we solve this?
Using Data Mesh as a data architecture that promotes decentralized data ownership and aims to align technology and business through a product-centric approach. It is designed to address the challenges of traditional data architecture models, which often result in data silos, lack of data governance, and slow decision-making due to a centralized data ownership structure.
In a Data Mesh architecture, data ownership is decentralized, with each product team having ownership over its own data. This allows for faster decision-making and more efficient data management, as well as improved data quality and easier data access for the teams that need it. The architecture is based on the principles of domain-driven design, event-driven architecture, and microservices, and it encourages collaboration and communication between teams.
Data Mesh Pillars:
Data Ownership: Decentralize the ownership of analytical data to business domains closest to the data.
Data as a Product: Domain-oriented data is shared as a product directly with data users. It would adhere to some guidelines to be shareable and reusable.
Self serve data platform : Data core team empowers domain’s cross functional teams to share data and consume data without causing dependency by creating a self serve data platform.
Federated governance : This model works on federated decision-making and accountability structure, with a team composed of domain representatives, data platform and SME’s.
Data Mesh :
Data mesh is a decentralized approach to share, access, and manage analytical data in complex and large scale environments. It enables domain teams to perform cross-platform data analytics without being dependant on Data engineers. The domain team ingests operational data and builds analytical data models as data products to perform their own analysis. It may also choose to publish data products with data contracts to serve other domains’ data needs.
The domain team agrees with others on global policies, such as interoperability, security, and documentation standards in a federated governance group, so that domain teams know how to discover, understand and use data products available in the data mesh. The self-serve domain-agnostic data platform, provided by the data platform team, enables domain teams to easily build their own data products and do their own analysis effectively. An enabling team guides domain teams on how to model analytical data, use the data platform, and build and maintain interoperable data products.
By Nitin Jain
