🤔What is Data Mesh: the essence of the concept
Data Mesh is a decentralized flexible approach to the work of various distributed teams and the dissemination of information. Data Mesh was born as a response to the dominant concepts of working with data in data-driven organizations - Data Warehouse and Data Lake. They are united by the idea of centralization. All data flows into a central repository, from where different teams can take it for their own purposes. However, all this needs to be supported by a team of data engineers with a special set of skills. Also, with the growth in the number of sources and the variety of data, it becomes more and more difficult to ensure their business quality, pipelines for transformation become more and more difficult.
Datamesh proposes to solve these and other problems based on four main principles:
1. Domain-oriented ownership - domain teams own data, not a centralized Data team. A domain is a part of an organization that performs a specific business function, for example, it can be product domains (mammography, fluorography, CT scan of the chest) or a domain for working with scribes.
2. Data as a product - data is perceived not as a static dataset, but as a dynamic product with its users, quality metrics, development backlog, which is monitored by a dedicated product-owner.
3. Self-serve data platform. The main function of the data platform in Data Mesh is to eliminate unnecessary cognitive load. This allows developers in domain teams (data product developers and data product consumers) who are not data scientists to conveniently create Data products, build, deploy, test, update, access and use them for their own purposes.
4. Federated computational governance - instead of centralized data management, a special federated body is created, consisting of representatives of domain teams, data platforms and experts (for example, lawyers and doctors), which sets global policies in the field of working with data and discusses the development of the data platform.