Innovation is often associated with physical end products, such as electric cars, hydrogen fusion, and fintech apps. However, data is also a product, and innovation in how a company accesses and uses its data can be a significant competitive advantage. One of the latest trends in enterprise data management is the data mesh.
QUICK TAKES
An estimated 79 zettabytes of data were produced in 2021—meaning that in every hour that passed that year, the world made more bytes of data than grains of sand on earth. (Statista)
70% of employees will be expected to use data heavily by 2025, up from 40% in 2018. (Tableau / Forrester Consulting)
The data mesh relies on four fundamental pillars to make using this data efficient: business domain ownership, data as a product, self-service infrastructure and federated governance. (Forbes)
A data mesh is a hybrid architecture that combines the benefits of both siloed and centralized data models, while trying to mitigate the disadvantages of each.
In the past, businesses were often composed of silos, where each team generated and used its own data. This approach had the advantage of allowing engineers and other staff to focus on the data that was most relevant to their work, but it also denied others in the company access to data they might find useful and led to duplicative efforts. It also proved cumbersome as the amount of data flowing into the system exploded. In response, companies adopted a centralized data model, where all data was stored in a data warehouse or data lake. However, this model created challenges for analysts and data scientists, who found it difficult to discover and access the right data in an often unorganized and continuously changing warehouse environment.
The storing of all data in one place also presented governance and security challenges. If everyone was allowed access, then how do you protect your most sensitive data? But if only a small IT group could access the data, then decision making was slowed. It also reduced the number of serendipitous discoveries that often lead to the best innovations because no one outside the IT team would see all the data and, thus, fresh insights were limited.
A data mesh addresses these challenges by viewing data as a product in and of itself, and by decentralizing data management and ownership to individual teams or functional business lines. A data mesh aims to allow teams to independently deploy and test new data services and features, without having to wait for approval or coordination from a centralized data team. Unlike previous siloed teams, domains using a data mesh architecture are interrelated in a federated network, which encourages sharing of data and fosters a culture of experimentation and innovation, while discouraging duplicative efforts. Everyone has access to data and is able to see what others are doing, but they aren’t overwhelmed by information that isn’t pertinent to their own team’s mission.
As efficient as it is, however, a data mesh is not suitable for every organization. Although any organization could find benefit in a data mesh system, in practice, it tends to be most useful for organizations that are large enough and sophisticated enough to house multiple data-driven domains. In addition, there should be a high level of data analytics and innovation expertise within each of those domains. Access to data is nearly useless if the team is not able to analyze and put it to use.
Furthermore, one of the most important factors for the successful implementation of a data mesh is company culture. A data mesh can only foster an environment of ownership and innovation if the company culture is set up to foster ownership and innovation. This means that the company must encourage experimentation, flexibility, and have the ability to embrace change. By giving everyone in the company the tools to make data-driven decisions, a data mesh can help a firm stay ahead of the curve, continuously improve processes, and maintain its competitive edge.