I am 100% sure that at least one time in the last 12 months you heard someone, a colleague, a business partner or maybe even a friend talking about Data Mesh in an enthusiastic way.
Data mesh has been such a hot word in the data engineering industry for the whole 2023 and if you still don’t have a clear idea about it or if you have not still fully understood its potential, well, this article is the best place to start catching up.
So, what Data Mesh is?
It is nothing else than a set of principles for building a modern data architecture.
Do you feel this definition is a little too abstract? Don’t worry. Data mesh is a really new idea to better organize and exploit data inside an organization, we are still in the early days of this concept, so it is totally fine if you are struggling to imagine it.
The more you will read, the more you will make sense out of it.
The best way to think about Data Mesh is to frame the problem that it solves.
A well implemented Data Mesh lets you scale a data architecture and make you able to scale even if your organization is growing fast or is an ever evolving company. So “To be able to scale” has a double meaning and both of them are extremely important.
From the first point of view, scaling is intended with a technological interpretation, something like: ‘Can I add enough computers and still run fast enough?’ or ‘Am I able to fulfil my need for bigger and bigger storage without too much trouble?’
The second interpretation, instead, relates to the organisational level: as business evolves, changes and grows, and the things people want from data change too, “Is my architecture flexible enough to adapt to all these changes without the need to rethink the entire structure again and again?”
This second interpretation is often forgotten or sometimes even not considered on purpose. Data Mesh paradigm brings back this alternative interpretation to the centrality it deserves and builds the future of data architecture on this concept.
Data Mesh Principles
Now that you understood why Data Mesh is important and which needs it addresses, let’s briefly explain the 4 core principles it relies upon:
- Data ownership by domain: data is owned by a specific domain or business entity, this means that the access to the data is decentralized. There is not going to exist a “data team,” or “analytics team” or whatever, taking the ownership for the data management of the entire company. Data is owned, controlled and accessible by the domain that has generated that data.
- Data as a product: each team that generates or publishes any data must consider it as a product. What does this mean? It means they are are wholly responsible for it and they should spend time thinking about its quality, its cohesiveness and its usefulness for everybody else in the company.
- Data available everywhere: data must be self-served anywhere in the company. So for example, if you are producing a report for a sales forecast in any country, you should be able to find and source all the data you need and ideally it would take you just a small amount of time to do it.
- Data governed whatever it is: data governance is alway a big concern; in the Mesh, this concern is going to be driven down to the place where the data is created and produced. Governance allows you to trust and more quickly navigate data in the mesh and believe that you can use the stuff you find.
So what is the biggest barrier that prevents your company from adopting the Data Mesh paradigm?
Basically, the most difficult obstacle to overcome for your company in order to finally embrace the Data Mesh journey is that the mesh paradigm, to deploy its benefits, requires that every person inside the organization is conscious, trained and ready to adopt this big shift in data management.
Data Mesh is not something that few people can deploy while the rest of the organization continues to work as usual. Data Mesh is a whole paradigm that must be accepted, understood and put in place by every employee in the company.
Without this precondition, it becomes impossible to develop a new and efficient data architecture.
Conclusions
To conclude, we all know data architectures usually lack rigor and they evolve in an “ad hoc” way. Often without as much discipline or structure as we would like.
Applied well, Data Mesh should take a messy architecture and transform it into something more uniform and more manageable, where everyone is responsible for the data they produce and everyone else can access them whenever they want without governance concerns.
