Business Intelligence Series |
One metaphor that can be used to understand the difference between monolith and distributed architectures, respectively between data warehouses and data mesh-based architectures as per Dehghani’s definition [1] - think that you need to accommodate 500 families (the data products to be built). There are several options: (1) build a skyscraper (developing on vertical) (2) build a complex of high buildings and develop by horizontal and vertical but finding a balance between the two; (3) to split (aka distribute) the second option and create several buildings; (4) build for each family a house, creating a village or a neighborhood.
Monolith vs. Distributed Architecture - 500 families |
If the land
is expensive, for example in big, overpopulated cities, when the infrastructure and the terrain allow it, one can build entirely on vertical, a skyscraper. If the land is cheap one can build a
house for each family. The other architectures can be considered for everything
in between.
A
skyscraper is easier for externals to find (mailmen, couriers, milkmen, and
other service providers) though will need a doorman to interact with them and
probably a few other resources. Everybody will have the same address except the
apartment number. There must be many elevators and the infrastructure must
allow the flux of utilities up and down the floors, which can be challenging to
achieve.
Within a
village every person who needs to deliver or pick up something needs to
traverse parts of the village. There are many services that need to be provided
for both scenarios though the difference it will be the time that's needed to
move in between addresses. In the virtual world this shouldn't matter unless one needs to inspect each house to check and/or retrieve
something. The network of streets and the flux of utilities must scale with the
population from the area.
A
skyscraper will need materials of high quality that resist the various forces
that apply on the building even in the most extreme situations. Not the same
can be said about a house, which in theory needs more materials though a less
solid foundation and the construction specifications are more relaxed.
Moreover, a house needs smaller tools and is easier to build, unless each house has own design.
A
skyscraper can host the families only when the construction is finished, and
the needed certificates were approved. The same can be said about houses but
the effort and time is considerably smaller, though the utilities must be also available,
and they can have their own timeline.
The model is far from perfect, though it allows us to reason how changing the architecture affects various aspects. It doesn't reflect the reality because there's a big difference between the physical and virtual world. E.g., parts of the monolith can be used productively much earlier (though the core functionality might become available later), one doesn't need construction material but needs tool, the infrastructure must be available first, etc. Conversely, functional prototypes must be available beforehand, the needed skillset and a set of assumptions and other requirements must be met, etc.
Previous Post <<||>> Next Post
References:
[1] Zhamak Dehghani (2021) Data Mesh: Delivering Data-Driven Value at Scale (book review)
No comments:
Post a Comment