🏭🗒️Microsoft Fabric: Domains [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 29-May-2024

Domains & Entities

[Microsoft Fabric] Domains

• {definition} a way of logically grouping together data in an organization that is relevant to a particular area or field [1]
• associated with workspaces 
• {benefit} allows to group data into business domains [1]
• all the items in the workspace are then associated with the domain, and they receive a domain attribute as part of their metadata [1]
• {benefit} enables a better consumption experience [1]
• simplify discovery and consumption 
• provide a management boundary between tenant and workspace enabling domain admins to have more granular control over multiple workspaces [6]
• some tenant-level settings for managing and governing data can be delegated to the domain level [2]
• allow to achieve federated governance [7]
• by delegating settings to domain admins
• ⇒ allow provide more granular control over business area [7]
• [security] domain roles
• Fabric admins (or higher)
• can create and edit domains
• can specify domain admins and contributors
• can associate workspaces with domains [4]
• can see, edit, and delete all domains in the admin portal [4]
• domain admins
• business owners or experts of a domain
• can update the domain description
• can define contributors
• can associate workspaces with the domain [4]
• can define and update the domain image
• can override tenant settings for any specific settings the tenant admin has delegated to the domain level [4]
• can't delete the domain, change the domain name, or add/delete other domain admins
• can only see and edit the domains they're admins of.
• domain contributors
• ⇐ must be a workspace admin
• can associate their workspaces with a domain or change the current domain association
• don’t have access to the Domains page in the admin portal
• domain users
• can share lakehouse with other domain users without giving access to workspace and other artifacts [4]
• {concept} default domain
• a domain that has been specified as the default domain for specific users and/or security groups [3]
• ⇒ when these users/security groups create/update a new/unassigned workspace, that workspace will automatically be assigned to that domain [3]
• ⇒ generally automatically become domain contributors of the workspaces that are assigned in this manner [3]
• {feature} subdomains
• a way for fine tuning the logical grouping data under a domain [1]
• subdivisions of a domain
• only one level is supported in the hierarchy
• visible as part of the domains filter and as part of the item location path
• no setup available
• [planned] some domain settings will be added to subdomains as well

Previous Post <<||>>  Next Post

Acronyms:
MF - Microsoft Fabric

Resources:
[1] Microsoft Learn (2023) Administer Microsoft Fabric (link)
[2] Microsoft Learn - Fabric (2024) Governance overview and guidance (link)
[3] Microsoft Learn: Fabric (2023) Fabric domains (link)
[4] Establishing Data Mesh architectural pattern with Domains and OneLake on Microsoft Fabric, by Maheswaran Arunachalam (link) 
[5] Microsoft Fabric Updates Blog (2024) Easily implement data mesh architecture with domains in Fabric, by Naama Tsafrir (link)
[6] Microsoft (2024) Microsoft Fabric Domains – Data Mesh [with Naama Tsafrir & Assaf Shemesh]
[7] Microsoft Fabric (2024) Fabric Analyst in a Day [course notes]

🧭🏭Business Intelligence: Microsoft Fabric (Part III: The Metrics Layer [new feature])

Introduction

One of the announcements of this year's Microsoft Fabric Community first conference was the introduction of a metrics layer in Fabric which "allows organizations to create standardized business metrics, that are rooted in measures and are discoverable and intended for reuse" [1]. As it seems, the information content provided at the conference was kept to a minimum given that the feature is still in private preview, though several webcasts start to catch up on the topic (see [2], [4]). Moreover, as part of their show, the Explicit Measures (@PowerBITips) hosts had Carly Newsome as invitee, the manager of the project, who unveiled more details about the project and the feature, details which became the main source for the information below. 

The idea of a metric layer or metric store is not new, data professionals occasionally refer to their structure(s) of metrics as such. The terms gained weight in their modern conception relatively recently in 2021-2022 (see [5], [6], [7], [8], [10]). Within the modern data stack, a metrics layer or metric store is an abstraction layer available between the data store(s) and end users. It allows to centrally define, store, and manage business metrics. Thus, it allows us to standardize and enforce a single source of truth (SSoT), respectively solve several issues existing in the data stacks. As Benn Stancil earlier remarked, the metrics layer is one of the missing pieces from the modern data stack (see [10]).

Microsoft's Solution

Microsoft's business case for metrics layer's implementation is based on three main ideas (1) duplicate measures contribute to poor data quality, (2) complex data models hinder self-service, (3) reduce data silos in Power BI. In Microsoft's conception the metric layer provides several benefits: consistent definitions and descriptions, easy management via management views, searchable and discoverable metrics, respectively assure trust through indicators. 

For this feature's implementation Microsoft introduces a new Fabric Item called a metric set that allows to group several (business) metrics together as part of a mini-model that can be tailored to the needs of a subset of end-users and accessed by them via the standard tools already available. The metric set becomes thus a mini-model. Such mini-models allow to break down and reduce the overall complexity of semantic models, while being easy to evolve and consume. The challenge will become then on how to break down existing and future semantic models into nonoverlapping mini-models, creating in extremis a partition (see the Lego metaphor for data products). The idea of mini-models is not new, [12] advocating the idea of using a Master Model, a technique for creating derivative tabular models based on a single tabular solution.

A (business) metric is a way to elevate the measures from the various semantic models existing in the organization within the mini-model defined by the metric set. A metric can be reused in other fabric artifacts - currently in new reports on the Power BI service, respectively in notebooks by copying the code. Reusing metrics in other measures can mean that one can chain metrics and the changes made will be further propagated downstream. 

The Metrics Layer in Microsoft Fabric (adapted diagram)

Every metric is tied to the original semantic model which allows thus to track how a metric is used across the solutions and, looking forward to Purview, to identify data's lineage. A measure is related to a "table", the source from which the measure came from.

Users' Perspective

The Metrics Layer feature is available in Microsoft Fabric service for Power BI within the Metrics menu element next to Scorecards. One starts by creating a metric set in an existing workspace, an operation which creates the actual artifact, to which the individual metrics are added. To create a metric, a user with build permissions can navigate through the semantic models across different workspaces he/she has access to, pick a measure from one of them and elevate it to a metric, copying in the process its measure's definition and description. In this way the metric will always point back to the measure from the semantic model, while the metrics thus created are considered as a related collection and can be shared around accordingly. 

Once a metric is added to the metric set, one can add in edit mode dimensions to it (e.g. Date, Category, Product Id, etc.). One can then further explore a metric's output and add filters (e.g. concentrate on only one product or category) point from which one can slice-and-dice the data as needed.

There is a panel where one can see where the metric has been used (e.g. in reports, scorecards, and other integrations), when was last time refreshed, respectively how many times was used. Thus, one has the most important information in one place, which is great for developers as well as for the users. Probably, other metadata will be added, such as whether an increase in the metric would be favorable or unfavorable (like in Tableau Pulse, see [13]) or maybe levels of criticality, an unit of measure, or maybe its type - simple metric, performance indicator (PI), result indicator (RI), KPI, KRI etc.

Metrics can be persisted to the OneLake by saving their output to a delta table into the lakehouse. As demonstrated in the presentation(s), with just a copy-paste and a small piece of code one can materialize the data into a lakehouse delta table, from where the data can be reused as needed. Hopefully, the process will be further automated. 

One can consume metrics and metrics sets also in Power BI Desktop, where a new menu element called Metric sets was added under the OneLake data hub, which can be used to connect to a metric set from a Semantic model and select the metrics needed for the project. 

Tapping into the available Power BI solutions is done via an integration feature based on Sempy fabric package, a dataframe for storage and propagation of Power BI metadata which is part of the python-based semantic Link in Fabric [11].

Further Thoughts

When dealing with a new feature, a natural idea comes to mind: what challenges does the feature involve, respectively how can it be misused? Given that the metrics layer can be built within a workspace and that it can tap into the existing measures, this means that one can built on the existing infrastructure. However, this can imply restructuring, refactoring, moving, and testing a lot of code in the process, hopefully with minimal implications for the solutions already available. Whether the process is as simple as imagined is another story. As misusage, in extremis, data professionals might start building everything as metrics, though the danger might come when the data is persisted unnecessarily. 

From a data mesh's perspective, a metric set is associated with a domain, though there will be metrics and data common to multiple domains. Moreover, a mini-model has the potential of becoming a data product. Distributing the logic across multiple workspaces and domains can add further challenges, especially in what concerns the synchronization and implemented of requirements in a way that doesn't lead to bottlenecks. But this is a general challenge for the development team(s). 

The feature will probably suffer further changes until is released in public review (probably by September or the end of the year). I subscribe to other data professionals' opinion that the feature was for long needed and that can have an important impact on the solutions built. 

Previous Post <<||>> Next Post

Resources:
[1] Microsoft Fabric Blog (2024) Announcements from the Microsoft Fabric Community Conference (link)
[2] Power BI Tips (2024) Explicit Measures Ep. 236: Metrics Hub, Hot New Feature with Carly Newsome (link)
[3] Power BI Tips (2024) Introducing Fabric Metrics Layer / Power Metrics Hub [with Carly Newsome] (link)
[4] KratosBI (2024) Fabric Fridays: Metrics Layer Conspiracy Theories #40 (link)
[5] Chris Webb's BI Blog (2022) Is Power BI A Semantic Layer? (link)
[6] The Data Stack Show (2022) TDSS 95: How the Metrics Layer Bridges the Gap Between Data & Business with Nick Handel of Transform (link)
[7] Sundeep Teki (2022) The Metric Layer & how it fits into the Modern Data Stack (link)
[8] Nick Handel (2021) A brief history of the metrics store (link)
[9] Aurimas (2022) The Jungle of Metrics Layers and its Invisible Elephant (link)
[10] Benn Stancil (2021) The missing piece of the modern data stack (link)
[11] Microsoft Learn (2024) Sempy fabric Package (link)
[12] Michael Kovalsky (2019) Master Model: Creating Derivative Tabular Models (link)
[13] Christina Obry (2023) The Power of a Metrics Layer - and How Your Organization Can Benefit From It (link) 
[14] KratosBI (2024) Introducing the Metrics Layer in #MicrosoftFabric with Carly Newsome [link]

🏭🗒️Microsoft Fabric: Data Governance [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 23-May-2024

[Microsoft Fabric] Data Governance

• {definition}set of capabilities that help organizations to manage, protect, monitor, and improve the discoverability of data, so as to meet data governance (and compliance) requirements and regulations [2]
• several built-in governance features are available to manage and control the data within Fabric (MF)  [1]
• {feature} endorsement [aka content endorsement] 
• {definition} formal process performed by admins to endorse MF items
• {benefit} allows admins to designate specific MF items as trusted and approved for use across the organization [1]
• establishes trust in data assets by promoting and certifying specific MF items [1]
• users know which assets they can trust and rely on for accurate information [1]
• endorsed assets are identified with a badge that indicates they have been reviewed and approved [1]
• {scope} applies to all MF items except dashboards [1]
• {benefit} helps admin manage the overall growth of items across your environment [1]
• {feature} promoting [aka content promoting] 
• {definition} formal process performed by contributors or admins to promote content
• promoted content appears with a Promoted badge in the MF portal [1]
• workspace members with the contributor or admin role can promote content within a workspace [1]
• MF admin can promote content across the organization [1]
• {feature} certification [aka content certification]
• {definition} formal process that involves a review of the content by a designated reviewer and managed by the admin [1]
• can be customized to meet organization’s needs [1]
• users can request item certification from an admin [1]
• via Request certification from the More menu [1]
• the certified content appears with a Certified badge in the Fabric portal [1]
• {benefit} allows organizations to label items considered to be quality items [1]
• an organization can certify items to identify them an as authoritative sources for critical information [1]
• ⇐ all Fabric items except Power BI dashboards can be certified [1]
• {benefit} allows to specify certifiers who are experts in the domain [1]
• domain level settings
• enable or disable certification of items that belong to the domain [1]
• provides a URL to documentation that is relevant to certification in the domain [1]
• {feature} tenant (aka Microsoft Fabric tenant, MF tenant)
• a single instance of Fabric for an organization that is aligned with a Microsoft Entra ID
• can contain any number of workspaces
• {feature} workspaces
• {definition} a collection of items that brings together different functionality in a single environment designed for collaboration
• can be assigned to teams or departments based on governance requirements and data boundaries [2]
• are associated with domains [3]
• ⇐ {benefit} allows to group data into business domains
• all the items in the workspace are then associated with the domain, and they receive a domain attribute as part of their metadata [3]
• ⇐ {benefit} enables a better consumption experience [1]
• {benefit} enables better discoverability and governance [2]
• {feature} domains [Notes]
• {definition} a way of logically grouping together data in an organization that is relevant to a particular area or field [1]
• allows to group data by business domains
• ⇒{benefit} allows business domains to manage their data according to their specific regulations, restrictions, and needs [3]
• {feature} subdomains
  • {definition} a way for fine tuning the logical grouping data under a domain [1]
  • ⇐ subdivisions of a domain
• {feature} labeling
• default labeling, label inheritance, and programmatic labeling, 
• {benefit} help achieve maximal sensitivity label coverage across MF [2]
• once labeled, data remains protected even when it's exported out of MF via supported export paths [2]
• [Purview Audit] compliance admins can monitor activities on sensitivity labels
• {feature|preview} folders
• {definition} a way of logically grouping MF items
• {feature|preview} tags
• {benefit} allow managing Fabric items for enhanced compliance, discoverability, and reuse
• {feature} scanner API
• a set of admin REST APIs 
• {benefit} allows to scan MF items for sensitive data [1]
• can be used to scan both structured and unstructured data [1]
• {concept} metadata scanning
• facilitates governance of data by enabling cataloging and reporting on all the metadata of organization's Fabric items [1]
• it needs to be set up by Admin before metadata scanning can be run [1]
• {concept} data lineage
• {definition} 
• {benefit} allows to track the flow of data through Fabric [1]
• {benefit} allows to see where data comes from, how it's transformed, and where it goes [1]
• {benefit} helps understand the data available in Fabric, and how it's being used [1]
• {concept} Fabric item (aka MF item)
• {definition} a set of capabilities within an experience
• form the building blocks of the Fabric platform
• {type} data warehouse
• {type} data pipeline
• {type} semantic model
• {type} reports
• {type} dashboards
• {type} notebook
• {type} lakehouse
• {type} metric set

Previous Post <<||>>  Next Post

Acronyms:
API - Application Programming Interface
MF - Microsoft Fabric

Resources:
[1] Microsoft Learn (2023) Administer Microsoft Fabric (link)
[2] Microsoft Learn - Fabric (2024) Governance overview and guidance (link)
[3] Microsoft Learn: Fabric (2023) Fabric domains (link)
[4] Establishing Data Mesh architectural pattern with Domains and OneLake on Microsoft Fabric, by Maheswaran Arunachalam (link) 

🧭Business Intelligence: Monolithic vs. Distributed Architecture (Part III: Architectural Applications)

 

Business Intelligence Series

Now considering the 500 houses and the skyscraper model introduced in thee previous post, which do you think will be built first? A skyscraper takes 2-10 years to build, depending on the city in which is built and the architecture characteristics. A house may take 6-12 months depending on similar factors. But one needs to build 500 houses. For sure the process can be optimized when the houses look the same, though there are many constraints one needs to consider - the number of workers, tools, and the construction material available at a given time, the volume of planning, etc. 

Within a rough estimate, it can take 2-5 years for each architecture to be built considering that on the average the advantages and disadvantages from the various areas can balance each other out. Historical data are in general needed for estimating the actual development time. One can start with a rough estimate and reevaluate the estimates up and down as more information are gathered. This usually happens in Software Engineering as well. 

Monolith vs. Distributed Architecture - 500 families

There are multiple ways in which the work can be assigned to the contractors. When the houses are split between domains, each domain can have its own contractor(s) or the contractors can be specialized by knowledge areas, or a combination of the two. Contractors’ performance should be the same, though in practice no two contractors are the same. Conversely, the chances are higher for some contractors to deliver at the expected quality. It would be useful to have worked before with the contractors and have a partnership that spans years back. There are risks on both sides, even if the risks might favor one architecture over the other, and this depends also on the quality of the contractors, designs, and planning. 

The planning must be good if not perfect to assure smooth development and each day can cost money when contractors are involved. The first planning must be done for the whole project and then split individually for each contractor and/or group of buildings. A back-and-forth check between the various plans is needed. Managing by exception can work, though it can also go terribly wrong. 

Lot of communication must occur between domains to make sure that everything fits together. Especially at the beginning, all the parties must plan together, must make sure that the rules of the games (best practices, policies, procedures, processes, methodologies) are agreed upon. Oversight (governance) needs to happen at a small scale as well on aggregate to makes sure that the rules of the game are followed. 

Now, which of the architectures do you think will fit a data warehouse (DWH)? Probably multiple voices will opt for the skyscraper, at least this is how a DWH looks from the outside. However, when one evaluates the architecture behind it, it can resemble a residential complex in which parts are bound together, but there are parts that can be distributed if needed. For example, in a DWH the HR department has its own area that's isolated from the other areas as it has higher security demands. There can be 2-3 other areas that don't share objects, and they can be distributed as well. The reasons why all infrastructure is on one machine are the costs associated with the licenses, respectively the reporting tools point to only one address. 

In data marts based DWHs, there are multiple buildings within the architecture, and thus the data marts can be distributed across a wider infrastructure, with each domain responsible for its own data mart(s). The data marts are by definition domain-dependent, and this is one of the downsides imputed to this architecture. 

Previous Post <<||>> Next Post

🧭Business Intelligence: Data Products (Part I: A Lego Exercise)

Business Intelligence Series

One can define a data product as the smallest unit of data-driven architecture that can be independently deployed and managed (aka product quantum) [1]. In other terms one can think of a data product like a box (or Lego piece) which takes data as inputs, performs several transformations on the data from which result several output data (or even data visualizations or a hybrid between data, visualizations and other content). 

At high-level each Data Analytics solution can be regarded as a set of inputs, a set of outputs and the transformations that must be performed on the inputs to generate the outputs. The inputs are the data from the operational systems, while the outputs are analytics data that can be anything from data to KPIs and other metrics. A data mart, data warehouse, lakehouse and data mesh can be abstracted in this way, though different scales apply. 

For creating data products within a data mesh, given a set of inputs, outputs and transformations, the challenge is to find horizontal and vertical partitions within these areas to create something that looks like a Lego structure, in which each piece of Lego represents a data product, while its color represents the membership to a business domain. Each such piece is self-contained and contains a set of transformations, respectively intermediary inputs and outputs. Multiple such pieces can be combined in a linear or hierarchical fashion to transform the initial inputs into the final outputs. 

Data Products with a Data Mesh

Finding such a partition is possible though it involves a considerable effort, especially in designing the whole thing - identifying each Lego piece uniquely. When each department is on its own and develops its own Lego pieces, there's no guarantee that the pieces from the various domains will fit together to built something cohesive, performant, secure or well-structured. Is like building a house from modules, the pieces must fit together. That would be the role of governance (federated computational governance) - to align and coordinate the effort. 

Conversely, there are transformations that need to be replicated for obtaining autonomous data products, and the volume of such overlapping can be considerable high. Consider for example the logic available in reports and how often it needs to be replicated. Alternatively, one can create intermediary data products, when that's feasible. 

It's challenging to define the inputs and outputs for a Lego piece. Now imagine in doing the same for a whole set of such pieces depending on each other! This might work for small pieces of data and entities quite stable in their lifetime (e.g. playlists, artists, songs), but with complex information systems the effort can increase by a few factors. Moreover, the complexity of the structure increases as soon the Lego pieces expand beyond their initial design. It's like the real Lego pieces would grow within the available space but still keep the initial structure - strange constructs may result, which even if they work, change the gravity center of the edifice in other directions. There will be thus limits to grow that can easily lead to duplication of functionality to overcome such challenges.

Each new output or change in the initial input for this magic boxes involves a change of all the intermediary Lego pieces from input to output. Just recollect the last experience of defining the inputs and the outputs for an important complex report, how many iterations and how much effort was involved. This might have been an extreme case, though how realistic is the assumption that with data products everything will go smoother? No matter of the effort involved in design, there will be always changes and further iterations involved.

Previous Post <<||>> Next Post

References:
[1] Zhamak Dehghani (2021) Data Mesh: Delivering Data-Driven Value at Scale (book review) 

🧊🗒️Data Warehousing: Data Mesh [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources. 

Last updated: 17-Mar-2024

Data Products with a Data Mesh

Data Mesh

• {definition} "a sociotechnical approach to share, access and manage analytical data in complex and large-scale environments - within or across organizations" [1]
• ⇐ there is no default standard or reference implementation of data mesh and its components [2]
• {definition} a type of decentralized data architecture that organizes data based on different business domains [2]
• ⇐ no centralized data architecture coexists with data mesh, unless in transition [1]
• distributes the modeling of analytical data, the data itself and its ownership [1]
• {characteristic} partitions data around business domains and gives data ownership to the domains [1]
• each domain can model their data according to their context [1]
• there can be multiple models of the same concept in different domains gives the data sharing responsibility to those who are most intimately familiar with the data [1]
• endorses multiple models of the data
• data can be read from one domain, transformed and stored by another domain [1]
• {characteristic} evolutionary execution process
• {characteristic} agnostic of the underlying technology and infrastructure [1]
• {aim} respond gracefully to change [1]
• {aim} sustain agility in the face of growth [1]
• {aim} increase the ratio of value from data to investment [1]
• {principle} data as a product
• {goal} business domains become accountable to share their data as a product to data users
• {goal} introduce a new unit of logical architecture that controls and encapsulates all the structural components needed to share data as a product autonomously [1]
• {goal} adhere to a set of acceptance criteria that assure the usability, quality, understandability, accessibility and interoperability of data products*
• usability characteristics
• {principle} domain-oriented ownership
• {goal} decentralize the ownership of sharing analytical data to business domains that are closest to the data [1]
• {goal} decompose logically the data artefacts based on the business domain they represent and manage their life cycle independently [1]
• {goal} align business, technology and analytical data [1]
• {principle} self-serve data platform
• {goal} provide a self-serve data platform to empower domain-oriented teams to manage and govern the end-to-end life cycle of their data products* [1]
• {goal} streamline the experience of data consumers to discover, access, and use the data products [1]
• {principle} federated computational governance
• {goal} implement a federated decision making and accountability structure that balances the autonomy and agility of domains, while respecting the global conformance, interoperability and security of the mesh* [1]
• {goal} codifying and automated execution of policies at a fine-grained level [1]
• ⇐ the principles represent a generalization and adaptation of practices that address the scale of organization digitization* [1]
• {concept} decentralization of data products
• {requirement} ability to compose data across different modes of access and topologies [1]
• data needs to be agnostic to the syntax of data, underlying storage type, and mode of access to it [1]
• many of the existing composability techniques that assume homogeneous data won’t work
• e.g.  defining primary and foreign key relationships between tables of a single schema [1]
• {requirement} ability to discover and learn what is relatable and decentral [1]
• {requirement} ability to seamlessly link relatable data [1]
• {requirement} ability to relate data temporally [1]
• {concept} data product 
• the smallest unit of data-based architecture that can be independently deployed and managed (aka product quantum) [1]
• provides a set of explicitly defined and data sharing contracts
• provides a truthful portion of the reality for a particular domain (aka single slice of truth) [1]
• constructed in alignment with the source domain [3]
• {characteristic} autonomous
• its life cycle and model are managed independently of other data products [1]
• {characteristic} discoverable
• via a centralized registry or catalog that list the available datasets with some additional information about each dataset, the owners, the location, sample data, etc. [1]
• {characteristic} addressable
• via a permanent and unique address to the data user to programmatically or manually access it [1] 
• {characteristic} understandable
• involves getting to know the semantics of its underlying data and the syntax in which the data is encoded [1]
• describes which entities it encapsulates, the relationships between them, and their adjacent data products [1]
• {characteristic} trustworthy and truthful
• represents the fact of the business correctly [1]
• provides data provenance and data lineage [1]
• {characteristic} natively accessible
• make it possible for various data users to access and read its data in their native mode of access [1]
• meant to be broadcast and shared widely [3]
• {characteristic} interoperable and composable
• follows a set of standards and harmonization rules that allow linking data across domains easily [1]
• {characteristic} valuable on its own
• must have some inherent value for the data users [1]
• {characteristic} secure
• the access control is validated by the data product, right in the flow of data, access, read, or write [1] 
• ⇐ the access control policies can change dynamically
• {characteristic} multimodal 
• there is no definitive 'right way' to create a data product, nor is there a single expected form, format, or mode that it is expected to take [3] 
• shares its logs, traces, and metrics while consuming, transforming, and sharing data [1]
• {concept} data quantum (aka product data quantum, architectural quantum) 
• unit of logical architecture that controls and encapsulates all the structural components needed to share a data product [1]
• {component} data
• {component} metadata
• {component} code
• {component} policies
• {component} dependencies' listing
• {concept} data product observability
• monitor the operational health of the mesh
• debug and perform postmortem analysis
• perform audits
• understand data lineage
• {concept} logs 
• immutable, timestamped, and often structured events that are produced as a result of processing and the execution of a particular task [1]
• used for debugging and root cause analysis
• {concept} traces
• records of causally related distributed events [1]
• {concept} metrics
• objectively quantifiable parameters that continue to communicate build-time and runtime characteristics of data products [1]
• artefacts 
• e.g. data, code, metadata, policies

References:
[1] Zhamak Dehghani (2021) Data Mesh: Delivering Data-Driven Value at Scale (book review)
[2] Zhamak Dehghani (2019) How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (link)
[3] Adam Bellemare (2023) Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures

🧭Business Intelligence: Architecture (Part I: Monolithic vs. Distributed and Zhamak Dehghani's Data Mesh - Debunked)

Business Intelligence Series

In [1] the author categorizes data warehouses (DWHs) and lakes as monolithic architectures, as opposed to data mesh's distributed architecture, which makes me circumspect about term's use. There are two general definitions of what monolithic means: (1) formed of a single large block (2) large, indivisible, and slow to change.

In software architecture one can differentiate between monolithic applications where the whole application is one block of code, multi-tier applications where the logic is split over several components with different functions that may reside on the same machine or are split non-redundantly between multiple machines, respectively distributed, where the application or its components run on multiple machines in parallel.

Distributed multi-tire applications are a natural evolution of the two types of applications, allowing to distribute redundantly components across multiple machines. Much later came the cloud where components are mostly entirely distributed within same or across distinct geo-locations, respectively cloud providers.

Data Warehouse vs. Data Lake vs. Lakehouse [2]

From licensing and maintenance convenience, a DWH resides typically on one powerful machine with many chores, though components can be moved to other machines and even distributed, the ETL functionality being probably the best candidate for this. In what concerns the overall schema there can be two or more data stores with different purposes (operational/transactional data stores, data marts), each of them with their own schema. Each such data store could be moved on its own machine though that's not feasible.

DWHs tend to be large because they need to accommodate a considerable number of tables where data is extracted, transformed, and maybe dumped for the various needs. With the proper design, also DWHs can be partitioned in domains (e.g. define one schema for each domain) and model domain-based perspectives, at least from a data consumer's perspective. The advantage a DWH offers is that one can create general dimensions and fact tables and build on top of them the domain-based perspectives, minimizing thus code's redundancy and reducing the costs.  

With this type of design, the DWH can be changed when needed, however there are several aspects to consider. First, it takes time until the development team can process the request, and this depends on the workload and priorities set. Secondly, implementing the changes should take a fair amount of time no matter of the overall architecture used, given that the transformations that need to be done on the data are largely the same. Therefore, one should not confuse the speed with which a team can start working on a change with the actual implementation of the change. Third, the possibility of reusing existing objects can speed up changes' implementation. 

Data lakes are distributed data repositories in which structured, unstructured and semi-structured data are dumped in raw form in standard file formats from the various sources and further prepared for consumption in other data files via data pipelines, notebooks and similar means. One can use the medallion architecture with a folder structure and adequate permissions for domains and build reports and other data artefacts on top. 

A data lake's value increases when is combined with the capabilities of a DWH (see dedicated SQL server pool) and/or analytics engine (see serverless SQL pool) that allow(s) building an enterprise semantic model on top of the data lake. The result is a data lakehouse that from data consumer's perspective and other aspects mentioned above is not much different than the DWH. The resulting architecture is distributed too. 

Especially in the context of cloud computing, referring to nowadays applications metaphorically (for advocative purposes) as monolithic or distributed is at most a matter of degree and not of distinction. Therefore, the reader should be careful!

Previous Post <<||>> Next Post

References:
[1] Zhamak Dehghani (2021) Data Mesh: Delivering Data-Driven Value at Scale (book review)
[2] Databricks (2022) Data Lakehouse (link)

🔖Book Review: Zhamak Dehghani's Data Mesh: Delivering Data-Driven Value at Scale (2021)

	Zhamak Dehghani's "Data Mesh: Delivering Data-Driven Value at Scale" (2021) is a must read book for the data professional. So, here I am, finally managing to read it and give it some thought, even if it will probably take more time and a few more reads for the ideas to grow. Working in the fields of Business Intelligence and Software Engineering for almost a quarter-century, I think I can understand the historical background and the direction of the ideas presented in the book. There are many good ideas but also formulations that make me circumspect about the applicability of some assumptions and requirements considered.  So, after data marts, warehouses, lakes and lakehouses, the data mesh paradigm seems to be the new shiny thing that will bring organizations beyond the inflection point with tipping potential from where organization's growth will have an exponential effect. At least this seems to be the first impression when reading the first chapters.
The book follows to some degree the advocative tone of promoting that "our shiny thing is much better than previous thing", or "how bad the previous architectures or paradigms were and how good the new ones are" (see [2]). Architectures and paradigms evolve with the available technologies and our perception of what is important for businesses. Old and new have their place in the order of things, and the old will continue to exist, at least until the new proves its feasibility.   The definition of the data mash as "a sociotechnical approach to share, access and manage analytical data in complex and large-scale environments - within or across organizations" [1] is too abstract even if it reflects at high level what the concept is about. Compared to other material I read on the topic, the book succeeds in explaining the related concepts as well the goals (called definitions) and benefits (called motivations) associated with the principles behind the data mesh, making the book approachable also by non-professionals.  Built around four principles "data as a product", "domain-oriented ownership", "self-serve data platform" and "federated governance", the data mesh is the paradigm on which data as products are developed; where the products are "the smallest unit of architecture that can be independently deployed and managed", providing by design the information necessary to be discovered, understood, debugged, and audited. It's possible to create Lego-like data products, data contracts and/or manifests that address product's usability characteristics, though unless the latter are generated automatically, put in the context of ERP and other complex systems, everything becomes quite an endeavor that requires time and adequate testing, increasing the overall timeframe until a data product becomes available.  The data mesh describes data products in terms of microservices that structure architectures in terms of a collection of services that are independently deployable and loosely coupled. Asking from data products to behave in this way is probably too hard a constraint, given the complexity and interdependency of the data models behind business processes and their needs. Does all the effort make sense? Is this the "agility" the data mesh solutions are looking for? Many pioneering organizations are still fighting with the concept of data mesh as it proves to be challenging to implement. At a high level everything makes sense, but the way data products are expected to function makes the concept challenging to implement to the full extent. Moreover, as occasionally implied, the data mesh is about scaling data analytics solutions with the size and complexity of organizations. The effort makes sense when the organizations have a certain size and the departments have a certain autonomy, therefore, it might not apply to small to medium businesses. Previous Post <<||>>  Next Post References: [1] Zhamak Dehghani (2021) "Data Mesh: Delivering Data-Driven Value at Scale" (link) [2] SQL-troubles (2024) Zhamak Dehghani's Data Mesh - Monolithic Warehouses and Lakes (link)

🧭🏭Business Intelligence: Microsoft Fabric (Part II: Domains and the Data Mesh I -The Challenge of Structure Matching)

Business Intelligence Series

The holy grail of building a Data Analytics infrastructure seems to be nowadays the creation of a data mesh, a decentralized data architecture that organizes data by specific business domains. This endeavor proves to be difficult to achieve given the various challenges faced  – data integration, data ownership, data product creation and ownership, enablement of data citizens, respectively enforcing security and governance in a federated manner. 

Microsoft Fabric promises to facilitate the creation of data mashes with the help of domains and subdomain by providing built-in security, administration, and governance features associated with them. A domain is a way of logically grouping together all the data in an organization that is relevant to a particular area or field. A subdomain is a way for fine tuning the logical grouping of the data.

Business domains & their entities

At high level the challenge of building a data mesh is on how to match or aggregate structures. On one side is the high-level structure of the data mesh, while on the other side is the structure of the business data entities. The data entities can be grouped within a taxonomy with multiple levels that expands to the departments. That’s why it seems somehow natural to consider the departments as the top-most domains of the data mesh. The issue is that if the segmentation starts from a high level, iI becomes inflexible in modeling. Moreover, one has only domains and subdomains, and thus a 2-level structure to model the main aspects of the data mesh.

Some organizations allow unrestricted access to the data belonging to a given department, while others breakdown the access to a more granular level. There are also organizations that don’t restrict the access at all, though this may change later. Besides permissions and a way of grouping together the entities, what value brings to set the domains as departments? 

Therefore, I’m not convinced about using an organizations’ departmental structure as domains, especially when such a structure may change and this would imply a full range of further changes. Moreover, such a structure doesn’t reflect the span of processes or how permissions are assigned for the various roles, which are better reflected on how information systems are structured. Most probably the solution needs to accommodate both perspective and be somehow in the middle. 

Take for example the internal structure of the modules from Dynamics 365 (D365). The Finance area is broken down in Accounts Payable, Accounts Receivables, Fixed Assets, General Ledger, etc. In some organizations the departments reflect this delimitation to some degree, while in others are just associated with finance-related roles. Moreover, the permissions are more granular and, reflecting the data entities the users work with. 

Conversely, SCM extends into Finance as Purchase orders, Sales orders and other business documents are the starting or intermediary points of processes that span modules. Similarly, there are processes that start in CRM or other systems. The span of processes seem to be more appropriate for structuring the data mesh, though the system overlapping with the roles involved in the processes and the free definition of process boundaries can overcomplicate the whole design.

It makes sense to define the domains at a level that resembles the structure of the modules available in D365, while the macro data-entities represent the subdomain. The subdomain would represent then master as well as transactional data entities from the perspective of the domains, with there will be entities that need to be shared between multiple domains. Such a structure has less chances to change over time, allowing more flexibility and smaller areas of focus and thus easier to design, develop, test, deploy and maintain.

Previous Post <<||>> Next Post

🔭Data Science: Domains (Just the Quotes)

"Methods perform well if the conditions of their derivation are met, with no method capable of covering all possible conditions. More strongly put, each good method has a domain on which it may be best, and different methods have different domains so the task is to characterize those domains and then figure out which domain a given problem’s solution is likely to be in. This of course, is extraordinarily difficult in its own right." (Bertrand Clarke et al, "Principles and Theory for Data Mining and Machine Learning", 2009)

"Much of machine learning is concerned with devising different models, and different algorithms to fit them. We can use methods such as cross validation to empirically choose the best method for our particular problem. However, there is no universally best model - this is sometimes called the no free lunch theorem. The reason for this is that a set of assumptions that works well in one domain may work poorly in another." (Kevin P Murphy, "Machine Learning: A Probabilistic Perspective", 2012)

"An attempt to use the wrong model for a given data set is likely to provide poor results. Therefore, the core principle of discovering outliers is based on assumptions about the structure of the normal patterns in a given data set. Clearly, the choice of the 'normal' model depends highly upon the analyst’s understanding of the natural data patterns in that particular domain." (Charu C Aggarwal, "Outlier Analysis", 2013)

"Bayesian networks provide a more flexible representation for encoding the conditional independence assumptions between the features in a domain. Ideally, the topology of a network should reflect the causal relationships between the entities in a domain. Properly constructed Bayesian networks are relatively powerful models that can capture the interactions between descriptive features in determining a prediction." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, worked examples, and case studies", 2015)

"Bayesian networks use a graph-based representation to encode the structural relationships - such as direct influence and conditional independence - between subsets of features in a domain. Consequently, a Bayesian network representation is generally more compact than a full joint distribution (because it can encode conditional independence relationships), yet it is not forced to assert a global conditional independence between all descriptive features. As such, Bayesian network models are an intermediary between full joint distributions and naive Bayes models and offer a useful compromise between model compactness and predictive accuracy." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, worked examples, and case studies", 2015)

"Big data is based on the feedback economy where the Internet of Things places sensors on more and more equipment. More and more data is being generated as medical records are digitized, more stores have loyalty cards to track consumer purchases, and people are wearing health-tracking devices. Generally, big data is more about looking at behavior, rather than monitoring transactions, which is the domain of traditional relational databases. As the cost of storage is dropping, companies track more and more data to look for patterns and build predictive models." (Neil Dunlop, "Big Data", 2015)

"The main advantage of decision tree models is that they are interpretable. It is relatively easy to understand the sequences of tests a decision tree carried out in order to make a prediction. This interpretability is very important in some domains. [...] Decision tree models can be used for datasets that contain both categorical and continuous descriptive features. A real advantage of the decision tree approach is that it has the ability to model the interactions between descriptive features. This arises from the fact that the tests carried out at each node in the tree are performed in the context of the results of the tests on the other descriptive features that were tested at the preceding nodes on the path from the root. Consequently, if there is an interaction effect between two or more descriptive features, a decision tree can model this."  (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies", 2015)

"Feature extraction is also the most creative part of data science and the one most closely tied to domain expertise. Typically, a really good feature will correspond to some real‐world phenomenon. Data scientists should work closely with domain experts and understand what these phenomena mean and how to distill them into numbers." (Field Cady, "The Data Science Handbook", 2017)

"Data analysis and data mining are concerned with unsupervised pattern finding and structure determination in data sets. The data sets themselves are explicitly linked as a form of representation to an observational or otherwise empirical domain of interest. 'Structure' has long been understood as symmetry which can take many forms with respect to any transformation, including point, translational, rotational, and many others. Symmetries directly point to invariants, which pinpoint intrinsic properties of the data and of the background empirical domain of interest. As our data models change, so too do our perspectives on analysing data." (Fionn Murtagh, "Data Science Foundations: Geometry and Topology of Complex Hierarchic Systems and Big Data Analytics", 2018)

"Data scientists should have some domain expertise. Most data science projects begin with a real-world, domain-specific problem and the need to design a data-driven solution to this problem. As a result, it is important for a data scientist to have enough domain expertise that they understand the problem, why it is important, an dhow a data science solution to the problem might fit into an organization’s processes. This domain expertise guides the data scientist as she works toward identifying an optimized solution." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"Using data science, we can uncover the important patterns in a data set, and these patterns can reveal the important attributes in the domain. The reason why data science is used in so many domains is that it doesn’t matter what the problem domain is: if the right data are available and the problem can be clearly defined, then data science can help."  (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"The ability to go beyond human domain knowledge is usually achieved by inductive learning methods that are unfettered from the imperfections in the domain knowledge of deductive methods." (Charu C Aggarwal, "Artificial Intelligence: A Textbook", 2021)

⛩️Eric Evans - Collected Quotes

"A domain model is not a particular diagram; it is the idea that the diagram is intended to convey. It is not just the knowledge in a domain expert’s head; it is a rigorously organized and selective abstraction of that knowledge." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Always remember that the model is not the diagram. The diagram’s purpose is to help communicate and explain the model. The code can serve as a repository of the details of the design." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"But code as a design document does have its limits. It can overwhelm the reader with detail. Although its behavior is unambiguous, that doesn’t mean it is obvious. And the meaning behind a behavior can be hard to convey. […] A document shouldn’t try to do what the code already does well. The code already supplies the detail. It is an exact specification of program behavior. Other documents need to illuminate meaning, to give insight into large-scale structures, and to focus attention on core elements. Documents can clarify design intent when the programming language does not support a straightforward implementation of a concept. Written documents should complement the code and the talking." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Diagrams are a means of communication and explanation, and they facilitate brainstorming. They serve these ends best if they are minimal. Comprehensive diagrams of the entire object model fail to communicate or explain; they overwhelm the reader with detail and they lack meaning." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Domain-driven design is both a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Domain experts are usually not aware of how complex their mental processes are as, in the course of their work, they navigate all these rules, reconcile contradictions, and fill in gaps with common sense. Software can’t do this. It is through knowledge crunching in close collaboration with software experts that the rules are clarified, fleshed out, reconciled, or placed out of scope." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Effective domain modelers are knowledge crunchers. They take a torrent of information and probe for the relevant trickle. They try one organizing idea after another, searching for the simple view that makes sense of the mass. Many models are tried and rejected or transformed. Success comes in an emerging set of abstract concepts that makes sense of all the detail. This distillation is a rigorous expression of the particular knowledge that has been found most relevant." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Extreme Programming recognizes the importance of design decisions, but it strongly resists upfront design. Instead, it puts an admirable effort into communication and improving the project’s ability to change course rapidly. With that ability to react, developers can use the “simplest thing that could work” at any stage of a project and then continuously refactor, making many small design improvements, ultimately arriving at a design that fits the customer’s true needs." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Continuous refactoring is a series of small redesigns; developers without solid design principles will produce a code base that is hard to understand or change—the opposite of agility. And although fear of unanticipated requirements often leads to overengineering, the attempt to avoid overengineering can develop into another fear: a fear of doing any deep design thinking at all." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"If the architecture isolates the domain-related code in a way that allows a cohesive domain design loosely coupled to the rest of the system, then that architecture can probably support domain-driven DESIGN." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"If the design, or some central part of it, does not map to the domain model, that model is of little value, and the correctness of the software is suspect. At the same time, complex mappings between models and design functions are difficult to understand and, in practice, impossible to maintain as the design changes. A deadly divide opens between analysis and design so that insight gained in each of those activities does not feed into the other." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"In fact, XP works best for developers with a sharp design sense. The XP process assumes that you can improve a design by refactoring, and that you will do this often and rapidly. But past design choices make refactoring itself either easier or harder. The XP process attempts to increase team communication, but model and design choices clarify or confuse communication." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Knowledge crunching is an exploration, and you can’t know where you will end up." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Many objects are not fundamentally defined by their attributes, but rather by a thread of continuity and identity." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Many things can put a project off course: bureaucracy, unclear objectives, and lack of resources, to name a few. But it is the approach to design that largely determines how complex software can become. When complexity gets out of hand, developers can no longer understand the software well enough to change or extend it easily and safely. On the other hand, a good design can create opportunities to exploit those complex features." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Maps are models, and every model represents some aspect of reality or an idea that is of interest. A model is a simplification. It is an interpretation of reality that abstracts the aspects relevant to solving the problem at hand and ignores extraneous detail." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Models come in many varieties and serve many roles, even those restricted to the context of a software development project. Domain-driven design calls for a model that doesn’t just aid early analysis but is the very foundation of the design […]  Tightly relating the code to an underlying model gives the code meaning and makes the model relevant." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Software design is a constant battle with complexity." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"The effectiveness of an overall design is very sensitive to the quality and consistency of fine-grained design and implementation decisions. With a MODEL-DRIVEN DESIGN, a portion of the code is an expression of the model; changing that code changes the model. Programmers are modelers, whether anyone likes it or not. So it is better to set up the project so that the programmers do good modeling work." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"The technical model that drives the software development process must be strictly pared down to the necessary minimum to fulfill its functions. An explanatory model can include aspects of the domain that provide context that clarifies the more narrowly scoped model. Explanatory models offer the freedom to create much more communicative styles tailored to a particular topic. Visual metaphors used by the domain experts in a field often present clearer explanations, educating developers and harmonizing experts. Explanatory models also present the domain in a way that is simply different, and multiple, diverse explanations help people learn." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"To create software that is valuably involved in users' activities, a development team must bring to bear a body of knowledge related to those activities. The breadth of knowledge required can be daunting. The volume and complexity of information can be overwhelming. Models are tools for grappling with this overload. A model is a selectively simplified and consciously structured form of knowledge. An appropriate model makes sense of information and focuses it on a problem." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Useful models seldom lie on the surface. As we come to understand the domain and the needs of the application, we usually discard superficial model elements that seemed important in the beginning, or we shift their perspective. Subtle abstractions emerge that would not have occurred to us at the outset but that pierce to the heart of the matter." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Well-written code can be very communicative, but the message it communicates is not guaranteed to be accurate. Oh, the reality of the behavior caused by a section of code is inescapable. But a method name can be ambiguous, misleading, or out of date compared to the internals of the method. The assertions in a test are rigorous, but the story told by variable names and the organization of the code is not. Good programming style keeps this connection as direct as possible, but it is still an exercise in self-discipline. It takes fastidiousness to write code that doesn’t just do the right thing but also says the right thing." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"When a design is based on a model that reflects the basic concerns of the users and domain experts, the bones of the design can BE revealed to the user to a greater extent than with other design approaches. Revealing the model gives the user more access to the potential of the software and yields consistent, predictable behavior." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"When we set out to write software, we never know enough. Knowledge on the project is fragmented, scattered among many people and documents, and it’s mixed with other information so that we don’t even know which bits of knowledge we really need. Domains that seem less technically daunting can be deceiving: we don’t realize how much we don’t know. This ignorance leads us to make false assumptions." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

⛩️Scott Millett - Collected Quotes

"A lack of focus on a shared language and knowledge of the problem domain results in a codebase that works but does not reveal the intent of the business. This makes codebases difficult to read and maintain because translations between the analysis model and the code model can be costly and error prone." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"All models are not created equal; the most appropriate design patterns are used based on the complexity needs of each subdomain rather than applying a blanket design to the whole system. Models for subdomains that are not core to the success of the product or that are not as complex need not be based on rich object‐oriented designs, and can instead utilize more procedural or data‐driven architectures." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Any team can write a software product to meet the needs of a set of use cases, but teams that put time and effort into the problem domain they are working on can consistently evolve the product to meet new business use cases." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Areas of low complexity or that are unlikely to be invested in can be built without the need for perfect code quality; working software is good enough. Sometimes feedback and first-to-market are core to the success of a product; in this instance, it can make business sense to get working software up as soon as possible, whatever the architecture." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"By sketching out high‐level logic before you write code, you are in a better position to build the component efficiently and properly because you understand what you are doing. This is where a component diagram provides a lot of benefit. A useful time to start creating component diagrams is during knowledge‐crunching sessions with domain experts. You can produce basic sketches together using just boxes and lines to communicate domain events and processes. When you then sit down to start coding, you already have an idea of what you need to build and terminology from the UL that needs to be modeled in your system." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Domain-Driven Design (DDD) is a process that aligns your code with the reality of your problem domain." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Don’t let design patterns and principles get in the way of getting things done and providing value to the business. Patterns and principles are guides for you to produce supple designs. Badges of honor will not be given out the more you use them in an application. DDD is about providing value, not producing elegant code." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"If key areas of the software are not in synergy with the business domain  then, over time, it is likely that the design will rot and turn into a big ball of mud, resulting in hard‐to‐maintain software." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"It’s important to understand that not all parts of a problem are equal. Some parts of the application are more important than others. Some parts need more attention and investment than others to make the application a success. During knowledge crunching with domain experts, it’s important to reduce the noise of what’s unimportant to enable you to focus on what is important. Model‐Driven Design is hard and should only be reserved to the areas of your systems that are vital to its success. This chapter covers how you can reveal the most important areas of a system and how by using distillation you can focus on those areas. With the knowledge of where to focus you can deeply model what is core, and focus on what will make a difference." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"It may seem sensible to model the entire problem domain using a single model. However, this can be problematic because it needs to cater to all the needs of your domain. This renders the model either too complex or overly generic and devoid of any behavior. If you have large systems, it is far better and more manageable to break down the problem space into smaller, more focused models that can be tied to a specific context. Remember DDD is all about reducing complexity; a single monolithic model would increase complexity. Instead you should break the problem domain down so that you are able to create smaller models in the solution space." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Knowledge crunching is key to bridging any knowledge gaps for the technical team when designing a solution for a problem domain based on a set of requirements. In order for a team to produce a useful model they need to have a deep insight of the problem domain to ensure important concepts are not overlooked or misunderstood. This can only be done through working in collaboration with the people that understand the domain the most; i.e., the business users, stakeholders, and subject matter experts. Without this there is a danger that a technical solution will be produced that is void of any real domain insight and something that cannot be understood by the business or by other developers during software maintenance or subsequent enhancements." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Model‐Driven Design is the process of binding an analysis model to a code implementation model, ensuring that both stay in sync and are useful during evolution. It is the process of validating and proving the model in practice, because it’s pointless to have an elaborate model if you can’t actually implement it. Model‐Driven Design differs from DDD in that it is focused on implementation and any constraints that may require changes to an initial model, whereas DDD focuses on language, collaboration, and domain knowledge. The two complement each other; a Model‐Driven Design approach enables domain knowledge and the shared language to be incorporated into a software model that mirrors the language and mental models of the business experts. This then supports collaboration because business experts and software developers are able to solve problems together as a result of their respective models being valid. Insights gained in either model are shared and knowledge is increased, leading to better problem solving and clearer communication between the business and development team." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Not all of a large software product needs be perfectly designed - in fact trying to do so would be a waste of effort." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"The DDD philosophy is not about following a set of rules or applying coding patterns. It is a process of learning. The journey is far more important than the destination, and the journey is all about exploring your problem domain in collaboration with domain experts rather than how many design patterns you can employ in your solution." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Unfortunately, many developers find it difficult to create effective diagrams. However, when drawing sketches, one basic principle can help you to create highly effective diagrams: keep your diagrams at a consistent level of detail. If you’re talking about high‐level concepts like the way independent software systems communicate to fulfill a business use case, try not to drop down into lower-level concepts like class or module names that will clutter the diagram. Keeping your diagrams at a consistent level of detail will prevent you from showing too much detail or too little detail, meaning everyone can understand what you are trying to convey. It’s often better to create multiple diagrams each at a different level of detail." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"When you have a sound understanding of the problem domain, strategic patterns of DDD can help you implement a technical solution in synergy with the problem space. Patterns enable core parts of your system that are crucial to the success of the product to be protected from the generic areas. Isolating integral components allows them to be modified without having a rippling effect throughout the system." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"You won’t get a useful model on the first attempt; you might not even get one on the second or third attempts. Don’t be afraid of experimentation. Get used to ripping up designs and starting over. Remember that there is not a correct model, only a model that is useful for the current context and the set of problems you are facing." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)