🎯Bennie Haelen - Collected Quotes

"A data lake is a cost-effective central repository to store structured, semi-structured, or unstructured data at any scale, in the form of files and blobs. The term 'data lake' came from the analogy of a real river or lake, holding the water, or in this case data, with several tributaries that are flowing the water (aka “data”) into the lake in real time." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data lakes have some very strong benefits. A data lake architecture enables the consolidation of an organization’s data assets into one central location. Data lakes are format agnostic and rely on open source formats, such as Parquet and Avro. These formats are well understood by a variety of tools, drivers, and libraries, enabling smooth interoperability. Data lakes are deployed on mature cloud storage subsystems, allowing them to benefit from the scalability, monitoring, ease of deployment, and low storage costs associated with these systems. Automated DevOps tools, such as Terraform, have well-established drivers, enabling automated deployments and maintenance." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data lakes require very large, scalable storage systems, like the ones typically offered in cloud environments. The storage needs to be durable and scalable and should offer interoperability with a variety of third-party tools, libraries, and drivers. Note that data lakes separate the concepts of storage and compute, allowing both to scale independently. Independent scaling of storage and compute allows for on-demand, elastic fine-tuning of resources, allowing our solution architectures to be more flexible. The ingress and egress channels to the storage systems should support high bandwidths, enabling the ingestion or consumption of large batch volumes, or the continuous flow of large volumes of streaming data, such as IoT and streaming media." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data silos often start to develop as the gap between data engineering activities and data science activities begins to grow. Data scientists frequently spend the majority of their time creating separate ETL and data pipelines that clean and transform data and prepare it into features for their models. These silos usually develop because the tools and technologies used for data engineering don’t support the same activities for data scientists." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Delta Lake brings capabilities such as transactional reliability and support for UPSERTs and MERGEs to data lakes while maintaining the dynamic horizontal scalability and separation of storage and compute of data lakes. Delta Lake is one solution for building data lakehouses, an open data architecture combining the best of data warehouses and data lakes." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Like data lakes, the lakehouse architecture leverages low-cost cloud storage systems with the inherent flexibility and horizontal scalability of those systems. The goal of a lakehouse is to use existing high-performance data formats, such as Parquet, while also enabling ACID transactions (and other features). To add these capabilities, lakehouses use an open-table format, which adds features like ACID transactions, record-level operations, indexing, and key metadata to those existing data formats. This enables data assets stored on low-cost storage systems to have the same reliability that used to be exclusive to the domain of an RDBMS. Delta Lake is an example of an open-table format that supports these types of capabilities." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"MLOps best practices include the need to reproduce and validate every stage of the ML workflow. The ability to reproduce a model reduces the risk of errors, and ensures the correctness and robustness of the ML solution. Consistent data is the most difficult challenge faced in reproducibility, and an ML model will only reproduce the exact same result if the exact same data is used. And since data is constantly changing over time, this can introduce significant challenges to ML reproducibility and MLOps." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Modern data platforms leveraging ETL for analytics will always be consumers of data as they ingest data from various data sources. And as organizations continue to collect, process, and analyze data from a growing number of data sources, the ability to swiftly handle schema evolution and data validation is a critical aspect of any data platform. [...] Delta Lake gives you flexibility to evolve a table’s schema through dynamic and explicit schema updates, while also enforcing schema validation." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Querying by timestamp makes it easy to perform time-series analysis because we can compare the data of the same table to itself at two different points in time. And while there are other ETL patterns we can follow to capture historical data and enable time-series analysis (e.g., slowly changing dimensions and change data feeds), time travel provides a quick and easy way to perform ad hoc analysis for tables that may not have these ETL patterns in place." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Schema evolution in Delta Lake refers to the ability to evolve the schema of a Delta table over time, while preserving the existing data in the table. In other words, schema evolution allows us to add, remove, or modify columns in an existing Delta table without losing any data or breaking any downstream jobs that depend on the table. This is important as your data and business needs change over time and you may need to add new columns to your table or modify the existing columns to support new use cases." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"The main goal of the transaction log is to enable multiple readers and writers to operate on a given version of a dataset file simultaneously and to provide additional information, like data skipping indexes to the execution engine for more performant operations. The Delta Lake transaction log always shows the user a consistent view of the data and serves as a single source of truth. It is the central repository that tracks all changes the user makes to a Delta table." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"The lakehouse is centered around the idea of unification and combining the best elements of different technologies in a single place. This means it is also important that the data flow within the lakehouse itself supports this unification of data. In order to support all use cases, this data flow requires merging batch and streaming data into a single data flow to support scenarios across the entire data lifecycle."(Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Traditionally, data lakes have always operated under the principle of schema on read, but have always had challenges enforcing schema on write. This means there is no predefined schema when data is written to storage, and a schema is only adapted when the data is processed. It is imperative for the case of analytics and data platforms that your table formats enforce the schema on write to prevent introducing change-breaking processes, and to maintain proper data quality and integrity." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Unlike data warehouses, data lakes support all data types, including semi-structured and unstructured data, enabling workloads such as media processing. Because of their high throughput ingress channels, they are very well suited for streaming use cases, such as ingesting IoT sensor data, media streaming, or web clickstreams. However, as data lakes become more popular and widely used, organizations started recognizing some challenges with traditional data lakes. While the underlying cloud storage is relatively inexpensive, building and maintaining an effective data lake requires expert skills, resulting in high-end staffing or increased consulting services costs." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Unstructured and semi-structured data are often critical for AI and machine learning use cases, whereas structured and semi-structured data are critical for BI use cases. Because it natively supports all three types of data classifications, you can create a unified system that supports these diverse workloads in a data lake. These workloads can complement each other in a well-designed processing architecture [...]. A data lake helps solve many of the challenges related to data volumes, types, and cost, and while Delta Lake runs on top of a data lake, it is optimized to run best on a cloud data lake." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"When changing clustered columns, liquid clustering does not require the entire table to be rewritten. This clustering evolution is due to the dynamic data layout feature of liquid clustering and offers a significant advantage over partition features mentioned earlier in the chapter. Traditional partitioning is a fixed data layout and does not support changing how a table is partitioned without having to rewrite the entire table. This clustering evolution can be essential as query patterns for a table can often change over time, and this allows you to dynamically adapt to new query patterns without any significant overhead or challenges." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"When you leverage Delta Lake with Structured Streaming, you get both the transactional guarantees of Delta Lake and the powerful programming model of Apache Spark Structured Streaming. With Delta Lake, you can now use Delta tables as both streaming sources and sinks, enabling a continuous processing model that processes your data through the Raw, Bronze, Silver, and Gold data lake layers in a streaming fashion, eliminating the need for batch jobs, resulting in a simplified solution architecture."(Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"With cloud data lakes you typically pay for what you use, so your costs always align with your data volumes. Since there is only a single storage layer, less data movement across different systems, availability settings, and decoupled storage versus compute, you have isolated and minimized costs for just data storage. For greater cost allocation, most cloud data lakes offer buckets, or containers (filesystems, not to be confused with application containers), to store different layers of the data (e.g., raw versus transformed data). These containers allow you to have finer-grained cost allocation for different areas of your organization. Since data sources and volumes are growing exponentially, it is extremely important to allocate and optimize costs without limiting the volume or variety of data that can be stored." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Artificial intelligence is the science of getting machines to accomplish tasks that match or exceed human intelligence. Intelligence is generally defined as the ability to learn, infer, reason, and perform problem-solving." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"As AI systems become more autonomous and integrated into decision-making processes, the question of accountability grows in importance. Businesses and developers must ensure that AI systems operate within predefined ethical boundaries and that there are mechanisms to identify and correct issues when things go wrong. Because AI models can make decisions that directly impact people’s lives (such as credit approvals, hiring decisions, or healthcare recommendations), organizations must maintain accountability for these outcomes. Clear guidelines should be in place to determine who is responsible when an AI system makes a mistake or causes harm." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Beyond intentionally misleading content, GenAI systems can produce inaccurate information unintentionally. LLMs are prone to hallucination, generating plausible but false statements with the same confidence as accurate ones. In enterprise contexts, this poses particular risks: an AI assistant might report incorrect financial figures, fabricate customer details, or misrepresent historical trends. Organizations deploying GenAI must implement validation mechanisms, human oversight, and retrieval-augmented approaches that ground model outputs in verified data sources." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Delta tables in the Lakehouse serve as the backbone for feature engineering, providing a scalable, transactional storage layer that ensures data reliability and performance. With ACID transactions, Delta tables guarantee data consistency and reliability. The time travel functionality allows teams to retrieve historical versions of data, enabling reproducible ML experiments and ensuring traceability. Seamless integration with Spark facilitates efficient and scalable data processing for ML workflows, while schema enforcement ensures that feature tables conform to predefined structures, reducing errors during training and inference." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Deep learning represents a specialized branch of ML that uses artificial neural networks, which are computational structures composed of multiple layers of interconnected nodes. Each layer extracts increasingly abstract and meaningful features from the input data, enabling the network to build hierarchical representations of complex patterns." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Ensuring that a language model reliably retrieves and presents correct information, often referred to as factual recall, is critical for any production-grade application. Whether you’re building an internal helpdesk assistant, a medical Q&A system, or an automated compliance auditor, users expect concise, accurate answers that align with up-to-date source material. Unfortunately, without explicit context, even the most powerful LLM can hallucinate or omit key facts." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Explainability refers to the ability to understand and articulate how AI models make decisions or generate outputs. Many AI models, particularly deep learning models, operate mysteriously, making it difficult to interpret their inner workings. This lack of clarity can raise ethical concerns, especially when AI decisions impact critical areas such as healthcare, finance, or law enforcement. It is essential to develop models that can be explained in understandable terms, ensuring that users and stakeholders can trust that the AI is making decisions based on clear, logical processes. Explainability is key to ensuring fairness, safety, and ethical use of AI." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"GANs consist of two neural networks competing against each other: a generator and a discriminator. The generator attempts to create realistic synthetic data, while the discriminator tries to distinguish between real and generated samples. This adversarial competition pushes the generator to produce increasingly convincing outputs. The training process continues until the discriminator can no longer reliably tell the difference between real and fake data." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026) 

"GenAI encompasses a variety of model architectures, including diffusion models, normalizing flows, and autoregressive models. However, three types of GenAI have become particularly dominant in practical applications. GANs pit two neural networks against each other to produce realistic data. VAEs encode input data into a latent space and decode it to generate new samples. Transformer-based models, like those in the GPT, Llama, and Gemini families, leverage attention mechanisms to generate coherent and contextually relevant sequences." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"GenAI is a type of AI that can generate new content, such as text, images, music, or video. It is a class of AI models that is designed to generate new data or content that resembles existing data. Traditional AI models tend to be primarily focused on classification, regression, or decision making, but GenAI models learn the underlying patterns and structure of the input data and use this knowledge to create new, synthetic instances that are similar to the original data." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"GenAI models are remarkably powerful, but they carry an inherent limitation rooted in how they are built. Every foundation model has a training cutoff, a point in time beyond which it has no awareness of world events, product changes, regulatory updates, or organizational developments. This knowledge gap is not a flaw that can be patched with better prompting. It’s a structural consequence of how models are trained, and it grows wider every day the model remains in production without retraining. Combined with the tendency to hallucinate when asked about topics outside their training distribution, models operating on stale knowledge can confidently deliver responses that are factually incorrect, dangerously outdated, or simply no longer relevant to the user’s context." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"If ethical lapses or AI failures occur, the impact on a business can be significant. Misinformation, biases, or harmful content generated by AI can lead to reputational damage, customer distrust, and potential regulatory scrutiny. The public relations fallout from an AI-driven error or ethical misstep can erode consumer confidence, resulting in lost revenue and lasting harm to brand image. Businesses, therefore, need to proactively address ethical considerations in AI implementation, not only to ensure compliance but also to protect and strengthen their reputation in a highly competitive, and increasingly transparent, marketplace." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"MCP follows a client-server architecture that prioritizes security and composability. A host application (such as an IDE or AI assistant) embeds a single MCP client, which can connect to multiple MCP servers simultaneously. Each server exposes distinct capabilities (database access, file operations, API integrations) without requiring the host to understand server-specific implementation details." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"MCP gives AI applications a common way to discover available capabilities, retrieve context, and invoke actions across local and remote systems. In a Lakehouse environment, that standardization is especially valuable. [...] With MCP, a host can connect to a server that exposes capabilities such as catalog discovery, schema inspection, SQL execution, or workflow orchestration through a consistent protocol layer. That makes the Lakehouse easier to expose to notebooks, IDEs, assistants, and agent frameworks in a way that is composable, inspectable, and easier to govern." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"MCP is an open standard that lets AI applications connect to external tools, data sources, and workflows through a consistent, discoverable interface. In practice, MCP enables LLM-powered hosts, such as IDEs, desktops, and cloud services, to discover capabilities at runtime. These hosts can then invoke tools through a shared schema without requiring bespoke adapters for each integration [..]. MCP addresses the long-standing M×N integration problem: without a common protocol, every model ends up integrating separately with every system. MCP collapses this complexity by standardizing how agents discover, call, and exchange context with external capabilities." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"RAG is a paradigm that combines the strengths of LLMs with the rich, often unstructured data stored in a lakehouse. Rather than asking an LLM to generate responses purely from its internal parameters and training data, where knowledge can be outdated or incomplete, RAG systems first retrieve relevant documents, records, or data slices from your lakehouse and then feed those pieces into the model as context for its generative step. The result is an AI that can speak confidently about the latest reports, proprietary datasets, or domain-specific knowledge you have stored without having to retrain the model each time your data changes." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026) 

"The foundational form of RAG, often called naive RAG, follows a straightforward pattern. A pipeline retrieves supporting context from external sources such as enterprise documents, knowledge bases, or structured datasets and appends that information to the model’s prompt before inference. In the most common implementation, each document is converted into an embedding, a numerical representation of its semantic meaning, using either the same foundation model or a specialized embedding model. When a user submits a query, the system performs a vector similarity search to find documents whose embeddings most closely match the query’s vector representation, and the retrieved content is concatenated with the user query before being passed to the language model." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026) 

"The value of reinforcement learning lies in its ability to solve highly complex problems by mimicking human learning processes. The model undergoes continuous learning and improvement through interaction with its environment, making it highly accurate over time. Reinforcement learning is especially valuable when applied to tasks like robotics, autonomous driving, and game playing, where the agent constantly learns from new experiences. It also integrates well with neural networks, creating the foundation for deep reinforcement learning, which allows models to handle even more intricate tasks." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Transformers dominate language for several reasons: their autoregressive framework provides a straightforward training objective (predict next token), self-attention captures long-range dependencies across thousands of tokens, parallel training processes entire sequences simultaneously (unlike RNNs’ sequential bottleneck), and scaling behavior produces emergent capabilities at billions of parameters. These properties make transformers uniquely suited for the language understanding and generation tasks central to enterprise AI applications [...]" (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Transformer-based models represent the dominant architecture for processing sequential data, particularly text. Unlike GANs and VAEs, which excel at generating visual content, transformers have revolutionized natural language processing (NLP) and are now being applied to code, music, and even protein structures. At their core, transformers use a mechanism called self-attention that allows the model to consider relationships between all elements in a sequence simultaneously. This enables transformers to capture long-range dependencies and contextual nuances that earlier sequential models struggled with." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026) 

"Unlike earlier neural network designs that processed text sequentially, transformers use a mechanism called self-attention that allows the model to consider relationships between all words in a passage simultaneously. This parallel processing enables transformers to capture long-range dependencies and contextual nuances that previous architectures struggled with. Positional encoding provides information about word order, while the attention mechanism allows the model to focus on relevant context when processing each word." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"VAEs take a fundamentally different approach from GANs. Rather than using adversarial competition, VAEs learn to compress data into a structured, interpretable latent space and then reconstruct it. This makes VAEs particularly valuable when understanding and manipulating the underlying structure of data is more important than producing pixel-perfect outputs. VAEs consist of two neural networks: an encoder and a decoder. The encoder compresses input data into a compact latent representation, a numerical vector that captures the essential characteristics of the data. The decoder then reconstructs the original input from this latent representation. Crucially, VAEs impose a probabilistic structure on the latent space, which enables smooth interpolation between data points and meaningful manipulation of generated outputs."  (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

🔢Bernard Marr - Collected Quotes

"A good data strategy is not determined by what data is readily or potentially available –​​​​​​​ it’​​​​​​​s about what your business wants to achieve, and how data can help you get there." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"A picture can paint a thousand words, as the saying goes. In this way, visuals are great for conveying information because they’​​​​​​​re quick and direct, they’​​​​​​​re memorable, and they add interest (being much more likely to hold the reader’​​​​​​​s attention than a full page of text). But unless we know how to decode its message, a picture can also be difficult to read." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"Analytics is the process of collecting, processing and analysing data to generate insights that help you improve the way you do business." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"Data for data’​​​​​​​s sake is meaningless. Therefore, instead of hoarding data, collect only what you really need and what makes business sense." (Bernard Marr, ​​​​​​​"Data Strategy", 2017) [?!] 

"Data is certainly exciting –​​​​​​​ revolutionary, even. But that doesn’​​​​​​​t always mean useful. To be truly useful, in a business sense, data must address a specific business need, help the organization reach its strategic goals, or generate real value." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"[…] from a data strategy point of view, you need to describe the ideal data sets that would help you achieve your strategic objectives. You can then choose the best options for you based on how well they help you achieve your objectives, how easy it is to access or gather that data, and how cost effective it is." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"However you plan to use data, even if you plan to treat data as a key business asset, it is never a good idea to capture huge mountains of data that you don’​​​​​​​t really need. Remember, the power of big data is not in the data it - self, it’​​​​​​​s in how you use it." (Bernard Marr, ​​​​​​​"Data Strategy", 2017) [?!] 

"I can’​​​​​​​t stress enough how important this stage is; ‘​​​​​​​selling’​​​​​​​ big data to your people is a crucial early step on your data journey. It instils confidence in data." (Bernard Marr, ​​​​​​​"Data Strategy", 2017) [?!] 

"[…] if companies want to avoid drowning in data, they need to develop a smart strategy that focuses on the data they really need to achieve their goals. In other words, this means defining the business-critical questions that need answering and then collecting and analysing only that data which will answer those questions." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"Structured data is any data or information that is located in a fixed field within a defined record or file, usually in databases or spreadsheets. Essentially, it is data that is organized in a predetermined way, usually in rows and columns." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"[…] the better insights are communicated, the more likely it is that data leads to positive action (in this case, better business decisions)." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"Unfortunately, the widespread perception among business executives is that data and analytics are purely IT matters. And as with all IT matters, this means they don’​​​​​​​t really need to understand how they work, or why." (Bernard Marr, ​​​​​​​"Data Strategy", 2017)

"When data isn’​​​​​​​t properly looked after, it becomes meaningless and valueless. Even worse, if the data is out of date, incorrectly categorized, or used out of context, it can lead to misinformed decisions that can damage the long-term health of the company." (Bernard Marr, ​​​​​​​"Data Strategy", 2017) [?!] 

🎯Pradeep Menon - Collected Quotes

"A central node acts as a hub in the hub-spoke pattern, and many edge nodes act as the spoke. The hub is the central node that orchestrates and governs the data sharing between each of the spokes. Each spoke node can have its own data lakehouse. The way the spoke nodes are constructed depends on the organizational structure. It can be at a department level, a separate organizational unit, or even a sub-department within a large department." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"A data sharing data service shares data, in any format and any size, from multiple sources within an organization or other organizations. This type of service provides the required control to share data and allows data-sharing policies to be created. It also enables data sharing in a structured manner and offers complete visibility into how the data is shared and how it is used. A data-sharing system uses APIs for data sharing." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"A data lakehouse stores a lot of data. It stores data in the data lake layer and the serving layer in structured and unstructured formats. The data needs to be processed with different types of compute engines. It can be a batch-based compute or a stream-based compute. A tightly coupled compute and storage layer strips off the flexibility required in a data lakehouse. Decoupling compute and storage also has a cost implication - storage is cheap and persistent but compute is expensive and ephemeral. It gives you the flexibility to spin up compute services on-demand and scale them as required, and also gives better cost control and cost predictability." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"A data warehouse service provides cleansed and transformed data that can be used for multiple purposes. First, it serves as a layer for reporting and BI. Second, it is a platform to query data for business or data analysis. Third, it serves as a repository to store historical data that needs to be online and available. Finally, it also acts as a source of transformed data for other downstream data marts that may cater to specific departmental requirements." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"Advanced analytics pivots around machine learning methods. Machine learning employs statistical learning methods to perform analysis on data. These statistical methods utilize algorithms that predict what may happen based on historical data or extract complex mathematical relationships from data to generate insights." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"An API is an interface that allows applications to interact with an external service using a simple set of commands. Data can also be served as part of API interaction. As the data is exposed to multiple external services, API-based methods can scale to share data securely with external services. Data through an API is served in JSON format, therefore the technology used to serve the data using APIs should be able to support JSON formats. For example, a NoSQL database can store such data." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"Any change in the reporting requirement had to go through a long-winded process of data model changes, ETL code changes, and respective changes to the reporting system. Often, the ETL process was a specialized skill and became a bottleneck for reducing data to insight turnover time. The nature of analytics is unique. The more you see the output, the more you demand. Many EDW projects were deemed a failure. The failure was not from a technical perspective, but from a business perspective. Operationally, the design changes required to cater to these fast-evolving requirements were too difficult to handle." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"Data architecture is the structure that enables the storage, transformation, exploitation, and governance of data." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"Descriptive analytics takes the form of on-demand queries, standard reporting, self-service business intelligence, and data lake exploration. This category of analytics performs analytics on historical data by providing different points of view on the data. The points of view are created by aggregating and filtering the quantitative data (also called the measure), and slicing the data across attributes of functional dimensions, such as sales, customers, and products. We can deliver descriptive analytics in multiple formats such as files, cross tab reports, visual reports, or dashboards." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"Each domain data lakehouse may opt to have its data catalog. However, the critical component in this architecture is the data mesh catalog. The data mesh catalog is the master catalog used to discover the data elements available in different nodes. Each domain-oriented node will donate its metadata to the data mesh catalog. This donation of metadata determines the effectiveness of the data mesh architecture. Once the metadata is contributed, other nodes can browse through the data mesh catalog. They can select the data of interest and mutually share data between the nodes through a governed data sharing process. The critical point to note here is that, unlike the hub-spoke architecture, the data mesh architecture enables data sharing between the 'spoke nodes'. There is no hub node in a data mesh architecture." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"The AI-ML service allows data scientists to build, train and deploy production-ready AI-ML models. This layer also provides the framework to maintain and monitor such models. In addition, it gives the ability for teams to collaborate as they go about building these models. This service should be able to scale up and down as required and should be able to facilitate automatic model deployment and operations." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"The data mesh pattern doesn't feature a central node and is loosely coupled compared to a hub-spoke architecture. It has different data lakehouse nodes that are independent of each other. The node data lakehouses are domain-driven. A domain can be oriented in multiple ways. The original idea of data mesh alludes to a source-oriented domain aligning to business processes. However, a more practical approach would be to define a domain based on the organizational setup and practicality; for example, a domain can be a product group, it can be separate organizational entities, and it can also be a specific business process, such as marketing. Each domain has its own data lakehouse that is managed and maintained by that domain." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is without having first to structure the data and run different types of analytics  - from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"A Data Lakehouse provides a unified platform for various data workloads, such as descriptive, predictive, and prescriptive analytics. It can handle structured and unstructured data and enforce schema at both read and write times, enabling traditional business intelligence tasks and advanced analytics on the same platform." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"A defining attribute of the domain-oriented ownership principle is the focus on context preservation in Data Management. This aspect accentuates the importance of keeping data within its native domain environment, allowing it to retain its original context, value, and meaning. When data is managed close to its source, its contextual richness is preserved. This sharply contrasts centralized models, where data is often abstracted from its source, leading to potential loss of signal or context. When data remains within its generating domain, it retains the nuances and specificities unique to its activities, challenges, and goals." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"A transformative perspective offered by the Data Mesh is envisioning data as a product. This section underscores the significance of curating data with the meticulousness and vision akin to product development, ensuring it delivers tangible value to its consumers. The ripple effects of this paradigm shift, spanning roles, processes, and technologies, are also meticulously unpacked." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Centralized governance structures often have an abstract view of data, focusing more on uniformity and compliance than context and relevance. While these are essential elements, the nuance often needs to be noticed. Decentralized governance flips the script by giving data ownership to the domain that generates it. The domain has the richest understanding of the data’s context, relevance, and potential impact, thereby being well-positioned to enforce governance policies that improve data quality." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Data Mesh also emphasizes aligning data products with business domains and use- cases to ensure that the data serves a clear business purpose and provides tangible value. Beforehand, we define the value proposition, target audience, quality attributes, and KPIs of each data product to ensure that it meets or exceeds the expectations of its consumers." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Data Mesh emphasizes ensuring reliable, consistent, and interoperable data products. When data is treated as a product, quality is non-negotiable. High-quality data must meet the expectations and requirements of its users, both internally and externally. Additionally, data products must be designed with other products in mind, adhering to principles like loose coupling for easy interchangeability and high cohesion for strong functional relatedness. This feature enables the integration of different data products, ensuring seamless interoperability and greater usability. Data products should be reliable, complete, accurate, and accurate. They should also be integrated, compatible, and consistent rather than isolated, incompatible, or conflicting." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"[...] domain-oriented ownership empowers individual domains to create and adapt their data strategies with agility, with a thorough understanding of their business needs and market demands. Whether pivoting due to a new competitor’s actions or adjusting to a sudden change in consumer behavior, domains can independently and swiftly modify their data strategies, providing them with a unique edge in the marketplace." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Domain-oriented ownership is a core principle of data mesh. It entails that data producers, experts in their business domains, are responsible for the entire lifecycle of their produced data. Specifically, they take ownership of the data from the point of ingestion through transformation, serving, quality assurance, and governance. Moreover, they are responsible for the data products created from their data, which serve as units of data consumption for other domains or users." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Domain autonomy should not be mistaken for a lack of governance or accountability. Autonomy, in this context, implies a higher level of responsibility. Domains are free to act and accountable for their actions, especially regarding how well their data strategies align with domain-specific and broader organizational objectives." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Empowering with self-serve data infrastructure: The Data Mesh champions the ethos of self-reliance. By empowering teams to construct and oversee their data infrastructure, organizations can foster a culture of speed, autonomy, and accountability." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Governance refers to an organization’s framework to exercise direction and control over a specific domain. In the context of data, it could include rules, protocols, and systems to manage data quality, security, and accessibility."  (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"In a centralized model, changes to data strategy often require navigating through bureaucratic layers and rigid governance structures. This delays adaptability and increases the risk of misalignment between what the data strategy aims to achieve and what the business needs. Centralized models are typically disconnected from the ground realities of individual business units, leading to a generic, one-size-fits-all approach that seldom caters to unique market challenges or opportunities." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Promoting domain-oriented ownership is to combat the common problem of organizational silos. Silos can significantly hinder the free flow of data and expertise, making decision-making and innovation more challenging. We aim to break down these barriers by advocating for domain-oriented ownership and creating a more dynamic and collaborative data management landscape." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"The allure of Data Lakes was their ability to store vast amounts of raw data. However, this advantage can become counterproductive without stringent governance and management protocols. In their zeal to harness the power of Big Data, some organizations indiscriminately dump data into their lakes. Without proper classification, curation, and quality checks, these lakes can become swamps - murky repositories filled with valuable data, redundant information, and outdated datasets. Navigating these data swamps becomes a significant challenge, leading to prolonged data retrieval times, increased chances of using obsolete or incorrect data, and a decline in the agility and efficiency of data-driven decision-making processes rather than facilitating quick and insightful analytics." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"When data is considered a product, it creates opportunities for collaboration across different domains. This collaboration involves working with other teams to create, share, and use data products that span multiple areas of expertise, interest, or value. Data Mesh promotes cross-domain collaboration by focusing on the consumers rather than the producers. Data products are made available through standardized interfaces and protocols that support various modes of consumption and are governed by domain experts who understand the context and nuances of their data." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

🎯Fadi Maali - Collected Quotes

"A common mistake when implementing a data catalog is to focus only on technical metadata. This limits its use and the potential value. It also excludes business users who have valuable related input or need to use the catalog. A catalog should in fact function as a two-way translation layer between technical and business users." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"A core premise of data mesh is federating data ownership among domain data owners who are responsible for their data as a product. Offering the data as a product requires the data to be discoverable and to have explicitly stated quality characteristics and a clearly defined access method. Such requirements are at the core of what data catalogs support. With support for data labeling, curation, and crowdsourced feedback, data catalogs are well positioned to offer data as a product. Furthermore, data catalogs support the enforcement of compliant data usage, which becomes more important when data ownership is not managed centrally." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Active governance guides users as they find and use data. A data catalog with active governance will surface compliance information about sensitive data at point of use, so as to encourage users to use canonical and high-quality data assets; it will also provide a way to ask domain experts for help. They actively help users to ensure compliant usage of data with features such as masking, which anonymizes PII for given user personas who are restricted from viewing it per the GDPR." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Build a community around the catalog. Make sure data producers, stewards, and consumers are all involved and empowered to enrich the content of the catalog. Establish a leader or a team to have clear ownership of the data catalog." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Data catalog platforms take a more holistic view to focus not only on data assets within an enterprise, but also on the surrounding ecosystem (including business and people elements). They are typically characterized by an extensible data model that can grow to define various assets and concepts, such as metrics, charts, AI features, and users. Data catalog platforms typically augment their data with a focus on business and users to support collaborative governance and enrichment of metadata and to interlink data with business glossaries and dictionaries. Moreover, they are architected to make them easily integrable with other systems." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Data catalogs that focus on governance are concerned mainly with controlling data access and ensuring that data is used according to defined policies; this includes external policies such as data privacy laws as well as policies defined with an enterprise. Those catalogs apply techniques to identify data assets with sensitive information and to monitor data flow and access." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Data catalogs that focus on search bring techniques and methods from information retrieval and web search engines to the data domain within enterprises. Some of those catalogs, such as Facebook Nemo, use advanced machine learning and NLP tools to provide personalized search of data within an enterprise. The search can also use data-specific signals such as usage, popularity, and freshness to rank data assets by usefulness." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Enterprises typically become interested in data catalogs when they have a specific use case or need in mind. Data governance, self-service analytics, and cloud data migration are common examples. Having a specific need or use case helps focus efforts and measure impact. However, as with other technical efforts within enterprises, it is essential to prepare for long-term sustainable success and to have a plan to maximize successful adoption." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Historically, for their analytics needs, enterprises relied upon a set of tightly coupled tools, typically provided by a single vendor. Nowadays, nearly all of the components of a traditional data warehouse are independent and interchangeable. Those independent tools can be flexibly combined to provide a modern data stack. It is common for current enterprises to have separate tools for data ingestion, data pipelines, data storage and querying, data visualization and business intelligence, and data quality. Furthermore, data can flow in the opposite direction out of the data warehouse in what is referred to as reverse extract, transform, and load (ETL)." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"In a self-service environment with multiple publishers, it’s impossible to completely avoid data redundancy and overlapping. Multiple data assets with similar content, but possibly with varying quality, will exist. A data catalog can guide users to trusted data that comes from a reliable source and is frequently used. A data catalog can also use various explicit and implicit quality signals when ranking datasets for recommendation. Some of those signals are discussed next. Furthermore, a data catalog can recommend domain experts who are automatically identified based on actual data usage." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"It is often said that data scientists and data analysts spend only 20% of their time doing data analysis work, with 80% consumed by data 'issues'. The bulk of their time is spent finding, evaluating, understanding, and preparing data before analysis can begin. A data catalog inverts this principle by enabling data analysts and data scientists to spend 20% of their time looking for data and 80% performing analysis." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Self-service BI initiatives help organizations become more data-driven and democratize access to data. But data can’t be used if it can’t be found. Search and discovery of trustworthy data is a core value of enterprise data catalogs, and the value extends well beyond business users." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

🎯🏭🗒️Sonia Mezzetta - Collected Quotes

"A data architecture needs to have the robustness and ability to support multiple data management and operational models to provide the necessary business value and agility to support an enterprise’s business strategy and capabilities." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"A data strategy must align with the business goals and overall framework of how data will be used and managed within an organization. It needs to include standards for how data will be discovered, integrated, accessed, shared, and protected. It needs to address how data will meet regulatory compliance policies, Master Data Management, and data democratization. There needs to be an assurance that both data and metadata have a quality control framework in place to achieve data trust. A data strategy needs to have a clear path on how an organization will accomplish data monetization." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"A data strategy is a living document that needs to be continuously updated to align with business goals. It should have a clear maintenance process with frequent reviews and identification of authors and stakeholders that will contribute to the data strategy. This also includes the handling of exceptions to a data strategy process for any one-off decisions in special circumstances. A data strategy document must always be easily assessable, to the point, and understandable." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Apply DataOps principles to the development and delivery of data. DataOps is a best practice framework that accelerates the development of data and quality across its entire life cycle with high efficiency and quality. This is especially important when integrating data across distributed complex systems and environments." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Automated data orchestration is a key DataOps principle. An example of orchestration can take ETL jobs and a Python script to ingest and transform data based on a specific sequence from different source systems. It can handle the versioning of data to avoid breaking existing data consumption pipelines already in place." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data Fabric’s building blocks represent groupings of different components and characteristics. They are high-level blocks that describe a package of capabilities that address specific business needs. The building blocks are Data Governance and its knowledge layer, Data Integration, and Self-Service." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data Fabric is a composable architecture made up of different tools, technologies, and systems. It has an active metadata and event-driven design that automates Data Integration while achieving interoperability. Data Governance, Data Privacy, Data Protection, and Data Security are paramount to its design and to enable Self-Service data sharing. The following figure summarizes the different characteristics that constitute a Data Fabric design." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data Fabric focuses on Self-Service data access via active metadata leveraging a composable set of tools and technologies. It offers the ability to discover, understand, and access data across hybrid and multi-cloud data landscapes with automation and Data Governance. It is primarily process and technology centric with flexibility in supporting diverse organizational models. On the other hand, Data Mesh is organizationally and process driven. It requires a technical implementation approach to execute its design. Data Mesh is at a higher level and Data Fabric is at a lower level. Data Fabric is capable of fulfilling Data Mesh’s key principles." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data Fabric is a distributed and composable architecture that is metadata and event driven. It’s use case agnostic and excels in managing and governing distributed data. It integrates dispersed data with automation, strong Data Governance, protection, and security. Data Fabric focuses on the Self-Service delivery of governed data." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data Fabric is a distributed data architecture that connects scattered data across tools and systems with the objective of providing governed access to fit-for-purpose data at speed. Data Fabric focuses on Data Governance, Data Integration, and Self-Service data sharing. It leverages a sophisticated active metadata layer that captures knowledge derived from data and its operations, data relationships, and business context. Data Fabric continuously analyzes data management activities to recommend value-driven improvements. Data Fabric works with both centralized and decentralized data systems and supports diverse operational models." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"[Data Fabric] is not a single technology, such as data virtualization. […] It is not a single tool like a data catalog and it doesn’t have to be a single data storage system like a data warehouse. It represents a diverse set of tools, technologies, and storage systems that work together in a connected ecosystem via a distributed data architecture, with active metadata as the glue. It doesn’t just support centralized data management but also federated and decentralized data management. It excels in connecting distributed data. Data Fabric is not the same as Data Mesh. They are different data architectures that tackle the complexities of distributed data management using different but complementary approaches." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data Fabric supports a federated, decentralized, or centralized organization. To participate in Data Fabric, metadata is contributed in an automated manner and knowledge is populated from it to propel data management. Data Fabric is different from a Data Mesh design in that it supports decentralized, federated, and centralized organizations. Data Fabric’s objectives are to help an organization to evolve to a more mature level of data management by leveraging active metadata, which is a core prerequisite." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data Mesh is a design concept based on federated data and business domains. It applies product management thinking to data management with the outcome being Data Products. It’s technology agnostic and calls for a domain-centric organization with federated Data Governance." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Establish an organization’s data maturity level and progress toward ongoing improvement. An organization needs to first understand what its current data maturity level is to determine the areas of improvement to create a forward-looking plan. A data maturity assessment offers a position on the current data maturity that serves as an indicator of the health of an organization. A data maturity assessment can be used as a tool to drive continuous improvement by measuring progress. The key thing here is to always strive for continuous improvement to achieve success." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"I emphasize this point as there are views in the industry that Data Fabric is a centralized storage architecture, which is not the case from my point of view. A Data Fabric architecture is driven by the needs and direction of the business architecture." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Manage data as a strategic asset that evolves into a data product. The premise here is to stop managing data as a byproduct and create an ecosystem that manages data as a valuable strategic asset that can evolve into a data product. Data producers are accountable for managing the life cycle of data from creation to end of life and ensuring it creates business value along the way for data consumers. This requires data that is governed, trusted, protected, secure, and easily accessible. Move data from technical data assets to Data Products by operationalizing data for high scale sharing." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Where Data Mesh differs from Data Fabric is that it has fixed requirements for the Self-Service platform focused on organizing and managing Data Products by business domain. Another difference is Data Fabric supports managing data as an asset and as a product. A Data Product can be composed of assets that have been governed and managed in a Data Fabric architecture. Data Fabric does not have these fixed requirements, although it inherently supports isolating data and Data Governance enforcement via metadata by business domain. You can think of a Data Mesh Self-Service data platform as supporting separate, independent companies (business domains), although the key criteria are that it does not create data silos and attains data sharing across these companies in a secure, quick, and easy manner. In Data Mesh, Data Products are created and managed by federated business domains and a data platform requires capabilities that enable data and policy federation. This is where a Data Fabric solution can also address Data Mesh’s requirements." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

🔢Charles D Tupper - Collected Quotes

"An architecture is the response to the integrated collections of models and views within the problem area being examined." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

"An architecture represents combined perspectives in a structured format that is easily viewable and explains the context of the area being analyzed to all those viewing it." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

"Analyzing and defining an area must be done prior to doing any activity within that area. Without understanding all that must be done, incorrect assumptions can be reached. Short-term vision may handicap future development. Inappropriate scoping may produce artificial boundaries where there should be none." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

"Data architecture allows strategic development of flexible modular designs by insulating the data from the business as well as the technology process." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

"Methodologies provide guidelines for the application development process. They specify analysis and design techniques as well as the stages in which they occur. They also develop event sequencing. Lastly, they specify milestones and work products that must be created and the appropriate documentation that should be generated." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

"Data architectures are the heart of business functionality. Given the proper data architecture, all possible functions can be completed within the enterprise easily and expeditiously." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

"Processes that use data change far more frequently than the data structures themselves." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

"The enterprise architecture delineates the data according to the inherent structure within the organization rather than by organizational function or use. In this manner it makes the data dependent on business objects but independent of business processes." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

"Using architecture leads to foundational stability, not rigidity. As long as the appropriate characteristics are in place to ensure positive architectural evolution, the architecture will remain a living construct. Well-developed architectures are frameworks that evolve as the business evolves." (Charles D Tupper, "Data Architecture: From Zen to Reality", 2011)

🔢Pearl Zhu - Collected Quotes

"A good strategy tells you not only what specifically needs to accomplish, but WHY." (Pearl Zhu, "Digitizing Boardroom: The Multifaceted Aspects of Digital Ready Boards", 2016)

"Agile is more a 'direction', than an 'end'. Transforming to Agile culture means the business knows the direction they want to go on." (Pearl Zhu, "Digital Agility: The Rocky Road from Doing Agile to Being Agile", 2016)

"Breaking rules is indeed an important part of creativity. Innovation needs a level of guidance." (Pearl Zhu,  "Digitizing Boardroom: The Multifaceted Aspects of Digital Ready Boards", 2016)

"Good governance is less about structure and rules than being focused, effective and accountable." (Pearl Zhu,  "Digitizing Boardroom: The Multifaceted Aspects of Digital Ready Boards", 2016)

"Governance is not about maximization, but about optimization." (Pearl Zhu, "Digitizing Boardroom: The Multifaceted Aspects of Digital Ready Boards", 2016)

"Selecting the right measure and measuring things right are both art and science. And KPIs influence management behavior as well as business culture." (Pearl Zhu, "CIO Master: Unleash the Digital Potential of It", 2016)

"Setting the right priorities or having superior time management skill means knowing the difference between 'must have', and 'nice to have'." (Pearl Zhu, "Thinkingaire: 100 Game Changing Digital Mindsets to Compete for the Future", 2016)

"The art of questioning is to ignite innovative thinking; the science of questioning is to frame system thinking, with the progressive pursuit of better solutions." (Pearl Zhu, "Leadership Master: Five Digital Trends to Leap Leadership Maturity", 2016)

"The 'result' of micromanagement is perhaps tangible in the short run, but more often causes damage for the long term." (Pearl Zhu, "Change Insight: Change as an Ongoing Capability to Fuel Digital Transformation", 2016)

"Using two-dimensional lenses to perceive the multi-faceted world can limit your ability to observe the world more objectively." (Pearl Zhu, "Thinkingaire: 100 Game Changing Digital Mindsets to Compete for the Future", 2016)

"A performance dashboard is a practical tool to improve management effectiveness and efficiency, not just a pretty retrospective picture in an annual report." (Pearl Zhu, "Performance Master: Take a Holistic Approach to Unlock Digital Performance", 2017)

"A 'roadmap' is simply a plan for moving or transitioning, from one state to another. A roadmap provides the direction to the future." (Pearl Zhu, "Digital Capability: Building Lego Like Capability Into Business Competency", 2017)

"A well-defined set of digital rules are not for limiting innovation, but for setting the frame of relevance and guide through changes and digital transformation." (Pearl Zhu, "100 Digital Rules: Setting Guidelines to Explore Digital New Normal", 2017)

"Building a comprehensive problem-solving framework is about leveraging a structured methodology that allows you to frame problems systematically and solve problems creatively." (Pearl Zhu, "Problem Solving Master: Frame Problems Systematically and Solve Problem Creatively", 2017)

"Decision makers with emotional excellence have the ability to dispassionately examine alternatives via fact finding, analysis, structured planning, objective evaluations, and comparison." (Pearl Zhu, "Decision Master: The Art and Science of Decision Making", 2017)

"Decision making is an art only until the person understands the science." (Pearl Zhu, "Decision Master: The Art and Science of Decision Making", 2017)

"Decision maturity is to ensure the right decisions have been made by the right people at the right time to solve the right problems." (Pearl Zhu, "Decision Master: The Art and Science of Decision Making", 2017)

"Digital synchronization and strategic alignment occur when all parts of the choir sing their respective parts in harmony to achieve a higher purpose." (Pearl Zhu, "12 CIO Personas: The Digital CIO's Situational Leadership Practices", 2017)

"Digitalization implies the full-scale changes in the way business is conducted so that it’s a multi-dimensional planning and orchestration." (Pearl Zhu, "Digital Capability: Building Lego Like Capability Into Business Competency", 2017)

"Framing the right problem is equally or even more important than solving it." (Pearl Zhu, “Change, Creativity and Problem-Solving”, 2017)

"Most organizations fail to manage performance effectively because they fail to look into the system holistically." (Pearl Zhu, "Performance Master: Take a Holistic Approach to Unlock Digital Performance", 2017)

"The science of decision-making is to make sure there is an effective decision process in place." (Pearl Zhu, "Decision Master: The Art and Science of Decision Making", 2017)

"It is important to strengthen the weakest link, to ensure all important business elements integrated and knitted into ongoing organizational capabilities and unique business competency." (Pearl Zhu, "Digital Capability: Building Lego Like Capability Into Business Competency", 2017)

"The simplicity and the complexity are just the opposite ends of the same spectrum." (Pearl Zhu, "Digital Gaps: Bridging Multiple Gaps to Run Cohesive Digital Business", 2017)

"We are moving slowly into an era where Big Data is the starting point, not the end." (Pearl Zhu, "Digital Master: Debunk the Myths of Enterprise Digital Maturity", 2017)

"You can’t improve what you are not managing, you can’t manage what you are not measuring, and you can’t measure what you are not focusing." (Pearl Zhu, "Digital Capability: Building Lego Like Capability Into Business Competency", 2017)

"A business ecosystem is just like the natural ecosystem; first, needs to be understood, then, needs to be well planned, and also needs to be thoughtfully renewed as well." (Pearl Zhu, "Digital Maturity: Take a Journey of a Thousand Miles from Functioning to Delight", 2018)

"A seamless digital transformation requires a vision to convey 'WHY', a solid strategy to clarify 'WHAT', and a technical specification to articulate 'HOW' you want to transform radically." (Pearl Zhu, "Digital Maturity: Take a Journey of a Thousand Miles from Functioning to Delight", 2018)

"An organizational structure carries inherent capabilities as to what can be achieved within its frame." (Pearl Zhu, Digital Maturity: Take a Journey of a Thousand Miles from Functioning to Delight, 2018)

"Change Management is a journey, not just a one-time project, riding ahead of change curve takes both strategy and methodology." (Pearl Zhu, "The Change Agent CIO: The CIO’s Dynamic Role of Leading Digitalization", 2018)

"Coherence improves business flow; resilience makes business robust and anti-fragile." (Pearl Zhu, "Digital Hybridity: How to Strike the Right Balance for Digital Paradigm Shift", 2018)

"Going digital is more like a journey than a destination. Predicting and preparing the next level of digitalization is an iterative learning and doing continuum." (Pearl Zhu, "Digital Maturity: Take a Journey of a Thousand Miles from Functioning to Delight", 2018)

"Ideally, the two structures - hierarchy, and relationship structure wrap around each other to ensure responsibility, to keep information flow and the creation of power." (Pearl Zhu, "Digital Maturity: Take a Journey of a Thousand Miles from Functioning to Delight", 2018)

"Taking the multidimensional hybrid models for going digital is all about how to strike the right balance of reaping quick wins and focusing on the long-term strategic goals." (Pearl Zhu, "Digital Hybridity: How to Strike the Right Balance for Digital Paradigm Shift", 2018)

"The most effective digital workplace is one where collaboration and sharing are the norms." (Pearl Zhu, "Digital Maturity: Take a Journey of a Thousand Miles from Functioning to Delight", 2018)

🎯Rukmani Gopalan - Collected Quotes

"A cloud data warehouse is an enterprise data warehouse offered as a managed service (PaaS) on public clouds with optimized integrations for data ingestion, analytics processing, and BI analytics." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022) 

"Churn refers to rapidly changing the activities and your plan when they are in flux - this is disruptive to your organization and slows your progress. Change refers to an inevitable movement in requirements and helps you plan for and execute this movement thoughtfully." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022) 

"Data mesh relies on a distributed architecture that consists of domains. Each domain is an independent unit of data and its associated storage and compute components. When an organization contains various product units, each with its own data needs, each product team owns a domain that is operated and governed independently by the product team. […] Data mesh has a unique value proposition, not just offering scale of infrastructure and scenarios but also helping shift the organization’s culture around data," (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022)

"If there is one thing I strongly recommend, it is to invest in a cloud data lake and start collecting and processing data that you believe is useful to your organization today." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022) 

"It’s true that data and data strategy are critical to the organization; however, it’s also true that data by itself is a means to the end of business or customer impact unless you’re a provider of data or data-related services." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022) 

"Plan for customer impact, and prepare to learn and fine-tune as you progress. Make choices based on the impact they offer to customers, and stay consistent in your implementation while keeping open-minded for learnings. Especially if you are an early adopter of a technology, you can help develop the technology with the provider and thus get ample support from the technology provider in return. Similarly, identify highly motivated early adopters within your customer base and offer to develop your solution with them." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022) 

"Real-time stream processing refers to the ingestion, processing, and consumption of data with a specific focus on speed, targeting near real time - that is, almost instantaneous results. […] Real-time stream processing pipelines involve data that is arriving from its source at very high velocity; in other words, it is data that is streaming into the system, just like rain or a waterfall." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022) 

"The lakehouse provides a key advantage over the modern data warehouse by eliminating the need to have two places to store the same data. [...] Data lakehouses offer the key benefit of being able to run performant BI/SQL-based scenarios directly on the data lake, right alongside the other exploratory data science and machine learning scenarios." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022) 

"The promise of a cloud data lake architecture lies in the boundless diversity of scenarios that it enables." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022) 

"The very simple definition of cloud data lake storage is a service available as a cloud offering that can serve as a central repository for all kinds of data (structured, unstructured, and semistructured) and can support data and transactions at a large scale." (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022)

"When it comes to data lakes, some things usually stay constant: the storage and processing patterns. Change could come in any of the following ways: Adding new components and processing or consumption patterns to respond to new requirements. […] Optimizing existing architecture for better cost or performance" (Rukmani Gopalan, "The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture", 2022)

🏗️Software Engineering: Continuous Integration & Continuous Deployment [CI/CD] (Just the Quotes)

"The deployment pipeline has its foundations in the process of continuous integration and is in essence the principle of continuous integration taken to its logical conclusion. The aim of the deployment pipeline is threefold. First, it makes every part of the process of building, deploying, testing, and releasing software visible to everybody involved, aiding collaboration. Second, it improves feedback so that problems are identified, and so resolved, as early in the process as possible. Finally, it enables teams to deploy and release any version of their software to any environment at will through a fully automated process." (David Farley & Jez Humble, "Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation", 2010)

"In essence, Continuous Integration is about reducing risk by providing faster feedback. First and foremost, it is designed to help identify and fix integration and regression issues faster, resulting in smoother, quicker delivery, and fewer bugs. By providing better visibility for both technical and non-technical team members on the state of the project, Continuous Integration can open and facilitate communication channels between team members and encourage collaborative problem solving and process improvement. And, by automating the deployment process, Continuous Integration helps you get your software into the hands of the testers and the end users faster, more reliably, and with less effort." (John F Smart, "Jenkins: The Definitive Guide", 2011)

"Remember, the most basic function of any Continuous Integration tool is to monitor source code in a version control system and to fetch and build the latest version of your source code whenever any changes are committed." (John F Smart, "Jenkins: The Definitive Guide", 2011)

"Continuous deployment is but one of many powerful tools at your disposal for increasing iteration speed. Other options include investing in time-saving tools, improving your debugging loops, mastering your programming workflows, and, more generally, removing any bottlenecks that you identify." (Edmond Lau, "The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact", 2015)

"Many problems stem from a premature attempt at scaling Agile within the organization. The nature of the transformation is such that it is unrealistic to plan upfront for an 18-month organization-wide change program to go from status quo to continuous delivery. People try nevertheless, and when the outcomes don’t materialize, they say Agile doesn’t work." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"Why is continuous deployment such a powerful tool? Fundamentally, it allows engineers to make and deploy small, incremental changes rather than the larger, batched changes typical at other companies. That shift in approach eliminates a significant amount of overhead associated with traditional release processes, making it easier to reason about changes and enabling engineers to iterate much more quickly." (Edmond Lau, "The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact", 2015)

"[…] the practice of continuous integration helps a development team fail-fast in integrating code under development. A corollary of failing fast is to aim for fast feedback. The practice of regularly showcasing (demoing) features under development to product owners and business stakeholders helps them verify whether it is what they asked for and decide whether it is what they really want." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"DevOps and Continuous Integration/Continuous Deployment (CI/CD) are vital to any software project that is developed by more than one developer and needs to uphold quality standards. A central code repository that offers versioning, branching, and merging for collaborative development and approval workflows and documentation features is the minimum requirement here." (Patrik Borosch, "Cloud Scale Analytics with Azure Data Services: Build modern data warehouses on Microsoft Azure", 2021)

"[...] ML engineers tend to focus on MLOps and DevOps, which are tradition‐ally focused on deployment automation, infrastructure-as-code, and CI/CD. On theother hand, data scientists tend to focus on training and evaluating ML models. While the two worlds have collided, there remains a competency gap between ML engineers (automation) and d? ata scientists (model evaluation) in many teams. We know how to set up CI pipelines and we know how to train and evaluate models, but not all teams have worked out how to bridge both practices to automate manual model evaluation procedures." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)

🔢Robert Hawker - Collected Quotes

"[...] a conceptual data model [...] is system-agnostic and is a diagrammatic business representation of how different types of data are associated with one another in the organization." (Robert Hawker, "Practical Data Quality", 2023)

"A data quality rule is logic that is applied to each row of a dataset, which can determine whether the row of data is correct or incorrect. Correct data is deemed to have passed the rule, and incorrect data is deemed to have failed the rule – hence, the term failed data [...]" (Robert Hawker, "Practical Data Quality", 2023)

"Correction of data in the secondary source is not recommended. However, it is important to recognize that sometimes, secondary source fixes are required." (Robert Hawker, "Practical Data Quality", 2023)

"Data discovery is the process where an organization obtains an understanding of which data matters the most and identifies challenges with that data. The outcome of data discovery is that the scope of a data quality initiative should be clear and data quality rules can be defined." (Robert Hawker, "Practical Data Quality", 2023)

"Data profiling assesses a set of data and provides information on the values, the length of strings, the level of completeness, and the distribution patterns of each column." (Robert Hawker, "Practical Data Quality", 2023)

"Data quality rules are only effective if they are tightly scoped. Generic rules tend to produce a lot of unwanted failed records, and business users start to ignore the results. Once business users lose faith in what they see from a data quality tool, it is hard to restore engagement." (Robert Hawker, "Practical Data Quality", 2023)

"Every data quality initiative is different, and senior stakeholders at different organizations will have different needs." (Robert Hawker, "Practical Data Quality", 2023)

"If an organization had a single overall data quality key performance indicator (KPI), then it might be appropriate to put a greater weighting on those rules which would impact regulatory compliance. A lack of regulatory compliance is a risk to the very existence of organizations like these, and therefore, a greater weighting might be needed." (Robert Hawker, "Practical Data Quality", 2023)

"It rarely makes sense to aim for what people might consider perfect data (every record is complete, accurate, and up to date). The investment required is usually prohibitive, and the gains made for the last 1% of data quality improvement effort become far too marginal." (Robert Hawker, "Practical Data Quality", 2023)

"In truth, no one knows how much bad data quality costs a company – even companies with mature data quality initiatives in place, who are measuring hundreds of data points for their quality struggle to accurately measure quantitative impact. This is often a deal-breaker for senior leaders when trying to get approval for a budget for data quality work. Data quality initiatives often seek substantial budgets and are up against projects with more tangible benefits." (Robert Hawker, "Practical Data Quality", 2023)

"Momentum is important in data quality initiatives. If an issue is problematic, even where the priority is high, it can be better to move on to an issue that can be progressed efficiently." (Robert Hawker, "Practical Data Quality", 2023)

"Most data quality issues will re-occur if the root cause is not fully understood [...]" (Robert Hawker, "Practical Data Quality", 2023)

"Organizations will always only have a limited amount of resources available to remediate data. It will almost certainly not be possible to tackle all the issues at the same time. Therefore, prioritization is key to ensuring that the most value is generated from the available resources." (Robert Hawker, "Practical Data Quality", 2023)

"Successful organizations try to put a holistic data culture in place. Everyone is educated on the basics of looking after data and the importance of having good data. They consider what they have learned when performing their day-to-day tasks. This is often referred to as the promotion of good data literacy." (Robert Hawker, "Practical Data Quality", 2023)

"The biggest mistake that can be made in a data quality initiative is focusing on the wrong data. If you fix data that does not impact a critical business process or drive important decisions, your initiative simply will not make the difference that you want it to." (Robert Hawker, "Practical Data Quality", 2023)

"The data should be monitored in the source, it should be corrected in the source, and it should then feed the secondary source(s) with high-quality data that can be used without workarounds. The reduction in workarounds will make the data engineers, scientists, and data visualization specialists much more productive." (Robert Hawker, "Practical Data Quality", 2023)

"The level of data quality in an organization is the extent to which data can be used for its intended purposes."  (Robert Hawker, "Practical Data Quality", 2023)

"Start with a business strategy. Too many organizations start their data quality initiative by looking at the details of the data and trying to see 'what is wrong with it'. The right approach is to understand what the business is trying to achieve and to work out where data issues might impede this. It ensures that data quality work will be truly impactful." (Robert Hawker, "Practical Data Quality", 2023)

🎯Hubert Dulay - Collected Quotes

"A data fabric is a pattern that is very similar to a data mesh in that both provide solutions encompassing data governance and self-service: discovery, access, security, integration, transformation, and lineage. [...] In simple terms, a data fabric is a metadriven means of connecting disparate sets of data and related tools to provide a cohesive data experience and to deliver data in a self-service manner." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"A data fabric is an architectural approach to provide data access across multiple technologies and platforms, and is based on a technology solution. One key contrast is that a data mesh is much more than just technology: it is a pattern that involves people and processes. Instead of taking ownership of an entire data platform, as in a data fabric, the data mesh allows data producers to focus on data production, allows data consumers to focus on consumption, and allows hybrid teams to consume other data products, blend other data to create even more interesting data products, and publish these data products - with some data governance considerations in place." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"A domain has two main roles: data product engineer (or just data engineer) and the data product owner (or data product manager, or data steward). These roles can be the same or dedicated people in the domain. Data product owners must have a deep understanding of who their data consumers are, how the data is used, and what methods are used to consume the data. This will help ensure that the data products meet the needs of their use cases. Data product engineers are responsible for creating data products that are high quality, reliable, and usable by consumers. It should be possible to extend existing domain roles to include these domain roles with minimal effort." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Consumability is a very important requirement because it will directly affect the experience domain consumers will have in a streaming data mesh. If other domains cannot easily consume streaming data products, then they may opt out of the streaming data mesh and decide to build their own integrations by hand, bypassing any issues they encounter with the data mesh. Some factors to consider when ingesting data derivatives that will affect the consumability of other domains are as follows: (*) Lack of scalability (*) Lack of interoperability" (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data governance creates access controls between the data product producer and consumer and provides metadata like schema definitions and lineages. In some cases, mastered data along with reference data may be relevant to the implementation. Data governance allows us to create appropriate access controls for these resources as well." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data governance is a set of policies, standards, processes, roles, and responsibilities that collectively ensure accountability and ownership of data across the business. Policies are the rules and regulations surrounding data defined by the business itself or, more importantly, externally by laws that, if broken, could cost a business a massive amount in fines. These policies also include enforcement of standards that enable interoperability and consumability of data between domains, especially in a decentralized data platform like a streaming data mesh. These policies are implemented as processes and controls on data by authorizing, authenticating, and safeguarding private or personal data. Policies are implemented using roles that represent groups, people, or systems to create access controls around data." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data lineage is the path the data took from its source origin, the stops it made along the way, and its destination. This includes information on all the systems it passed through, how it was cleansed, what it was enriched with, and how it was secured. Capturing all that metadata is difficult because many of those systems and applications don’t share information. It’s up to you to assemble the data’s path by pulling metadata from all those systems/applications and assembling them in hopes that you find the path your data took from its current location (destination) to its source system. Lineage is probably the hardest piece of metadata to acquire for either streaming or batching data pipelines." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data lineage provides the entire history of the data: origin, transformations, enrichments, users who engineered the transformations, etc. Consumers of the data need to trust that the data they will be using is the correct data. Data lineage provides a perspective that creates trust. It does so by mapping out the steps for policies, standards, processes, roles, and responsibilities that were involved with the sourcing, transformation, enrichment, and cleansing of the data." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data mesh is not completely decentralized. The data is decentralized in domains, but the mesh part of data mesh is not. Data governance is critical in building the mesh in a data mesh." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data tags are a simple way of providing the consuming domains with more information about the streaming data product: how it was built and what to expect when consuming it. Many of the streaming data characteristics are hard to measure, like quality and security, so it’s sometimes tough to provide that important information to the consuming domain. Instead of providing a number or a score, we can provide tags that represent levels of quality and security." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Domain-driven design (DDD) is the methodology that helps us understand complex domain models by connecting the data model itself to core business concepts. The understanding that emerges from DDD creates a foundation to designing distributed, microservice-based, client-facing applications. DDD connects the implementation of software and its components to an evolving and ever-changing data model. The domain is the world of the business you are working with and the problems you are trying to solve. This typically involves rules, processes, and existing systems that need to be integrated as part of your solution." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"In a data mesh, data is decentralized, while in a data fabric, centralization of data is allowed. And with data centralization like data lakes, you get the monolithic problems that come with it. Data mesh tries to apply a microservices approach to data by decomposing data domains into smaller and more agile groups." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Since domains are used to create data products, and sharing data products across many domains ultimately builds a mesh of data, we need to ensure that the data being served follows some guidelines. Data governance involves creating and adhering to a set of global rules, standards, and policies applied to all data products and their interfaces to ensure a collaborative and interoperable data mesh community. These guidelines must be agreed upon among the participating data mesh domains." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"The best approach to building self-services for domains is to follow what many SaaS services do. They follow a serverless model that is easier for their users to understand and utilize in their applications. The intention of the serverless SaaS providers is to not require their users to worry about the 'servers' that are allocated on their behalf. Users can focus more on their business rather than managing and tuning servers. This should be the same model for self-services: to make a streaming data mesh serverless so domains need to focus only on their business."(Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"To overcome ambiguous domain challenges, each domain boundary must be distinct and explicit. Business area, processes, and data that belong together need to stay together. Additionally, each data domain should belong to one, and only one, Agile or DevOps team. Data integration points within a data domain should be manageable and understood by all team members." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"We recommend making domain boundaries concrete and immutable. This helps avoid lengthy discussions about who owns what data, and also prohibits teams from freely interpreting domain boundaries to suit their own needs. Creating a domain-oriented structure is a transition - not only for data, but for people and resources. When creating domain boundaries, resources may eventually align with other teams, disrupting and evolving the current team structure. The entire concept of data mesh is just as much about resource alignment as it is about data, so the realignment of resources should not be considered a roadblock as you go through this process." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"When building a data mesh, it is necessary to enable existing engineers in a domain to perform the tasks required. Domains have to capture data from their operational stores, transform (join or enrich, aggregate, balance) that data, and publish their data products to the data mesh. Self-service services are the “easy buttons” necessary to make data mesh easy to adopt with high usability. In summary, the selfservices enable the domain engineers to take on many of the tasks the data engineer was responsible for across all lines of the business. A data mesh not only breaks up the monolithic data lake, but also breaks up the monolithic role of the data engineer into simple tasks the domain engineers can perform." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"While a data mesh seeks to solve many of the same problems that a data fabric addresses - namely, the ability to address data in a single, composite data environment—the approach is different. While a data fabric enables users to create a single, virtual layer on top of distributed data, a data mesh further empowers distributed groups of data producers to manage and publish data as they see fit. Data fabrics allow for a low-to-no-code data virtualization experience by applying data integration within APIs that reside within the data fabric. The data mesh, however, allows for data engineers to write code for APIs with which to interface further. Without clearly defined boundaries, domains appear to be too interconnected, and ownership becomes either political or subject to interpretation. For instance, a large retailer most likely has multiple domains. [...]" (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)