🏗️Software Engineering: Feedback (Just the Quotes)

"The system should always keep users informed about what is going on, through appropriate feedback within reasonable time." (Jakob Nielsen, "Usability Engineering", 1993)

"Optimism is an occupational hazard of programming: feedback is the treatment." (Kent Beck, "Extreme Programming Explained", 2000) 

"Extreme Programming is a discipline of software development with values of simplicity, communication, feedback and courage. We focus on the roles of customer, manager, and programmer and accord key rights and responsibilities to those in those roles." (Ron Jeffries, "Extreme Programming Installed", 2001)

"The values of XP are simplicity, communication, feedback, and courage. [...] Use simple design and programming practices, and simple methods of planning, tracking, and reporting. Test your program and your practices, using feedback to decide how to steer the project. Working together in this way gives the team courage."" (Ron Jeffries, "Extreme Programming Installed", 2001)

"XP isn't slash and burn programming, not code and fix, not at all. Extreme Programming is about careful and continuous design, rapid  feedback from extensive testing, and the maintenance of relentlessly clear and high-quality code." (Ron Jeffries, "Extreme Programming Installed, 2001)

"Prototypes should command only as much time, effort, and investment as is necessary to generate useful feedback and drive an idea forward. The greater the complexity and expense, the more 'finished' it is likely to seem and the less likely its creators will be to profit from constructive feedback - or even to listen to it. The goal of prototyping is not to create a working model. It is to give form to an idea to learn about its strengths and weaknesses and to identify new directions for the next generation of more detailed, more refined prototypes. A prototype's scope should be limited. The purpose of early prototypes might be to understand whether an idea has functional value." (Tim Brown, "Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation", 2009)

"The deployment pipeline has its foundations in the process of continuous integration and is in essence the principle of continuous integration taken to its logical conclusion. The aim of the deployment pipeline is threefold. First, it makes every part of the process of building, deploying, testing, and releasing software visible to everybody involved, aiding collaboration. Second, it improves feedback so that problems are identified, and so resolved, as early in the process as possible. Finally, it enables teams to deploy and release any version of their software to any environment at will through a fully automated process." (David Farley & Jez Humble, "Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation", 2010)

"In essence, Continuous Integration is about reducing risk by providing faster feedback. First and foremost, it is designed to help identify and fix integration and regression issues faster, resulting in smoother, quicker delivery, and fewer bugs. By providing better visibility for both technical and non-technical team members on the state of the project, Continuous Integration can open and facilitate communication channels between team members and encourage collaborative problem solving and process improvement. And, by automating the deployment process, Continuous Integration helps you get your software into the hands of the testers and the end users faster, more reliably, and with less effort." (John F Smart, "Jenkins: The Definitive Guide", 2011)

"Agile development methods require a disciplined approach to ensure that customer feedback, continuous testing, and iterative development actually lead to frequent deliveries of working, valuable software." (Michael Hűttermann et al, "DevOps for Developers", 2013)

"DevOps is essentially about gaining fast feedback and decreasing the risk of releases through a holistic approach that is meaningful for both development and operations. One major step for achieving this approach is to improve the fl ow of features from their inception to availability. This process can be refined to the point that it becomes important to reduce batch size" (the size of one package of changes or the amount of work that is done before the new version is shipped) without changing capacity or demand." (Michael Hűttermann et al, "DevOps for Developers", 2013)

"A value stream is a series of activities required to deliver an outcome. The software development value stream may be described as: validate business case, analyze, design, build, test, deploy, learn from usage analytics and other feedback - rinse and repeat." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"Areas of low complexity or that are unlikely to be invested in can be built without the need for perfect code quality; working software is good enough. Sometimes feedback and first-to-market are core to the success of a product; in this instance, it can make business sense to get working software up as soon as possible, whatever the architecture." (Scott Millett, "Patterns Principles and Practices of Domain Driven Design", 2015)

"Feedback is what makes it iterative; otherwise, it is just mini-waterfall. Merely splitting use cases into stories does not make for iterative development if we wait until all stories are developed before we seek feedback. The point of splitting is to get feedback faster so that it can be incorporated into ongoing development. However, seeking stakeholder/user feedback for small batches of functionality" (stories) is often not feasible with formal stage-gate processes. They were conceived with linear flows of large batches in mind." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"This is what the Agile Manifesto means when it says responding to change over following a plan. To maximize adaptability, it is essential to have good, fast feedback loops. This is why there is so much emphasis on iterative development." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"But perhaps the biggest problem is that the longer you spend working on something - whether it's a prototype or a real product - the more attached you'll become, and the less likely you'll be to take negative test results to heart. After one day, you're receptive to feedback. After three months, you're committed." (Jake Knapp et al, "Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days", 2016)

"No methodology can guarantee success. But a good methodology can provide a feedback loop for continual improvement and learning." (Ash Maurya, "Scaling Lean: Mastering the Key Metrics for Startup Growth", 2016)

📉Graphical Representation: Prototyping (Just the Quotes)

"An important rule in the drafting of curve charts is that the amount scale should begin at zero. In comparisons of size the omission of the zero base, unless clearly indicated, is likely to give a misleading impression of the relative values and trend." (Rufus R Lutz, "Graphic Presentation Simplified", 1949)

"Of course statistical graphics, just like statistical calculations, are only as good as what goes into them. An ill-specified or preposterous model or a puny data set cannot be rescued by a graphic" (or by calculation), no matter how clever or fancy. A silly theory means a silly graphic." (Edward R Tufte, "The Visual Display of Quantitative Information", 1983)

"The information on a plot should be relevant to the goals of the analysis. This means that in choosing graphical methods we should match the capabilities of the methods to our needs in the context of each application. [...] Scatter plots, with the views carefully selected as in draftsman's displays, casement displays, and multiwindow plots, are likely to be more informative. We must be careful, however, not to confuse what is relevant with what we expect or want to find. Often wholly unexpected phenomena constitute our most important findings." (John M Chambers et al, "Graphical Methods for Data Analysis", 1983)

"A good way to evaluate a model is to look at a visual representation of it. After all, what is easier to understand - a table full of mathematical relationships or a graphic displaying a decision tree with all of its splits and branches?" (Seth Paul et al. "Preparing and Mining Data with Microsoft SQL Server 2000 and Analysis", 2002)

"There are two main reasons for using graphic displays of datasets: either to present or to explore data. Presenting data involves deciding what information you want to convey and drawing a display appropriate for the content and for the intended audience. [...] Exploring data is a much more individual matter, using graphics to find information and to generate ideas.Many displays may be drawn. They can be changed at will or discarded and new versions prepared, so generally no one plot is especially important, and they all have a short life span." (Antony Unwin, "Good Graphics?" [in "Handbook of Data Visualization"], 2008)

"Although it might seem as though frittering away valuable time on sketches and models and simulations will slow work down, prototyping generates results faster." (Tim Brown, "Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation", 2009)

"Just as it can accelerate the pace of a project, prototyping allows the exploration of many ideas in parallel. Early prototypes should be fast, rough, and cheap. The greater the investment in an idea, the more committed one becomes to it. Overinvestment in a refined prototype has two undesirable consequences: First, a mediocre idea may go too far toward realization - or even, in the worst case, all the way. Second, the prototyping process itself creates the opportunity to discover new and better ideas at minimal cost." (Tim Brown, "Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation", 2009)

"In analytics, it’s more important for individuals to be able to formulate problems well, to prototype solutions quickly, to make reasonable assumptions in the face of ill-structured problems, to design experiments that represent good investments, and to analyze results." (Foster Provost & Tom Fawcett, "Data Science for Business", 2013)

"If I had to pick a single go-to graph for categorical data, it would be the horizontal bar chart, which flips the vertical version on its side. Why? Because it is extremely easy to read. The horizontal bar chart is especially useful if your category names are long, as the text is written from left to right, as most audiences read, making your graph legible for your audience." (Cole N Knaflic, "Storytelling with Data: A Data Visualization Guide for Business Professionals", 2015)

"The intention behind prototypes is to explore the visualization design space, as opposed to the data space. A typical project usually entails a series of prototypes; each is a tool to gather feedback from stakeholders and help explore different ways to most effectively support the higher-level questions that they have. The repeated feedback also helps validate the operationalization along the way." (Danyel Fisher & Miriah Meyer, "Making Data Visual", 2018)

"Rapid prototyping is a process of trying out many visualization ideas as quickly as possible and getting feedback from stakeholders on their efficacy. […] The design concept of 'failing fast' informs this: by exploring many different possible visual representations, it quickly becomes clear which tasks are supported by which techniques." (Danyel Fisher & Miriah Meyer, "Making Data Visual", 2018)

🔭Data Science: Prototyping (Just the Quotes)

"A model is a qualitative or quantitative representation of a process or endeavor that shows the effects of those factors which are significant for the purposes being considered. A model may be pictorial, descriptive, qualitative, or generally approximate in nature; or it may be mathematical and quantitative in nature and reasonably precise. It is important that effective means for modeling be understood such as analog, stochastic, procedural, scheduling, flow chart, schematic, and block diagrams." (Harold Chestnut, "Systems Engineering Tools", 1965)

"A model is an attempt to represent some segment of reality and explain, in a simplified manner, the way the segment operates." (E Frank Harrison,The managerial decision-making process", 1975)

"A mathematical model is any complete and consistent set of mathematical equations which are designed to correspond to some other entity, its prototype. The prototype may be a physical, biological, social, psychological or conceptual entity, perhaps even another mathematical model." (Rutherford Aris, :Mathematical Modelling", 1978)

"Modeling in its broadest sense is the cost-effective use of something in place of something else for some [cognitive] purpose. It allows us to use something that is simpler, safer, or cheaper than reality instead of reality for some purpose. A model represents reality for the given purpose; the model is an abstraction of reality in the sense that it cannot represent all aspects of reality. This allows us to deal with the world in a simplified manner, avoiding the complexity, danger and irreversibility of reality." (Jeff Rothenberg,The Nature of Modeling. In: Artificial Intelligence, Simulation, and Modeling", 1989)

"Model building is the art of selecting those aspects of a process that are relevant to the question being asked. As with any art, this selection is guided by taste, elegance, and metaphor; it is a matter of induction, rather than deduction. High science depends on this art." (John H Holland, "Hidden Order: How Adaptation Builds Complexity", 1995)

"A model is a deliberately simplified representation of a much more complicated situation. […] The idea is to focus on one or two causal or conditioning factors, exclude everything else, and hope to understand how just these aspects of reality work and interact." (Robert M Solow,How Did Economics Get That Way and What Way Did It Get?", Daedalus, Vol. 126, No. 1, 1997)

"More generally, a data scientist is someone who knows how to extract meaning from and interpret data, which requires both tools and methods from statistics and machine learning, as well as being human. She spends a lot of time in the process of collecting, cleaning, and munging data, because data is never clean. This process requires persistence, statistics, and software engineering skills - skills that are also necessary for understanding biases in the data, and for debugging logging output from code. Once she gets the data into shape, a crucial part is exploratory data analysis, which combines visualization and data sense. She’ll find patterns, build models, and algorithms - some with the intention of understanding product usage and the overall health of the product, and others to serve as prototypes that ultimately get baked back into the product. She may design experiments, and she is a critical part of data-driven decision making. She’ll communicate with team members, engineers, and leadership in clear language and with data visualizations so that even if her colleagues are not immersed in the data themselves, they will understand the implications." (Rachel Schutt,Doing Data Science: Straight Talk from the Frontline", 2013)

"A mockup shows what we should expect to take away from a project. In contrast, an argument sketch tells us roughly what we need to do to be convincing at all. It is a loose outline of the statements that will make our work relevant and correct. While they are both collections of sentences, mockups and argument sketches serve very different purposes. Mockups give a flavor of the finished product, while argument sketches give us a sense of the logic behind the solution." (Max Shron, "Thinking with Data: How to Turn Information into Insights", 2014)

"Keep in mind that a mockup is not the actual answer we expect to arrive at. Instead, a mockup is an example of the kind of result we would expect, an illustration of the form that results might take. Whether we are designing a tool or pulling data together, concrete knowledge of what we are aiming at is incredibly valuable. Without a mockup, it’s easy to get lost in abstraction, or to be unsure what we are actually aiming toward. We risk missing our goals completely while the ground slowly shifts beneath our feet. Mockups also make it much easier to focus in on what is important, because mockups are shareable. We can pass our few sentences, idealized graphs, or user interface sketches off to other people to solicit their opinion in a way that diving straight into source code and spreadsheets can never do." (Max Shron, "Thinking with Data: How to Turn Information into Insights", 2014)

"Creating mockups to communicate is not intrinsically a bad idea. But, as we are subject to confirmation bias, there’s always a risk that we will stop at our first design attempt and become reluctant to ask if there are better ways to achieve the same goals. Making these first ideas very detailed; putting them into a document; and especially blessing that document with the label 'requirements' are all moves which make further revision less likely, and put us more at risk from confirmation bias." (Laurent Bossavit, "The Leprechauns of Software Engineering", 2015)

🖍️Alessandro Negro - Collected Quotes

"A graph is a simple and quite old mathematical concept: a data structure consisting of a set of vertices (or nodes/points) and edges (or relationships/lines) that can be used to model relationships among a collection of objects." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"A widely adopted technique for solving the data sparsity issue and the cold-start problem is based on graph representation, navigation, and processing. Graph navigation methods (like the pathfinding example [...]) and graph algorithms (such as PageRank) are applied to fill some gaps and create a denser representation of [a] dataset."

"Deep learning approaches the problem of representation learning by introducing representations that are expressed in terms of other, simpler representations. In deep learning, the machine builds multiple levels of increasing complexity over the underlying simpler concepts." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"GNNs are capable of generating representations of nodes that depend on the structure of the graph as well as on any feature information we have. These features could be nodes’ properties, relationship types, and relationship properties. That’s why GNNs could drive the final tasks to better results. These embeddings represent the input for tasks such as node classification, link prediction, and graph classification." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"[...] graphs are extremely useful for encoding information, and data in graph format is increasingly plentiful. In many areas of machine learning - including natural language processing, computer vision, and recommendations - graphs are used to model local relationships between isolated data items (users, items, events, and so on) and to construct global structures from local information. Representing data as graphs is often a necessary step (and at other times only a desirable one) in dealing with problems arising from applications in machine learning or data mining." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"Graphs are powerful structures useful not only for representing connected information, but also for supporting multiple types of analysis. Their simple data model, consisting of two basic concepts such as nodes and relationships, is flexible enough to store complex information. If you also store properties in nodes and relationships, it is possible to represent practically everything of any size." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"Graphs are useful for representing how things are either physically or logically linked in simple or complex structures. A graph in which we assign names and meanings to the edges and vertices becomes what is known as a network. In these cases, a graph is the mathematical model for describing a network, whereas a network is a set of relations between objects, which could include people, organizations, nations, items found in a Google search, brain cells, or electrical transformers." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"Graphs can be used to model and analyze the relationships between entities as well as their properties. This aspect brings an additional dimension of information that graph-powered machine learning can harness for prediction and categorization. The schema flexibility provided by graphs also allows different models to coexist in the same dataset." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"Graphs can support machine learning by doing what they do best: representing data in a way that is easily understandable and easily accessible. Graphs make all the necessary processes faster, more accurate, and much more effective. Moreover, graph algorithms are powerful tools for machine learning practitioners. Graph community detection algorithms can help identify groups of people, page rank can reveal the most relevant keywords in a text, and so on." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"Graphs, with multiple node types and different types of relationships, are far from being a Euclidean space. This task is where graph neural networks (GNNs) comes in. GNNs are deep learning-based methods that operate on a graph domain to perform complex tasks such as node classification (the bot example), link prediction (the disease example), and so on. Due to its convincing performance, GNN has become a widely applied graph analysis method." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"In many areas of machine learning, graphs are used to model local relationships between data elements and to build global structures from local information. Building graphs is sometimes necessary for dealing with problems arising from applications in machine learning or data mining, and at other times, it's helpful for managing data. It's important to note that the transformation from the original data to a graph data representation can always be performed in a lossless manner. The opposite is not always true." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"One of the main goals of machine learning is to make sense of data and deliver some sort of predictive capability to the end user ([...] data analysis in general aims at extracting knowledge, insights, and finally wisdom from raw data sources, and prediction represents a small portion of possible uses). In this learning path, data visualization plays a key role because it allows us to access and analyze data from a different perspectiv." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"The performance of machine learning algorithms, both in terms of accuracy and speed, is affected almost directly from the way in which we represent our training data and store our prediction model. The quality of algorithm prediction is as good as the quality of the training dataset. Data cleansing and feature selection, among other tasks, are mandatory if we would like to achieve a reasonable level of trust in the prediction. The speed at which the system provides prediction affects the usability of the entire product." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

"To compute the similarity between items, we must define a similarity measure. Cosine similarity is the standard metric in item-based recommendation approaches: it determines the similarity between two vectors by calculating the cosine of the angle between them In machine learning applications, this measure is often used to compare two text documents, which are represented as vectors of terms [...] prediction represents a small portion of possible uses). In this learning path, data visualization plays a key role because it allows us to access and analyze data from a different perspective." (Alessandro Negro, "Graph-Powered Machine Learning", 2021)

🖍️Aldo Marzullo - Collected Quotes

"[...] a graph is a mathematical model that is used for describing relationships between entities. However, each complex network presents intrinsic properties. Such properties can be measured by particular metrics, and each measure may characterize one or several local and global aspects of the graph." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"A point of caution when dealing with projections: be aware of the dimension of the projected graph.  In certain cases, such as the one we are considering here, projection may create extremely large numbers of edges, which makes the graph hard to be analyzed." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"A small-world network is characterized by a high clustering coefficient and a short average path length, meaning that most nodes can be reached from any other node through a small number of intermediate connections. This structure often mirrors real-world social networks, where individuals are typically connected through a few mutual acquaintances, allowing for rapid information dissemination." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Although creating simple subgraphs and merging them is a way to generate new graphs of increasing complexity, networks may also be generated by means of probabilistic models   and/or generative models that let a graph grow by itself. Such graphs usually share   interesting properties with real networks and have long been used to create benchmarks and synthetic graphs, especially in times when the amount of data available was not as overwhelming as today." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"As with many other deep learning-based approaches, another major challenge is in interpretability. While knowledge graphs provide a structured and transparent way to store relationships, LLMs operate as a black box, making it difficult to understand how specific outputs are generated. [...] Data alignment is also a key issue, as structured knowledge graphs and unstructured text data must be carefully preprocessed to ensure consistency.  Differences in data formats, ontology mismatches, and information redundancy can create inefficiencies when integrating these two paradigms. Developing robust pipelines that seamlessly connect graph-based insights with LLM-generated text remains an open challenge." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Assortativity is used to quantify the tendency of nodes being connected to similar nodes, which can impact the network’s ability to withstand failures or 'attacks'. High assortativity indicates that nodes of similar degrees are more likely to be connected, leading to a resilient structure where the failure of some nodes does not significantly disrupt overall connectivity. Conversely, networks with low assortativity tend to have nodes connecting with dissimilar degrees, making them more vulnerable to targeted attacks on high-degree nodes [...]" (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Besides allowing us to compress a sparse representation into a denser vector, autoencoders are also widely used to process a signal in order to filter out noise and extract only a relevant (characteristic) signal. This can be very useful in many applications, especially when identifying anomalies and outliers." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Despite their impressive capabilities, LLMs are not without limitations. One of the most significant challenges is the problem of hallucination, where an LLM generates factually incorrect or misleading information that appears plausible. This is particularly problematic in domains requiring high factual accuracy, such as healthcare, finance, and legal applications. To mitigate hallucinations and enhance the reliability of LLM outputs,  Retrieval-Augmented Generation (RAG) has emerged as a powerful technique. RAG works by dynamically retrieving relevant information from an external knowledge source (such as a knowledge graph) at inference time, rather than just relying on pre-trained knowledge. This approach ensures that the model has access to up-to-date and accurate data, grounding answers in verified information rather than generating content purely from its internal representations." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Generally speaking, all the unsupervised embedding algorithms based on matrix factorization use the same principle. They all factorize an input graph expressed as a matrix in different components. The main difference between each method lies in the loss function used during the optimization process and the constraints/formulations posed for the V and H matrices. Indeed, different loss functions allow the creation of an embedding space that emphasizes specific properties of the input graph." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Graph analytics is generally very effective in clustering users, merchants, and communities to provide an effective implementation of behavior analytics. On the other hand, second-party fraud can be identified with the implementation of monitoring employee behavior, as well as compliance checks. Graph analytics can indeed be useful for these use cases. Similar to the first-party models, employee behavior can also be analyzed using graph machine learning, although the dataset may need to encode a number of other sources of information besides transactional data. From a compliance standpoint, process mining techniques that still rely on a graph representation of the various procedural steps/pathways can be effective in identifying fraudulent behavior or non-compliant processes. Finally, third-party fraud, especially in the form of phishing attacks, can also be addressed using graph machine learning. In this context, understanding the network from which the phishing attack comes as well as the URLs being used (which can also benefit from a graph representation) can be critical for building an effective phishing classifier." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Graphs are mathematical structures that are used for describing relationships between entities, and they are used almost everywhere." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"In a nutshell, training LLMs involves optimizing a large number of parameters on very large datasets. This process, known as pretraining, typically employs unsupervised learning objectives, such as predicting missing words in a sentence (masked language modeling) or forecasting subsequent words (causal language modeling). As a side effect, the pre-training phase lets the model learn and 'understand' a language, resulting in a remarkable ability to generalize across various tasks, often achieving state-of-the-art performance. LLMs have demonstrated proficiency in a diverse array of applications, reflecting their versatility and depth of language understanding. Key areas include text generation, language translation, question answering, and summarization, among many others." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Machine learning is a subset of artificial intelligence that aims to provide systems with the ability to learn and improve from data. It has achieved impressive results in many different applications, especially where it is difficult or unfeasible to explicitly define rules to solve a specific task." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Most of the complexity indeed arises from the presence of entities that appear only once or very few times, but still generate cliques within the graph. Such entities are not very informative to capture patterns and provide insights. Besides, they are possibly strongly affected by statistical variability. On the other hand, we should focus on strong correlations that are supported by larger occurrences and provide more reliable statistical results." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Overfitting is one of the main problems that affect machine learning practitioners. It can occur due to several reasons. Some of the reasons can be as follows: The dataset can be ill-defined or not sufficiently representative of the task. In this case, adding more data could help to mitigate the problem. The mathematical model used for addressing the problem is too powerful for the task. In this case, proper constraints can be added to the loss function in order to reduce the model’s 'power'. Such constraints are called regularization terms.(Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"RAG is a framework that combines the strengths of traditional information retrieval systems with the generative capabilities of LLMs. In this setup, an LLM is augmented with a retrieval component that fetches relevant information from external data sources, such as knowledge bases or databases, to produce more accurate and contextually relevant responses. This method enhances the LLM’s output by grounding it in authoritative, up-to-date information." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Reinforcement learning is used for training machine learning agents to make a sequence of decisions. The artificial intelligence algorithm faces a game-like situation, where the agent gets penalties or rewards based on the actions performed The goal of the agent is to understand how to act in order to maximize rewards and minimize penalties". (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"The concept of temporal graphs is useful in all the real-world problems that can be represented as a graph, where the nodes and edges of the graph may change over time. For example, temporal graphs are extensively applied in modeling social networks. By capturing the evolving relationships between individuals, temporal graphs enable a more accurate representation of social dynamics. This is particularly useful for predicting changes in friendships, community structures, and the information diffusion over time." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"The label spreading algorithm is another semi-supervised shallow embedding algorithm.  It was built in order to overcome one big limitation of the label propagation method: the   initial labeling. Indeed, according to the label propagation algorithm, the initial labels cannot be modified in the training process and, in each iteration, they are forced to be equal to their original value. This constraint could generate incorrect results when the initial labeling is affected by errors or noise. As a consequence, the error will be propagated in all nodes of the input graph." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"The main difference between unsupervised and supervised embedding methods essentially lies in the task they attempt to solve. Indeed, if unsupervised shallow embedding algorithms try to learn a good graph, node, or edge representation in order to understand the underlying structure, the supervised algorithms try to find the best solution for a prediction task such as node classification, label prediction, or graph classification." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

🎯Jean-Georges Perrin - Collected Quotes

"Accessibility is another key principle. It’s not enough for data to be findable; In fact, once found, a data product must also be easily accessible. Accessibility includes providing comprehensive documentation that explains how to use the data, as well as ensuring that the data can be easily integrated into various applications and workflows. A good data product should be as straightforward to use as a well-designed software application, with clear instructions and support."  (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Consider data silos. Data silos hinder data accessibility and collaboration, making it difficult to gain a holistic view and leverage the full potential of the available data. They present a real, present, and formidable challenge that almost all data practitioners experience in modern enterprises. Data silos, much like isolated islands in an immense ocean, are repositories of data that are confined within specific departments or systems, disconnected from the broader organizational data landscape. This segregation results in a fragmented data ecosystem, where valuable insights remain untapped, and the collective intelligence of the enterprise is underutilized." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Data Mesh addresses data governance challenges by advocating for a federated governance model, which positions accountability for governance with the data owners who are most knowledgeable about the data. In this model, governance is decentralized, with each domain team responsible for the governance of its data products. This approach ensures that governance decisions are made by those who have the deepest understanding of the data’s context, use, and risks. It leads to more relevant, efficient, and effective governance practices that are closely aligned with the specific needs of each domain." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Data Mesh advocates for domain-driven ownership of data, enabling individual teams to manage and share their data effectively while aligning with the overall organizational objectives. By embracing this paradigm, enterprises can gradually dismantle the barriers of data silos, paving the way for a more integrated, agile, and data-centric organizational culture." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"[...] Data Mesh introduces agility into the data landscape, emphasizing decentralized ownership, responsive data management, and collaborative cross-functional teams. Just as Agile promotes self-organizing teams, Data Mesh advocates for domain-oriented decentralized ownership, putting the power of data in the hands of individual domain teams. In an Agile context, customer collaboration involves continuous engagement with stakeholders to understand their evolving needs. Likewise, Data Mesh encourages domain teams to engage with data consumers within their organization, gathering feedback, and iterating on their data products to meet their specific requirements." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"[...] Data Mesh is, then, at its foundation, a conceptual framework in the realm of data architecture, which emphasizes decentralized data ownership and architecture. It recognizes that in large organizations, data is vast and varied, where each business domain has autonomy over its own data. By decentralizing control, it empowers individual domains to manage and make decisions about their data while maintaining a cohesive overall structure. And presumably, with this autonomy comes better, more localized, and faster decisions, which in-turn, leads to speed and agility." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Federated computational governance is essential for maintaining consistency and compatibility across the Data Mesh. It ensures that despite the decentralized nature of data ownership, there is a unified framework governing how data is managed, used, and shared. This unified approach is crucial in preventing data silos, ensuring data interoperability, and maintaining the overall integrity of the data ecosystem." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Implementing federated computational governance requires a delicate balance. It involves creating governance structures that are robust enough to ensure consistency and compliance, yet flexible enough to accommodate the unique needs and contexts of different data products. This balance is key to fostering an environment where innovation can thrive without compromising the standards and protocols essential for a cohesive data ecosystem." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"In the context of a Data Mesh, a data product is a package of data that is self-contained, self-descriptive, and oriented towards a specific business purpose or function. But, it is not just a mere collection of data, rather it is a coherent unit that provides value to its consumers, akin to a product in any other industry." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Observability extends the concept of reliability. It’s about having the ability to monitor the health and performance of the data product. By using tools to track various metrics like response times and error rates, organizations can proactively manage the data product’s health. This proactive management plays a crucial role in maintaining the product’s reliability, as it allows for the early identification and resolution of potential issues before they escalate." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"[...] scalability and maintainability are key aspects of a valuable data product. It should be capable of handling increasing volumes of data or user demands without necessitating significant redesign or rework. Alongside scalability, maintainability - the ease with which a data product can be updated, modified, or repaired - is critical for its long-term utility. This also includes the product’s ability to evolve based on user feedback and changing business needs, ensuring that it remains relevant and valuable over time." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Self-serve capability in a Data Mesh not only empowers users but also fosters a culture of innovation and agility. It enables individuals to leverage data for their specific needs, encouraging experimentation and personalized analysis. This capability reduces bottlenecks typically associated with centralized data systems, where requests for data access and analysis can slow down decision-making processes." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The fourth principle, Reuse, focuses on the ability to apply data in multiple contexts. This principle is particularly important in maximizing the value of data. By designing data products to be modular and reusable, they can be used across different projects and applications. For instance, a data product containing customer demographic information can be used by marketing teams for campaign planning, by sales teams for sales strategy development, and by product development teams for market analysis."  (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The principle of a clear boundary in a Data Mesh ensures that every data product is a well-defined entity within the larger ecosystem. This clarity prevents overlap and confusion, establishing a clear understanding of the data product’s purpose and scope. It aids in managing expectations and directs efforts and resources appropriately, ensuring that each data product can effectively fulfill its intended role." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The ramifications of data silos extend beyond mere inefficiencies; they actively hinder collaboration and innovation within an organization. When data is trapped in silos, it becomes difficult for teams to access the information they need to collaborate effectively. This lack of accessibility and visibility leads to duplicated efforts, inconsistent data practices, and a general sense of organizational disjointedness." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The role of an empowered owner is multifaceted. They are tasked with ensuring that the data product aligns with both specific business requirements and the overarching governance framework. This alignment is crucial for maintaining the integrity and usefulness of the data product, ensuring it remains a valuable asset within the organization’s data landscape." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The structure of a data product is self-contained, meaning that it includes everything necessary for its effective utilization. It adheres to strict standards of quality and governance, thereby ensuring reliability, security, and compliance with relevant regulations. This comprehensive approach makes data products a trusted and dependable resource within the organization. They are designed with user accessibility in mind, offering interfaces and documentation that are easily navigable by a wide range of users, from data experts to those with minimal technical expertise." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Usability is a key determinant of a data product’s value - if it is complex or unintuitive, its potential utility diminishes irrespective of the underlying data quality. Therefore, the design and interface of a data product should facilitate ease of use to ensure that it can be effectively employed by its target users. Somewhat related to this is interoperability - in other words it is also usable from an operations perspective. A valuable data product should not only function in isolation but also integrate seamlessly with other data products. This interoperability is vital for comprehensive analytics and insight generation, as it allows for the combination and analysis of data across various domains. Additionally, compliance with regulatory requirements and security standards is non-negotiable."  (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

⛩️Eric Broda - Collected Quotes

"Agents, unlike workflows, dynamically create their own plan to fulfill a task - they select their tools, pick execution paths, and control how they accomplish tasks. Unlike workflows, an agent has a built-in capacity to figure out how best to accomplish a task without predefined static implementation. That means the agent can decide on the fly when to perform a calculation, consult a database, or otherwise adapt its plan." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"An agent is a program powered by LLMs that can independently make decisions, plan iteratively, and execute tasks to achieve complex goals." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"An agent’s memory draws from multiple sources: the native knowledge encoded in its LLM weights, the transient information provided in its immediate context, and external repositories accessed through retrieval techniques such as retrieval-augmented generation (RAG). Together, these form a dynamic hierarchy of recall, reasoning, and adaptation that defines how an agent perceives, interprets, and acts in the world." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"An agentic mesh is an interconnected ecosystem that makes it easy for agents to find each other, collaborate, interact, and transact." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"As agent autonomy and sophistication grow, we believe that agents need to become enterprise grade. They must integrate easily into an enterprise’s technology and application landscape. They must adhere to enterprise processes - DevSecOps and MLOps, for example - that provide the rigor needed to move mission-critical applications, and soon agents, into production. They must adhere to the expectations of all enterprise applications to become discoverable, observable, operable, secure, and trustworthy." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"Context engineering is the practice of selecting, structuring, and delivering the right information to an agent’s LLM at the right time. Because LLMs do not “know” your current situation beyond the text you provide and the tokens they can access, performance depends heavily on what context you place in the prompt: instructions, constraints, facts, prior steps, and goals. Good context engineering turns a general model into a task-competent assistant by shaping what it sees and how it should reason." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"For agents, principles are especially important because agents can act autonomously and operate with minimal human oversight and often influence critical business outcomes. Well-defined agent principles steer agents toward outcomes that reflect organizational values, regulatory requirements, and ethical norms. By embedding these principles into agent design, we - society, organizations, developers - can understand and then manage risk, build stakeholder trust, and ensure that AI-driven processes remain aligned with strategic objectives." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"In an agentic mesh, agents are the core participants, designed with governance, interoperability, and trust so they can collaborate, interact, and even transact in a broader ecosystem of agents. The key distinction, however, lies between the needs of an individual agent and those of the larger ecosystem. Ecosystems exist to enable collaboration at scale, raising questions of how thousands of agents, each operating independently, can plan, execute, and deliver consistent outcomes. These are the challenges that agentic mesh is designed to address." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"LLMs enable an agent to interact with people and other agents using natural language. LLMs are designed to interpret and convert human inputs into data that can be used to plan and execute complex operations. By translating the words that humans type or speak into usable information, LLMs give agents the ability to interact with people, to reason, and to plan and execute all sorts of tasks." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

"The concept of explainability in autonomous agents is rooted in the need to understand how agents reach their decisions. Given that LLMs underpin many agent functions, their inherent nondeterminism makes it critical to have a system that clearly outlines each operational step. Briefly, nondeterminism for our purposes refers to the variability in the output of systems like LLMs, meaning that identical inputs can yield different outputs upon each execution. This variability arises from probabilistic decision-making processes embedded within these models, where multiple plausible responses exist for a given prompt." (Eric Broda & Davis Broda,"Agentic Mesh: The GenAI-Powered Autonomous Agent Ecosystem", 2026)

🎯Bennie Haelen - Collected Quotes

"A data lake is a cost-effective central repository to store structured, semi-structured, or unstructured data at any scale, in the form of files and blobs. The term 'data lake' came from the analogy of a real river or lake, holding the water, or in this case data, with several tributaries that are flowing the water (aka “data”) into the lake in real time." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data lakes have some very strong benefits. A data lake architecture enables the consolidation of an organization’s data assets into one central location. Data lakes are format agnostic and rely on open source formats, such as Parquet and Avro. These formats are well understood by a variety of tools, drivers, and libraries, enabling smooth interoperability. Data lakes are deployed on mature cloud storage subsystems, allowing them to benefit from the scalability, monitoring, ease of deployment, and low storage costs associated with these systems. Automated DevOps tools, such as Terraform, have well-established drivers, enabling automated deployments and maintenance." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data lakes require very large, scalable storage systems, like the ones typically offered in cloud environments. The storage needs to be durable and scalable and should offer interoperability with a variety of third-party tools, libraries, and drivers. Note that data lakes separate the concepts of storage and compute, allowing both to scale independently. Independent scaling of storage and compute allows for on-demand, elastic fine-tuning of resources, allowing our solution architectures to be more flexible. The ingress and egress channels to the storage systems should support high bandwidths, enabling the ingestion or consumption of large batch volumes, or the continuous flow of large volumes of streaming data, such as IoT and streaming media." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data silos often start to develop as the gap between data engineering activities and data science activities begins to grow. Data scientists frequently spend the majority of their time creating separate ETL and data pipelines that clean and transform data and prepare it into features for their models. These silos usually develop because the tools and technologies used for data engineering don’t support the same activities for data scientists." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Delta Lake brings capabilities such as transactional reliability and support for UPSERTs and MERGEs to data lakes while maintaining the dynamic horizontal scalability and separation of storage and compute of data lakes. Delta Lake is one solution for building data lakehouses, an open data architecture combining the best of data warehouses and data lakes." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Like data lakes, the lakehouse architecture leverages low-cost cloud storage systems with the inherent flexibility and horizontal scalability of those systems. The goal of a lakehouse is to use existing high-performance data formats, such as Parquet, while also enabling ACID transactions (and other features). To add these capabilities, lakehouses use an open-table format, which adds features like ACID transactions, record-level operations, indexing, and key metadata to those existing data formats. This enables data assets stored on low-cost storage systems to have the same reliability that used to be exclusive to the domain of an RDBMS. Delta Lake is an example of an open-table format that supports these types of capabilities." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"MLOps best practices include the need to reproduce and validate every stage of the ML workflow. The ability to reproduce a model reduces the risk of errors, and ensures the correctness and robustness of the ML solution. Consistent data is the most difficult challenge faced in reproducibility, and an ML model will only reproduce the exact same result if the exact same data is used. And since data is constantly changing over time, this can introduce significant challenges to ML reproducibility and MLOps." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Modern data platforms leveraging ETL for analytics will always be consumers of data as they ingest data from various data sources. And as organizations continue to collect, process, and analyze data from a growing number of data sources, the ability to swiftly handle schema evolution and data validation is a critical aspect of any data platform. [...] Delta Lake gives you flexibility to evolve a table’s schema through dynamic and explicit schema updates, while also enforcing schema validation." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Querying by timestamp makes it easy to perform time-series analysis because we can compare the data of the same table to itself at two different points in time. And while there are other ETL patterns we can follow to capture historical data and enable time-series analysis (e.g., slowly changing dimensions and change data feeds), time travel provides a quick and easy way to perform ad hoc analysis for tables that may not have these ETL patterns in place." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Schema evolution in Delta Lake refers to the ability to evolve the schema of a Delta table over time, while preserving the existing data in the table. In other words, schema evolution allows us to add, remove, or modify columns in an existing Delta table without losing any data or breaking any downstream jobs that depend on the table. This is important as your data and business needs change over time and you may need to add new columns to your table or modify the existing columns to support new use cases." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"The main goal of the transaction log is to enable multiple readers and writers to operate on a given version of a dataset file simultaneously and to provide additional information, like data skipping indexes to the execution engine for more performant operations. The Delta Lake transaction log always shows the user a consistent view of the data and serves as a single source of truth. It is the central repository that tracks all changes the user makes to a Delta table." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"The lakehouse is centered around the idea of unification and combining the best elements of different technologies in a single place. This means it is also important that the data flow within the lakehouse itself supports this unification of data. In order to support all use cases, this data flow requires merging batch and streaming data into a single data flow to support scenarios across the entire data lifecycle."(Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Traditionally, data lakes have always operated under the principle of schema on read, but have always had challenges enforcing schema on write. This means there is no predefined schema when data is written to storage, and a schema is only adapted when the data is processed. It is imperative for the case of analytics and data platforms that your table formats enforce the schema on write to prevent introducing change-breaking processes, and to maintain proper data quality and integrity." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Unlike data warehouses, data lakes support all data types, including semi-structured and unstructured data, enabling workloads such as media processing. Because of their high throughput ingress channels, they are very well suited for streaming use cases, such as ingesting IoT sensor data, media streaming, or web clickstreams. However, as data lakes become more popular and widely used, organizations started recognizing some challenges with traditional data lakes. While the underlying cloud storage is relatively inexpensive, building and maintaining an effective data lake requires expert skills, resulting in high-end staffing or increased consulting services costs." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Unstructured and semi-structured data are often critical for AI and machine learning use cases, whereas structured and semi-structured data are critical for BI use cases. Because it natively supports all three types of data classifications, you can create a unified system that supports these diverse workloads in a data lake. These workloads can complement each other in a well-designed processing architecture [...]. A data lake helps solve many of the challenges related to data volumes, types, and cost, and while Delta Lake runs on top of a data lake, it is optimized to run best on a cloud data lake." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"When changing clustered columns, liquid clustering does not require the entire table to be rewritten. This clustering evolution is due to the dynamic data layout feature of liquid clustering and offers a significant advantage over partition features mentioned earlier in the chapter. Traditional partitioning is a fixed data layout and does not support changing how a table is partitioned without having to rewrite the entire table. This clustering evolution can be essential as query patterns for a table can often change over time, and this allows you to dynamically adapt to new query patterns without any significant overhead or challenges." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"When you leverage Delta Lake with Structured Streaming, you get both the transactional guarantees of Delta Lake and the powerful programming model of Apache Spark Structured Streaming. With Delta Lake, you can now use Delta tables as both streaming sources and sinks, enabling a continuous processing model that processes your data through the Raw, Bronze, Silver, and Gold data lake layers in a streaming fashion, eliminating the need for batch jobs, resulting in a simplified solution architecture."(Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"With cloud data lakes you typically pay for what you use, so your costs always align with your data volumes. Since there is only a single storage layer, less data movement across different systems, availability settings, and decoupled storage versus compute, you have isolated and minimized costs for just data storage. For greater cost allocation, most cloud data lakes offer buckets, or containers (filesystems, not to be confused with application containers), to store different layers of the data (e.g., raw versus transformed data). These containers allow you to have finer-grained cost allocation for different areas of your organization. Since data sources and volumes are growing exponentially, it is extremely important to allocate and optimize costs without limiting the volume or variety of data that can be stored." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

🎯Fadi Maali - Collected Quotes

"A common mistake when implementing a data catalog is to focus only on technical metadata. This limits its use and the potential value. It also excludes business users who have valuable related input or need to use the catalog. A catalog should in fact function as a two-way translation layer between technical and business users." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"A core premise of data mesh is federating data ownership among domain data owners who are responsible for their data as a product. Offering the data as a product requires the data to be discoverable and to have explicitly stated quality characteristics and a clearly defined access method. Such requirements are at the core of what data catalogs support. With support for data labeling, curation, and crowdsourced feedback, data catalogs are well positioned to offer data as a product. Furthermore, data catalogs support the enforcement of compliant data usage, which becomes more important when data ownership is not managed centrally." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Active governance guides users as they find and use data. A data catalog with active governance will surface compliance information about sensitive data at point of use, so as to encourage users to use canonical and high-quality data assets; it will also provide a way to ask domain experts for help. They actively help users to ensure compliant usage of data with features such as masking, which anonymizes PII for given user personas who are restricted from viewing it per the GDPR." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Build a community around the catalog. Make sure data producers, stewards, and consumers are all involved and empowered to enrich the content of the catalog. Establish a leader or a team to have clear ownership of the data catalog." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Data catalog platforms take a more holistic view to focus not only on data assets within an enterprise, but also on the surrounding ecosystem (including business and people elements). They are typically characterized by an extensible data model that can grow to define various assets and concepts, such as metrics, charts, AI features, and users. Data catalog platforms typically augment their data with a focus on business and users to support collaborative governance and enrichment of metadata and to interlink data with business glossaries and dictionaries. Moreover, they are architected to make them easily integrable with other systems." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Data catalogs that focus on governance are concerned mainly with controlling data access and ensuring that data is used according to defined policies; this includes external policies such as data privacy laws as well as policies defined with an enterprise. Those catalogs apply techniques to identify data assets with sensitive information and to monitor data flow and access." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Data catalogs that focus on search bring techniques and methods from information retrieval and web search engines to the data domain within enterprises. Some of those catalogs, such as Facebook Nemo, use advanced machine learning and NLP tools to provide personalized search of data within an enterprise. The search can also use data-specific signals such as usage, popularity, and freshness to rank data assets by usefulness." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Enterprises typically become interested in data catalogs when they have a specific use case or need in mind. Data governance, self-service analytics, and cloud data migration are common examples. Having a specific need or use case helps focus efforts and measure impact. However, as with other technical efforts within enterprises, it is essential to prepare for long-term sustainable success and to have a plan to maximize successful adoption." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Historically, for their analytics needs, enterprises relied upon a set of tightly coupled tools, typically provided by a single vendor. Nowadays, nearly all of the components of a traditional data warehouse are independent and interchangeable. Those independent tools can be flexibly combined to provide a modern data stack. It is common for current enterprises to have separate tools for data ingestion, data pipelines, data storage and querying, data visualization and business intelligence, and data quality. Furthermore, data can flow in the opposite direction out of the data warehouse in what is referred to as reverse extract, transform, and load (ETL)." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"In a self-service environment with multiple publishers, it’s impossible to completely avoid data redundancy and overlapping. Multiple data assets with similar content, but possibly with varying quality, will exist. A data catalog can guide users to trusted data that comes from a reliable source and is frequently used. A data catalog can also use various explicit and implicit quality signals when ranking datasets for recommendation. Some of those signals are discussed next. Furthermore, a data catalog can recommend domain experts who are automatically identified based on actual data usage." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"It is often said that data scientists and data analysts spend only 20% of their time doing data analysis work, with 80% consumed by data 'issues'. The bulk of their time is spent finding, evaluating, understanding, and preparing data before analysis can begin. A data catalog inverts this principle by enabling data analysts and data scientists to spend 20% of their time looking for data and 80% performing analysis." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Self-service BI initiatives help organizations become more data-driven and democratize access to data. But data can’t be used if it can’t be found. Search and discovery of trustworthy data is a core value of enterprise data catalogs, and the value extends well beyond business users." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

🎯Christopher Maneu - Collected Quotes

"A data lake is a distributed repository of raw and unprocessed data stored in its original format, without a predefined schema or structure. A data lake is designed to support a wide range of data types, sources, and use cases, such as exploration, discovery, and data experimentation. A data lake follows a 'schema on read' approach. Data is structured and processed only when it is accessed or consumed by a user or application (Extract, Load, Transform (ELT)). A data lake also enables data democratization, meaning data is accessible and available to anyone who needs it, without barriers or restrictions." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"A data warehouse is a centralized repository of structured, cleaned, and verified data that has been extracted, transformed, and loaded from various sources. These steps are commonly called ETL, which stands for Extract, Transform, Load. This data processing methodology involves extracting data from multiple sources, transforming it to meet business needs, and loading it into a destination for analysis and consultation." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"A lake based on the medallion architecture combines the best of lakes and data warehouses. By breaking down silos and eliminating data duplication, it becomes a standard for building data platform architecture." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"A lakehouse is a data storage space that hosts and manages all types of data in one place (structured, semi-struc-tured, and unstructured), allowing different tools to normalize and examine this data according to organizational requirements and/or individual choices. A lakehouse thus combines the best aspects of a data lake and a data warehouse by eliminating data duplication and friction related to ingestion, transformation, and sharing of data within the organization, all in the open format, Delta Lake." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Considered by many companies as the next generation of data architecture, the data mesh represents the natural evolution of traditional data lakes and data warehouses. While the latter are often limited by their centralized and monolithic structure, the data mesh aims to enable companies to deploy a more flexible, responsive, and massively scalable data strategy." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"[...] the data mesh architecture of Microsoft Fabric primarily supports the organization of data into domains and federated governance [...]  Hierarchizing data within OneLake by domain simplifies organizing data, allowing a data producer to easily identify where to deposit data or a data consumer to filter and discover content by functional domain. But it also enables the distribution of governance responsibilities by defining roles and responsibilities for teams in charge of specific domains."  (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Data transformation sits at the heart of every successful data platform, serving as the critical bridge between data ingestion and data consumption. While basic transformations might involve simple cleaning and formatting, advanced transformation techniques encompass complex operations such as data enrichment, sophisticated deduplication, machine learning-based predictions, and the creation of derived metrics that weren’t present in the original data sources. These processes are essential for organizations looking to extract maximum value from their data investments." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Data virtualization is a technique that allows users and applications to access and interact with data stored in multiple, physically separate locations as if it were all in one place. Instead of moving or duplicating data, virtualization creates a logical layer that connects to the original sources and presents them in a unified view. This means users can query, analyze, or combine data from different systems - cloud storage, databases, or other platforms - without needing to know where or how the data is stored." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Fabric integrates the various technologies needed for an end-to-end data project (namely, ingestion, preparation, storage, processing, enrichment, analysis, visualization, and data sharing) within a single platform accessible as Software as a Service (SaaS), meaning via a simple connection on a web browser. This reduces complexity, costs, and delays related to using multiple tools and technologies, and eliminates all the operational maintenance of infrastructure serving data analytics needs." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Fabric Pipelines provide reliable and efficient end-to-end orchestration of data flows, managing ingestion, transformation, and loading through a sequence of steps that can leverage various data processing engines. They allow centralizing and orchestrating data movements from various sources, thanks to advanced connectivity features, and with great scalability. Built-in monitoring tools enable real-time tracking of data flow status and quick detection of anomalies or errors." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Fabric relies on a lakehouse, a data storage model that combines the benefits of a data lake and a data warehouse. Within Fabric, the various data analytics and processing tools rely on a data lake that collects and stores data in its original format, whether structured, semi-structured, or unstructured, without the need to transform or normalize it beforehand. The lakehouse approach then enables converting these diverse data formats into a single format (i.e., compatible with all the data processing engines offered by Fabric) and in an open format, allowing other market vendors to interact with data in the Fabric lakehouse." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"In Fabric, a domain represents a way to logically group data corresponding to specific functional areas. Domains are frequently used to organize data by business sector in order to manage it according to each sector’s regulations, specifics, and requirements." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"It should be noted that, unlike Dataflow Gen2, in pipelines, it is not mandatory to enable staging to load data into a warehouse. Indeed, pipelines are designed for more general orchestration scenarios where you can combine various activities such as transformations, API calls, and so on to create complex workflows. They are not specifically focused on data preparation but rather on end-to-end process automation. Pipelines are more flexible and used for a variety of orchestration tasks, whereas Dataflow Gen2 is specifically designed for data preparation and transformation, hence the requirement for staging in that case." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"One of the most powerful enhancements in Real-Time Intelligence is the integration of anomaly detection capabilities, enabling systems to automatically flag unusual deviations in real time. Rather than relying on predefined thresholds or periodic audits, these AI-driven agents continuously monitor data streams, learning normal behavior patterns and surfacing outliers or unexpected shifts the moment they appear. This proactive approach transforms what was once passive reporting into active surveillance, allowing operational teams to respond instantly when something deviates from the norm." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"The hub and spoke, or 'star network', is a data architecture model that centralizes data from various sources into a single hub, such as a data warehouse or data lake. The hub serves as the source of truth for data and provides standardized schemas and formats. The spokes are the various applications or services that consume data from the hub for different purposes, such as analytics, reporting, or ma-chine learning. Spokes can also perform transformations or aggregations on data before presenting it to end users. The hub and spoke architecture aims to simplify data integration and management by reducing complexity and redundancy in data pipelines" (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"The problem with data lakes is that they have several drawbacks preventing them from being the perfect or ideal solution. The first drawback is an organizational problem: (•) How to organize data in the lake (•) How to classify, catalog, secure, document, and find it (•) How to avoid the lake turning into a swamp where data is mixed, duplicated, obsolete, or inaccessible (•) How to manage quality, governance, and traceability in the lake."(Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"The transformation phase represents the most resource-intensive stage of most data projects, often consuming 60-80% of total project time and effort. This significant investment stems from the inherent complexity of converting raw, inconsistent data into clean, structured, and enriched information ready for business use. Every data quality issue must be identified and resolved, every business rule must be correctly implemented, and every integration point must be properly validated. This meticulous work serves as the essential bridge between raw data ingestion and meaningful business insights." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"This transition to OneDrive highlights the importance of governance adapted to new methods of collaborative work and data sharing. The idea of OneLake is, therefore, based on this same concept: rather than subscribing to a data lake technology that must be maintained, why not simply subscribe to a storage service that offers a layer of abstraction over the complexities of these data storage infrastructures? As a result, the data lake becomes a controlled or governed environment, but still accessible to users who can view it as a simple and intuitive way to securely share data with their colleagues and IT teams."(Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Traditionally, data engineers are responsible for the first steps of data transformation, commonly referred to as the transition from the 'bronze' stage to the 'silver' stage. This phase includes the normalization of raw data to clean and organize it into a structured and accessible format. Data Engineers ensure that data is properly ingested, stored, and prepared for subsequent steps. Their work focuses on building robust data pipelines and applying basic transformations that make the data usable. Next, responsibility may be handed over to an analytics engineer, who takes charge of the transition from the 'silver' stage to the 'gold' stage. This step involves more complex transformations aimed at refining, enriching, and modeling the data to meet specific analytical needs. The analytics engineer ensures that the data is ready to be used in reports, dashboards, and advanced analyses. The transition to the 'gold' stage means that the data is fully prepared for analytic use, providing strategic insights from consolidated data sources." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"We are now witnessing the rise of a new paradigm in technology, the age of agentic AI, where intelligence moves beyond automation and prediction to autonomy and intent. In this new world, operations across industries are no longer passive systems waiting for human input or post-event analysis. Instead, they have evolved into dynamic ecosystems of intelligence, continuously learning from every signal that flows through the organization. [...] Agentic AI marks the fourth great evolution of software, after client-server, cloud, and SaaS - and perhaps the most transformative of all. It represents the moment when technology stops being a tool we use and becomes a collaborator that thinks, learns, and acts alongside us." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"While Fabric provides all the traditional tools that data specialists use daily to work on data integration and processing projects, it also offers new intuitive interfaces to enable business users, citizen analysts, or business analysts to interact with their data regardless of their skill level. The primary goal is to meet the needs and expectations of these users, who often do not benefit from data analytics and processing tools because they are too complex to use, even though they are themselves the main consumers and producers of data within organizations." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"With Fabric, organizations can unlock the full potential of AI and machine learning in their data workflows. First, it provides users with all the tools necessary to create and deploy AI and machine learning models; users can use the frameworks and languages of their choice. Next, it enables these users to benefit from native integration of models that enrich the data present within Fabric with advanced cognitive analytics, such as vision and language, for example, and leverage the new capabilities of generative AI. Finally, it supports users at every stage of their data project with intelligent assistants that help create data integration flows, develop transformations or analyses, build data visualization reports, and even answer business questions by leveraging existing reports and semantic models to deliver contextual insights instantly." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

🤖〽️Prompt Engineering: Copilot Unabridged (Part 212: How Multi‑Modal Stressors Enable Holistic Evaluation Through Incomplete or Corrupted Inputs in AI Models)

Prompt Engineering Series

Prompt: "write a post of 600 words on how to use multi‑modal stressors for holistic evaluation in which stress testing reflects the complexity through incomplete or corrupted inputs in AI models"

Introduction

As Artificial Intelligence (AI) systems expand into multi‑modal architectures - processing text, images, audio, diagrams, tables, and code - their vulnerabilities become more complex. Real‑world environments rarely present clean, perfectly aligned inputs. Instead, models must interpret incomplete, corrupted, or partially contradictory signals across modalities. This is where multi‑modal stressors become essential. By deliberately introducing degraded or inconsistent inputs, evaluators can observe how the model prioritizes signals, how it compensates for missing information, and where its reasoning begins to break down.

Incomplete or corrupted inputs matter because each modality activates different representational pathways. Text relies on linguistic priors; images rely on spatial embeddings; audio relies on temporal patterns; code relies on structural logic. When one modality is degraded, the model must decide whether to rely more heavily on the remaining modalities or attempt to reconstruct the missing information. That decision exposes its internal hierarchy of cues, a central theme in instruction‑priority testing.

One of the simplest multi‑modal stressors is the partially corrupted image. For example, an image may be blurred, occluded, or missing key regions, while the accompanying text describes a scene that may or may not match the visible content. This tests whether the model over‑trusts visual fragments or defaults to textual interpretation. The result reveals how the model resolves conflicts between incomplete sensory input and linguistic cues - an essential capability for real‑world robustness.

A more advanced technique involves cross‑signal incompleteness, where each modality is missing different pieces of information. For example:

• The text describes an event but omits the key actor.
• The image shows the actor but hides the action.
• The audio clip provides environmental noise but no speech.

The model must integrate these partial signals to form a coherent interpretation. This exposes whether the model can perform multi‑modal reconstruction, or whether it collapses into hallucination or over‑generalization - patterns often surfaced through weak‑point analysis.

Another powerful stressor is corrupted‑modality contradiction, where the corruption itself creates misleading cues. For example, a distorted audio clip may sound angry even though the text describes a calm conversation. Or a corrupted diagram may misalign labels, contradicting the accompanying explanation. These stressors force the model to determine whether the corruption is noise or signal. The model’s behavior reveals whether it can distinguish reliable from unreliable modalities, a key insight for holistic evaluation.

Incomplete inputs can also be used to test temporal resilience. A video clip may drop frames, skip segments, or freeze mid‑action, while the text describes a continuous sequence. The model must decide whether to trust the visual timeline or the textual narrative. This exposes how the model handles temporal reasoning, a capability often overlooked in single‑modality evaluation.

The most challenging multi‑modal stressors involve hybrid corrupted inputs, where multiple modalities degrade in different ways. For example:

• A table with missing values contradicts a narrative summary.
• A diagram with corrupted labels conflicts with a code snippet.
• An audio clip with static obscures key words while the text misidentifies the speaker.

These hybrid contradictions push the model into conceptual regions where no training example exists. The resulting behavior reveals the model’s cross‑modal arbitration strategy, a crucial insight for understanding its robustness.

Ultimately, multi‑modal stressors that use incomplete or corrupted inputs allow evaluators to move beyond surface‑level robustness. By introducing degradation across text, images, audio, diagrams, and structured data, we can map the deep architecture of model reasoning - how it prioritizes modalities, how it compensates for missing information, and where its internal logic becomes unstable. This is the next frontier of boundary‑stress evaluation: not just testing what the model can do, but testing how it behaves when the world becomes noisy, partial, and imperfect.

Disclaimer: The whole text was generated by Copilot (under Windows 11) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post