🪙Business Intelligence: Problem Solving (Just the Quotes)

"If we view organizations as adaptive, problem-solving structures, then inferences about effectiveness have to be made, not from static measures of output, but on the basis of the processes through which the organization approaches problems. In other words, no single measurement of organizational efficiency or satisfaction - no single time-slice of organizational performance can provide valid indicators of organizational health." (Warren G Bennis, "General Systems Yearbook", 1962)

"90 percent of all problems can be solved by using the techniques of data stratification, histograms, and control charts. Among the causes of nonconformance, only one-fifth or less are attributable to the workers." (Kaoru Ishikawa, The Quality Management Journal Vol. 1, 1993)

"Delay time, the time between causes and their impacts, can highly influence systems. Yet the concept of delayed effect is often missed in our impatient society, and when it is recognized, it’s almost always underestimated. Such oversight and devaluation can lead to poor decision making as well as poor problem solving, for decisions often have consequences that don’t show up until years later. Fortunately, mind mapping, fishbone diagrams, and creativity/brainstorming tools can be quite useful here." (Stephen G Haines, "The Manager's Pocket Guide to Strategic and Business Planning", 1998)

"Mostly we rely on stories to put our ideas into context and give them meaning. It should be no surprise, then, that the human capacity for storytelling plays an important role in the intrinsically human-centered approach to problem solving, design thinking." (Tim Brown, "Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation", 2009)

"Even with simple and usable models, most organizations will need to upgrade their analytical skills and literacy. Managers must come to view analytics as central to solving problems and identifying opportunities - to make it part of the fabric of daily operations." (Dominic Barton & David Court, "Making Advanced Analytics Work for You", 2012)

"Data warehousing, as we are aware, is the traditional approach of consolidating data from multiple source systems and combining into one store that would serve as the source for analytical and business intelligence reporting. The concept of data warehousing resolved the problems of data heterogeneity and low-level integration. In terms of objectives, a data lake is no different from a data warehouse. Both are primary advocates of terms like 'single source of truth' and 'central data repository'." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"In the world of data and analytics, people get enamored by the nice, shiny object. We are pulled around by the wind of the latest technology, but in so doing we are pulled away from the sound and intelligent path that can lead us to data and analytical success. The data and analytical world is full of examples of overhyped technology or processes, thinking this thing will solve all of the data and analytical needs for an individual or organization. Such topics include big data or data science. These two were pushed into our minds and down our throats so incessantly over the past decade that they are somewhat of a myth, or people finally saw the light. In reality, both have a place and do matter, but they are not the only solution to your data and analytical needs. Unfortunately, though, organizations bit into them, thinking they would solve everything, and were left at the alter, if you will, when it came time for the marriage of data and analytical success with tools." (Jordan Morrow, "Be Data Literate: The data literacy skills everyone needs to succeed", 2021)

"Data warehousing, as we are aware, is the traditional approach of consolidating data from multiple source systems and combining into one store that would serve as the source for analytical and business intelligence reporting. The concept of data warehousing resolved the problems of data heterogeneity and low-level integration. In terms of objectives, a data lake is no different from a data warehouse. Both are primary advocates of terms like 'single source of truth' and 'central data repository'." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"A data pipeline is an artifact of a data engineering process. It transforms raw data into data ready for analytics. These in turn help solve problems, aid support decisions, and make our lives more convenient. In some ways, it can be thought of as the stitch between the OLTP and OLAP systems. Data pipelines are sometimes referred to as ETL, which stands for extract, transform, load, and it has a variation called extract, load, transform (ELT). The main difference between the two is whether the incoming data is first saved to disk and then transformed (data wrangling) or vice versa. The processing is loosely referred to as ETL. Although, it is fair to say ELT is relevant in the context of Data Lakes and unstructured data, whereas ETL is used for Data Warehouses." (Anindita Mahapatra, "Simplifying Data Engineering and Analytics with Delta", 2022)

"Conversational repair is the process people use to detect and resolve problems in communicating, receiving, and understanding. Through repair, participants in social interaction display how they establish and maintain communication and mutual understanding. Language interpretation formalizes multiple levels of repair, from monitoring and evaluating various benchmarks of accuracy to proper ways to intervene and seek clarification." (Vidya Setlur & Bridget Cogley, "Functional Aesthetics for data visualization", 2022)

"Unstructured and semi-structured data are often critical for AI and machine learning use cases, whereas structured and semi-structured data are critical for BI use cases. Because it natively supports all three types of data classifications, you can create a unified system that supports these diverse workloads in a data lake. These workloads can complement each other in a well-designed processing architecture [...]. A data lake helps solve many of the challenges related to data volumes, types, and cost, and while Delta Lake runs on top of a data lake, it is optimized to run best on a cloud data lake." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Domain-driven design (DDD) is the methodology that helps us understand complex domain models by connecting the data model itself to core business concepts. The understanding that emerges from DDD creates a foundation to designing distributed, microservice-based, client-facing applications. DDD connects the implementation of software and its components to an evolving and ever-changing data model. The domain is the world of the business you are working with and the problems you are trying to solve. This typically involves rules, processes, and existing systems that need to be integrated as part of your solution." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"While a data mesh seeks to solve many of the same problems that a data fabric addresses - namely, the ability to address data in a single, composite data environment—the approach is different. While a data fabric enables users to create a single, virtual layer on top of distributed data, a data mesh further empowers distributed groups of data producers to manage and publish data as they see fit. Data fabrics allow for a low-to-no-code data virtualization experience by applying data integration within APIs that reside within the data fabric. The data mesh, however, allows for data engineers to write code for APIs with which to interface further. Without clearly defined boundaries, domains appear to be too interconnected, and ownership becomes either political or subject to interpretation. For instance, a large retailer most likely has multiple domains. [...]" (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"The transformation phase represents the most resource-intensive stage of most data projects, often consuming 60-80% of total project time and effort. This significant investment stems from the inherent complexity of converting raw, inconsistent data into clean, structured, and enriched information ready for business use. Every data quality issue must be identified and resolved, every business rule must be correctly implemented, and every integration point must be properly validated. This meticulous work serves as the essential bridge between raw data ingestion and meaningful business insights." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Generative AI is essential because it can make creative processes quicker and better, which saves time and money and pushes the limits of what machines can do. For example, it can transform text descriptions into realistic pictures, write articles, make music, or even code software. This technology is changing how content is made and making it possible to have personalized experiences like custom learning materials or marketing campaigns. Generative AI is also a big part of the progress that is being made in other areas of AI, like computer vision and natural language processing (NLP). This is why it is an important tool for solving hard problems in the real world." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Generative artificial intelligence (AI) is a type of AI that learns from existing data and uses that knowledge to make new things, like text, images, music, or code. Generative models make new data points that look like the training data, while discriminative models only focus on classifying or predicting outcomes. This ability is game-changing because it lets machines copy how people are creative and solve problems in ways that were thought to be impossible before. Generative AI is a key part of modern AI applications and is pushing new ideas in fields like healthcare, entertainment, education, and more." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"The value of reinforcement learning lies in its ability to solve highly complex problems by mimicking human learning processes. The model undergoes continuous learning and improvement through interaction with its environment, making it highly accurate over time. Reinforcement learning is especially valuable when applied to tasks like robotics, autonomous driving, and game playing, where the agent constantly learns from new experiences. It also integrates well with neural networks, creating the foundation for deep reinforcement learning, which allows models to handle even more intricate tasks." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

🤖Prompt Engineering: Knowledge (Just the Quotes)

"A major advantage of probabilistic models is that they can be easily applied to virtually any data type (or mixed data type), as long as an appropriate generative model is available for each mixture component. [...] A downside of probabilistic models is that they try to fit the data to a particular kind of distribution, which may often not be appropriate for the underlying data. Furthermore, as the number of model parameters increases, over-fitting becomes more common. In such cases, the outliers may fit the underlying model of normal data. Many parametric models are also harder to interpret in terms of intensional knowledge, especially when the parameters of the model cannot be intuitively presented to an analyst in terms of underlying attributes. This can defeat one of the important purposes of anomaly detection, which is to provide diagnostic understanding of the abnormal data generative process." (Charu C Aggarwal, "Outlier Analysis", 2013)

"Fine-tuning involves training the LLM on a smaller, task-specific dataset to adjust its parameters for the specific task at hand. This allows the LLM to leverage its pre-trained knowledge of the language to improve its accuracy for the specific task. Fine-tuning has been shown to drastically improve performance on domain-specific and task-specific tasks and lets LLMs adapt quickly to a wide variety of NLP applications." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"Transfer learning is a technique used in machine learning to leverage the knowledge gained from one task to improve performance on another related task. Transfer learning for LLMs involves taking an LLM that has been pre-trained on one corpus of text data and then fine-tuning it for a specific 'downstream' task, such as text classification or text generation, by updating themodel’s parameters with task-specific data." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"Transfer learning is a technique used in machine learning to leverage the knowledge gained from one task to improve performance on another related task. Transfer learning for LLMs involves taking an LLM that has been pre-trained on one corpus of text data and then fine-tuning it for a specific 'downstream' task, such as text classification or text generation, by updating the model’s parameters with task-specific data. The idea behind transfer learning is that the pre-trained model has already learned a lot of information about the language and relationships between words, and this information can be used as a starting point to improve performance on a new task. Transfer learning allows LLMs to be fine-tuned for specific tasks with much smaller amounts of task-specific data than would be required if the model were trained from scratch. This greatly reduces the amount of time and resources needed to train LLMs." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"An important capability for our LLM app to become smarter is to maintain a working memory of its interactions with us - otherwise, it will approach each prompt with no knowledge of our previous interactions. For example, it won’t remember details like where we live or what our interests are, which would make it more challenging to develop useful LLM assistants that can use personal information about us to provide more engaging, relevant responses. It also makes it practically more challenging to code a personalized application if we have to explicitly pass context for this personalized information with each interaction, rather than maintaining it 'for free' through LangChain’s memory functionality. It can also allow us to make the LLM specialized for different users by maintaining different memories on different 'threads' that we can visualize and retrieve from LangSmith."(Joseph Babcock & Raghav Bali, "Generative AI with Python and PyTorch" 2nd. Ed., 2025)

"Context is crucial for how language models understand and generate code. The model processes your input by analyzing relationships between different parts of the code and documentation to determine meaning and intent. [...] The model evaluates context by calculating mathematical relationships between elements in your input. However, it may miss important domain knowledge, coding standards, or architectural patterns that experienced developers understand implicitly." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"Despite their impressive capabilities, LLMs are not without limitations. One of the most significant challenges is the problem of hallucination, where an LLM generates factually incorrect or misleading information that appears plausible. This is particularly problematic in domains requiring high factual accuracy, such as healthcare, finance, and legal applications. To mitigate hallucinations and enhance the reliability of LLM outputs,  Retrieval-Augmented Generation (RAG) has emerged as a powerful technique. RAG works by dynamically retrieving relevant information from an external knowledge source (such as a knowledge graph) at inference time, rather than just relying on pre-trained knowledge. This approach ensures that the model has access to up-to-date and accurate data, grounding answers in verified information rather than generating content purely from its internal representations." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"LLMs excel at understanding context and making associations among words, phrases, and concepts to provide relevant information based on the input query or prompt. While structured knowledge bases rely on humancurated data, LLMs can  automatically extract knowledge from unstructured text. When trained on diverse textual sources, they can process a vast amount of information without explicit human intervention. However, this also introduces a challenge, as the model can learn biased or incorrect information from the training data." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"Semantic Kernel is a framework designed to simplify integrating LLMs into applications that require dynamic knowledge, reasoning, and state tracking. It’s particularly useful when you want to build complex, modular AI systems that can interact with external APIs, knowledge bases, or decision-making processes. Semantic Kernel focuses on building more flexible AI systems that can handle a variety of tasks beyond just generating text. It allows for modularity, enabling developers to easily combine different components - such as embeddings, prompt templates, and custom functions - in a cohesive manner." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"A 'hallucination' in the context of generative AI refers to the phenomenon where a model produces information that is factually incorrect, nonsensical, or not grounded in its input data or pre-existing knowledge. These are not mere typos or minor inaccuracies; they are confident, coherent, and often persuasive fabrications. In high-stakes domains like healthcare, law, or finance, a single hallucination can have severe consequences, eroding user trust and leading to catastrophic decision-making. While all LLMs are prone to this, the RAG architecture is specifically designed to combat it by tethering the model’s output to an external, verifiable knowledge base. Understanding why hallucinations occur is the essential first step to building more reliable and truthful AI systems." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"GenAI models are remarkably powerful, but they carry an inherent limitation rooted in how they are built. Every foundation model has a training cutoff, a point in time beyond which it has no awareness of world events, product changes, regulatory updates, or organizational developments. This knowledge gap is not a flaw that can be patched with better prompting. It’s a structural consequence of how models are trained, and it grows wider every day the model remains in production without retraining. Combined with the tendency to hallucinate when asked about topics outside their training distribution, models operating on stale knowledge can confidently deliver responses that are factually incorrect, dangerously outdated, or simply no longer relevant to the user’s context." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Generative artificial intelligence (GenAI), powered by large language models (LLMs) like Google’s Gemini and OpenAI’s GPT, has transformed how we work and live, revolutionizing business after business. Despite this success, generative AI falls short in domains where specific domain knowledge, high accuracy, and explainability are essential. And it has other significant limitations, including hallucinations and a lack of context and relations. This is where knowledge graphs (KGs) come in, provid-ing contextual information - such as experiences, environmental characteristics, cultural aspects, and social normsneeded to build the 'third wave of AI' for mission-critical applications." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026) 

"KGs are sophisticated graph structures that represent real-world entities (people, places, diseases, proteins), define meaningful connections between them, and provide context. KGs provide structured, explainable knowledge representation but are challenging to build and query; LLMs offer natural language processing capabilities but suffer from hallucinations, stale information, and a lack of domain-specific grounding. Together, they are a 'killer combination': LLMs can extract entities and relationships from unstructured text to build KGs more efficiently, providing more autonomous and powerful graph querying and analysis. Meanwhile, KGs provide reliable, up-to-date domain knowledge to ground LLM responses and prevent hallucinations." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"Knowledge representation is the 'language' that AI systems use to structure, encode, and communicate information about the domain they are modelling. AI systems need it to reason or predict, and we use it to interpret the results they produce. The choice of how to represent knowledge can significantly affect the capabilities and limitations of the AI system." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"Intelligent systems connect users to AI and ML to achieve meaningful objectives. An intelligent system is one in which intelligence evolves and improves over time, particularly when it improves by watching how users interact with the system.[...] The primary objective of the intelligent system is to support users in accomplishing complex tasks - not by replacing them, but by enhancing their decision-making capabilities. [...] An intelligent system must also have the ability to learn from user interactions and explicit feedback, as well as utilize contextual information. The system should contin-uously develop, use, and maintain an evolving knowledge base. This evolution is driven not only by data sources but also by ongoing interactions with users." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"RAG is a framework that combines the strengths of generative models and retrieval mechanisms to produce more accurate and contextually relevant outputs. The process begins with an input query or prompt, which is used to retrieve relevant information from an external knowledge source, such as a database, document repository, or the internet. This retrieval step ensures that the model has access to up-to-date and verified information, addressing the limitations of traditional generative models that rely solely on pretrained knowledge." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"RAG models offer several advantages over standard generative models, addressing many of their limitations. Traditional generative models, like GPT, rely solely on patterns learned during training, which can lead to issues such as factual inaccuracies, model hallucinations, and lack of contextual relevance. These models generate content based on pre-existing knowledge, often without the ability to verify or update the information, making them less reliable for tasks requiring high accuracy. In contrast, RAG models integrate retrieval mechanisms that allow them to access external knowledge sources in real time. This ensures that the generated content is grounded in verified data, significantly improving factual consistency and relevance. For example, while a standard generative model might produce a plausible but incorrect answer to a factual question, a RAG model can retrieve and incorporate accurate information from a trusted source, reducing errors." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

🏗️Software Engineering: Metrics (Just the Quotes)

"Encapsulation is the key to increasing the value of such software metrics as reusability, refinability, testability, maintainability, and extensibility. Object-oriented languages provide a number of mechanisms for improving encapsulation, but it is during the design phase that the greatest leverage can be realized." (Rebecca Wirfs-Brock, "Object-oriented design: a responsibility-driven approach", 1989)

"Object-oriented programming increases the value of these metrics by managing this complexity. The most effective tool available for dealing with complexity is abstraction. Many types of abstraction can be used, but encapsulation is the main form of abstraction by which complexity is managed in object-oriented programming. Programming in an object-oriented language, however, does not ensure that the complexity of an application will be well encapsulated. Applying good programming techniques can improve encapsulation, but the full benefit of object-oriented programming can be realized only if encapsulation is a recognized goal of the design process." (Rebecca Wirfs-Brock," Object-Oriented Design: A responsibility-driven approach", 1989)

"Agile development teams often view metrics as a onetime pointer instead of a continuous measurement. The pointer makes the current state of the software’s internal quality visible. It is then up to the team to decide when to adjust the code base or whether to do so at all. These pointers provide indicators that there’s something worth investigating, but they don’t provide the context and understanding needed to make critical decisions." (Michael Hüttermann et al, "DevOps for Developers", 2013)

"Adopting a mindset of instrumentation means ensuring we have a set of dashboards that surface key health metrics and that enable us to drill down to the relevant data. However, many of the questions we want to answer tend to be exploratory, since we often don't know everything that we want to measure ahead of time. Therefore, we need to build flexible tools and abstractions that make it easy to track additional metrics." (Edmond Lau, "The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact", 2015)

"[…] an overall green status indicator doesn’t mean anything most of the time. All it says is that the things under measurement seem okay. But there always will be many more things not under measurement. To celebrate green indicators is to ignore the unknowns. […] The tendency to roll up metrics into dashboards promotes ignorance of the real situation on the ground. We forget that we only see what is under measurement. We only act when something is not green." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"But 'average cost to fix one defect' is a stupid metric [...] It makes bad projects look good, and good projects look bad. How? By failing to divide the costs of fixing into two categories: fixed costs of detecting and fixing defects - costs which are the same no matter how buggy or how good the product is - and variable costs, those which you pay for each defect." (Laurent Bossavit, "The Leprechauns of Software Engineering", 2015)

"Good metrics accomplish a number of goals. First, they help you focus on the right things. […] Second, when visualized over time, good metrics help guard against future regressions. […] Third, good metrics can drive forward progress. […] Fourth, a good metric lets you measure your effectiveness over time and compare the leverage of what you’re doing against other activities you could be doing instead." (Edmond Lau, "The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact", 2015)

"Metrics are a common adjunct to the deployment pipeline in incremental change environments. If teams use this effort as a proof-of-concept, developers should gather appropriate metrics for both before and after scenarios. Gathering concrete data is the best way to for developers to vet the approach; remember the adage that demonstration defeats discussion." (Neal Ford, "Building Evolutionary Architectures: Support Constant Change", 2017)

"When establishing our goals, it’s important to choose carefully what core metrics to measure (or not measure) and optimize. When it comes to day-to-day operations, however, you should be less discriminatory: measure and instrument as much as possible. Although these two principles may seem contradictory, they actually complement each other. The first describes a high-level, big-picture activity, whereas the second is about gaining insight into what’s going on with systems that we’ve built." (Edmond Lau, "The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact", 2015)

🪙Business Intelligence: Audience (Just the Quotes)

"If you want to hide data, try putting it into a larger group and then use the average of the group for the chart. The basis of the deceit is the endearingly innocent assumption on the part of your readers that you have been scrupulous in using a representative average: one from which individual values do not deviate all that much. In scientific or statistical circles, where audiences tend to take less on trust, the 'quality' of the average (in terms of the scatter of the underlying individual figures) is described by the standard deviation, although this figure is itself an average." (Nicholas Strange, "Smoke and Mirrors: How to bend facts and figures to your advantage", 2007)

"Any presentation of data, whether a simple calculated metric or a complex predictive model, is going to have a set of assumptions and choices that the producer has made to get to the output. The more that these can be made explicit, the more the audience of the data will be open to accepting the message offered by the presenter." (Zach Gemignani et al, "Data Fluency", 2014)

"In fact, the analogy to storytelling is limited when applied to communicating with data. Data visualization has fundamental characteristics missing from traditional storytelling. For example, interactive data visualizations let audiences explore information to find insights that resonate with them. Visualizations take shape based to a large extent on the underlying data. And as this data changes, the emphasis and message of the visualization is likely to change." (Zach Gemignani et al, "Data Fluency", 2014)

"If you simply present data, it’s easy for your audience to say, Oh, that’s interesting, and move on to the next thing. But if you ask for action, your audience has to make a decision whether to comply or not. This elicits a more productive reaction from your audience, which can lead to a more productive conversation - one that might never have been started if you hadn’t recommended the action in the first place." (Cole N Knaflic, "Storytelling with Data: A Data Visualization Guide for Business Professionals", 2015)

"Infographics combine art and science to produce something that is not unlike a dashboard. The main difference from a dashboard is the subjective data and the narrative or story, which enhances the data-driven visual and engages the audience quickly through highlighting the required context." (Travis Murphy, "Infographics Powered by SAS®: Data Visualization Techniques for Business Reporting", 2018

"Analysis is a two-step process that has an exploratory and an explanatory phase. In order to create a powerful data story, you must effectively transition from data discovery (when you’re finding insights) to data communication (when you’re explaining them to an audience). If you don’t properly traverse these two phases, you may end up with something that resembles a data story but doesn’t have the same effect. Yes, it may have numbers, charts, and annotations, but because it’s poorly formed, it won’t achieve the same results." (Brent Dykes, "Effective Data Storytelling: How to Drive Change with Data, Narrative and Visuals", 2019)

"Analysis is a two-step process that has an exploratory and an explanatory phase. In order to create a powerful data story, you must effectively transition from data discovery (when you’re finding insights) to data communication (when you’re explaining them to an audience). If you don’t properly traverse these two phases, you may end up with something that resembles a data story but doesn’t have the same effect. Yes, it may have numbers, charts, and annotations, but because it’s poorly formed, it won’t achieve the same results." (Brent Dykes, "Effective Data Storytelling: How to Drive Change with Data, Narrative and Visuals", 2019)

"First, from an ethos perspective, the success of your data story will be shaped by your own credibility and the trustworthiness of your data. Second, because your data story is based on facts and figures, the logos appeal will be integral to your message. Third, as you weave the data into a convincing narrative, the pathos or emotional appeal makes your message more engaging. Fourth, having a visualized insight at the core of your message adds the telos appeal, as it sharpens the focus and purpose of your communication. Fifth, when you share a relevant data story with the right audience at the right time (kairos), your message can be a powerful catalyst for change." (Brent Dykes, "Effective Data Storytelling: How to Drive Change with Data, Narrative and Visuals", 2019)

"As presenters of data visualizations, often we just want our audience to understand something about their environment – a trend, a pattern, a breakdown, a way in which things have been progressing. If we ask ourselves what we want our audience to do with that information, we might have a hard time coming up with a clear answer sometimes. We might just want them to know something." (Ben Jones, "Avoiding Data Pitfalls: How to Steer Clear of Common Blunders When Working with Data and Presenting Analysis and Visualizations", 2020)

"The lack of focus and commitment to color is a perplexing thing. When used correctly, color has no equal as a visualization tool - in advertising, in branding, in getting the message across to any audience you seek. Data analysts can make numbers dance and sing on command, but they sometimes struggle to create visually stimulating environments that convince the intended audience to tap their feet in time." (Kate Strachnyi, "ColorWise: A Data Storyteller’s Guide to the Intentional Use of Color", 2023)

"Data Mesh also emphasizes aligning data products with business domains and use- cases to ensure that the data serves a clear business purpose and provides tangible value. Beforehand, we define the value proposition, target audience, quality attributes, and KPIs of each data product to ensure that it meets or exceeds the expectations of its consumers." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"Data Storytelling allows you to fill the gap between simply visualizing data and communicating it to an audience. Data storytelling improves your communication skills and standardizes and simplifies the process of communicating results, making it easier for people to understand and remember information. Data storytelling also helps you learn to communicate more effectively with others, improving personal and professional relationships." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"[...] data storytelling builds compelling stories supported by data, allowing analysts and data scientists to present and share their insights interestingly and interactively. The ultimate goal of data storytelling is to engage the audience and inspire them to make decisions. In some cases, including business scenarios, you do not start from data but from a narrative or a hypothesis you have in mind and then search for data that confirm or negate it. In this case, you can still have data storytelling, but you must pay attention not to alter your data to support your hypothesis." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Data storytelling is about more than just communicating data. It's about inspiring your audience and inviting them to take action. Good data storytelling requires a mix of art and science. Art comes in finding the right story, while science understands how to use data to support that story. When done well, data storytelling can be a potent tool for change." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Data visualization is primarily concerned with presenting data in a way that is easy to understand, usually through charts and graphs. On the other hand, data storytelling is more concerned with using data to tell a story that will engage the audience and help them understand the problem. In addition, data storytelling often uses narratives to explain the data, while data visualization focuses on presenting the data itself. Anyway, good data storytelling includes visualization elements." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

🤖Prompt Engineering: Improvement (Just the Quotes)

"Attention is a mechanism used in deep learning models (not just Transformers) that assigns different weights to different parts of the input, allowing the model to prioritize and emphasize the most important information while performing tasks like translation or summarization. Essentially, attention allows a model to 'focus' on different parts of the input dynamically, leading to improved performance and more accurate results. Before the popularization of attention, most neural networks processed all inputs equally and the models relied on a fixed representation of the input to make predictions. Modern LLMs that rely on attention can dynamically focus on different parts of input sequences, allowing them to weigh the importance of each part in making predictions." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"Fine-tuning involves training the LLM on a smaller, task-specific dataset to adjust its parameters for the specific task at hand. This allows the LLM to leverage its pre-trained knowledge of the language to improve its accuracy for the specific task. Fine-tuning has been shown to drastically improve performance on domain-specific and task-specific tasks and lets LLMs adapt quickly to a wide variety of NLP applications." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"The idea behind transfer learning is that the pre-trained model has already learned a lot of information about the language and relationships between words, and this information can be used as a starting point to improve performance on a new task. Transfer learning allows LLMs to be fine-tuned for specific tasks with much smaller amounts of task-specific data than would be required if the model were trained from scratch. This greatly reduces the amount of time and resources needed to train LLMs." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"Transfer learning is a technique used in machine learning to leverage the knowledge gained from one task to improve performance on another related task. Transfer learning for LLMs involves taking an LLM that has been pre-trained on one corpus of text data and then fine-tuning it for a specific 'downstream' task, such as text classification or text generation, by updating themodel’s parameters with task-specific data." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"Chain-of-thought prompting is a method that forces LLMs to reason through a series of steps, resulting in more structured, transparent, and precise outputs. The goal is to break down complex tasks into smaller, interconnected subtasks, allowing the LLM to address each subtask in a stepby-step manner. This not only helps the model to 'focus' on specific aspects of the problem, but also encourages it to generate intermediate outputs, making it easier to identify and debug potential issues along the way. Another significant advantage of chain-of-thought prompting is the improved interpretability and transparency of the LLM-generated response. By offering insights into the model’s reasoning process, we, as users, can better understand and qualify how the final output was derived, which promotes trust in the model’s decision-making abilities." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"In prompt engineering, we customize the prompts or questions we give the model to get more accurate or insightful responses. The way a prompt is structured has a massive impact on how well a model understands the task at hand and, ultimately, how well it performs. Given LLMs’ versatility, prompt engineering has become an important skill for getting the most out of these models across different domains and tasks. The key is to understand how different prompt structures lead to different model behaviors. There are various strategies - ranging from simple one-shot prompting to more complex techniques like chain-of-thought prompting - that can significantly improve the effectiveness of LLMs." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"LLM-centric workloads change everything. Now the raw material is heterogeneous text, code, images, audio, and chat logs whose value depends on semantic richness - that is, the informational value of the content - rather than a rigid structure. Pipelines must tokenize, chunk, embed, and version this content; store it in vector indexes for similarity search; and apply filters for personally identifiable information, toxicity, and licensing constraints. Instead of ETL jobs, teams run continuous ingestion and reembedding loops so that RAG systems stay fresh, and they log every prompt–response pair so that the inputs and outputs can be evaluated and improve the future performance of this system. Data quality in this context is judged by grounding, factuality, and bias metrics - attributes that require automated red-teaming and humanin-the-loop (HITL) review rather than the data structure violation checks of the past." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"Prompt engineering is a crucial aspect of working with large language models (LLMs) like OpenAI's GPT, Google's PaLM, and others in the space of AI and machine learning. It involves the art and science of designing inputs (prompts) in a way that maximizes the quality, relevance, and accuracy of the AI-generated output. As the capabilities of AI continue to improve, the task of crafting effective prompts has become an essential skill for anyone leveraging these tools for real-world applications, including natural language understanding, translation, summarization, code generation, and more." (Code Planet, "Python for Large Language Models", 2025)

"Recursive prompting is a systematic approach to achieving higher-quality outputs through iterative refinement. Rather than accepting the first response, it uses a step-by-step process of evaluation and improvement, making it particularly valuable for complex tasks such as code development, writing, and problem-solving. Our example demonstrated how a basic factorial function evolved from a simple implementation to a robust, optimized solution through multiple iterations of targeted refinements." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"The pre-training step is by far the biggest in terms of data and compute requirements for the whole of the LLM’s lifecycle. Yet fine-tuning is quite resource-intensive when we compare it to traditional machine learning and deep learning workflows. Fine-tuning is also a very important step in improving the quality of the models; hence, it makes sense to understand how we can optimize this step without impacting the performance. Efficiencies in this step also enable us to iterate faster, thereby improving adaptability in many fast-moving domains. In this section, we will focus on some interesting efficient method." (Joseph Babcock & Raghav Bali, "Generative AI with Python and PyTorch" 2nd. Ed., 2025)

"Unlike traditional code completion, which operates on predefined rules, generative AI creates a continuous improvement cycle, which includes the following five basic steps: (1) Developer input: You provide source code, comments, or natural language requirements. (2) Context analysis: The model analyzes patterns in your existingcode and requirements. (3) Prediction: Based on training data and your specific context, the model generates probable code. (4) Developer feedback: You accept, modify, or reject suggestions. (5) Model adaptation: The system incorporates your feedback to improve future suggestions." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"As AI systems become more integrated into critical decision-making processes, the 'black box' problem – the inability to understand how a model arrived at a specific output – becomes a major barrier to trust and adoption. For RAG systems, this is particularly crucial; a user needs to know not just the answer, but why the system believes that answer to be true. Transparency and explainability (explainable AI, XAI) are the disciplines focused on making AI reasoning understandable to humans. A transparent RAG system allows users to verify the accuracy of its responses, builds trust by demonstrating a logical process, and enables developers to debug and improve the system. It transforms the AI from an oracle that must be blindly trusted into a tool for augmented intelligence, where the human remains the ultimate arbiter of truth." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Intelligent systems connect users to AI and ML to achieve meaningful objectives. An intelligent system is one in which intelligence evolves and improves over time, particularly when it improves by watching how users interact with the system.[...] The primary objective of the intelligent system is to support users in accomplishing complex tasks - not by replacing them, but by enhancing their decision-making capabilities. [...] An intelligent system must also have the ability to learn from user interactions and explicit feedback, as well as utilize contextual information. The system should contin-uously develop, use, and maintain an evolving knowledge base. This evolution is driven not only by data sources but also by ongoing interactions with users." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"[...] RAG models excel in dynamic environments where information changes frequently, such as news generation or customer support. Standard generative models, constrained by their training data, may provide outdated or irrelevant responses. RAG, however, can pull the latest information, ensuring up-to-date and contextually appropriate outputs. While RAG models may require more computational resources due to the retrieval step, the trade-off is often justified by the substantial improvements in accuracy and reliability, making them a superior choice for many real-world applications." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"RAG models offer several advantages over standard generative models, addressing many of their limitations. Traditional generative models, like GPT, rely solely on patterns learned during training, which can lead to issues such as factual inaccuracies, model hallucinations, and lack of contextual relevance. These models generate content based on pre-existing knowledge, often without the ability to verify or update the information, making them less reliable for tasks requiring high accuracy. In contrast, RAG models integrate retrieval mechanisms that allow them to access external knowledge sources in real time. This ensures that the generated content is grounded in verified data, significantly improving factual consistency and relevance. For example, while a standard generative model might produce a plausible but incorrect answer to a factual question, a RAG model can retrieve and incorporate accurate information from a trusted source, reducing errors." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

🖍️Angelica Lo Duca - Collected Quotes

"Artificial Intelligence (AI) is a discipline, like physics and mathematics, aiming to construct intelligent agents capable of independent reasoning, learning, and decision-making. AI focuses on developing theories and methodologies to create agents that can emulate human-like thinking and behavior." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Data Storytelling allows you to fill the gap between simply visualizing data and communicating it to an audience. Data storytelling improves your communication skills and standardizes and simplifies the process of communicating results, making it easier for people to understand and remember information. Data storytelling also helps you learn to communicate more effectively with others, improving personal and professional relationships." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"[...] data storytelling builds compelling stories supported by data, allowing analysts and data scientists to present and share their insights interestingly and interactively. The ultimate goal of data storytelling is to engage the audience and inspire them to make decisions. In some cases, including business scenarios, you do not start from data but from a narrative or a hypothesis you have in mind and then search for data that confirm or negate it. In this case, you can still have data storytelling, but you must pay attention not to alter your data to support your hypothesis." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Data storytelling is about more than just communicating data. It's about inspiring your audience and inviting them to take action. Good data storytelling requires a mix of art and science. Art comes in finding the right story, while science understands how to use data to support that story. When done well, data storytelling can be a potent tool for change." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Data visualization is primarily concerned with presenting data in a way that is easy to understand, usually through charts and graphs. On the other hand, data storytelling is more concerned with using data to tell a story that will engage the audience and help them understand the problem. In addition, data storytelling often uses narratives to explain the data, while data visualization focuses on presenting the data itself. Anyway, good data storytelling includes visualization elements." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Deep Learning (DL) is a subfield of ML that focuses on training artificial neural networks to process and understand complex patterns in data. Inspired by the human brain, DL models consist of multiple hidden layers of interconnected nodes, known as artificial neurons or units. The greater the number of layers, the more complex the data structure the network can recognize. DL is quite resource-intensive because it requires high-performance GPUs and large amounts of storage to train models. You can use DL models to solve tasks such as image and speech recognition, natural language processing, and even game-playing." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024) 

"Machine Learning (ML) is a subfield of AI that enables computers to acquire knowledge and improve performance without explicit programming. Different from traditional programming, where you must write the code line by line to provide instructions, ML allows machines to learn from data and make predictions or take actions based on that learned information. This approach empowers machines to discover patterns, relationships, and insights that may not be immediately apparent to human programmers. ML relies on two types of learning: supervised learning and unsupervised learning." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Prompt Engineering is a discipline aiming at carefully crafting and refining the prompts to achieve the desired results. The goal of prompt engineering is to elicit responses that are coherent, relevant, and aligned with the desired outcome while minimizing any unintended biases or undesired behavior." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"RAG is an advanced Natural Language Processing (NLP) technique that combines elements of information retrieval and text generation. First, RAG performs a retrieval step, which queries an external knowledge source, such as a vast text corpus or a structured database. Next, RAG uses this knowledge source to enhance its response generation. RAG integrates the retrieved facts into its generated text." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"To turn information into knowledge, add context. Context involves all circumstances around data, such as events, situations, etc. Adding context helps the audience to understand data better." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

🏭🗒️Microsoft Fabric: Direct Lake [Notes] 🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 13-Oct-2026

[MF] Direct Lake

• {def} Power BI semantic model table storage mode option available in MF [1]
• optimized for large volumes of data to be quickly loaded into memory from Delta tables available in the OneLake [1] 
• leverages VertiPaq query engine [1]
• {benefit} deliver query performance comparable to Import mode 
• without the management overhead of data refresh cycles to load the entire data volume [1]
• similar to Import mode
• {difference}refreshes copies only metadata (aka framing)
•  [Import mode] replicates the data and creates an entire cached copy of the data for the semantic model [1]
•  depends on data preparation being done in the data lake [1]
• {benefit} enables high-performance interactive analysis [1]
•  ideal for semantic models connecting to large data sources
• e.g. lakehouse, warehouse and other data source with Delta tables [1]
• performance depends on well-tuned Delta tables [1]
•  {benefit} performance
• normally outperforms DirectQuery queries when loading and interacting with visuals in reports [1]
• {benefit}maximizes ROI because 
• analyzed data volumes can exceed capacity's max memory limits [1]
• only the needed data to answer a query is loaded into memory [1]  
• {benefit} minimizes data latencies 
•  makes new data available without refresh schedules [1]
• quickly and automatically synchronizing a semantic model with its sources [1]
• {operation} refresh 
• low-cost operation that analyzes the metadata of the latest version of the Delta tables and updates references to the latest files in OneLake
• DL moves data preparation to OneLake
• <-- uses the full breadth of Fabric technologies for data prep
• incl. Spark jobs, T-SQL DML statements, dataflows, pipelines, etc.
• {operation} transcoding
• the process of loading data from OneLake only when a query requests a column for the first time 
• {action} the semantic model receives a DAX (or MDX) query 
• {step} determines what columns are needed to produce the result
• {step} determines which columns are already resident in memory [2]
• {step} loads the missing columns from OneLake [2]
•  typically a fast operation, however it can depend on factors 
•  e.g. the cardinality of data stored in the columns
•  remains resident until there's reason for it to be removed (evicted) from memory
• {reason} the model or table was refreshed after a Delta table update at the source [2]
• {reason} no query used the column for some time [2]
• {reason} other memory management reasons [2]
•  e.g. memory pressure in the capacity due to other, concurrent operations [2]
•  SKU determines the maximum available memory for each Direct Lake semantic model on the capacity [2]
• {operation} framing 
• an operation triggered by a refresh of a semantic model [2]
• it usually takes only a few seconds to complete
• <-- because it's a low-cost operation where the semantic model analyzes the metadata of the latest version of the Delta Lake tables and updates to reference the latest Parquet files in OneLake [2]
• the process might evict resident table column segments and dictionaries from memory if the underlying data changed [2]
• the point in time of the refresh becomes the new baseline for all future transcoding events [2]
• DL queries only consider data in the Delta tables as of the time of the most recent framing operation [2]
• DL tables are queried to return data based on the state of the Delta table at the point of the most recent successful framing operation [2]
• that time isn't necessarily the latest state of the Delta tables [2]
• the semantic model analyzes the Delta log of each Delta table during framing to drop only the affected column segments and to reload newly added data during transcoding [2]
• {optimization} dictionaries usually aren't dropped when incremental framing takes effect, and new values are added to the existing dictionaries [2]
• helps reduce the reload burden and benefits query performance [2]
• when a Delta table receives no updates, no reload is necessary for columns already resident in memory [2]
•  queries show far less performance impact after framing because incremental framing essentially enables the semantic model to update substantial portions of the existing in-memory data in place [2]
• can fail if a Delta table exceeds the Fabric capacity guardrails [2]
•  e.g. when a Delta table has more than 10,000 parquet files [2]
•  {architecture} Direct Lake on OneLake
•  isn't coupled with the SQL analytics endpoint [2]
•  offers tighter integration with OneLake features such as OneLake security [2]
•  offers more efficient DAX query plans because [2]
• <-- isn’t required to check for SQL based security [2]
•  fallback isn't supported [2]
• {architecture} Direct Lake on SQL analytics endpoint
•  a DAX query might use DirectQuery fallback [2]
•  involves seamlessly switching to DirectQuery mode [2]
•  DirectQuery fallback
• retrieves data directly from the SQL analytics endpoint of the lakehouse or the warehouse [2]
•  fallback occurs when SQL based security is detected in the SQL analytics endpoints
•  a DirectQuery operation sends a query to the SQL analytics endpoint. Fallback operations might result in slower query performance [2]
•  {setting} Keep your Direct Lake data up to date
• performs automatic updates of Direct Lake tables [2]
• via semantic model settings >> Refresh 
• {enabled|default} ensures that data changes in OneLake are automatically reflected in the Direct Lake semantic model [2]
•  the semantic model performs a framing operation whenever data modifications in the underlying Delta tables are detected [2]
• {recommendation} leave the setting on, especially when you have a small or medium-sized semantic model
• useful in case of low-latency reporting requirements and Delta tables are modified regularly [2]
•  {disabled} it might be needed to disable automatic updates to allow completion of data preparation jobs or an ETL process before exposing any new data to consumers of the semantic model [2]
•  users can trigger a refresh manually in the Fabric portal [2]
•  via refresh schedule, or programmatically by using the Power BI REST API or TOM [2]
• 
  

References: 
[1] Microsoft Learn (2026) Direct Lake overview [link]
[2] Microsoft Learn (2026) 

Acronyms:
API - Application Programming Interface
DL - Direct Lake 
MDX - Multidimensional Expressions
MF - Microsoft Fabric
REST
SKU - Stock Keeping Unit
TOM - Tabular Object Model
T-SQL - Transact SQL

 

⚡️🗒️Power BI: ALLSELECTED in DAX [Notes]🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 13-Oct-2026

[DAX] ALLSELECTED()

• {def} function that returns all the rows in a table, or all the values in a column [1]
• ignores any filters that might have been applied inside the query [1]
•  removes the corresponding filters from the filter context, restoring the last shadow filter context
• keeps filters that come from outside [1] 
• gets the context that represents all rows and columns in the query, while keeping explicit filters and contexts other than row and column filters 
• does not materialize the resulting table when called directly in a filter argument of CALCULATE or CALCULATETABLE 
• via ALLSELECTED([<tableName> | <columnName>[, <columnName>[, <columnName>[,…]]
• {parameter|optional} tableName
• the name of an existing table 
• cannot be an expression
• {parameter|optional} columnName
• the name of an existing column using standard DAX syntax, usually fully qualified
• cannot be an expression 
• if there is one argument, the argument is either tableName or columnName [1]
• if there is more than one argument, they must be columns from the same table [1] 
• {scenario} used to obtain visual totals in queries [1] 
• {scenario} used as a table expression when it has at least one argument [2] 
• {scenario} used without arguments only as a CALCULATE or CALCULATETABLE modifier [2]  
• {distinction} different from ALL() because 
• it retains all filters explicitly set within the query [1] 
• it retains all context filters other than row and column filters [1] 
• {limitation}[DirectQuery mode] not supported when used in calculated columns or RLS rules [1] 
• only DAX function that leverages shadow filter contexts [3] 
• has a slightly different behavior when
• used in SUMMARIZECOLUMNS 
• mostly produces the expected result [3] 
• removes filters for the currently-iterated group-by column, restoring the original filter set by SUMMARIZECOLUMNS itself [3] 
• SUMMARIZECOLUMNS  can be used in nearly all the visuals [3] 
• inside an iterator
• {warning} can produce weird results [3] 
• does not return the values of a column (or table) filtered outside a visual [3] 
• displays different behaviors, depending on whether it is used in SUMMARIZECOLUMNS or not, and on the presence of shadow filter contexts [3]
• {best practice} don’t create a measure that uses ALLSELECTED within an iteration [3]
•  it can produce unexpected results, poor performance, and difficult-to-understand calculations.
• {best practice} minimize the use of measures that contain ALLSELECTED [3]
• can be safely used only in measures that are directly placed in the reports [3] 
•  

Previous Post <<||>> Next Post

References: 
[1] Microsoft Learn (2026) DAX: ALLSELECTED [link]
[2] DAX Guide (2026) ALLSELECTED DAX Function (Filter) SQL[link]
[3] SQLBI (2026) ALLSELECTED Best Practices [link]

🤖Prompt Engineering: Patterns (Just the Quotes)

"Another problem that can be confusing is that LLMs seldom put out the same thing twice. [...] Traditional databases are straightforward - you ask for something specific, and you get back exactly what was stored. Search engines work similarly, finding existing information. LLMs work differently. They analyze massive amounts of text data to understand statistical patterns in language. The model processes information through multiple layers, each capturing different aspects - from simple word patterns to complex relationships between ideas." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"Context is crucial for how language models understand and generate code. The model processes your input by analyzing relationships between different parts of the code and documentation to determine meaning and intent. [...] The model evaluates context by calculating mathematical relationships between elements in your input. However, it may miss important domain knowledge, coding standards, or architectural patterns that experienced developers understand implicitly." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"Generative AI for coding and language tools is based on the LLM concept. A large language model is a type of neural network that processes and generates text in a humanlike way. It does this by being trained on a massive dataset of text, which allows it to learn human language patterns, as described previously. It lets LLMs translate, write, and answer questions with text. LLMs can contain natural language, source code, and  more." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"This ability to zero in on important code is why modern AI coding assistants can offer meaningful suggestions for your specific needs. It’s similar to how skilled developers know which code sections affect a new implementation the most. Each transformer layer learns about various code patterns, ranging from syntax validation to understanding the relationships among functions, classes, and modules." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"Unlike traditional code completion, which operates on predefined rules, generative AI creates a continuous improvement cycle, which includes the following five basic steps: (1) Developer input: You provide source code, comments, or natural language requirements. (2) Context analysis: The model analyzes patterns in your existingcode and requirements. (3) Prediction: Based on training data and your specific context, the model generates probable code. (4) Developer feedback: You accept, modify, or reject suggestions. (5) Model adaptation: The system incorporates your feedback to improve future suggestions." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025) 

"Generative AI has a lot of problems because it needs a lot of data to work. Generative models like GPT and GANs need a lot of training data to learn patterns and make good outputs. This data dependency can cause problems like overfitting, which happens when the model does well on training data but doesn’t work well with new, unseen data. Also, the quality of the content that is generated is directly related to the diversity and representativeness of the training data. This means that biased or incomplete datasets can lead to outputs that are wrong or unfair." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"[...] KGs and LLMs can be the foundation for different types of reasoning, complementing each other in intelligent systems. We can use KGs for tasks that require precise, rule-based reasoning and explicit knowledge representation, and LLMs for tasks involving pattern recognition, context understanding, handling ambiguity or incomplete information, and reasoning about graph structures and their derived metrics. However, neither approach inherently possesses common-sense reasoning capabilities comparable to those of humans, and they often fail to make intuitive leaps or understand the implicit context that would be obvious to a person. These limitations underscore the importance of carefully considering the strengths and weaknesses of each approach when designing intelligent systems and potentially developing a powerful hybrid IAS." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"Machine learning was a big change because it let systems learn patterns from data instead of having to follow hardcoded rules. AI could generalize better and change to new inputs thanks to techniques like decision trees and support vector machines. These models still needed a lot of work to get the features right, though, and they couldn’t handle high-dimensional data like images or text very well. The breakthrough happened when deep learning, a type of machine learning that uses neural networks with multiple layers to automatically learn hierarchical representations of data, became popular. Convolutional neural networks (CNNs) for processing images and recurrent neural networks (RNNs) for sequential data changed what AI could do. Architectures like GANs and transformers took things even further by making it possible to do things like make images, understand natural language, and more." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026) 

"RAG models offer several advantages over standard generative models, addressing many of their limitations. Traditional generative models, like GPT, rely solely on patterns learned during training, which can lead to issues such as factual inaccuracies, model hallucinations, and lack of contextual relevance. These models generate content based on pre-existing knowledge, often without the ability to verify or update the information, making them less reliable for tasks requiring high accuracy. In contrast, RAG models integrate retrieval mechanisms that allow them to access external knowledge sources in real time. This ensures that the generated content is grounded in verified data, significantly improving factual consistency and relevance. For example, while a standard generative model might produce a plausible but incorrect answer to a factual question, a RAG model can retrieve and incorporate accurate information from a trusted source, reducing errors." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Retrieval mechanisms are essential for addressing some of the key limitations of traditional generative AI models, such as factual inaccuracies, lack of context awareness, and model hallucinations. While generative models excel at creating coherent and fluent content, they often struggle to produce outputs that are factually correct or contextually relevant. This is because these models rely solely on patterns learned during training, without access to real-time or external information. For example, a generative model might generate a plausible-sounding but incorrect answer to a factual question, as it cannot verify the accuracy of its response." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"The foundational form of RAG, often called naive RAG, follows a straightforward pattern. A pipeline retrieves supporting context from external sources such as enterprise documents, knowledge bases, or structured datasets and appends that information to the model’s prompt before inference. In the most common implementation, each document is converted into an embedding, a numerical representation of its semantic meaning, using either the same foundation model or a specialized embedding model. When a user submits a query, the system performs a vector similarity search to find documents whose embeddings most closely match the query’s vector representation, and the retrieved content is concatenated with the user query before being passed to the language model." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Traditional paradigms build systems for specific purposes with structured, homogeneous databases. This approach works for tailored needs but is impractical for complex domains that need to adapt to user characteristics and integrate heterogeneous data. KGs capture connections, enabling relationship discovery through graph pattern matching and traversal. Both the Resource Description Framework (RDF) and Labeled Property Graphs (LPGs) provide machine-readable formats that humans can interpret. KGs emphasize rich, meaningful data representations usable by both humans and machines, enabling a paradigm shift where intelligent behavior is encoded in a unique source of truth." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026) 

💎🏭SQL Reloaded: QUALIFY in Microsoft Fabric (Before/After)

One recent addition to T-SQL is the QUALIFY clause, which filters rows returned by a query based on a search condition that can reference window (analytic) functions [1]. The QUALIFY clause allows filtering directly in the block in which the window function is defined. According to the documentation QUALIFY can be used directly after HAVING clause:

SELECT select_list
FROM   table_source
[ WHERE <search_condition> ]
[ GROUP BY group_by_specification ]
[ HAVING <search_condition> ]
[ QUALIFY <filter_condition> ]
[ ORDER BY <order_by_expression> [ , ...n ] ]
[ FOR JSON <options> ]

So, for example, if we consider a fictitious Sales Invoices table, to retrieve the customers who bought a product more than once, we had to apply the constraint outside the block in which the window function was defined:

-- product bought more than once by same customer (before)
SELECT *
FROM (-- calculating resultset
	SELECT customer_id
	, product_id
	, SUM(quantity) OVER(PARTITION BY invoice_id, product_id) total_quantity
	FROM [Sales].[sales_invoices]
) DAT
WHERE total_quantity>1

Now, a constraint can be applied directly in the block with the window function via QUALIFY followed by one or more constraints:

-- product bought more than once by same customer (after) 
SELECT customer_id
, product_id
, SUM(quantity) OVER(PARTITION BY invoice_id, product_id) total_quantity
FROM [Sales].[sales_invoices]
QUALIFY SUM(quantity) OVER(PARTITION BY invoice_id, product_id)>1

Given the order in which the constraints are processed, QUALIFY allows to reduce redundancy by referencing a new column defined in the SELECT statement (update 19-Oct-2026):

-- product bought more than once by same customer (after) 
SELECT customer_id
, product_id
, SUM(quantity) OVER(PARTITION BY invoice_id, product_id) total_quantity
FROM [Sales].[sales_invoices]
QUALIFY total_quantity>1

Hopefully this is available on all supported platforms. Otherwise, the logic must be repeated if a second constraint is needed for checking whether the value is within a boundary. Upon case, the logic can become even more complex (e.g. multiple intervals are checked):

-- product bought more than once by same customer (after) 
SELECT customer_id
, product_id
, SUM(quantity) OVER(PARTITION BY invoice_id, product_id) total_quantity
FROM [Sales].[sales_invoices]
QUALIFY SUM(quantity) OVER(PARTITION BY invoice_id, product_id)>1

  AND SUM(quantity) OVER(PARTITION BY invoice_id, product_id)<10

Imagine how complex can become a query like the following (update 19-Oct-2026):

-- product bought more than once by same customer with interval testing
SELECT customer_id
, product_id
, SUM(quantity) OVER(PARTITION BY invoice_id, product_id) total_quantity
FROM [Sales].[sales_invoices]
QUALIFY (total_quantity>1 AND total_quantity)<10)

    OR (total_quantity>21 AND total_quantity)<30)

    OR (total_quantity>41 AND total_quantity)<50)

    OR (total_quantity>61 AND total_quantity)<70)  

Notes:
1) Having fewer blocks of code improves in theory code's readability and hopefully maintainability when the code doesn't need to be replicated as in the last example. This approach should improve also performance, however further tests are needed to validate this. 
2) It's useful to include the same logic used in QUALIFY also in the main SELECT statement, at least for troubleshooting purposes. 
3) The syntax seems to be supported only in Fabric warehouse, respectively SQL Endpoints for warehouse, lakehouse and SQL database.4) QUALIFY clause requires at least one window function in the SELECT list, ORDER BY list or QUALIFY expression (see Msg 8826, Level 16).
4) QUALIFY clause requires at least one window function in the SELECT list, ORDER BY list or QUALIFY expression (see Msg 8826, Level 16). (update 19-Oct-2026)
5) One should check also whether the syntax is supported by all the features. For example, it's not supported by UPDATE and DELETE statements in warehouse. The restrictions for SQL Endpoints should be considered as well.
6) Developers must consider also portability of code when it's the case.
7) According to Copilot, QUALIFY is supported in Teradata (originator of the feature), Snowflake, Google BigQuery, Amazon Redshift, Databricks SQL, DuckDB and ClickHouse. (update 19-Oct-2026)
8) Of course, the window functions can be mixed to address different business questions, though it's maybe a good idea to move the DENSE_RANK in SELECT and reference the new column within QUALIFY (update 19-Oct-2026):

-- first/last customer by product incl. total
SELECT product_id
, SUM(quantity) OVER(PARTITION BY product_id) quantity_total
, FIRST_VALUE(customer_id) OVER(PARTITION BY product_id ORDER BY invoice_date ASC) first_status
, LAST_VALUE(customer_id) OVER(PARTITION BY product_id ORDER BY invoice_date DESC) last_status
FROM [LH_CRM_Data].ERP.[sales_invoices_partitioned4]
QUALIFY DENSE_RANK() OVER(PARTITION BY product_id ORDER BY invoice_date)=1

Happy coding!

References:
[1] Microsoft Learn (2026) SELECT - QUALIFY clause (Transact-SQL) [link]