SQL Troubles

14 October 2026

🤖Prompt Engineering: Improvement (Just the Quotes)

"Attention is a mechanism used in deep learning models (not just Transformers) that assigns different weights to different parts of the input, allowing the model to prioritize and emphasize the most important information while performing tasks like translation or summarization. Essentially, attention allows a model to 'focus' on different parts of the input dynamically, leading to improved performance and more accurate results. Before the popularization of attention, most neural networks processed all inputs equally and the models relied on a fixed representation of the input to make predictions. Modern LLMs that rely on attention can dynamically focus on different parts of input sequences, allowing them to weigh the importance of each part in making predictions." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"Fine-tuning involves training the LLM on a smaller, task-specific dataset to adjust its parameters for the specific task at hand. This allows the LLM to leverage its pre-trained knowledge of the language to improve its accuracy for the specific task. Fine-tuning has been shown to drastically improve performance on domain-specific and task-specific tasks and lets LLMs adapt quickly to a wide variety of NLP applications." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"The idea behind transfer learning is that the pre-trained model has already learned a lot of information about the language and relationships between words, and this information can be used as a starting point to improve performance on a new task. Transfer learning allows LLMs to be fine-tuned for specific tasks with much smaller amounts of task-specific data than would be required if the model were trained from scratch. This greatly reduces the amount of time and resources needed to train LLMs." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"Transfer learning is a technique used in machine learning to leverage the knowledge gained from one task to improve performance on another related task. Transfer learning for LLMs involves taking an LLM that has been pre-trained on one corpus of text data and then fine-tuning it for a specific 'downstream' task, such as text classification or text generation, by updating themodel’s parameters with task-specific data." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"Chain-of-thought prompting is a method that forces LLMs to reason through a series of steps, resulting in more structured, transparent, and precise outputs. The goal is to break down complex tasks into smaller, interconnected subtasks, allowing the LLM to address each subtask in a stepby-step manner. This not only helps the model to 'focus' on specific aspects of the problem, but also encourages it to generate intermediate outputs, making it easier to identify and debug potential issues along the way. Another significant advantage of chain-of-thought prompting is the improved interpretability and transparency of the LLM-generated response. By offering insights into the model’s reasoning process, we, as users, can better understand and qualify how the final output was derived, which promotes trust in the model’s decision-making abilities." (Sinan Ozdemir, "Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs", 2024)

"In prompt engineering, we customize the prompts or questions we give the model to get more accurate or insightful responses. The way a prompt is structured has a massive impact on how well a model understands the task at hand and, ultimately, how well it performs. Given LLMs’ versatility, prompt engineering has become an important skill for getting the most out of these models across different domains and tasks. The key is to understand how different prompt structures lead to different model behaviors. There are various strategies - ranging from simple one-shot prompting to more complex techniques like chain-of-thought prompting - that can significantly improve the effectiveness of LLMs." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"LLM-centric workloads change everything. Now the raw material is heterogeneous text, code, images, audio, and chat logs whose value depends on semantic richness - that is, the informational value of the content - rather than a rigid structure. Pipelines must tokenize, chunk, embed, and version this content; store it in vector indexes for similarity search; and apply filters for personally identifiable information, toxicity, and licensing constraints. Instead of ETL jobs, teams run continuous ingestion and reembedding loops so that RAG systems stay fresh, and they log every prompt–response pair so that the inputs and outputs can be evaluated and improve the future performance of this system. Data quality in this context is judged by grounding, factuality, and bias metrics - attributes that require automated red-teaming and humanin-the-loop (HITL) review rather than the data structure violation checks of the past." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"Prompt engineering is a crucial aspect of working with large language models (LLMs) like OpenAI's GPT, Google's PaLM, and others in the space of AI and machine learning. It involves the art and science of designing inputs (prompts) in a way that maximizes the quality, relevance, and accuracy of the AI-generated output. As the capabilities of AI continue to improve, the task of crafting effective prompts has become an essential skill for anyone leveraging these tools for real-world applications, including natural language understanding, translation, summarization, code generation, and more." (Code Planet, "Python for Large Language Models", 2025)

"Recursive prompting is a systematic approach to achieving higher-quality outputs through iterative refinement. Rather than accepting the first response, it uses a step-by-step process of evaluation and improvement, making it particularly valuable for complex tasks such as code development, writing, and problem-solving. Our example demonstrated how a basic factorial function evolved from a simple implementation to a robust, optimized solution through multiple iterations of targeted refinements." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"The pre-training step is by far the biggest in terms of data and compute requirements for the whole of the LLM’s lifecycle. Yet fine-tuning is quite resource-intensive when we compare it to traditional machine learning and deep learning workflows. Fine-tuning is also a very important step in improving the quality of the models; hence, it makes sense to understand how we can optimize this step without impacting the performance. Efficiencies in this step also enable us to iterate faster, thereby improving adaptability in many fast-moving domains. In this section, we will focus on some interesting efficient method." (Joseph Babcock & Raghav Bali, "Generative AI with Python and PyTorch" 2nd. Ed., 2025)

"Unlike traditional code completion, which operates on predefined rules, generative AI creates a continuous improvement cycle, which includes the following five basic steps: (1) Developer input: You provide source code, comments, or natural language requirements. (2) Context analysis: The model analyzes patterns in your existingcode and requirements. (3) Prediction: Based on training data and your specific context, the model generates probable code. (4) Developer feedback: You accept, modify, or reject suggestions. (5) Model adaptation: The system incorporates your feedback to improve future suggestions." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"As AI systems become more integrated into critical decision-making processes, the 'black box' problem – the inability to understand how a model arrived at a specific output – becomes a major barrier to trust and adoption. For RAG systems, this is particularly crucial; a user needs to know not just the answer, but why the system believes that answer to be true. Transparency and explainability (explainable AI, XAI) are the disciplines focused on making AI reasoning understandable to humans. A transparent RAG system allows users to verify the accuracy of its responses, builds trust by demonstrating a logical process, and enables developers to debug and improve the system. It transforms the AI from an oracle that must be blindly trusted into a tool for augmented intelligence, where the human remains the ultimate arbiter of truth." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Intelligent systems connect users to AI and ML to achieve meaningful objectives. An intelligent system is one in which intelligence evolves and improves over time, particularly when it improves by watching how users interact with the system.[...] The primary objective of the intelligent system is to support users in accomplishing complex tasks - not by replacing them, but by enhancing their decision-making capabilities. [...] An intelligent system must also have the ability to learn from user interactions and explicit feedback, as well as utilize contextual information. The system should contin-uously develop, use, and maintain an evolving knowledge base. This evolution is driven not only by data sources but also by ongoing interactions with users." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"[...] RAG models excel in dynamic environments where information changes frequently, such as news generation or customer support. Standard generative models, constrained by their training data, may provide outdated or irrelevant responses. RAG, however, can pull the latest information, ensuring up-to-date and contextually appropriate outputs. While RAG models may require more computational resources due to the retrieval step, the trade-off is often justified by the substantial improvements in accuracy and reliability, making them a superior choice for many real-world applications." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"RAG models offer several advantages over standard generative models, addressing many of their limitations. Traditional generative models, like GPT, rely solely on patterns learned during training, which can lead to issues such as factual inaccuracies, model hallucinations, and lack of contextual relevance. These models generate content based on pre-existing knowledge, often without the ability to verify or update the information, making them less reliable for tasks requiring high accuracy. In contrast, RAG models integrate retrieval mechanisms that allow them to access external knowledge sources in real time. This ensures that the generated content is grounded in verified data, significantly improving factual consistency and relevance. For example, while a standard generative model might produce a plausible but incorrect answer to a factual question, a RAG model can retrieve and incorporate accurate information from a trusted source, reducing errors." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

13 October 2026

🖍️Angelica Lo Duca - Collected Quotes

"Artificial Intelligence (AI) is a discipline, like physics and mathematics, aiming to construct intelligent agents capable of independent reasoning, learning, and decision-making. AI focuses on developing theories and methodologies to create agents that can emulate human-like thinking and behavior." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Data Storytelling allows you to fill the gap between simply visualizing data and communicating it to an audience. Data storytelling improves your communication skills and standardizes and simplifies the process of communicating results, making it easier for people to understand and remember information. Data storytelling also helps you learn to communicate more effectively with others, improving personal and professional relationships." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"[...] data storytelling builds compelling stories supported by data, allowing analysts and data scientists to present and share their insights interestingly and interactively. The ultimate goal of data storytelling is to engage the audience and inspire them to make decisions. In some cases, including business scenarios, you do not start from data but from a narrative or a hypothesis you have in mind and then search for data that confirm or negate it. In this case, you can still have data storytelling, but you must pay attention not to alter your data to support your hypothesis." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Data storytelling is about more than just communicating data. It's about inspiring your audience and inviting them to take action. Good data storytelling requires a mix of art and science. Art comes in finding the right story, while science understands how to use data to support that story. When done well, data storytelling can be a potent tool for change." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Data visualization is primarily concerned with presenting data in a way that is easy to understand, usually through charts and graphs. On the other hand, data storytelling is more concerned with using data to tell a story that will engage the audience and help them understand the problem. In addition, data storytelling often uses narratives to explain the data, while data visualization focuses on presenting the data itself. Anyway, good data storytelling includes visualization elements." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Deep Learning (DL) is a subfield of ML that focuses on training artificial neural networks to process and understand complex patterns in data. Inspired by the human brain, DL models consist of multiple hidden layers of interconnected nodes, known as artificial neurons or units. The greater the number of layers, the more complex the data structure the network can recognize. DL is quite resource-intensive because it requires high-performance GPUs and large amounts of storage to train models. You can use DL models to solve tasks such as image and speech recognition, natural language processing, and even game-playing." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Machine Learning (ML) is a subfield of AI that enables computers to acquire knowledge and improve performance without explicit programming. Different from traditional programming, where you must write the code line by line to provide instructions, ML allows machines to learn from data and make predictions or take actions based on that learned information. This approach empowers machines to discover patterns, relationships, and insights that may not be immediately apparent to human programmers. ML relies on two types of learning: supervised learning and unsupervised learning." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"Prompt Engineering is a discipline aiming at carefully crafting and refining the prompts to achieve the desired results. The goal of prompt engineering is to elicit responses that are coherent, relevant, and aligned with the desired outcome while minimizing any unintended biases or undesired behavior." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"RAG is an advanced Natural Language Processing (NLP) technique that combines elements of information retrieval and text generation. First, RAG performs a retrieval step, which queries an external knowledge source, such as a vast text corpus or a structured database. Next, RAG uses this knowledge source to enhance its response generation. RAG integrates the retrieved facts into its generated text." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"To turn information into knowledge, add context. Context involves all circumstances around data, such as events, situations, etc. Adding context helps the audience to understand data better." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

🏭🗒️Microsoft Fabric: Direct Lake [Notes] 🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)!

Last updated: 13-Oct-2026

[MF] Direct Lake

{def} Power BI semantic model table storage mode option available in MF [1]

optimized for large volumes of data to be quickly loaded into memory from Delta tables available in the OneLake [1]
leverages VertiPaq query engine [1]

{benefit} deliver query performance comparable to Import mode

without the management overhead of data refresh cycles to load the entire data volume [1]

similar to Import mode

{difference}refreshes copies only metadata (aka framing)

[Import mode] replicates the data and creates an entire cached copy of the data for the semantic model [1]

depends on data preparation being done in the data lake [1]
{benefit} enables high-performance interactive analysis [1]

ideal for semantic models connecting to large data sources

e.g. lakehouse, warehouse and other data source with Delta tables [1]

performance depends on well-tuned Delta tables [1]

{benefit} performance

normally outperforms DirectQuery queries when loading and interacting with visuals in reports [1]

{benefit}maximizes ROI because

analyzed data volumes can exceed capacity's max memory limits [1]

only the needed data to answer a query is loaded into memory [1]

{benefit} minimizes data latencies

makes new data available without refresh schedules [1]

quickly and automatically synchronizing a semantic model with its sources [1]

{operation} refresh

low-cost operation that analyzes the metadata of the latest version of the Delta tables and updates references to the latest files in OneLake
DL moves data preparation to OneLake

<-- uses the full breadth of Fabric technologies for data prep

incl. Spark jobs, T-SQL DML statements, dataflows, pipelines, etc.

{operation} transcoding

the process of loading data from OneLake only when a query requests a column for the first time
{action} the semantic model receives a DAX (or MDX) query

{step} determines what columns are needed to produce the result
{step} determines which columns are already resident in memory [2]
{step} loads the missing columns from OneLake [2]

typically a fast operation, however it can depend on factors

e.g. the cardinality of data stored in the columns

remains resident until there's reason for it to be removed (evicted) from memory

{reason} the model or table was refreshed after a Delta table update at the source [2]
{reason} no query used the column for some time [2]
{reason} other memory management reasons [2]

e.g. memory pressure in the capacity due to other, concurrent operations [2]

SKU determines the maximum available memory for each Direct Lake semantic model on the capacity [2]

{operation} framing

an operation triggered by a refresh of a semantic model [2]
it usually takes only a few seconds to complete

<-- because it's a low-cost operation where the semantic model analyzes the metadata of the latest version of the Delta Lake tables and updates to reference the latest Parquet files in OneLake [2]

the process might evict resident table column segments and dictionaries from memory if the underlying data changed [2]

the point in time of the refresh becomes the new baseline for all future transcoding events [2]

DL queries only consider data in the Delta tables as of the time of the most recent framing operation [2]
DL tables are queried to return data based on the state of the Delta table at the point of the most recent successful framing operation [2]

that time isn't necessarily the latest state of the Delta tables [2]

the semantic model analyzes the Delta log of each Delta table during framing to drop only the affected column segments and to reload newly added data during transcoding [2]

{optimization} dictionaries usually aren't dropped when incremental framing takes effect, and new values are added to the existing dictionaries [2]

helps reduce the reload burden and benefits query performance [2]

when a Delta table receives no updates, no reload is necessary for columns already resident in memory [2]

queries show far less performance impact after framing because incremental framing essentially enables the semantic model to update substantial portions of the existing in-memory data in place [2]

can fail if a Delta table exceeds the Fabric capacity guardrails [2]

e.g. when a Delta table has more than 10,000 parquet files [2]

{architecture} Direct Lake on OneLake

isn't coupled with the SQL analytics endpoint [2]
offers tighter integration with OneLake features such as OneLake security [2]
offers more efficient DAX query plans because [2]

<-- isn’t required to check for SQL based security [2]
fallback isn't supported [2]

{architecture} Direct Lake on SQL analytics endpoint

a DAX query might use DirectQuery fallback [2]

involves seamlessly switching to DirectQuery mode [2]
DirectQuery fallback

retrieves data directly from the SQL analytics endpoint of the lakehouse or the warehouse [2]

fallback occurs when SQL based security is detected in the SQL analytics endpoints

a DirectQuery operation sends a query to the SQL analytics endpoint. Fallback operations might result in slower query performance [2]

{setting} Keep your Direct Lake data up to date

performs automatic updates of Direct Lake tables [2]

via semantic model settings >> Refresh

{enabled|default} ensures that data changes in OneLake are automatically reflected in the Direct Lake semantic model [2]
the semantic model performs a framing operation whenever data modifications in the underlying Delta tables are detected [2]
{recommendation} leave the setting on, especially when you have a small or medium-sized semantic model

useful in case of low-latency reporting requirements and Delta tables are modified regularly [2]

{disabled} it might be needed to disable automatic updates to allow completion of data preparation jobs or an ETL process before exposing any new data to consumers of the semantic model [2]

users can trigger a refresh manually in the Fabric portal [2]

via refresh schedule, or programmatically by using the Power BI REST API or TOM [2]

References:
[1] Microsoft Learn (2026) Direct Lake overview [link]
[2] Microsoft Learn (2026)

Acronyms:
API - Application Programming Interface
DL - Direct Lake
MDX - Multidimensional Expressions
MF - Microsoft Fabric
REST
SKU - Stock Keeping Unit
TOM - Tabular Object Model
T-SQL - Transact SQL

⚡️🗒️Power BI: ALLSELECTED in DAX [Notes]🆕

Last updated: 13-Oct-2026

[DAX] ALLSELECTED()

{def} function that returns all the rows in a table, or all the values in a column [1]

ignores any filters that might have been applied inside the query [1]

removes the corresponding filters from the filter context, restoring the last shadow filter context

keeps filters that come from outside [1]
gets the context that represents all rows and columns in the query, while keeping explicit filters and contexts other than row and column filters
does not materialize the resulting table when called directly in a filter argument of CALCULATE or CALCULATETABLE

via ALLSELECTED([<tableName> | <columnName>[, <columnName>[, <columnName>[,…]]

{parameter|optional} tableName

the name of an existing table
cannot be an expression

{parameter|optional} columnName

the name of an existing column using standard DAX syntax, usually fully qualified
cannot be an expression

if there is one argument, the argument is either tableName or columnName [1]
if there is more than one argument, they must be columns from the same table [1]

{scenario} used to obtain visual totals in queries [1]
{scenario} used as a table expression when it has at least one argument [2]
{scenario} used without arguments only as a CALCULATE or CALCULATETABLE modifier [2]
{distinction} different from ALL() because

it retains all filters explicitly set within the query [1]
it retains all context filters other than row and column filters [1]

{limitation}[DirectQuery mode] not supported when used in calculated columns or RLS rules [1]
only DAX function that leverages shadow filter contexts [3]
has a slightly different behavior when

used in SUMMARIZECOLUMNS

mostly produces the expected result [3]
removes filters for the currently-iterated group-by column, restoring the original filter set by SUMMARIZECOLUMNS itself [3]
SUMMARIZECOLUMNS can be used in nearly all the visuals [3]

inside an iterator

{warning} can produce weird results [3]

does not return the values of a column (or table) filtered outside a visual [3]
displays different behaviors, depending on whether it is used in SUMMARIZECOLUMNS or not, and on the presence of shadow filter contexts [3]
{best practice} don’t create a measure that uses ALLSELECTED within an iteration [3]

it can produce unexpected results, poor performance, and difficult-to-understand calculations.

{best practice} minimize the use of measures that contain ALLSELECTED [3]

can be safely used only in measures that are directly placed in the reports [3]

Previous Post <<||>> Next Post

References:
[1] Microsoft Learn (2026) DAX: ALLSELECTED [link]
[2] DAX Guide (2026) ALLSELECTED DAX Function (Filter) SQL[link]
[3] SQLBI (2026) ALLSELECTED Best Practices [link]

🤖Prompt Engineering: Patterns (Just the Quotes)

"Another problem that can be confusing is that LLMs seldom put out the same thing twice. [...] Traditional databases are straightforward - you ask for something specific, and you get back exactly what was stored. Search engines work similarly, finding existing information. LLMs work differently. They analyze massive amounts of text data to understand statistical patterns in language. The model processes information through multiple layers, each capturing different aspects - from simple word patterns to complex relationships between ideas." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"Context is crucial for how language models understand and generate code. The model processes your input by analyzing relationships between different parts of the code and documentation to determine meaning and intent. [...] The model evaluates context by calculating mathematical relationships between elements in your input. However, it may miss important domain knowledge, coding standards, or architectural patterns that experienced developers understand implicitly." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"Generative AI for coding and language tools is based on the LLM concept. A large language model is a type of neural network that processes and generates text in a humanlike way. It does this by being trained on a massive dataset of text, which allows it to learn human language patterns, as described previously. It lets LLMs translate, write, and answer questions with text. LLMs can contain natural language, source code, and more." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"This ability to zero in on important code is why modern AI coding assistants can offer meaningful suggestions for your specific needs. It’s similar to how skilled developers know which code sections affect a new implementation the most. Each transformer layer learns about various code patterns, ranging from syntax validation to understanding the relationships among functions, classes, and modules." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"Generative AI has a lot of problems because it needs a lot of data to work. Generative models like GPT and GANs need a lot of training data to learn patterns and make good outputs. This data dependency can cause problems like overfitting, which happens when the model does well on training data but doesn’t work well with new, unseen data. Also, the quality of the content that is generated is directly related to the diversity and representativeness of the training data. This means that biased or incomplete datasets can lead to outputs that are wrong or unfair." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"[...] KGs and LLMs can be the foundation for different types of reasoning, complementing each other in intelligent systems. We can use KGs for tasks that require precise, rule-based reasoning and explicit knowledge representation, and LLMs for tasks involving pattern recognition, context understanding, handling ambiguity or incomplete information, and reasoning about graph structures and their derived metrics. However, neither approach inherently possesses common-sense reasoning capabilities comparable to those of humans, and they often fail to make intuitive leaps or understand the implicit context that would be obvious to a person. These limitations underscore the importance of carefully considering the strengths and weaknesses of each approach when designing intelligent systems and potentially developing a powerful hybrid IAS." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"Machine learning was a big change because it let systems learn patterns from data instead of having to follow hardcoded rules. AI could generalize better and change to new inputs thanks to techniques like decision trees and support vector machines. These models still needed a lot of work to get the features right, though, and they couldn’t handle high-dimensional data like images or text very well. The breakthrough happened when deep learning, a type of machine learning that uses neural networks with multiple layers to automatically learn hierarchical representations of data, became popular. Convolutional neural networks (CNNs) for processing images and recurrent neural networks (RNNs) for sequential data changed what AI could do. Architectures like GANs and transformers took things even further by making it possible to do things like make images, understand natural language, and more." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Retrieval mechanisms are essential for addressing some of the key limitations of traditional generative AI models, such as factual inaccuracies, lack of context awareness, and model hallucinations. While generative models excel at creating coherent and fluent content, they often struggle to produce outputs that are factually correct or contextually relevant. This is because these models rely solely on patterns learned during training, without access to real-time or external information. For example, a generative model might generate a plausible-sounding but incorrect answer to a factual question, as it cannot verify the accuracy of its response." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"The foundational form of RAG, often called naive RAG, follows a straightforward pattern. A pipeline retrieves supporting context from external sources such as enterprise documents, knowledge bases, or structured datasets and appends that information to the model’s prompt before inference. In the most common implementation, each document is converted into an embedding, a numerical representation of its semantic meaning, using either the same foundation model or a specialized embedding model. When a user submits a query, the system performs a vector similarity search to find documents whose embeddings most closely match the query’s vector representation, and the retrieved content is concatenated with the user query before being passed to the language model." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Traditional paradigms build systems for specific purposes with structured, homogeneous databases. This approach works for tailored needs but is impractical for complex domains that need to adapt to user characteristics and integrate heterogeneous data. KGs capture connections, enabling relationship discovery through graph pattern matching and traversal. Both the Resource Description Framework (RDF) and Labeled Property Graphs (LPGs) provide machine-readable formats that humans can interpret. KGs emphasize rich, meaningful data representations usable by both humans and machines, enabling a paradigm shift where intelligent behavior is encoded in a unique source of truth." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

12 October 2026

💎🏭SQL Reloaded: QUALIFY in Microsoft Fabric (Before/After)

One recent addition to T-SQL is the QUALIFY clause, which filters rows returned by a query based on a search condition that can reference window (analytic) functions [1]. The QUALIFY clause allows filtering directly in the block in which the window function is defined. According to the documentation QUALIFY can be used directly after HAVING clause:

SELECT select_list
FROM   table_source
[ WHERE <search_condition> ]
[ GROUP BY group_by_specification ]
[ HAVING <search_condition> ]
[ QUALIFY <filter_condition> ]
[ ORDER BY <order_by_expression> [ , ...n ] ]
[ FOR JSON <options> ]

So, for example, if we consider a fictitious Sales Invoices table, to retrieve the customers who bought a product more than once, we had to apply the constraint outside the block in which the window function was defined:

-- product bought more than once by same customer (before)
SELECT *
FROM (-- calculating resultset
	SELECT customer_id
	, product_id
	, SUM(quantity) OVER(PARTITION BY invoice_id, product_id) total_quantity
	FROM [Sales].[sales_invoices]
) DAT
WHERE total_quantity>1

Now, a constraint can be applied directly in the block with the window function via QUALIFY followed by one or more constraints:

-- product bought more than once by same customer (after) 
SELECT customer_id
, product_id
, SUM(quantity) OVER(PARTITION BY invoice_id, product_id) total_quantity
FROM [Sales].[sales_invoices]
QUALIFY SUM(quantity) OVER(PARTITION BY invoice_id, product_id)>1

The window function's logic must be repeated if a second constraint is needed for checking whether the value is within a boundary and, upon case, the logic can become even more complex (e.g. multiple intervals are checked):

-- product bought more than once by same customer (after) 
SELECT customer_id
, product_id
, SUM(quantity) OVER(PARTITION BY invoice_id, product_id) total_quantity
FROM [Sales].[sales_invoices]
QUALIFY SUM(quantity) OVER(PARTITION BY invoice_id, product_id)>1

  AND SUM(quantity) OVER(PARTITION BY invoice_id, product_id)<10

Notes:
1) Having fewer blocks of code improves in theory code's readability and hopefully maintainability when the code doesn't need to be replicated as in the last example. This approach should improve also performance, however further tests are needed to validate this.
2) It's useful to include the same logic used in QUALIFY also in the main SELECT statement, at least for troubleshooting purposes. This makes the code a bit redundant and developers must make sure the code is synchronized when making changes. This is the advantage of the first version.
3) The syntax seems to be supported only in Fabric warehouse, respectively SQL Endpoints for warehouse, lakehouse and SQL database.
4) One should check also whether the syntax is supported by all the features. For example, it's not supported by UPDATE and DELETE statements in warehouse. The restrictions for SQL Endpoints should be considered as well.
5) Developers must consider also portability of code when it's the case.

Happy coding!

References:
[1] Microsoft Learn (2026) SELECT - QUALIFY clause (Transact-SQL) [link]

🤖Prompt Engineering: Hallucinations (Just the Quotes)

"Despite their impressive capabilities, LLMs are not without limitations. One of the most significant challenges is the problem of hallucination, where an LLM generates factually incorrect or misleading information that appears plausible. This is particularly problematic in domains requiring high factual accuracy, such as healthcare, finance, and legal applications. To mitigate hallucinations and enhance the reliability of LLM outputs, Retrieval-Augmented Generation (RAG) has emerged as a powerful technique. RAG works by dynamically retrieving relevant information from an external knowledge source (such as a knowledge graph) at inference time, rather than just relying on pre-trained knowledge. This approach ensures that the model has access to up-to-date and accurate data, grounding answers in verified information rather than generating content purely from its internal representations." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Generative AI tools for coding are sometimes inaccurate. They can produce results that look good but are wrong. This is common with LLMs. They can write code or chat like a person. And sometimes, they share information that’s just plain wrong. Not just a bit off, but totally backwards or nonsense. And they say it so confidently! We call this 'hallucinating', which is a funny term, but it makes sense." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"LLMs are trained on large volumes of data, which inherently provides them with an immense knowledge base and understanding of different languages. Yet, LLMs at their core are complex text completion engines. Since this knowledge and understanding of language is compressed in a very high-dimensional latent space. LLMs end up using these in a very fluid and intelligible way (which often leads to hallucinations). In order to guide LLMs to focus on specific topics or pieces of information to solve certain tasks, (for instance, question-answering from a given piece of text), it is important to provide contextual information explicitly. While most current generations of LLMs have extremely wide context windows, it is recommended to preprocess context into overlapping smaller chunks for better results, reduced latency, and so on. For similar reasons, it is also recommended to preprocess contextual information in clear and task-specific formats. This aspect of context preprocessing is extremely useful in Retrieval-Gugmented Generation (RAG) scenarios." (Joseph Babcock & Raghav Bali, "Generative AI with Python and PyTorch" 2nd. Ed., 2025)

"A 'hallucination' in the context of generative AI refers to the phenomenon where a model produces information that is factually incorrect, nonsensical, or not grounded in its input data or pre-existing knowledge. These are not mere typos or minor inaccuracies; they are confident, coherent, and often persuasive fabrications. In high-stakes domains like healthcare, law, or finance, a single hallucination can have severe consequences, eroding user trust and leading to catastrophic decision-making. While all LLMs are prone to this, the RAG architecture is specifically designed to combat it by tethering the model’s output to an external, verifiable knowledge base. Understanding why hallucinations occur is the essential first step to building more reliable and truthful AI systems." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Another big problem is model hallucinations, which happen when generative models make content that seems real but is actually wrong or made up. For instance, a language model could write a news story or a medical diagnosis that has wrong information. This happens because these models value coherence and fluency more than factual accuracy. When the training data is not enough or is not clear, they often 'fill in the gaps' with made-up information." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Beyond intentionally misleading content, GenAI systems can produce inaccurate information unintentionally. LLMs are prone to hallucination, generating plausible but false statements with the same confidence as accurate ones. In enterprise contexts, this poses particular risks: an AI assistant might report incorrect financial figures, fabricate customer details, or misrepresent historical trends. Organizations deploying GenAI must implement validation mechanisms, human oversight, and retrieval-augmented approaches that ground model outputs in verified data sources." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Ensuring that a language model reliably retrieves and presents correct information, often referred to as factual recall, is critical for any production-grade application. Whether you’re building an internal helpdesk assistant, a medical Q&A system, or an automated compliance auditor, users expect concise, accurate answers that align with up-to-date source material. Unfortunately, without explicit context, even the most powerful LLM can hallucinate or omit key facts." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"GenAI models are remarkably powerful, but they carry an inherent limitation rooted in how they are built. Every foundation model has a training cutoff, a point in time beyond which it has no awareness of world events, product changes, regulatory updates, or organizational developments. This knowledge gap is not a flaw that can be patched with better prompting. It’s a structural consequence of how models are trained, and it grows wider every day the model remains in production without retraining. Combined with the tendency to hallucinate when asked about topics outside their training distribution, models operating on stale knowledge can confidently deliver responses that are factually incorrect, dangerously outdated, or simply no longer relevant to the user’s context." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Generative artificial intelligence (GenAI), powered by large language models (LLMs) like Google’s Gemini and OpenAI’s GPT, has transformed how we work and live, revolutionizing business after business. Despite this success, generative AI falls short in domains where specific domain knowledge, high accuracy, and explainability are essential. And it has other significant limitations, including hallucinations and a lack of context and relations. This is where knowledge graphs (KGs) come in, provid-ing contextual information - such as experiences, environmental characteristics, cultural aspects, and social normsneeded to build the 'third wave of AI' for mission-critical applications." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"KGs are sophisticated graph structures that represent real-world entities (people, places, diseases, proteins), define meaningful connections between them, and provide context. KGs provide structured, explainable knowledge representation but are challenging to build and query; LLMs offer natural language processing capabilities but suffer from hallucinations, stale information, and a lack of domain-specific grounding. Together, they are a 'killer combination': LLMs can extract entities and relationships from unstructured text to build KGs more efficiently, providing more autonomous and powerful graph querying and analysis. Meanwhile, KGs provide reliable, up-to-date domain knowledge to ground LLM responses and prevent hallucinations." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"[...] LLMs raise serious concerns about ethics, bias and fairness, errors in reasoning, hallucinations, and misuse (e.g., misinformation and disinformation). These concerns are exacerbated by modern LLMs being both literal and figurative 'black boxes': Literal black boxes because many advanced AI systems are proprietary and the weights (trained parameters of the models) are not released to the public; and figurative black boxes because even the open-source AI models are so complicated that understanding them and developing safety guardrails has thus far proven extremely difficult." (Mike X Cohen,"50 ML Projects To Understand LLMs", 2026)

"Misinformation is false or misleading information, and its generation by AI is particularly dangerous because of the aura of credibility these systems can project. In a RAG system, misinformation primarily arises from two failure points: Retrieval of inaccurate content from the knowledge base and fabrication or distortion by the large language model (LLM) during generation, even when given good context. The first line of defense is ensuring the integrity of the knowledge base. A RAG system is only as reliable as the documents it has access to. If non-credible, manipulated, or satirical sources are ingested, the system will retrieve and use them as fact. This makes rigorous data curation and source validation the most critical step in combating misinformation. The second line of defense is strengthening the connection between retrieval and generation to prevent the LLM from 'going off script'. The LLM, based on its pre-trained knowledge, might confidently generate an answer that contradicts the provided evidence or adds unsupported details – a phenomenon known as 'hallucination'. To mitigate this, thesystem must be designed to strictly adhere to the retrieved context." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"RAG applications must be built with semantics, metadata, and governance in mind. The retrieved information must be high-quality, secure, and appropriate for the user’s role. Equally important is monitoring and management: checking whether source data has changed, ensuring vector stores remain accurate, and watching for hallucinations or data leakage. Organizations are definitely starting to experiment with RAG models today; some are putting them into production applications. Some believe that using RAG helps mitigate hallucinations because it is grounded in trusted organizational data." (Fern Halper, "Data Makes the World Go 'Round", 2026)

"The foundational premise of RAG is that the most effective way to reduce hallucinations is to provide the LLM with the correct, explicit information needed to answer a query, thereby minimizing its need to rely on fallible parametric knowledge. Therefore, the quality, relevance, and accuracy of the retrieval step are the most significant factors in determining the truthfulness of the final output. Better retrieval is the most powerful antidote to hallucination. If the retriever fails to find the correct information, the generator is essentially left to guess, making hallucinations almost inevitable. The goal is to create a tight, unambiguous link between the user’s question and the evidence in the knowledge base." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

11 October 2026

🪙Business Intelligence: Accountability (Just the Quotes)

"The chaos without a data strategy is not as obvious, but the indicators abound: dirty data, redundant data, inconsistent data, the inability to integrate, poor performance, terrible availability, little accountability, users who are increasingly dissatisfied with the performance of IT, and the general feeling that things are out of control." (Sid Adelman et al, "Data Strategy", 2005)

"Good governance is less about structure and rules than being focused, effective and accountable." (Pearl Zhu, "Digitizing Boardroom: The Multifaceted Aspects of Digital Ready Boards", 2016)

"Data governance presents a clear shift in approach, signals a dedicated focus on data management, distinctly identifies accountability for data, and improves communication through a known escalation path for data questions and issues. In fact, data governance is central to data management in that it touches on essentially every other data management function. In so doing, organizational change will be brought to a group is newly - and seriously - engaging in any aspect of data management." (Mike Fleckenstein & Lorraine Fellows, "Modern Data Strategy", 2018)

"Data is owned by the enterprise, not by systems or individuals. The enterprise should recognize and formalize the responsibilities of roles, such as data stewards, with specific accountabilities for managing data. A data governance framework and guidelines must be developed to allow data stewards to coordinate with their peers and to communicate and escalate issues when needed. Data should be governed cooperatively to ensure that the interests of data stewards and users are represented and also that value to the enterprise is maximized." (Mike Fleckenstein & Lorraine Fellows, "Modern Data Strategy", 2018)

"[...] the governance function is accountable to define what constitutes data quality and how each data product communicates that in a standard way. It’s no longer accountable for the quality of each data product. The platform team is accountable to build capabilities to validate the quality of the data and communicate its quality metrics, and each domain (data product owner) is accountable to adhere to the quality standards and provide quality data products." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"The premise of OKRs is to keep objectives and results simple and flexible, ensuring they align with business goals and enterprise initiatives guided by regular reviews to assess progress during the quarter. The intent is to keep OKRs clear and accountable, as well as measurable, with between three and five objectives recommended at a high level that can each be tracked by three to five key measures. They should be ambitious goals, even uncomfortable, in challenging aspirations, making them stretch targets." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"Manage data as a strategic asset that evolves into a data product. The premise here is to stop managing data as a byproduct and create an ecosystem that manages data as a valuable strategic asset that can evolve into a data product. Data producers are accountable for managing the life cycle of data from creation to end of life and ensuring it creates business value along the way for data consumers. This requires data that is governed, trusted, protected, secure, and easily accessible. Move data from technical data assets to Data Products by operationalizing data for high scale sharing." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data Mesh addresses data governance challenges by advocating for a federated governance model, which positions accountability for governance with the data owners who are most knowledgeable about the data. In this model, governance is decentralized, with each domain team responsible for the governance of its data products. This approach ensures that governance decisions are made by those who have the deepest understanding of the data’s context, use, and risks. It leads to more relevant, efficient, and effective governance practices that are closely aligned with the specific needs of each domain." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Data mesh fundamentally reframes data governance and validation by distributing accountability to domain-oriented teams who act as custodians and producers of their respective data products. These teams possess intimate domain knowledge, which is essential for nuanced validation criteria that adapt to the semantics, context, and evolution of their datasets. By treating datasets as first-class products with clear ownership, interfaces, and service-level objectives, data mesh encourages autonomous validation workflows embedded directly within the domains where data originates and is consumed." (William Smith, "Great Expectations for Modern Data Quality: The Complete Guide for Developers and Engineers", 2025)

"Effective management of data quality at scale requires a clear delineation of organizational roles and operational frameworks that ensure accountability, consistency, and continuous improvement. Central to this structure are the interrelated concepts of data governance, data stewardship, and operational ownership. Each serves distinct, yet complementary purposes in embedding responsibility within technology platforms, business processes, and organizational culture." (William Smith, "Great Expectations for Modern Data Quality: The Complete Guide for Developers and Engineers", 2025)

10 October 2026

🪙Business Intelligence: Availability (Just the Quotes)

"Unfortunately, just collecting the data in one place and making it easily available isn’t enough. When operational data from transactions is loaded into the data warehouse, it often contains missing or inaccurate data. How good or bad the data is a function of the amount of input checking done in the application that generates the transaction. Unfortunately, many deployed applications are less than stellar when it comes to validating the inputs. To overcome this problem, the operational data must go through a 'cleansing' process, which takes care of missing or out-of-range values. If this cleansing step is not done before the data is loaded into the data warehouse, it will have to be performed repeatedly whenever that data is used in a data mining operation." (Joseph P Bigus,"Data Mining with Neural Networks: Solving business problems from application development to decision support", 1996)

"Dashboards and visualization are cognitive tools that improve your 'span of control' over a lot of business data. These tools help people visually identify trends, patterns and anomalies, reason about what they see and help guide them toward effective decisions. As such, these tools need to leverage people's visual capabilities. With the prevalence of scorecards, dashboards and other visualization tools now widely available for business users to review their data, the issue of visual information design is more important than ever." (Richard Brath & Michael Peters, "Dashboard Design: Why Design is Important," DM Direct, 2004)

"A good data strategy is not determined by what data is readily or potentially available – it’s about what your business wants to achieve, and how data can help you get there." (Bernard Marr, "Data Strategy", 2017)

"Data Lake induces accessibility and catalyzes availability. It warrants data discovery platforms to soak the data trends at a horizontal scale and produce visual insights. It largely cuts down the time that goes into data preparation and exhaustive data analysis." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"The desire to collect as much data as possible must be balanced with an approximation of which data sources are useful to address a business issue. It is worth mentioning that often the value of internal data is high. Most internal data has been cleansed and transformed to suit the mission. It should not be overlooked simply because of the excitement of so much other available data." (Mike Fleckenstein & Lorraine Fellows, "Modern Data Strategy", 2018)

"Using data science, we can uncover the important patterns in a data set, and these patterns can reveal the important attributes in the domain. The reason why data science is used in so many domains is that it doesn’t matter what the problem domain is: if the right data are available and the problem can be clearly defined, then data science can help." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"It’s important to look at multiple hypotheses. If your initial hypothesis isn’t supported, you’ll have little direction for moving forward. […] Thinking through viable alternative hypotheses will help you get a sense of where to find data in your organization. […] start by brainstorming. Think about the information that may be available. Is there a way it might be relevant? What sources are available to you?" (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"DataOps requires that data scientists, data analysts, and data engineers have quick access to data, tools, and infrastructure to eliminate bottlenecks. That is, they need to be able to access, add, or modify data quickly by themselves. We term this availability to data self-service. Through self-service, data analytics professionals can create data products in far less time than with traditional approaches." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020)

"Timeliness means that information is available when it is needed. Most managers function in a dynamic environment of change, demands updated and current information. Computerised information systems have the ability to gather, sort, analyse, store, retrieve, and transmit large amounts of information in a very short period of time. Completeness of information is the extent to which information is all there." (C S V Murthy, "Data and Businesss Analytics", 2020)

"At first, we threw all of this data into a pit called the 'data lake'. But we soon discovered that merely throwing data into a pit was a pointless exercise. To be useful - to be analyzed - data needed to (1) be related to each other and (2) have its analytical infrastructure carefully arranged and made available to the end user. Unless we meet these two conditions, the data lake turns into a swamp, and swamps start to smell after a while. [...] In a data swamp, data just sits there are no one uses it. In the data swamp, data just rots over time." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"The term data governance is used for the processes and responsibilities that define, manage, and enforce access, privacy, availability, and security of the organization’s data. It typically includes a set of policies, rules, and data classifications and functionality to monitor and enforce compliance. As stated earlier, we use the term AI governance in a broader sense, also including AI artefacts." (Eberhard Hechler et al, "Data Fabric and Data Mesh Approaches with AI", 2023)

"Be aware that data product is not the same thing as data as a product. Data as a product describes the idea that data owners treat data as a fully contained product that they are responsible for, rather than a byproduct of a process that others manage, and should make the data available to other domains and consumers. Data product refers to the architecture of implementing data as a product." (James Serra, "Deciphering Data Architectures", 2024)

"The data fabric architecture is an evolution of the modern data warehouse (MDW) architecture: an advanced layer built onto the MDW to enhance data accessibility, security, discoverability, and availability. [...] The most important aspect of the data fabric philosophy is that a data fabric solution can consume any and all data within the organization." (James Serra, "Deciphering Data Architectures", 2024)

"Timeliness captures the degree to which data is available when needed and reflects the relevant time frame of the underlying phenomena." (William Smith, "Soda Core for Modern Data Quality and Observability: The Complete Guide for Developers and Engineers", 2025)

🤖Prompt Engineering: Issues (Just the Quotes)

"Agentic workflows break when the logic is messy - if, say, the plans don’t decompose or memory is poorly structured. However, infrastructure-level LLM applications introduce even more failure points and complexity. If the protocols don’t sync with each other, or the data flows start leaking, or the model boundaries are unclear... there are far too many failure points to count. While most people have been jumping on the bandwagon to adopt MCPs or A2A, very few are equipped to handle the LLMOps issues these tools introduce." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"As with many other deep learning-based approaches, another major challenge is in interpretability. While knowledge graphs provide a structured and transparent way to store relationships, LLMs operate as a black box, making it difficult to understand how specific outputs are generated. [...] Data alignment is also a key issue, as structured knowledge graphs and unstructured text data must be carefully preprocessed to ensure consistency. Differences in data formats, ontology mismatches, and information redundancy can create inefficiencies when integrating these two paradigms. Developing robust pipelines that seamlessly connect graph-based insights with LLM-generated text remains an open challenge." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"The art of mega-prompts spanning multiple written pages and looking like essays has become commonplace for complex tasks when building applications to get things `just right'. Unfortunately, they bring with them lots of issues: errors, portability, complexity, and more. The GenAI world didn’t plan for mega-prompts. They have simply evolved into what they’ve become today because practitioners kept wanting to do more and more complex things, and their only way to express those intents was with a prompt. But step back and look at some of these prompts [...] Lurking just below the surface are a bunch of classical computing concepts like data, programming instructions, control flows, memory, and stora - all the components typically associated with classical computing elements." (Rob Thomas et al, "AI Value Creators: Beyond the Generative AI User Mindset", 2025)

"A solid data foundation is critical for AI success. The foundation can grow in phases, but it needs to be there. AI will add pressure to the data foundation. Modern workloads require data that is connected, interpretable, high quality, and governed. Modern workloads utilize diverse data, and this is especially true with the popularity of GenAI. Trustworthy AI depends on knowing where the data came from and how it has changed. Modern architectures including the lakehouse and the data fabric exist because the traditional ways of managing data no longer fit the scale or complexity of what organizations need to do today. In other words, there is no path toward enterprise AI without a strong data foundation. This was echoed in the expert advices. Experts spoke about the dangers of 'vanity projects', particularly with GenAI, when foundational data issues were ignored. Others described how their companies moved faster specifically because they had already invested in governance, semantic layers, lineage, and observability." (Fern Halper, "Data Makes the World Go 'Round", 2026)

"As AI systems become more autonomous and integrated into decision-making processes, the question of accountability grows in importance. Businesses and developers must ensure that AI systems operate within predefined ethical boundaries and that there are mechanisms to identify and correct issues when things go wrong. Because AI models can make decisions that directly impact people’s lives (such as credit approvals, hiring decisions, or healthcare recommendations), organizations must maintain accountability for these outcomes. Clear guidelines should be in place to determine who is responsible when an AI system makes a mistake or causes harm." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Driven by a fear of missing out (FOMO), many companies have launched AI initiatives. But many failed. For example, customer service chatbots have been introduced to handle inquiries, reduce costs and improve efficiency. But chatbots failed to grasp the complexity of customer issues; they created frustration rather than solving problems. Customers were stuck in repetitive loops, being asked the same questions, and unable to reach a human when needed. Instead of enhancing the customer experience, they delivered spikes in customer complaints instead." (Alan Watkins & G C Cooke, "Smarter than You Winning in Business with Superintelligent AI", 2026)

"With autonomous agents there is the risk that they can take action that isn’t governed. These systems introduce autonomy, interdependence between agents, and possibly emergent behavior. [...] Because agents act autonomously, a single misconfigured or misaligned agent can propagate errors at scale. In multiagent environments, one faulty output can trigger many downstream mistakes. Other risks include tool misuse, conflicting goals among agents (and other interoperability issues), and operational opacity, when teams cannot easily determine which agent took which action or why. Governance must extend to agent registration, version control, permissioning, and simulation testing before deployment. Likewise, accountability may also blur. If an agent takes a dangerous action, who is responsible? There are, of course, cybersecurity risks as agents pose a new attack surface." (Fern Halper, "Data Makes the World Go 'Round", 2026)

09 October 2026

🪙Business Intelligence: Single Source of Truth [SSoT] (Just the Quotes)

"It is important to remember that the 'single version of the truth' - or enterprise logical data model - is not and should not be built all at once (that would take too long), but that it evolves over time as the project-specific logical data models are merged, one-by-one, a project at a time." (Sid Adelman et al, "Data Strategy", 2005

"Having multiple data lakes replicates the same problems that were created with multiple data warehouses - disparate data siloes and data fiefdoms that don't facilitate sharing of the corporate data assets across the organization. Organizations need to have a single data lake from which they can source the data for their BI/data warehousing and analytic needs. The data lake may never become the 'single version of the truth' for the organization, but then again, neither will the data warehouse. Instead, the data lake becomes the 'single or central repository for all the organization's data' from which all the organization's reporting and analytic needs are sourced." (Billl Schmarzo, "Driving Business Strategies with Data Science: Big Data MBA" 1st Ed., 2015)

"Data warehousing, as we are aware, is the traditional approach of consolidating data from multiple source systems and combining into one store that would serve as the source for analytical and business intelligence reporting. The concept of data warehousing resolved the problems of data heterogeneity and low-level integration. In terms of objectives, a data lake is no different from a data warehouse. Both are primary advocates of terms like 'single source of truth' and 'central data repository'." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Another myth is that we shall have a single source of truth for each concept or entity. […] This is a wonderful idea, and is placed to prevent multiple copies of out-of-date and untrustworthy data. But in reality it’s proved costly, an impediment to scale and speed, or simply unachievable. Data Mesh does not enforce the idea of one source of truth. However, it places multiple practices in place that reduces the likelihood of multiple copies of out-of-date data." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data mesh [...] reduces points of centralization that act as coordination bottlenecks. It finds a new way of decomposing the data architecture without slowing the organization down with synchronizations. It removes the gap between where the data originates and where it gets used and removes the accidental complexities - aka pipelines - that happen in between the two planes of data. Data mesh departs from data myths such as a single source of truth, or one tightly controlled canonical data model." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Unlike other analytical data management paradigms, data mesh does not embrace the concept of the mythical single source of truth. Every data product provides a truthful portion of the reality - for a particular domain - to the best of its ability, a single slice of truth." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Lakehouse is a new architecture and data storage paradigm that combines the characteristics of both data warehouses and data lakes to create a unified basis for all types of use cases to be built on top of it. There is no need to move data around. Data is curated and remains in an open format and serves as the single source of truth (SSOT) for all the consumption layers. A modern data platform has needs that span traditional data warehouses, data lakes, machine learning systems, and streaming systems and there is some overlap among these systems. A Lakehouse offers features that span all four systems [...]" (Anindita Mahapatra, "Simplifying Data Engineering and Analytics with Delta", 2022)

"The Data Fabric architecture needs to guarantee this single version of the truth within the application and transactional landscape, which – depending on the deployment option of an MDM solution – could also mean to assemble this single version of the truth based on core information that is dispersed and maintained in various data stores." (Eberhard Hechler et al, "Data Fabric and Data Mesh Approaches with AI", 2023)

"The main goal of the transaction log is to enable multiple readers and writers to operate on a given version of a dataset file simultaneously and to provide additional information, like data skipping indexes to the execution engine for more performant operations. The Delta Lake transaction log always shows the user a consistent view of the data and serves as a single source of truth. It is the central repository that tracks all changes the user makes to a Delta table." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"The hub and spoke, or 'star network', is a data architecture model that centralizes data from various sources into a single hub, such as a data warehouse or data lake. The hub serves as the source of truth for data and provides standardized schemas and formats. The spokes are the various applications or services that consume data from the hub for different purposes, such as analytics, reporting, or ma-chine learning. Spokes can also perform transformations or aggregations on data before presenting it to end users. The hub and spoke architecture aims to simplify data integration and management by reducing complexity and redundancy in data pipelines" (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

🤖Prompt Engineering: Trust (Just the Quotes)

"If ethical lapses or AI failures occur, the impact on a business can be significant. Misinformation, biases, or harmful content generated by AI can lead to reputational damage, customer distrust, and potential regulatory scrutiny. The public relations fallout from an AI-driven error or ethical misstep can erode consumer confidence, resulting in lost revenue and lasting harm to brand image. Businesses, therefore, need to proactively address ethical considerations in AI implementation, not only to ensure compliance but also to protect and strengthen their reputation in a highly competitive, and increasingly transparent, marketplace." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"The power of RAG systems stems from their ability to access and process vast amounts of data. However, this very capability introduces significant risks regarding the privacy of individuals and the security of sensitive information. Unlike a simple chatbot, a RAG system often has access to proprietary corporate data, internal documentation, and potentially personal user information within its knowledge base. A data breach or misuse of this information can lead to severe financial, legal, and reputational damage. Furthermore, a global patchwork of stringent regulations now governs how personal data must be handled, making compliance a central pillar of AI system design, not an afterthought. Ethical deployment requires an architecture built on privacy by design and by default, ensuring user trust is maintained through robust technical and procedural safeguards." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"The promise of AI is its ability to process information objectively and at scale. However, this promise is fundamentally threatened by the twin challenges of bias and misinformation. AI systems are not born in a vacuum; they are created by humans and trained on data produced by humans. Consequently, they are prone to inheriting and even amplifying our prejudices, errors, and the systemic inequalities present in that data. In a RAG system, this risk is a two-fold problem: first in the retrieval of information, and second in the generation of a response based on that retrieval. A failure to address these issues doesn’t just lead to technically incorrect outputs; it can perpetuate social harm, erode public trust, and lead to the widespread dissemination of falsehoods. Therefore, understanding and mitigating bias and misinformation is not an optional add-on but a core requirement for any ethically deployed AI system." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)