🪙Business Intelligence: Ownership (Just the Quotes)

"[...] the governance function is accountable to define what constitutes data quality and how each data product communicates that in a standard way. It’s no longer accountable for the quality of each data product. The platform team is accountable to build capabilities to validate the quality of the data and communicate its quality metrics, and each domain (data product owner) is accountable to adhere to the quality standards and provide quality data products." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"[...] data mesh introduces a fundamental shift that the owners of the data products must communicate and guarantee an acceptable level of quality and trustworthiness - specific to their domain - as an intrinsic characteristic of their data product. This means cleansing and running automated data integrity tests at the point of the creation of a data product." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"A core premise of data mesh is federating data ownership among domain data owners who are responsible for their data as a product. Offering the data as a product requires the data to be discoverable and to have explicitly stated quality characteristics and a clearly defined access method. Such requirements are at the core of what data catalogs support. With support for data labeling, curation, and crowdsourced feedback, data catalogs are well positioned to offer data as a product. Furthermore, data catalogs support the enforcement of compliant data usage, which becomes more important when data ownership is not managed centrally." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"A Data Mesh views data primarily as organized around domain owners who create business-focused data products, which can be aggregated and consumed across distributed consumers, organizations, and Line of Business (LoBs) in a self-service and shopping-for-data fashion. Transforming data from disparate data sources to be consumed as data-as-a-product is an essential paradigm of any Data Mesh." (Eberhard Hechler et al, "Data Fabric and Data Mesh Approaches with AI", 2023)

"A domain has two main roles: data product engineer (or just data engineer) and the data product owner (or data product manager, or data steward). These roles can be the same or dedicated people in the domain. Data product owners must have a deep understanding of who their data consumers are, how the data is used, and what methods are used to consume the data. This will help ensure that the data products meet the needs of their use cases. Data product engineers are responsible for creating data products that are high quality, reliable, and usable by consumers. It should be possible to extend existing domain roles to include these domain roles with minimal effort." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Building a data product is enabled by the data domain owner; however, building a data product itself is primarily driven by the data product owner, which can be a marketing or a customer care organization, an after-sales team, or even an individual business user. The data product owner is collaborating with data engineers, data scientists, and other subject matter experts throughout the entire data product build process." (Eberhard Hechler et al, "Data Fabric and Data Mesh Approaches with AI", 2023)

"Data mesh architectures are inherently decentralized, and significant responsibility is delegated to the data product owners. A data mesh also benefits from a degree of centralization in the form of data product compatibility and common self-service tooling. Differing opinions, preferences, business requirements, legal constraints, technologies, and technical debt are just a few of the many factors that influence how we work together." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"In Exploiting semantic knowledge graphs can support interpretability and explainability of nearly all AI model types (including DL models) by discovering and depicting semantic and non-obvious relationships or depicting an ML model in a simplified and more readable, explainable way., a Data Mesh solution organizes data around business domain owners and transforms relevant data assets (data sources) to data products that can be consumed by distributed business users from various business domains or functions. These data products are created, governed, and used in an autonomous, decentralized, and self-service manner. Self-service capabilities, which we have already referenced as a Data Fabric capability, enable business organizations to entertain a data marketplace with shopping-for-data characteristics." (Eberhard Hechler et al, "Data Fabric and Data Mesh Approaches with AI", 2023)

"Be aware that data product is not the same thing as data as a product. Data as a product describes the idea that data owners treat data as a fully contained product that they are responsible for, rather than a byproduct of a process that others manage, and should make the data available to other domains and consumers. Data product refers to the architecture of implementing data as a product." (James Serra, "Deciphering Data Architectures", 2024)

"Data Mesh advocates for domain-driven ownership of data, enabling individual teams to manage and share their data effectively while aligning with the overall organizational objectives. By embracing this paradigm, enterprises can gradually dismantle the barriers of data silos, paving the way for a more integrated, agile, and data-centric organizational culture." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"[...] Data Mesh introduces agility into the data landscape, emphasizing decentralized ownership, responsive data management, and collaborative cross-functional teams. Just as Agile promotes self-organizing teams, Data Mesh advocates for domain-oriented decentralized ownership, putting the power of data in the hands of individual domain teams. In an Agile context, customer collaboration involves continuous engagement with stakeholders to understand their evolving needs. Likewise, Data Mesh encourages domain teams to engage with data consumers within their organization, gathering feedback, and iterating on their data products to meet their specific requirements." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"[...] Data Mesh is, then, at its foundation, a conceptual framework in the realm of data architecture, which emphasizes decentralized data ownership and architecture. It recognizes that in large organizations, data is vast and varied, where each business domain has autonomy over its own data. By decentralizing control, it empowers individual domains to manage and make decisions about their data while maintaining a cohesive overall structure. And presumably, with this autonomy comes better, more localized, and faster decisions, which in-turn, leads to speed and agility." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Federation is about providing autonomy to each data product owner to make their own decisions about the storage, computing, and sharing of data. However, this autonomy cannot come at a risk to the security and compliance standards of the company." (Aniruddha Deswandikar, "Engineering Data Mesh in Azure Cloud", 2024)

"First, good data products align to decentralized domain ownership - they should align to a domain (large or small) with a clear boundary, and should have an empowered owner. Second, good data products are treated as their name implies - as a product and not a project. Good data products, like other products, have a life cycle, have clear consumers, and a clear value proposition. Good data products are self-serve, meaning that users can get what they need from a data product without undue participation from third parties. Good data products also have a federated governance mechanism, that is to say, that provides for local autonomy and decision making - at the data product level by the data product owner and their team. It also means that while being federated, data product owners/teams are responsible and accountable to ensuring their data product adheres to enterprise guidelines and standards as needed." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Promoting domain-oriented ownership is to combat the common problem of organizational silos. Silos can significantly hinder the free flow of data and expertise, making decision-making and innovation more challenging. We aim to break down these barriers by advocating for domain-oriented ownership and creating a more dynamic and collaborative data management landscape." (Pradeep Menon, "Data Mesh Principles, patterns, architecture, and strategies for data-driven decision making", 2024)

"The role of an empowered owner is multifaceted. They are tasked with ensuring that the data product aligns with both specific business requirements and the overarching governance framework. This alignment is crucial for maintaining the integrity and usefulness of the data product, ensuring it remains a valuable asset within the organization’s data landscape." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Data mesh fundamentally reframes data governance and validation by distributing accountability to domain-oriented teams who act as custodians and producers of their respective data products. These teams possess intimate domain knowledge, which is essential for nuanced validation criteria that adapt to the semantics, context, and evolution of their datasets. By treating datasets as first-class products with clear ownership, interfaces, and service-level objectives, data mesh encourages autonomous validation workflows embedded directly within the domains where data originates and is consumed." (William Smith, "Great Expectations for Modern Data Quality: The Complete Guide for Developers and Engineers", 2025)

🤖Prompt Engineering: Hallucinations (Just the Quotes)

"Despite their impressive capabilities, LLMs are not without limitations. One of the most significant challenges is the problem of hallucination, where an LLM generates factually incorrect or misleading information that appears plausible. This is particularly problematic in domains requiring high factual accuracy, such as healthcare, finance, and legal applications. To mitigate hallucinations and enhance the reliability of LLM outputs,  Retrieval-Augmented Generation (RAG) has emerged as a powerful technique. RAG works by dynamically retrieving relevant information from an external knowledge source (such as a knowledge graph) at inference time, rather than just relying on pre-trained knowledge. This approach ensures that the model has access to up-to-date and accurate data, grounding answers in verified information rather than generating content purely from its internal representations." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Generative AI tools for coding are sometimes inaccurate. They can produce results that look good but are wrong. This is common with LLMs. They can write code or chat like a person. And sometimes, they share information that’s just plain wrong. Not just a bit off, but totally backwards or nonsense. And they say it so confidently! We call this 'hallucinating', which is a funny term, but it makes sense." (Jeremy C Morgan, "Coding with AI: Examples in Python", 2025)

"LLMs are trained on large volumes of data, which inherently provides them with an immense knowledge base and understanding of different languages. Yet, LLMs at their core are complex text completion engines. Since this knowledge and understanding of language is compressed in a very high-dimensional latent space. LLMs end up using these in a very fluid and intelligible way (which often leads to hallucinations). In order to guide LLMs to focus on specific topics or pieces of information to solve certain tasks, (for instance, question-answering from a given piece of text), it is important to provide contextual information explicitly. While most current generations of LLMs have extremely wide context windows, it is recommended to preprocess context into overlapping smaller chunks for better results, reduced latency, and so on. For similar reasons, it is also recommended to preprocess contextual information in clear and task-specific formats. This aspect of context preprocessing is extremely useful in Retrieval-Gugmented Generation (RAG) scenarios." (Joseph Babcock & Raghav Bali, "Generative AI with Python and PyTorch" 2nd. Ed., 2025)

"A 'hallucination' in the context of generative AI refers to the phenomenon where a model produces information that is factually incorrect, nonsensical, or not grounded in its input data or pre-existing knowledge. These are not mere typos or minor inaccuracies; they are confident, coherent, and often persuasive fabrications. In high-stakes domains like healthcare, law, or finance, a single hallucination can have severe consequences, eroding user trust and leading to catastrophic decision-making. While all LLMs are prone to this, the RAG architecture is specifically designed to combat it by tethering the model’s output to an external, verifiable knowledge base. Understanding why hallucinations occur is the essential first step to building more reliable and truthful AI systems." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026) 

"Another big problem is model hallucinations, which happen when generative models make content that seems real but is actually wrong or made up. For instance, a language model could write a news story or a medical diagnosis that has wrong information. This happens because these models value coherence and fluency more than factual accuracy. When the training data is not enough or is not clear, they often 'fill in the gaps' with made-up information." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Beyond intentionally misleading content, GenAI systems can produce inaccurate information unintentionally. LLMs are prone to hallucination, generating plausible but false statements with the same confidence as accurate ones. In enterprise contexts, this poses particular risks: an AI assistant might report incorrect financial figures, fabricate customer details, or misrepresent historical trends. Organizations deploying GenAI must implement validation mechanisms, human oversight, and retrieval-augmented approaches that ground model outputs in verified data sources." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Ensuring that a language model reliably retrieves and presents correct information, often referred to as factual recall, is critical for any production-grade application. Whether you’re building an internal helpdesk assistant, a medical Q&A system, or an automated compliance auditor, users expect concise, accurate answers that align with up-to-date source material. Unfortunately, without explicit context, even the most powerful LLM can hallucinate or omit key facts." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"GenAI models are remarkably powerful, but they carry an inherent limitation rooted in how they are built. Every foundation model has a training cutoff, a point in time beyond which it has no awareness of world events, product changes, regulatory updates, or organizational developments. This knowledge gap is not a flaw that can be patched with better prompting. It’s a structural consequence of how models are trained, and it grows wider every day the model remains in production without retraining. Combined with the tendency to hallucinate when asked about topics outside their training distribution, models operating on stale knowledge can confidently deliver responses that are factually incorrect, dangerously outdated, or simply no longer relevant to the user’s context." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"Generative artificial intelligence (GenAI), powered by large language models (LLMs) like Google’s Gemini and OpenAI’s GPT, has transformed how we work and live, revolutionizing business after business. Despite this success, generative AI falls short in domains where specific domain knowledge, high accuracy, and explainability are essential. And it has other significant limitations, including hallucinations and a lack of context and relations. This is where knowledge graphs (KGs) come in, provid-ing contextual information - such as experiences, environmental characteristics, cultural aspects, and social normsneeded to build the 'third wave of AI' for mission-critical applications." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"KGs are sophisticated graph structures that represent real-world entities (people, places, diseases, proteins), define meaningful connections between them, and provide context. KGs provide structured, explainable knowledge representation but are challenging to build and query; LLMs offer natural language processing capabilities but suffer from hallucinations, stale information, and a lack of domain-specific grounding. Together, they are a 'killer combination': LLMs can extract entities and relationships from unstructured text to build KGs more efficiently, providing more autonomous and powerful graph querying and analysis. Meanwhile, KGs provide reliable, up-to-date domain knowledge to ground LLM responses and prevent hallucinations." (Alessandro Negro et al, "Knowledge Graphs and LLMs in Action", 2026)

"[...] LLMs raise serious concerns about ethics, bias and fairness, errors in reasoning, hallucinations, and misuse (e.g., misinformation and disinformation). These concerns are exacerbated by modern LLMs being both literal and figurative 'black boxes': Literal black boxes because many advanced AI systems are proprietary and the weights (trained parameters of the models) are not released to the public; and figurative black boxes because even the open-source AI models are so complicated that understanding them and developing safety guardrails has thus far proven extremely difficult." (Mike X Cohen,"50 ML Projects To Understand LLMs", 2026)

"Misinformation is false or misleading information, and its generation by AI is particularly dangerous because of the aura of credibility these systems can project. In a RAG system, misinformation primarily arises from two failure points: Retrieval of inaccurate content from the knowledge base and fabrication or distortion by the large language model (LLM) during generation, even when given good context. The first line of defense is ensuring the integrity of the knowledge base. A RAG system is only as reliable as the documents it has access to. If non-credible, manipulated, or satirical sources are ingested, the system will retrieve and use them as fact. This makes rigorous data curation and source validation the most critical step in combating misinformation. The second line of defense is strengthening the connection between retrieval and generation to prevent the LLM from 'going off script'. The LLM, based on its pre-trained knowledge, might confidently generate an answer that contradicts the provided evidence or adds unsupported details – a phenomenon known as 'hallucination'. To mitigate this, thesystem must be designed to strictly adhere to the retrieved context." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"RAG applications must be built with semantics, metadata, and governance in mind. The retrieved information must be high-quality, secure, and appropriate for the user’s role. Equally important is monitoring and management: checking whether source data has changed, ensuring vector stores remain accurate, and watching for hallucinations or data leakage. Organizations are definitely starting to experiment with RAG models today; some are putting them into production applications. Some believe that using RAG helps mitigate hallucinations because it is grounded in trusted organizational data." (Fern Halper, "Data Makes the World Go 'Round", 2026)

"RAG models offer several advantages over standard generative models, addressing many of their limitations. Traditional generative models, like GPT, rely solely on patterns learned during training, which can lead to issues such as factual inaccuracies, model hallucinations, and lack of contextual relevance. These models generate content based on pre-existing knowledge, often without the ability to verify or update the information, making them less reliable for tasks requiring high accuracy. In contrast, RAG models integrate retrieval mechanisms that allow them to access external knowledge sources in real time. This ensures that the generated content is grounded in verified data, significantly improving factual consistency and relevance. For example, while a standard generative model might produce a plausible but incorrect answer to a factual question, a RAG model can retrieve and incorporate accurate information from a trusted source, reducing errors." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Retrieval mechanisms are essential for addressing some of the key limitations of traditional generative AI models, such as factual inaccuracies, lack of context awareness, and model hallucinations. While generative models excel at creating coherent and fluent content, they often struggle to produce outputs that are factually correct or contextually relevant. This is because these models rely solely on patterns learned during training, without access to real-time or external information. For example, a generative model might generate a plausible-sounding but incorrect answer to a factual question, as it cannot verify the accuracy of its response." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"The foundational premise of RAG is that the most effective way to reduce hallucinations is to provide the LLM with the correct, explicit information needed to answer a query, thereby minimizing its need to rely on fallible parametric knowledge. Therefore, the quality, relevance, and accuracy of the retrieval step are the most significant factors in determining the truthfulness of the final output. Better retrieval is the most powerful antidote to hallucination. If the retriever fails to find the correct information, the generator is essentially left to guess, making hallucinations almost inevitable. The goal is to create a tight, unambiguous link between the user’s question and the evidence in the knowledge base." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Traditional RAG systems 'guide' LLMs in generating responses by providing document chunks that are related to the user’s initial query. Incorporating external knowledge from a document store helps to reduce hallucinations, but its effectiveness relies heavily on the relevance of the retrieved document chunks. This can be problematic if the retrieval step produces irrelevant document chunks or incorrect information. The key goal of corrective retrieval-augmented generation (CRAG) is to detect and correct hallucinations before the LLM attempts response generation. It adds an evaluator to the process that evaluates the quality of the retrieved chunks for a given query before passing on the documents to the response generation step. The evaluator returns a confidence score that triggers different knowledge retrieval actions." (Valliappa Lakshmanan & Hannes Hapke, "Generative AI Design Patterns: Solutions to Common Challenges When Building GenAI Agents and Applications", 2026)

🤖Prompt Engineering: Retrieval Augmented Generation [RAG] (Just the Quotes)

"RAG is an advanced Natural Language Processing (NLP) technique that combines elements of information retrieval and text generation. First, RAG performs a retrieval step, which queries an external knowledge source, such as a vast text corpus or a structured database. Next, RAG uses this knowledge source to enhance its response generation. RAG integrates the retrieved facts into its generated text." (Angelica Lo Duca, "Data Storytelling with Generative AI: Using Python and Altair", 2024)

"As the tech industry moves from non-generative models to generative models, it is shifting away from feature engineering, or creating features to model the data and experimenting with different hyperparameters to optimize performance. Generative models, and specifically LLMs, do not require feature engineering. Today, the core requirements are usually prompt engineering or building a RAG pipeline - skills that lie within the domain of AI engineers." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"Despite their impressive capabilities, LLMs are not without limitations. One of the most significant challenges is the problem of hallucination, where an LLM generates factually incorrect or misleading information that appears plausible. This is particularly problematic in domains requiring high factual accuracy, such as healthcare, finance, and legal applications. To mitigate hallucinations and enhance the reliability of LLM outputs,  Retrieval-Augmented Generation (RAG) has emerged as a powerful technique. RAG works by dynamically retrieving relevant information from an external knowledge source (such as a knowledge graph) at inference time, rather than just relying on pre-trained knowledge. This approach ensures that the model has access to up-to-date and accurate data, grounding answers in verified information rather than generating content purely from its internal representations." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

 "There are three techniques for model domain adaptation: prompt engineering, RAG, and fine-tuning. Strictly speaking, RAG is a form of dynamic prompt engineering where developers use a retrieval system to add content to an existing prompt, but RAG systems are used so often that it’s worth discussing them separately. One critical difference with fine-tuning is that you must have access to the model’s weights, information that is usually not available with cloud-based, proprietary LLMs." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"RAG is a framework that combines the strengths of traditional information retrieval systems with the generative capabilities of LLMs. In this setup, an LLM is augmented with a retrieval component that fetches relevant information from external data sources, such as knowledge bases or databases, to produce more accurate and contextually relevant responses. This method enhances the LLM’s output by grounding it in authoritative, up-to-date information." (Aldo Marzullo et al, "Graph Machine Learning" 2nd Ed., 2025)

"Vector databases are designed to store and index high-dimensional embeddings - dense numeric vectors that capture the semantic meaning of text, images, audio, or other content. Instead of looking for exact matches, they use approximate nearest neighbor (ANN) algorithms to return the items whose vectors lie closest to a query vector in that multidimensional space. This makes them the engine behind semantic search, recommendation systems, image-or-audio similarity matching, and retrieval augmented generation (RAG) pipelines that supply LLM prompts with relevant context in milliseconds." (Abi Aryan, "LLMOps: Managing Large Language Models in Production", 2025)

"A RAG system needs to identify relevant text chunks from the knowledge base. It does that by building a datastore of chunks and organizing the chunks in a way that makes searching for chunks that are relevant to a query efficient and fast. This preparatory step of building an efficient data store is called indexing, and the runtime step of searching the datastore for relevant chunks is called retrieval. [...] Whenever a query is received, a question-answering pipeline is invoked. This consists of two steps: retrieval and generation. In the retrieval step, the retriever finds chunks in the document store that are the most relevant to the query. In the generation step, these relevant chunks are added to the context of the prompt and used along with the user query to generate the answer. In this way, the generated answer is grounded in the relevant information from trusted knowledge sources. The response consists of the generated text as well as the relevant chunks that were used to generate the answer." (Valliappa Lakshmanan & Hannes Hapke, "Generative AI Design Patterns: Solutions to Common Challenges When Building GenAI Agents and Applications", 2026)

"In RAG methods, the AI model itself doesn’t actually 'remember' or learn the proprietary data directly. Instead, the proprietary data are stored separately in what’s called a vector database. When the model is asked a question, it first performs a quick search of the proprietary database, finds relevant pieces of information, and then uses these to generate its response. The model’s core parameters remain entirely unchanged and are never updated with this private data. In this sense, the model hasn’t learned or 'seen' your proprietary data in its internal parameters, it only temporarily consults it as a reference to formulate an answer." (Alan Watkins & G C Cooke, "Smarter than You Winning in Business with Superintelligent AI", 2026)

"Misinformation is false or misleading information, and its generation by AI is particularly dangerous because of the aura of credibility these systems can project. In a RAG system, misinformation primarily arises from two failure points: Retrieval of inaccurate content from the knowledge base and fabrication or distortion by the large language model (LLM) during generation, even when given good context. The first line of defense is ensuring the integrity of the knowledge base. A RAG system is only as reliable as the documents it has access to. If non-credible, manipulated, or satirical sources are ingested, the system will retrieve and use them as fact. This makes rigorous data curation and source validation the most critical step in combating misinformation. The second line of defense is strengthening the connection between retrieval and generation to prevent the LLM from 'going off script'. The LLM, based on its pre-trained knowledge, might confidently generate an answer that contradicts the provided evidence or adds unsupported details – a phenomenon known as 'hallucination'. To mitigate this, the system must be designed to strictly adhere to the retrieved context." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"RAG applications must be built with semantics, metadata, and governance in mind. The retrieved information must be high-quality, secure, and appropriate for the user’s role. Equally important is monitoring and management: checking whether source data has changed, ensuring vector stores remain accurate, and watching for hallucinations or data leakage. Organizations are definitely starting to experiment with RAG models today; some are putting them into production applications. Some believe that using RAG helps mitigate hallucinations because it is grounded in trusted organizational data." (Fern Halper, "Data Makes the World Go 'Round", 2026)

"RAG is a framework that combines the strengths of generative models and retrieval mechanisms to produce more accurate and contextually relevant outputs. The process begins with an input query or prompt, which is used to retrieve relevant information from an external knowledge source, such as a database, document repository, or the internet. This retrieval step ensures that the model has access to up-to-date and verified information, addressing the limitations of traditional generative models that rely solely on pretrained knowledge." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"RAG is a paradigm that combines the strengths of LLMs with the rich, often unstructured data stored in a lakehouse. Rather than asking an LLM to generate responses purely from its internal parameters and training data, where knowledge can be outdated or incomplete, RAG systems first retrieve relevant documents, records, or data slices from your lakehouse and then feed those pieces into the model as context for its generative step. The result is an AI that can speak confidently about the latest reports, proprietary datasets, or domain-specific knowledge you have stored without having to retrain the model each time your data changes." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"RAG models offer several advantages over standard generative models, addressing many of their limitations. Traditional generative models, like GPT, rely solely on patterns learned during training, which can lead to issues such as factual inaccuracies, model hallucinations, and lack of contextual relevance. These models generate content based on pre-existing knowledge, often without the ability to verify or update the information, making them less reliable for tasks requiring high accuracy. In contrast, RAG models integrate retrieval mechanisms that allow them to access external knowledge sources in real time. This ensures that the generated content is grounded in verified data, significantly improving factual consistency and relevance. For example, while a standard generative model might produce a plausible but incorrect answer to a factual question, a RAG model can retrieve and incorporate accurate information from a trusted source, reducing errors." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"The foundational form of RAG, often called naive RAG, follows a straightforward pattern. A pipeline retrieves supporting context from external sources such as enterprise documents, knowledge bases, or structured datasets and appends that information to the model’s prompt before inference. In the most common implementation, each document is converted into an embedding, a numerical representation of its semantic meaning, using either the same foundation model or a specialized embedding model. When a user submits a query, the system performs a vector similarity search to find documents whose embeddings most closely match the query’s vector representation, and the retrieved content is concatenated with the user query before being passed to the language model." (Bennie Haelen, "ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale", 2026)

"The promise of AI is its ability to process information objectively and at scale. However, this promise is fundamentally threatened by the twin challenges of bias and misinformation. AI systems are not born in a vacuum; they are created by humans and trained on data produced by humans. Consequently, they are prone to inheriting and even amplifying our prejudices, errors, and the systemic inequalities present in that data. In a RAG system, this risk is a two-fold problem: first in the retrieval of information, and second in the generation of a response based on that retrieval. A failure to address these issues doesn’t just lead to technically incorrect outputs; it can perpetuate social harm, erode public trust, and lead to the widespread dissemination of falsehoods. Therefore, understanding and mitigating bias and misinformation is not an optional add-on but a core requirement for any ethically deployed AI system." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"The foundational premise of RAG is that the most effective way to reduce hallucinations is to provide the LLM with the correct, explicit information needed to answer a query, thereby minimizing its need to rely on fallible parametric knowledge. Therefore, the quality, relevance, and accuracy of the retrieval step are the most significant factors in determining the truthfulness of the final output. Better retrieval is the most powerful antidote to hallucination. If the retriever fails to find the correct information, the generator is essentially left to guess, making hallucinations almost inevitable. The goal is to create a tight, unambiguous link between the user’s question and the evidence in the knowledge base." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"The power of RAG systems stems from their ability to access and process vast amounts of data. However, this very capability introduces significant risks regarding the privacy of individuals and the security of sensitive information. Unlike a simple chatbot, a RAG system often has access to proprietary corporate data, internal documentation, and potentially personal user information within its knowledge base. A data breach or misuse of this information can lead to severe financial, legal, and reputational damage. Furthermore, a global patchwork of stringent regulations now governs how personal data must be handled, making compliance a central pillar of AI system design, not an afterthought. Ethical deployment requires an architecture built on privacy by design and by default, ensuring user trust is maintained through robust technical and procedural safeguards." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Traditional NLP systems rely on static, pretrained knowledge embedded within their parameters, limiting their responses to information available during training. In contrast, RAG systems dynamically access external knowledge bases in real time, enabling them to provide up-to-date and contextually relevant answers. This fundamental distinction leads to key differences in architecture, performance, and adaptability." (Saloni Garg et al, "RAG Artificial Intelligence: Retrieval-Augmented Generation in Generative AI", 2026)

"Traditional RAG systems 'guide' LLMs in generating responses by providing document chunks that are related to the user’s initial query. Incorporating external knowledge from a document store helps to reduce hallucinations, but its effectiveness relies heavily on the relevance of the retrieved document chunks. This can be problematic if the retrieval step produces irrelevant document chunks or incorrect information. The key goal of corrective retrieval-augmented generation (CRAG) is to detect and correct hallucinations before the LLM attempts response generation. It adds an evaluator to the process that evaluates the quality of the retrieved chunks for a given query before passing on the documents to the response generation step. The evaluator returns a confidence score that triggers different knowledge retrieval actions." (Valliappa Lakshmanan & Hannes Hapke, "Generative AI Design Patterns: Solutions to Common Challenges When Building GenAI Agents and Applications", 2026)

🏭🗒️Microsoft Fabric: Git [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 6-Oct-2025

[Microsoft Fabric] Git

• {def} an open source, distributed version control platform
• enables developers commit their work to a local repository and then sync their copy of the repository with the copy on the server [1]
• to be differentiated from centralized version control 
• where clients must synchronize code with a server before creating new versions of code [1]
• provides tools for isolating changes and later merging them back together
• {benefit} simultaneous development
• everyone has their own local copy of code and works simultaneously on their own branches
•  Git works offline since almost every operation is local
• {benefit} faster release
• branches allow for flexible and simultaneous development
• {benefit} built-in integration
• integrates into most tools and products
•  every major IDE has built-in Git support
• this integration simplifies the day-to-day workflow
• {benefit} strong community support
• the volume of community support makes it easy to get help when needed
• {benefit} works with any team
• using Git with a source code management tool increases a team's productivity 
• by encouraging collaboration, enforcing policies, automating processes, and improving visibility and traceability of work
• the team can either
• settle on individual tools for version control, work item tracking, and continuous integration and deployment
• choose a solution that supports all of these tasks in one place
• e.g. GitHub, Azure DevOps
• {benefit} pull requests
• used to discuss code changes with the team before merging them into the main branch
• allows to ensure code quality and increase knowledge across team
• platforms like GitHub and Azure DevOps offer a rich pull request experience
• {benefit} branch policies
• protect important branches by preventing direct pushes, requiring reviewers, and ensuring clean build
•  used to ensure that pull requests meet requirements before completion
•  teams can configure their solution to enforce consistent workflows and process across the team
• {feature} continuous integration
• {feature} continuous deployment
• {feature} automated testing
• {feature} work item tracking
• {feature} metrics
• {feature} reporting 
• {operation} commit
• snapshot of all files at a point in time [1]
•  every time work is saved, Git creates a commit [1]
•  identified by a unique cryptographic hash of the committed content [1]
•  everything is hashed
•  it's impossible to make changes, lose information, or corrupt files without Git detecting it [1]
•  create links to other commits, forming a graph of the development history [2A]
• {operation} revert code to a previous commit [1]
• {operation} inspect how files changed from one commit to the next [1]
• {operation} review information e.g. where and when changes were made [1]
• {operation} branch
•  lightweight pointers to work in progress
•  each developer saves changes to their own local code repository
• there can be many different changes based on the same commit
•  branches manage this separation
• once work created in a branch is finished, it can be merged back into the team's main (or trunk) branch
• main branch
• contains stable, high-quality code from which programmers release
• feature branches 
• contain work in progress, which are merged into the main branch upon completion
•  allows to isolate development work and minimize conflicts among multiple developers [2]
•  release branch
•  by separating the release branch from development in progress, it's easier to manage stable code and ship updates more quickly
• if a file hasn't changed from one commit to the next, Git uses the previously stored file [1]
• files are in one of three states
• {state}modified
• when a file is first modified, the changes exist only in the working directory
•  they aren't yet part of a commit or the development history
•  the developer must stage the changed files to be included in the commit
•  the staging area contains all changes to include in the next commit
•  {state}committed
•  once the developer is happy with the staged files, the files are packaged as a commit with a message describing what changed
•  this commit becomes part of the development history
•  {state}staged
•  staging lets developers pick which file changes to save in a commit to break down large changes into a series of smaller commits
•   by reducing the scope of commits, it's easier to review the commit history to 
• {best practice} set up a shared Git repository and CI/CD pipelines [2]
• enables effective collaboration and deployment in PBIP [2]
• enables implementing version control in PBIP [2]
• it’s essential for managing project history and collaboration [2]
• allows to track changes throughout the model lifecycle [2]
• allows to enable effective governance and collaboratin
•  provides robust version tracking and collaboration features, ensuring traceability
• {best practice} use descriptive commit messages [2]
• allows to ensure clarity and facilitate collaboration in version control [2]
• {best practice} avoid sharing Git credentials [2]
• compromises security and accountability [2]
•  can lead to potential breaches [2]
• {best practice} define a naming conventions for files and communicated accordingly [2]
• {best practice} avoid merging changes directly into the master branch [2]
• {risk} this can lead to integration issues [2]
• {best practice} use git merge for integrating changes from one branch to another [2]
• {benefit} ensures seamless collaboration [2]
• {best practice} avoid skipping merges [2]
• failing to merge regularly can lead to complex conflicts and integration challenges [2]

Previous Post <<||>> Next Post 

References:

[1] Microsoft Learn (2022) DeveOps: What is Git? [link]
[2] M Anand, Microsoft Fabric Analytics Engineer Associate: Implementing Analytics Solutions Using Microsoft Fabric (DP-600), 2025 

Acronyms:

PBIP - Power BI Project
CI/CD - Continuous Integration and Continuous Deployment
IDE - Integrated Development Environments

 

🤖〽️Prompt Engineering: Copilot Unabridged (Part 52: Will AI Make Project Managers Obsolete?)

Prompt Engineering Series

Prompt: "write a blogpost of 600 words on how AI may start making project managers obsolete"

Introduction

Artificial intelligence (AI) is revolutionizing industries, automating tasks, and optimizing workflows. As AI-driven tools become more sophisticated, many professionals wonder whether their roles will be replaced by automation. One such profession under scrutiny is project management - a field that relies on leadership, organization, and decision-making. Could AI eventually make project managers obsolete, or will it simply reshape their responsibilities?

The Rise of AI in Project Management

AI-powered tools are already transforming project management by automating administrative tasks, analyzing data, and predicting project outcomes. AI-driven platforms can:

• Automate Scheduling and Task Allocation: AI can optimize project timelines, assign tasks based on team members’ skills, and adjust schedules dynamically.
• Enhance Risk Management: AI can analyze historical data to predict potential risks and suggest mitigation strategies.
• Improve Communication and Collaboration: AI-powered chatbots and virtual assistants streamline communication, ensuring teams stay informed and aligned.
• Optimize Resource Allocation: AI can assess workload distribution and recommend adjustments to maximize efficiency.

These advancements suggest that AI is becoming an indispensable tool for project managers, but does that mean it will replace them entirely?

Why AI Won’t Fully Replace Project Managers

Despite AI’s capabilities, project management is more than just scheduling and data analysis. Here’s why human project managers will remain essential:

• Leadership and Emotional Intelligence: AI lacks the ability to motivate teams, resolve conflicts, and inspire collaboration. Project managers provide emotional intelligence, guiding teams through challenges and fostering a positive work environment.
• Strategic Decision-Making: AI can analyze data, but it cannot make complex, high-stakes decisions that require human intuition, ethical considerations, and industry expertise.
• Adaptability and Crisis Management: Projects often face unexpected challenges, such as budget cuts, stakeholder conflicts, or shifting priorities. AI can suggest solutions, but human judgment is crucial in navigating unpredictable situations.
• Stakeholder Management: Project managers negotiate with clients, executives, and team members, balancing expectations and ensuring alignment. AI cannot replicate the nuanced interpersonal skills required for stakeholder engagement.

How AI Will Reshape Project Management

Rather than replacing project managers, AI will augment their capabilities, allowing them to focus on higher-level responsibilities. The future of project management will likely involve:

• AI-Assisted Decision-Making: Project managers will use AI-generated insights to make informed strategic choices.
• Automated Administrative Tasks: AI will handle routine tasks, freeing up managers to focus on leadership and innovation.
• Enhanced Predictive Analytics: AI will provide real-time data analysis, helping managers anticipate challenges and optimize project outcomes.
• Collaboration Between AI and Humans: AI will act as a co-pilot, assisting project managers rather than replacing them.

Conclusion: AI as a Tool, Not a Replacement

AI is transforming project management, but it is unlikely to make project managers obsolete. Instead, it will enhance efficiency, automate repetitive tasks, and provide valuable insights, allowing managers to focus on leadership, strategy, and human-centric decision-making.

Rather than fearing AI, project managers should embrace it as a powerful tool that elevates their role and enables them to drive successful projects in an increasingly digital world.

Disclaimer: The whole text was generated by Copilot (under Windows 10) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🏭🗒️Microsoft Fabric: Git Repository [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 4-Jul-2025

[Microsoft Fabric] Git Repository

• {def} set of features that enable developers to integrate their development processes, tools, and best practices straight into the Fabric platform [2]
• 
  
• {goal} the repo serves as single-source-of-truth
• {feature} backup and version control [2]
• {feature} revert to previous stages [2]
• {feature} collaborate with others or work alone using Git branches [2]
• {feature} source control 
• provides tools to manage Fabric items [2]
• supported for Azure DevOps and GitHub [3]
• {configuration} tenant switches 
• ⇐ must be enabled from the Admin portal 
• by the tenant admin, capacity admin, or workspace admin
• dependent on organization's settings [3]
• users can create Fabric items
• users can synchronize workspace items with their Git repositories
• create workspaces
• only if is needed to branch out to a new workspace [3]
• users can synchronize workspace items with GitHub repositories
• for GitHub users only [3]
• {concept} release process 
• begins once new updates complete a Pull Request process and merge into the team’s shared branch [3]
• {concept} branch
• {operation} switch branches
• the workspace syncs with the new branch and all items in the workspace are overridden [3]
• if there are different versions of the same item in each branch, the item is replaced [3]
• if an item is in the old branch, but not the new one, it gets deleted [3]
• one can't switch branches if there are any uncommitted changes in the workspace [3]
• {action} branch out to another workspace 
• creates a new workspace, or switches to an existing workspace based on the last commit to the current workspace, and then connects to the target workspace and branch [4]
• {permission} contributor and above
• {action} checkout new branch )
• creates a new branch based on the last synced commit in the workspace [4]
• changes the Git connection in the current workspace [4]
• doesn't change the workspace content [4]
• {permission} workspace admin
• {action} switch branch
• syncs the workspace with another new or existing branch and overrides all items in the workspace with the content of the selected branch [4]
• {permission} workspace admin
• {limitation} maximum length of branch name: 244 characters.
• {limitation} maximum length of full path for file names: 250 characters
• {limitation} maximum file size: 25 MB
• {operation} connect a workspace to a Git Repos 
• can be done only by a workspace admin [4]
• once connected, anyone with permissions can work in the workspace [4]
• synchronizes the content between the two (aka initial sync)
• {scenario} either of the two is empty while the other has content
• the content is copied from the nonempty location to the empty on [4]
• {scenario}both have content
• one must decide which direction the sync should go [4]
• overwrite the content from the destination [4]
• includes folder structures [4]
• workspace items in folders are exported to folders with the same name in the Git repo [4]
• items in Git folders are imported to folders with the same name in the workspace [4]
• if the workspace has folders and the connected Git folder doesn't yet have subfolders, they're considered to be different [4]
• leads to uncommitted changes status in the source control panel [4]
• one must to commit the changes to Git before updating the workspace [4]
• update first, the Git folder structure overwrites the workspace folder structure [4]
• {limitation} empty folders aren't copied to Git
• when creating or moving items to a folder, the folder is created in Git [4]
• {limitation} empty folders in Git are deleted automatically [4]
• {limitation} empty folders in the workspace aren't deleted automatically even if all items are moved to different folders [4]
• {limitation} folder structure is retained up to 10 levels deep [4]
• {limitation} the folder structure is maintained up to 10 levels deep
•  Git status
• synced 
• the item is the same in the workspace and Git branch [4]
•  conflict 
• the item was changed in both the workspace and Git branch [4]
•  unsupported item
•  uncommitted changes in the workspace
•  update required from Git [4]
•  item is identical in both places but needs to be updated to the last commit [4]
• source control panel
• shows the number of items that are different in the workspace and Git branch
• when changes are made, the number is updated
• when the workspace is synced with the Git branch, the Source control icon displays a 0
• commit and update panel 
• {section} changes 
• shows the number of items that were changed in the workspace and need to be committed to Git [4]
• changed workspace items are listed in the Changes section
• when there's more than one changed item, one can select which items to commit to the Git branch [4]
• if there were updates made to the Git branch, commits are disabled until you update your workspace [4]
• {section} updates 
• shows the number of items that were modified in the Git branch and need to be updated to the workspace [4]
• the Update command always updates the entire branch and syncs to the most recent commit [4]
• {limitation} one can’t select specific items to update [4]
• if changes were made in the workspace and in the Git branch on the same item, updates are disabled until the conflict is resolved [4]
• in each section, the changed items are listed with an icon indicating the status
•  new
•  modified
•  deleted
•  conflict
•  same-changes
• {concept} related workspace
• workspace with the same connection properties as the current branch [4]
• e.g.  the same organization, project, repository, and git folder [4] 

Previous Post <<||>> Next Post

References:

[2] Microsoft Learn (2025) Fabric: What is Microsoft Fabric Git integration? [link] 

What is lifecycle management in Microsoft Fabric? [link]

[3] Microsoft Fabric Updates Blog (2025) Fabric: Introducing New Branching Capabilities in Fabric Git Integration [link] 

[4] Microsoft Learn (2025) Fabric: Basic concepts in Git integration [link]
[5]  [link]

Resources:
[R1] Microsoft Learn (2025) Fabric: 

Acronyms:

CI/CD - Continuous Integration and Continuous Deployment

#️⃣Software Engineering: Mea Culpa (Part VII: A Look Forward)

Software Engineering Series

I worked for more than 20 years in various areas related to ERP systems - Data Migrations, Business Intelligence/Analytics, Data Warehousing, Data Management, Project Management, (data) integrations, Quality Assurance, and much more, having experience with IFS IV, Oracle e-Business Suite, MS Dynamics AX 2009 and during the past 3-7 years also with MS Dynamics 365 Finance, SCM & HR (in that order). Much earlier, I started to work with SQL Server (2000-2019), Oracle, and more recently with Azure Synapse and MS Fabric, writing over time more than 800 ad-hoc queries and reports for the various stakeholders, covering all the important areas, respectively many more queries for monitoring the various environments. 

In the areas where I couldn’t acquire experience on the job, I tried to address this by learning in my free time. I did it because I take seriously my profession, and I want to know how (some) things work. I put thus a lot of time into trying to keep actual with what’s happening in the MS Fabric world, from Power BI to KQL, Python, dataflows, SQL databases and much more. These technologies are Microsoft’s bet, though at least from German’s market perspective, all bets are off! Probably, many companies are circumspect or need more time to react to the political and economic impulses, or probably some companies are already in bad shape. 

Unfortunately, the political context has a broad impact on the economy, on what’s happening in the job market right now! However, the two aspects are not the only problem. Between candidates and jobs, the distance seems to grow, a dense wall of opinion being built, multiple layers based on presumptions filtering out voices that (still) matter! Does my experience matter or does it become obsolete like the technologies I used to work with? But I continued to learn, to keep actual… Or do I need to delete everything that reminds the old?

To succeed or at least be hired today one must fit a pattern that frankly doesn’t make sense! Yes, soft skills are important though not all of them are capable of compensating for the lack of technical skills! There seems to be a tendency to exaggerate some of the qualities associated with skills, or better said, of hiding behind big words. Sometimes it feels like a Shakespearian inaccurate adaptation of the stage on which we are merely players.

More likely, this lack of pragmatism will lead to suboptimal constructions that will tend to succumb under their own structure. All the inefficiencies need to be corrected, or somebody (or something) must be able to bear their weight. I saw this too often happening in ERP implementations! Big words don’t compensate for the lack of pragmatism, skills, knowledge, effort or management! For many organizations the answer to nowadays problems is more management, which occasionally might be the right approach, though this is not a universal solution for everything that crosses our path(s).

One of society’s answers to nowadays’ problem seems to be the refuge in AI. So, I wonder – where I’m going now? Jobless, without an acceptable perspective, with AI penetrating the markets and making probably many jobs obsolete. One must adapt, but adapt to what? AI is brainless even if it can mimic intelligence! Probably, it can do more in time to the degree that many more jobs will become obsolete (and I’m wondering what will happen to all those people). 

Conversely, to some trends there will be probably other trends against them, however it’s challenging to depict in clear terms the future yet in making. Society seems to be at a crossroad, more important than mine.

Previous Post <<||>> Next Post

🏭🗒️Microsoft Fabric: Deployment Pipelines [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 26-Apr-2025

[Microsoft Fabric] Deployment Pipelines

• {def} a structured process that enables content creators to manage the lifecycle of their organizational assets [5]
• enable creators to develop and test content in the service before it reaches the users [5]
• can simplify the deployment process to development, test, and production workspaces [5]
• one Premium workspace is assigned to each stage [5]
• each stage can have 
• different configurations [5]
• different databases or different query parameters [5]
• {action} create pipeline
• from the deployment pipelines entry point in Fabric [5]
• creating a pipeline from a workspace automatically assigns it to the pipeline [5]
• {action} define how many stages it should have and what they should be called [5]
• {default} has three stages
• e.g. Development, Test, and Production
• the number of stages can be changed anywhere between 2-10 
• {action} add another stage,
• {action} delete stage
• {action} rename stage 
• by typing a new name in the box
• {action} share a pipeline with others
• users receive access to the pipeline and become pipeline admins [5]
• ⇐ the number of stages are permanent [5]
• can't be changed after the pipeline is created [5]
• {action} add content to the pipeline [5]
• done by assigning a workspace to the pipeline stage [5]
• the workspace can be assigned to any stage [5]
• {action|optional} make a stage public
• {default} the final stage of the pipeline is made public
• a consumer of a public stage without access to the pipeline sees it as a regular workspace [5]
• without the stage name and deployment pipeline icon on the workspace page next to the workspace name [5]
• {action} deploy to an empty stage
• when finishing the work in one pipeline stage, the content can be deployed to the next stage [5] 
• deployment can happen in any direction [5]
• {option} full deployment 
• deploy all content to the target stage [5]
• {option} selective deployment 
• allows select the content to deploy to the target stage [5]
• {option} backward deployment 
• deploy content from a later stage to an earlier stage in the pipeline [5] 
• {restriction} only possible when the target stage is empty [5]
• {action} deploy content between pages [5]
• content can be deployed even if the next stage has content
• paired items are overwritten [5]
• {action|optional} create deployment rules
• when deploying content between pipeline stages, allow changes to content while keeping some settings intact [5] 
• once a rule is defined or changed, the content must be redeployed
• the deployed content inherits the value defined in the deployment rule [5]
• the value always applies as long as the rule is unchanged and valid [5]
• {feature} deployment history 
• allows to see the last time content was deployed to each stage [5]
• allows to to track time between deployments [5]
• {concept} pairing
• {def} the process by which an item in one stage of the deployment pipeline is associated with the same item in the adjacent stage
• applies to reports, dashboards, semantic models
• paired items appear on the same line in the pipeline content list [5]
• ⇐ items that aren't paired, appear on a line by themselves [5]
• the items remain paired even if their name changes
• items added after the workspace is assigned to a pipeline aren't automatically paired [5]
• ⇐ one can have identical items in adjacent workspaces that aren't paired [5]
• [lakehouse]
• can be removed as a dependent object upon deployment [3]
• supports mapping different Lakehouses within the deployment pipeline context [3]
• {default} a new empty Lakehouse object with same name is created in the target workspace [3]
• ⇐ if nothing is specified during deployment pipeline configuration
• notebook and Spark job definitions are remapped to reference the new lakehouse object in the new workspace [3]
• {warning} a new empty Lakehouse object with same name still is created in the target workspace [3]
• SQL Analytics endpoints and semantic models are provisioned
• no object inside the Lakehouse is overwritten [3]
• updates to Lakehouse name can be synchronized across workspaces in a deployment pipeline context [3] 
• [notebook] deployment rules can be used to customize the behavior of notebooks when deployed [4]
• e.g. change notebook's default lakehouse [4]
• {feature} auto-binding
• binds the default lakehouse and attached environment within the same workspace when deploying to next stage [4]
• [environment] custom pool is not supported in deployment pipeline
• the configurations of Compute section in the destination environment are set with default values [6]
• ⇐ subject to change in upcoming releases [6]
• [warehouse]
• [database project] ALTER TABLE to add a constraint or column
• {limitation} the table will be dropped and recreated when deploying, resulting in data loss
• {recommendation} do not create a Dataflow Gen2 with an output destination to the warehouse
• ⇐ deployment would be blocked by a new item named DataflowsStagingWarehouse that appears in the deployment pipeline [10]
• SQL analytics endpoint is not supported
• [Eventhouse]
• {limitation} the connection must be reconfigured in destination that use Direct Ingestion mode [8]
• [EventStream]
• {limitation} limited support for cross-workspace scenarios
• {recommendation} make sure all EventStream destinations within the same workspace [8]
• KQL database
• applies to tables, functions, materialized views [7]
• KQL queryset
• ⇐ tabs, data sources [7]
• [real-time dashboard]
• data sources, parameters, base queries, tiles [7]
• [SQL database]
• includes the specific differences between the individual database objects in the development and test workspaces [9]
• can be also used with
• Copy Job
• dataflows gen2
• data pipeline
• mirrored database
• activator 
• Power BI items

Previous Post <<||>> Next Post 

References:

[1] Microsoft Learn (2024) Get started with deployment pipelines [link]

[2] Microsoft Learn (2024) Implement continuous integration and continuous delivery (CI/CD) in Microsoft Fabric [link]

[3] Microsoft Learn (2024)  Lakehouse deployment pipelines and git integration (Preview) [link]

[4] Microsoft Learn (2024) Notebook source control and deployment [link] 

[5] Microsoft Learn (2024) Introduction to deployment pipelines [link]

[6] Environment Git integration and deployment pipeline [link]

[7] Microsoft Learn (2024) Microsoft Learn (2024) Real-Time Intelligence: Git integration and deployment pipelines (Preview) [link]

[8] Microsoft Learn (2024) Eventstream CI/CD - Git Integration and Deployment Pipeline [link]

[9] Microsoft Learn (2024) Get started with deployment pipelines integration with SQL database in Microsoft Fabric [link]

[10] Microsoft Learn (2025) Source control with Warehouse (preview) [link] 

Resources:

Acronyms:
CLM - Content Lifecycle Management
UAT - User Acceptance Testing

💫🗒️ERP Systems: Microsoft Dynamics 365's Business Process Catalog (BPC) [Notes]

Disclaimer: This is work in progress intended to consolidate information from the various sources and not to provide a complete overview of all the features. Please refer to the documentation for a complete overview!

Last updated: 25-Apr-2025

Business Process Catalog - End-to-End Scenarios

[Dynamics 365] Business Process Catalog (BPC)

• {def} lists of end-to-end processes that are commonly used to manage or support work within an organization [1]
• agnostic catalog of business processes contained within the entire D365 solution space [3]
• {benefit} efficiency and time savings [3]
• {benefit} best practices [3]
• {benefit} reduced risk [3]
• {benefit} technology alignment [3]
• {benefit} scalability [3]
• {benefit} cross-industry applicability [3]
• stored in an Excel workbook
• used to organize and prioritize the work on the business process documentation [1]
• {recommendation} check the latest versions (see [R1])
• assigns unique IDs to 
• {concept} end-to-end scenario
• describe in business terms 
• not in terms of software technology
• includes the high-level products and features that map to the process [3]
• covers two or more business process areas
• {purpose} map products and features to benefits that can be understood in business contexts [3]
• {concept} business process areas
• combination of business language and basic D365 terminology [3]
• groups business processes for easier searching and navigation [1]
• separated by major job functions or departments in an organization [1]
• {purpose} map concepts to benefits that can be understood in business context [3]
• more than 90 business process areas defined [1]
• {concept} business processes
• a series of structured activities and tasks that organizations use to achieve specific goals and objectives [3]
• efficiency and productivity
• consistency and quality
• cost reduction
• risk management
• scalability
• data-driven decision-making
• a set of tasks in a sequence that is completed to achieve a specific objective [5]
• define when each step is done in the implementation [5] [3]
• define how many are needed [5] [3]
• covers a wide range of structured, often sequenced, activities or tasks to achieve a predetermined organizational goal
• can refer to the cumulative effects of all steps progressing toward a business goal
• describes a function or process that D365 supports
• more than 700 business processes identified
• {goal} provide a single entry point with links to relevant product-specific content [1]
• {concept} business process guide
• provides documentation on the structure and patterns of the process along with guidance on how to use them in a process-oriented implementation [3]
• based on a catalog of business process supported by D365 [3]
• {concept} process steps 
• represented sequentially, top to bottom
• can include hyperlinks to the product documentation [5] 
• {recommendation} avoid back and forth in the steps as much as possible [5]
• can be
• forms used in D365 [5]
• steps completed in LCS, PPAC, Azure or other Microsoft products [5]
• steps that are done outside the system (incl. third-party system) [5]
• steps that are done manually [5]
• are not 
• product documentation [5]
• a list of each click to perform a task [5]
• {concept} process states
• include
• project phase 
• e.g. strategize, initialize, develop, prepare, operate
• configuration 
• e.g. base, foundation, optional
• process type
• e.g. configuration, operational
• {concept} patterns
• repeatable configurations that support a specific business process [1]
• specific way of setting up D365 to achieve an objective [1]
• address specific challenges in implementations and are based on a specific scenario or best practice [6]
• the solution is embedded into the application [6]
• includes high-level process steps [6]
• include the most common use cases, scenarios, and industries [1]
• {goal} provide a baseline for implementations
• more than 2000 patterns, and we expect that number to grow significantly over time [1]
• {activity} naming a new pattern
• starts with a verb
• describes a process
• includes product names
• indicate the industry
• indicate AppSource products
• {concept} reference architecture 
• acts as a core architecture with a common solution that applies to many scenarios [6]
• typically used for integrations to external solutions [6]
• must include an architecture diagram [6]
• {concept} process governance
• {benefit} improved quality
• {benefit} enhanced decision making
• {benefit} agility adaptability
• {benefit{ Sbd alignment
• {goal} enhance efficiency 
• {goal} ensure compliance 
• {goal} facilitate accountability 
• {concept} policy
• {concept} procedure
• {concept} control
• {concept} scope definition
• {recommendation} avoid replicating current processes without considering future needs [4]
• {risk} replicating processes in the new system without re-evaluating and optimizing [4] 
• {impact} missed opportunities for process improvement [4]
• {recommendation} align processes with overarching business goals rather than the limitations of the current system [4]
• {concept} guidance hub
• a central landing spot for D365 guidance and tools
• contains cross-application documentations
• {purpose} provide considerations and best practices for implementation [6]
• {purpose} provide technical information for implementation [6]
• {purpose} provide link to product documentation to achieve the tasks in scope [6]

Previous Post <<||>> Next Post 

References:

[1] Microsoft Learn (2024) Dynamics 365: Overview of end-to-end scenarios and business processes in Dynamics 365 [link]
[2] Microsoft Dynamics 365 Community (2023) Business Process Guides - Business Process Guides [link]
[3] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance - Part 2 Introduction to Business Processes [link]

[4] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance - Part 3: Using the Business Process Catalog to Manage Project Scope and Estimation [link]

[5] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance - Part 4: Authoring Business Processes [link]

[6] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance - Part 5:  Authoring Business Processes Patterns and Use Cases [link]
[7] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance  - Part 6: Conducting Process-Centric Discovery [link]
[8] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance  - Part 7: Introduction to Process Governance [link]

Resources:

[R1] GitHub (2024) Business Process Catalog [link]

[R2] Microsoft Learn (2024) Dynamics 365 guidance documentation and other resources [link]
[R3] Dynamics 365 Blog (2025) Process, meet product: The business process catalog for Dynamics 365 [link]

Acronyms:
3T - Tools, Techniques, Tips

ADO - 
BPC - Business Process Catalog

D365 - Dynamics 365

LCS - Lifecycle Services

PPAC - Power Platform admin center
RFI - Request for Information
RFP - Request for Proposal