🎯Christopher Maneu - Collected Quotes

"A data lake is a distributed repository of raw and unprocessed data stored in its original format, without a predefined schema or structure. A data lake is designed to support a wide range of data types, sources, and use cases, such as exploration, discovery, and data experimentation. A data lake follows a 'schema on read' approach. Data is structured and processed only when it is accessed or consumed by a user or application (Extract, Load, Transform (ELT)). A data lake also enables data democratization, meaning data is accessible and available to anyone who needs it, without barriers or restrictions." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"A data warehouse is a centralized repository of structured, cleaned, and verified data that has been extracted, transformed, and loaded from various sources. These steps are commonly called ETL, which stands for Extract, Transform, Load. This data processing methodology involves extracting data from multiple sources, transforming it to meet business needs, and loading it into a destination for analysis and consultation." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"A lake based on the medallion architecture combines the best of lakes and data warehouses. By breaking down silos and eliminating data duplication, it becomes a standard for building data platform architecture." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"A lakehouse is a data storage space that hosts and manages all types of data in one place (structured, semi-struc-tured, and unstructured), allowing different tools to normalize and examine this data according to organizational requirements and/or individual choices. A lakehouse thus combines the best aspects of a data lake and a data warehouse by eliminating data duplication and friction related to ingestion, transformation, and sharing of data within the organization, all in the open format, Delta Lake." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Considered by many companies as the next generation of data architecture, the data mesh represents the natural evolution of traditional data lakes and data warehouses. While the latter are often limited by their centralized and monolithic structure, the data mesh aims to enable companies to deploy a more flexible, responsive, and massively scalable data strategy." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"[...] the data mesh architecture of Microsoft Fabric primarily supports the organization of data into domains and federated governance [...]  Hierarchizing data within OneLake by domain simplifies organizing data, allowing a data producer to easily identify where to deposit data or a data consumer to filter and discover content by functional domain. But it also enables the distribution of governance responsibilities by defining roles and responsibilities for teams in charge of specific domains."  (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Data transformation sits at the heart of every successful data platform, serving as the critical bridge between data ingestion and data consumption. While basic transformations might involve simple cleaning and formatting, advanced transformation techniques encompass complex operations such as data enrichment, sophisticated deduplication, machine learning-based predictions, and the creation of derived metrics that weren’t present in the original data sources. These processes are essential for organizations looking to extract maximum value from their data investments." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Data virtualization is a technique that allows users and applications to access and interact with data stored in multiple, physically separate locations as if it were all in one place. Instead of moving or duplicating data, virtualization creates a logical layer that connects to the original sources and presents them in a unified view. This means users can query, analyze, or combine data from different systems - cloud storage, databases, or other platforms - without needing to know where or how the data is stored." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Fabric integrates the various technologies needed for an end-to-end data project (namely, ingestion, preparation, storage, processing, enrichment, analysis, visualization, and data sharing) within a single platform accessible as Software as a Service (SaaS), meaning via a simple connection on a web browser. This reduces complexity, costs, and delays related to using multiple tools and technologies, and eliminates all the operational maintenance of infrastructure serving data analytics needs." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Fabric Pipelines provide reliable and efficient end-to-end orchestration of data flows, managing ingestion, transformation, and loading through a sequence of steps that can leverage various data processing engines. They allow centralizing and orchestrating data movements from various sources, thanks to advanced connectivity features, and with great scalability. Built-in monitoring tools enable real-time tracking of data flow status and quick detection of anomalies or errors." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Fabric relies on a lakehouse, a data storage model that combines the benefits of a data lake and a data warehouse. Within Fabric, the various data analytics and processing tools rely on a data lake that collects and stores data in its original format, whether structured, semi-structured, or unstructured, without the need to transform or normalize it beforehand. The lakehouse approach then enables converting these diverse data formats into a single format (i.e., compatible with all the data processing engines offered by Fabric) and in an open format, allowing other market vendors to interact with data in the Fabric lakehouse." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"In Fabric, a domain represents a way to logically group data corresponding to specific functional areas. Domains are frequently used to organize data by business sector in order to manage it according to each sector’s regulations, specifics, and requirements." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"It should be noted that, unlike Dataflow Gen2, in pipelines, it is not mandatory to enable staging to load data into a warehouse. Indeed, pipelines are designed for more general orchestration scenarios where you can combine various activities such as transformations, API calls, and so on to create complex workflows. They are not specifically focused on data preparation but rather on end-to-end process automation. Pipelines are more flexible and used for a variety of orchestration tasks, whereas Dataflow Gen2 is specifically designed for data preparation and transformation, hence the requirement for staging in that case." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"One of the most powerful enhancements in Real-Time Intelligence is the integration of anomaly detection capabilities, enabling systems to automatically flag unusual deviations in real time. Rather than relying on predefined thresholds or periodic audits, these AI-driven agents continuously monitor data streams, learning normal behavior patterns and surfacing outliers or unexpected shifts the moment they appear. This proactive approach transforms what was once passive reporting into active surveillance, allowing operational teams to respond instantly when something deviates from the norm." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"The hub and spoke, or 'star network', is a data architecture model that centralizes data from various sources into a single hub, such as a data warehouse or data lake. The hub serves as the source of truth for data and provides standardized schemas and formats. The spokes are the various applications or services that consume data from the hub for different purposes, such as analytics, reporting, or ma-chine learning. Spokes can also perform transformations or aggregations on data before presenting it to end users. The hub and spoke architecture aims to simplify data integration and management by reducing complexity and redundancy in data pipelines" (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"The problem with data lakes is that they have several drawbacks preventing them from being the perfect or ideal solution. The first drawback is an organizational problem: (•) How to organize data in the lake (•) How to classify, catalog, secure, document, and find it (•) How to avoid the lake turning into a swamp where data is mixed, duplicated, obsolete, or inaccessible (•) How to manage quality, governance, and traceability in the lake."(Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"The transformation phase represents the most resource-intensive stage of most data projects, often consuming 60-80% of total project time and effort. This significant investment stems from the inherent complexity of converting raw, inconsistent data into clean, structured, and enriched information ready for business use. Every data quality issue must be identified and resolved, every business rule must be correctly implemented, and every integration point must be properly validated. This meticulous work serves as the essential bridge between raw data ingestion and meaningful business insights." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"This transition to OneDrive highlights the importance of governance adapted to new methods of collaborative work and data sharing. The idea of OneLake is, therefore, based on this same concept: rather than subscribing to a data lake technology that must be maintained, why not simply subscribe to a storage service that offers a layer of abstraction over the complexities of these data storage infrastructures? As a result, the data lake becomes a controlled or governed environment, but still accessible to users who can view it as a simple and intuitive way to securely share data with their colleagues and IT teams."(Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Traditionally, data engineers are responsible for the first steps of data transformation, commonly referred to as the transition from the 'bronze' stage to the 'silver' stage. This phase includes the normalization of raw data to clean and organize it into a structured and accessible format. Data Engineers ensure that data is properly ingested, stored, and prepared for subsequent steps. Their work focuses on building robust data pipelines and applying basic transformations that make the data usable. Next, responsibility may be handed over to an analytics engineer, who takes charge of the transition from the 'silver' stage to the 'gold' stage. This step involves more complex transformations aimed at refining, enriching, and modeling the data to meet specific analytical needs. The analytics engineer ensures that the data is ready to be used in reports, dashboards, and advanced analyses. The transition to the 'gold' stage means that the data is fully prepared for analytic use, providing strategic insights from consolidated data sources." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"We are now witnessing the rise of a new paradigm in technology, the age of agentic AI, where intelligence moves beyond automation and prediction to autonomy and intent. In this new world, operations across industries are no longer passive systems waiting for human input or post-event analysis. Instead, they have evolved into dynamic ecosystems of intelligence, continuously learning from every signal that flows through the organization. [...] Agentic AI marks the fourth great evolution of software, after client-server, cloud, and SaaS - and perhaps the most transformative of all. It represents the moment when technology stops being a tool we use and becomes a collaborator that thinks, learns, and acts alongside us." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"While Fabric provides all the traditional tools that data specialists use daily to work on data integration and processing projects, it also offers new intuitive interfaces to enable business users, citizen analysts, or business analysts to interact with their data regardless of their skill level. The primary goal is to meet the needs and expectations of these users, who often do not benefit from data analytics and processing tools because they are too complex to use, even though they are themselves the main consumers and producers of data within organizations." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"With Fabric, organizations can unlock the full potential of AI and machine learning in their data workflows. First, it provides users with all the tools necessary to create and deploy AI and machine learning models; users can use the frameworks and languages of their choice. Next, it enables these users to benefit from native integration of models that enrich the data present within Fabric with advanced cognitive analytics, such as vision and language, for example, and leverage the new capabilities of generative AI. Finally, it supports users at every stage of their data project with intelligent assistants that help create data integration flows, develop transformations or analyses, build data visualization reports, and even answer business questions by leveraging existing reports and semantic models to deliver contextual insights instantly." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

🏭🗒️Microsoft Fabric: Git [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 6-Oct-2025

[Microsoft Fabric] Git

• {def} an open source, distributed version control platform
• enables developers commit their work to a local repository and then sync their copy of the repository with the copy on the server [1]
• to be differentiated from centralized version control 
• where clients must synchronize code with a server before creating new versions of code [1
• provides tools for isolating changes and later merging them back together
• {benefit} simultaneous development
• everyone has their own local copy of code and works simultaneously on their own branches
•  Git works offline since almost every operation is local
• {benefit} faster release
• branches allow for flexible and simultaneous development
• {benefit} built-in integration
• integrates into most tools and products
•  every major IDE has built-in Git support
• this integration simplifies the day-to-day workflow
• {benefit} strong community support
• the volume of community support makes it easy to get help when needed
• {benefit} works with any team
• using Git with a source code management tool increases a team's productivity 
• by encouraging collaboration, enforcing policies, automating processes, and improving visibility and traceability of work
• the team can either
• settle on individual tools for version control, work item tracking, and continuous integration and deployment
• choose a solution that supports all of these tasks in one place
• e.g. GitHub, Azure DevOps
• {benefit} pull requests
• used to discuss code changes with the team before merging them into the main branch
• allows to ensure code quality and increase knowledge across team
• platforms like GitHub and Azure DevOps offer a rich pull request experience
• {benefit} branch policies
• protect important branches by preventing direct pushes, requiring reviewers, and ensuring clean build
•  used to ensure that pull requests meet requirements before completion
•  teams can configure their solution to enforce consistent workflows and process across the team
• {feature} continuous integration
• {feature} continuous deployment
• {feature} automated testing
• {feature} work item tracking
• {feature} metrics
• {feature} reporting 
• {operation} commit
• snapshot of all files at a point in time [1]
•  every time work is saved, Git creates a commit [1]
•  identified by a unique cryptographic hash of the committed content [1]
•  everything is hashed
•  it's impossible to make changes, lose information, or corrupt files without Git detecting it [1]
•  create links to other commits, forming a graph of the development history [2A]
• {operation} revert code to a previous commit [1]
• {operation} inspect how files changed from one commit to the next [1]
• {operation} review information e.g. where and when changes were made [1]
• {operation} branch
•  lightweight pointers to work in progress
•  each developer saves changes to their own local code repository
• there can be many different changes based on the same commit
•  branches manage this separation
• once work created in a branch is finished, it can be merged back into the team's main (or trunk) branch
• main branch
• contains stable, high-quality code from which programmers release
• feature branches 
• contain work in progress, which are merged into the main branch upon completion
•  allows to isolate development work and minimize conflicts among multiple developers [2]
•  release branch
•  by separating the release branch from development in progress, it's easier to manage stable code and ship updates more quickly
• if a file hasn't changed from one commit to the next, Git uses the previously stored file [1]
• files are in one of three states
• {state}modified
• when a file is first modified, the changes exist only in the working directory
•  they aren't yet part of a commit or the development history
•  the developer must stage the changed files to be included in the commit
•  the staging area contains all changes to include in the next commit
•  {state}committed
•  once the developer is happy with the staged files, the files are packaged as a commit with a message describing what changed
•  this commit becomes part of the development history
•  {state}staged
•  staging lets developers pick which file changes to save in a commit to break down large changes into a series of smaller commits
•   by reducing the scope of commits, it's easier to review the commit history to 
• {best practice} set up a shared Git repository and CI/CD pipelines [2]
• enables effective collaboration and deployment in PBIP [2]
• enables implementing version control in PBIP [2]
• it’s essential for managing project history and collaboration [2]
• allows to track changes throughout the model lifecycle [2]
• allows to enable effective governance and collaboratin
•  provides robust version tracking and collaboration features, ensuring traceability
• {best practice} use descriptive commit messages [2]
• allows to ensure clarity and facilitate collaboration in version control [2]
• {best practice} avoid sharing Git credentials [2]
• compromises security and accountability [2]
•  can lead to potential breaches [2]
• {best practice} define a naming conventions for files and communicated accordingly [2]
• {best practice} avoid merging changes directly into the master branch [2]
• {risk} this can lead to integration issues [2]
• {best practice} use git merge for integrating changes from one branch to another [2]
• {benefit} ensures seamless collaboration [2]
• {best practice} avoid skipping merges [2]
• failing to merge regularly can lead to complex conflicts and integration challenges [2]

Previous Post <<||>> Next Post 

References:

[1] Microsoft Learn (2022) DeveOps: What is Git? [link]
[2] M Anand, Microsoft Fabric Analytics Engineer Associate: Implementing Analytics Solutions Using Microsoft Fabric (DP-600), 2025 

Acronyms:

PBIP - Power BI Project
CI/CD - Continuous Integration and Continuous Deployment
IDE - Integrated Development Environments

 

🏭🗒️Microsoft Fabric: Git Repository [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 4-Jul-2025

[Microsoft Fabric] Git Repository

• {def} set of features that enable developers to integrate their development processes, tools, and best practices straight into the Fabric platform [2]
• 
  
• {goal} the repo serves as single-source-of-truth
• {feature} backup and version control [2]
• {feature} revert to previous stages [2]
• {feature} collaborate with others or work alone using Git branches [2]
• {feature} source control 
• provides tools to manage Fabric items [2]
• supported for Azure DevOps and GitHub [3]
• {configuration} tenant switches 
• ⇐ must be enabled from the Admin portal 
• by the tenant admin, capacity admin, or workspace admin
• dependent on organization's settings [3]
• users can create Fabric items
• users can synchronize workspace items with their Git repositories
• create workspaces
• only if is needed to branch out to a new workspace [3]
• users can synchronize workspace items with GitHub repositories
• for GitHub users only [3]
• {concept} release process 
• begins once new updates complete a Pull Request process and merge into the team’s shared branch [3]
• {concept} branch
• {operation} switch branches
• the workspace syncs with the new branch and all items in the workspace are overridden [3]
• if there are different versions of the same item in each branch, the item is replaced [3]
• if an item is in the old branch, but not the new one, it gets deleted [3]
• one can't switch branches if there are any uncommitted changes in the workspace [3]
• {action} branch out to another workspace 
• creates a new workspace, or switches to an existing workspace based on the last commit to the current workspace, and then connects to the target workspace and branch [4]
• {permission} contributor and above
• {action} checkout new branch )
• creates a new branch based on the last synced commit in the workspace [4]
• changes the Git connection in the current workspace [4]
• doesn't change the workspace content [4]
• {permission} workspace admin
• {action} switch branch
• syncs the workspace with another new or existing branch and overrides all items in the workspace with the content of the selected branch [4]
• {permission} workspace admin
• {limitation} maximum length of branch name: 244 characters.
• {limitation} maximum length of full path for file names: 250 characters
• {limitation} maximum file size: 25 MB
• {operation} connect a workspace to a Git Repos 
• can be done only by a workspace admin [4]
• once connected, anyone with permissions can work in the workspace [4]
• synchronizes the content between the two (aka initial sync)
• {scenario} either of the two is empty while the other has content
• the content is copied from the nonempty location to the empty on [4]
• {scenario}both have content
• one must decide which direction the sync should go [4]
• overwrite the content from the destination [4]
• includes folder structures [4]
• workspace items in folders are exported to folders with the same name in the Git repo [4]
• items in Git folders are imported to folders with the same name in the workspace [4]
• if the workspace has folders and the connected Git folder doesn't yet have subfolders, they're considered to be different [4]
• leads to uncommitted changes status in the source control panel [4]
• one must to commit the changes to Git before updating the workspace [4]
• update first, the Git folder structure overwrites the workspace folder structure [4]
• {limitation} empty folders aren't copied to Git
• when creating or moving items to a folder, the folder is created in Git [4]
• {limitation} empty folders in Git are deleted automatically [4]
• {limitation} empty folders in the workspace aren't deleted automatically even if all items are moved to different folders [4]
• {limitation} folder structure is retained up to 10 levels deep [4]
• {limitation} the folder structure is maintained up to 10 levels deep
•  Git status
• synced 
• the item is the same in the workspace and Git branch [4]
•  conflict 
• the item was changed in both the workspace and Git branch [4]
•  unsupported item
•  uncommitted changes in the workspace
•  update required from Git [4]
•  item is identical in both places but needs to be updated to the last commit [4]
• source control panel
• shows the number of items that are different in the workspace and Git branch
• when changes are made, the number is updated
• when the workspace is synced with the Git branch, the Source control icon displays a 0
• commit and update panel 
• {section} changes 
• shows the number of items that were changed in the workspace and need to be committed to Git [4]
• changed workspace items are listed in the Changes section
• when there's more than one changed item, one can select which items to commit to the Git branch [4]
• if there were updates made to the Git branch, commits are disabled until you update your workspace [4]
• {section} updates 
• shows the number of items that were modified in the Git branch and need to be updated to the workspace [4]
• the Update command always updates the entire branch and syncs to the most recent commit [4]
• {limitation} one can’t select specific items to update [4]
• if changes were made in the workspace and in the Git branch on the same item, updates are disabled until the conflict is resolved [4]
• in each section, the changed items are listed with an icon indicating the status
•  new
•  modified
•  deleted
•  conflict
•  same-changes
• {concept} related workspace
• workspace with the same connection properties as the current branch [4]
• e.g.  the same organization, project, repository, and git folder [4] 

Previous Post <<||>> Next Post

References:

[2] Microsoft Learn (2025) Fabric: What is Microsoft Fabric Git integration? [link] 

What is lifecycle management in Microsoft Fabric? [link]

[3] Microsoft Fabric Updates Blog (2025) Fabric: Introducing New Branching Capabilities in Fabric Git Integration [link] 

[4] Microsoft Learn (2025) Fabric: Basic concepts in Git integration [link]
[5]  [link]

Resources:
[R1] Microsoft Learn (2025) Fabric: 

Acronyms:

CI/CD - Continuous Integration and Continuous Deployment

🏭🗒️Microsoft Fabric: Result Set Caching in SQL Analytics Endpoints [Notes] 🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 21-Jun-2025

[Microsoft Fabric] Result Set Caching in SQL Analytics Endpoints

• {def} built-in performance optimization for Warehouse and Lakehouse that improves read latency [1]
• fully transparent to the user [3]
• persists the final result sets for applicable SELECT T-SQL queries
• caches all the data accessed by a query [3]
• subsequent runs that "hit" cache will process just the final result set
• can bypass complex compilation and data processing of the original query[1]
• ⇐ returns subsequent queries faster [1]
• the cache creation and reuse is applied opportunistically for queries
• works on
• warehouse tables
• shortcuts to OneLake sources
• shortcuts to non-Azure sources
• the management of cache is handled automatically [1]
• regularly evicts cache as needed
• as data changes, result consistency is ensured by invalidating cache created earlier [1]
• {operation} enable setting
• via ALTER DATABASE <database_name> SET RESULT_SET_CACHING ON
• {operation} validate setting
• via SELECT name, is_result_set_caching_on FROM sys.databases
• {operation} configure setting
• configurable at item level
• once enabled, it can then be disabled 
• at the item level
• for individual queries
• e.g. debugging or A/B testing a query
• via OPTION ( USE HINT ('DISABLE_RESULT_SET_CACHE') 
• {default} during the preview, result set caching is off for all items [1]
• [monitoring] 
• via Message Output
• applicable to Fabric Query editor, SSMS
• the statement "Result set cache was used" is displayed after query execution if the query was able to use an existing result set cache
• via queryinsights.exec_requests_history system view
• result_cache_hit displays indicates result set cache usage for each query execution [1]
• {value} 2: the query used result set cache (cache hit)
• {value} 1: the query created result set cache
• {value} 0: the query wasn't applicable for result set cache creation or usage [1]
• {reason} the cache no longer exists
• {reason} the cache was invalidated by a data change, disqualifying it for reuse [1]
• {reason} query isn't deterministic
• isn't eligible for cache creation [1]
• {reason} query isn't a SELECT statement
• [warehousing] 
• {scenario} analytical queries that process large amounts of data to produce a relatively small result [1]
• {scenario} workloads that trigger the same analytical queries repeatedly [1]
• the same heavy computation can be triggered multiple times, even though the final result remains the same [1]

Previous Post <<||>> Next Post

References:

[1] Microsoft Learn (2025) Result set caching (preview) [link]

[2] Microsoft Fabric Update Blog (2025) Result Set Caching for Microsoft Fabric Data Warehouse (Preview) [link|aka]

[3] Microsoft Learn (2025) In-memory and disk caching [link]

[4] Microsoft Learn (2025) Performance guidelines in Fabric Data Warehouse [link] 

Resources:
[R1] Microsoft Fabric (2025) Fabric Update - June 2025 [link]

Acronyms:

MF - Microsoft Fabric

SSMS - SQL Server Management Studio

🏭🗒️Microsoft Fabric: Materialized Lake Views (MLV) [Notes] 🆕🗓️

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 27-Jul-2025

-- create schema
CREATE SCHERA IF NOT EXISTS <lakehouse_name>.<schema_name>

-- create a materialized view
CREATE MATERIALIZED VIEW IF NOT EXISTS <lakehouse_name>.<schema_name>.<view_name> 
[(
    CONSTRAINT <constraint_name> CHECK (<constraint>) ON MISMATCH DROP 
)] 

[PARTITIONED BY (col1, col2, ... )] 
[COMMENT “description or comment”] 
[TBLPROPERTIES (“key1”=”val1”, “key2”=”val2”, 

AS 
SELECT ...
FROM ...
-- WHERE ...
--GROUP BY ...

[Microsoft Fabric] Materialized Lake Views (MLVs)

• {def} persisted, continuously updated view of data [1]
• {benefit} allows to build declarative data pipelines using SQL, complete with built-in data quality rules and automatic monitoring of data transformations
• simplifies the implementation of multi-stage Lakehouse processing [1]
• ⇐ aids in the creation, management, and monitoring of views [3]
• ⇐ improves transformations through a declarative approach [3]
• streamline data workflows
• enable developers to focus on business logic [1]
• ⇐ not on infrastructural or data quality-related issues [1]
• the views can be created in a notebook [2]
• {benefit} allows developers visualize lineage across all entities in lakehouse, view the dependencies, and track its execution progress [3]
• can have data quality constraints enforced and visualized for every run, showing completion status and conformance to data quality constraints defined in a single view [1]
• empowers developers to set up complex data pipelines with just a few SQL statements and then handle the rest automatically [1]
• faster development cycles 
• trustworthy data
• quicker insights
• {goal} process only the new or changed data instead of reprocessing everything each time [1]
• ⇐  leverages Delta Lake’s CDF under the hood
• ⇒ it can update just the portions of data that changed rather than recompute the whole view from scratch [1]
• {operation} creation
• allows defining transformations at each layer [1]
• e.g. aggregation, projection, filters
• allows specifying certain checks that the data must meet [1]
• incorporate data quality constraints directly into the pipeline definition
• via CREATE MATERIALIZED LAKE VIEW
• the SQL syntax is declarative and Fabric figures out how to produce and maintain it [1]
• {operation} refresh
• refreshes only when its source has new data [1]
• if there’s no change, it can skip running entirely (saving time and resources) [1]
• via REFRESH MATERIALIZED LAKE VIEW [workspace.lakehouse.schema].MLV_Identifier [FULL];
• {operation} list views from schema [3]
• via SHOW MATERIALIZED LAKE VIEWS <IN/FROM> Schema_Name;
• {opetation} retrieve definition
• via SHOW CREATE MATERIALIZED LAKE VIEW MLV_Identifier;
• {operstion} update definition
• via ALTER MATERIALIZED LAKE VIEW MLV_Identifier RENAME TO MLV_Identifier_New;
• {operstion} drop view
• via DROP MATERIALIZED LAKE VIEW MLV_Identifier;
• {warning} dropping or renaming a materialized lake view affects the lineage view and scheduled refresh [3]
• {recommendation} update the reference in all dependent materialized lake views [3]
• {operation} schedule view run
• lets users set how often the MLV should be refreshed based on business needs and lineage execution timing [5]
• depends on
• data update frequency: the frequency with which the data is updated [5]
• query performance requirements: Business requirement to refresh the data in defined frequent intervals [5]
• system load: optimizing the time to run the lineage without overloading the system [5]
• {operation} view run history
• users can access the last 25 runs including lineage and run metadata
• available from the dropdown for monitoring and troubleshooting
• {concept} lineage
• the sequence of MLV that needs to be executed to refresh the MLV once new data is available [5]
• {feature} automatically generate a visual report that shows trends on data quality constraints 
• {benefit} allows to easily identify the checks that introduce maximum errors and the associated MLVs for easy troubleshooting [1]
• {feature} can be combined with Shortcut Transformation feature for CSV ingestion 
• {benefit} facilitate the building of end-to-end Medallion architectures
• {feature} dependency graph
• allows to see the dependencies existing between the various objects [2]
• ⇐ automatically generated [2]
• {feature} data quality
• {benefit} allows to compose precise queries to exclude poor quality data from the source tables [5]
• [medallion architecture] ensuring data quality is essential at every stage of the architecture [5]
• maintained by setting constraints when defining the MLVs [5]
• {action} FAIL
• stops refreshing an MLV if any constraint is violated [5]
• {default} halt is at the first instance
• even without specifying the FAIL keyword [5]
• takes precedence over DROP
• {action} DROP
• processes the MLV and removes records that don't meet the specified constrain [5]
• provides the count of removed records in the lineage view [5]
• {constraint} updating data quality constraints after creating an MLV isn't supported [5]
• ⇐ the MLV must be recreated
• {constraint} the use of functions and pattern search with operators in constraint condition is restricted [5]
• e.g. LIKE, regex 
• {known issue} the creation and refresh of an MLV with a FAIL action in constraint may result in a "delta table not found" error
• {recommendation} recreate the MLV and avoid using the FAIL action [5] 
• {feature} data quality report
• built-in Power BI dashboard that shows several aggregated metrics [2]
• {feature} monitor hub
• centralized portal to browse MLV runs in the lakehouse [7]
• {operation} view runs' status [7]
• {operation} search and filter the runs [7]
• based on different criteria
• {operation} cancel in-progress run [7]
• {operation} drill down run execution details [7]
• doesn't support
• {feature|planned} PySpark [3]
• {feature|planned} incremental refresh [3]
• {feature|planned} integration with Data Activator [3]
• {feature|planned} API [3]
• {feature|planned} cross-lakehouse lineage and execution [3]
• {limitation} Spark properties set at the session level aren't applied during scheduled lineage refresh [4]
• {limitation} creation with delta time-travel [4]
• {limitation} DML statements [4]
• {limitation} UDFs in CTAS [4] 
• {limitation} temporary views can't be used to define MLVs [4]
•  {recommendation} create all MLVs for a workspace within a single Lakehouss [8]
• allows to ensure a consolidated and clear representation in the DAG view [8]

Previous Post <<||>> Next Post

References:

[1] Microsoft Fabric Update Blog (2025) Simplifying Medallion Implementation with Materialized Lake Views in Fabric [link|aka]

[2] Power BI Tips (2025) Microsoft Fabric Notebooks with Materialized Views - Quick Tips [link]

[3] Microsoft Learn (2025) What are materialized lake views in Microsoft Fabric? [link]

[4] Microsoft Learn (2025) Materialized lake views Spark SQL reference [link]

[5] Microsoft Learn (2025) Manage Fabric materialized lake views lineage [link] 

[6] Microsoft Learn (2025) Data quality in materialized lake views [link]

[7] Microsoft Learn (2025) Monitor materialized lake views [link] 
[8] Microsoft Learn (2025) Development and Monitoring [link]

Resources:
[R1] Databricks (2025) Use materialized views in Databricks SQL [link]

[R2] Microsoft Learn (2025) Implement medallion architecture with materialized lake views [link]

[R3] Mastering Declarative Data Transformations with Materialized Lake Views [link]

Acronyms:

API - Application Programming Interface

CDF - Change Data Feed

CTAS - Create Tables As Select

DML - Data Manipulation Language 

ETL - Extract, Transfer, Load

MF - Microsoft Fabric
MLV - Materialized Lake views

UDF - User-defined functions

🏭🗒️Microsoft Fabric: Warehouse Snapshots [Notes] 🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 23-May-2025

[Microsoft Fabric] Warehouse Snapshots

• {def} read-only representation of a warehouse at a specific point in time [1]
• allows support for analytics, reporting, and historical analysis scenarios without worrying about the volatility of live data updates [1]
• provide a consistent and stable view of data [1]
• ensuring that analytical workloads remain unaffected by ongoing changes or ETL  operations [1]
• {benefit} guarantees data consistency
• the dataset remains unaffected by ongoing ETL processes [1]
• {benefit} immediate roll-Forward updates
• can be seamlessly rolled forward on demand to reflect the latest state of the warehouse
• ⇒ {benefit} consumers access the same snapshot using a consistent connection string, even from third-party tools [1]
• ⇐ updates are applied immediately, as if in a single, atomic transaction [1]
• {benefit} facilitates historical analysis
• snapshots can be created on an hourly, daily, or weekly basis to suit their business requirements [1]
• {benefit} enhanced reporting
• provides a point-in-time reliable dataset for precise reporting [1]
• ⇐ free from disruptions caused by data modifications [1]
• {benefit} doesn't require separate storage [1]
• relies on source Warehouse [1]
• {limit} doesn't support database objects 
• {limit} capture a state within the last 30 days
• {operation} create snapshot
• via New warehouse snapshot
• multiple snapshots can be created for the same parent warehouse [1]
• appear as child items of the parent warehouse in the workspace view [1]
• the queries run against provide the current version of the data being accessed [1]
• {operation} read properties 
• via 
• GET https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{warehousesnapshotId} Authorization: Bearer <bearer token>
• {operation} update snapshot timestamp
• allows users to roll forward data instantly, ensuring consistency [1]
• use current state
• via ALTER DATABASE [<snapshot name>] SET TIMESTAMP = CURRENT_TIMESTAMP; 
• use point in time
• ALTER DATABASE snapshot SET TIMESTAMP = 'YYYY-MM-DDTHH:MM:SS.SS'//UTC time
• queries that are in progress during point in time update will complete against the version of data they were started against [1]
• {operation} rename snapshot
• {operation} delete snapshot
• via DELETE
• when the parent warehouse gets deleted, the snapshot is also deleted [1]
• {operation} modify source table
• DDL changes to source will only impact queries in the snapshot against tables affected [1]
• {operation} join multiple snapshots
• the resulting snapshot date will be applied to each warehouse connection [1]
• {operation} retrieve metadata
• via sys.databases [1]
• [permissions] inherited from the source warehouse [1]
• ⇐ any permission changes in the source warehouse applies instantly to the snapshot [1]
• security updates on source database will be rendered immediately to the snapshot databases [1]
• {limitation} can only be created against new warehouses [1]
• created after Mar-2025
• {limitation} do not appear in SSMS Object Explorer but will show up in the database selection dropdown [1]
• {limitation} datetime can be set to any date in the past up to 30 days or database creation time (whichever is later)  [1]
• {limitation} modified objects after the snapshot timestamp become invalid in the snapshot [1]
• applies to tables, views, and stored procedures [1]
• {limitation} must be recreated if the data warehouse is restored [1]
• {limitation} aren’t supported on the SQL analytics endpoint of the Lakehouse [1]
• {limitation} aren’t supported as a source for OneLake shortcuts [1]
•  [Power BI]{limitation} require Direct Query or Import mode [1]
• don’t support Direct Lake

Previous Post <<||>> Next Post

References:

[1] Microsoft Learn (2025) Fabric: Warehouse Snapshots in Microsoft Fabric (Preview) [link]

[2] Microsoft Learn (2025) Warehouse snapshots (preview) [link]

[3] Microsoft Learn (2025) Create and manage a warehouse snapshot (preview) [link]

Resources:

Acronyms:

DDL - Data Definition Language
ETL - Extract, Transfer, Load

MF - Microsoft Fabric
SSMS - SQL Server Management Studio

#️⃣Software Engineering: Mea Culpa (Part VII: A Look Forward)

Software Engineering Series

I worked for more than 20 years in various areas related to ERP systems - Data Migrations, Business Intelligence/Analytics, Data Warehousing, Data Management, Project Management, (data) integrations, Quality Assurance, and much more, having experience with IFS IV, Oracle e-Business Suite, MS Dynamics AX 2009 and during the past 3-7 years also with MS Dynamics 365 Finance, SCM & HR (in that order). Much earlier, I started to work with SQL Server (2000-2019), Oracle, and more recently with Azure Synapse and MS Fabric, writing over time more than 800 ad-hoc queries and reports for the various stakeholders, covering all the important areas, respectively many more queries for monitoring the various environments. 

In the areas where I couldn’t acquire experience on the job, I tried to address this by learning in my free time. I did it because I take seriously my profession, and I want to know how (some) things work. I put thus a lot of time into trying to keep actual with what’s happening in the MS Fabric world, from Power BI to KQL, Python, dataflows, SQL databases and much more. These technologies are Microsoft’s bet, though at least from German’s market perspective, all bets are off! Probably, many companies are circumspect or need more time to react to the political and economic impulses, or probably some companies are already in bad shape. 

Unfortunately, the political context has a broad impact on the economy, on what’s happening in the job market right now! However, the two aspects are not the only problem. Between candidates and jobs, the distance seems to grow, a dense wall of opinion being built, multiple layers based on presumptions filtering out voices that (still) matter! Does my experience matter or does it become obsolete like the technologies I used to work with? But I continued to learn, to keep actual… Or do I need to delete everything that reminds the old?

To succeed or at least be hired today one must fit a pattern that frankly doesn’t make sense! Yes, soft skills are important though not all of them are capable of compensating for the lack of technical skills! There seems to be a tendency to exaggerate some of the qualities associated with skills, or better said, of hiding behind big words. Sometimes it feels like a Shakespearian inaccurate adaptation of the stage on which we are merely players.

More likely, this lack of pragmatism will lead to suboptimal constructions that will tend to succumb under their own structure. All the inefficiencies need to be corrected, or somebody (or something) must be able to bear their weight. I saw this too often happening in ERP implementations! Big words don’t compensate for the lack of pragmatism, skills, knowledge, effort or management! For many organizations the answer to nowadays problems is more management, which occasionally might be the right approach, though this is not a universal solution for everything that crosses our path(s).

One of society’s answers to nowadays’ problem seems to be the refuge in AI. So, I wonder – where I’m going now? Jobless, without an acceptable perspective, with AI penetrating the markets and making probably many jobs obsolete. One must adapt, but adapt to what? AI is brainless even if it can mimic intelligence! Probably, it can do more in time to the degree that many more jobs will become obsolete (and I’m wondering what will happen to all those people). 

Conversely, to some trends there will be probably other trends against them, however it’s challenging to depict in clear terms the future yet in making. Society seems to be at a crossroad, more important than mine.

Previous Post <<||>> Next Post

🏭🗒️Microsoft Fabric: Purview [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 29-Apr-2025

[Microsoft Purview] Purview

• {def} comprehensive data governance and security platform designed to help organizations manage, protect, and govern their data across various environments [1]
• incl. on-premises, cloud & SaaS applications [1]
• provides the highest and most flexible level of functionality for data governance in MF [1]
• offers comprehensive tools for 
• data discovery
• data classification
• data cataloging
• {capability} managing the data estate
• {tool} dedicated portal
• aka Fabric Admin portal
• used to control tenant settings, capacities, domains, and other objects, typically reserved for administrators
• {type} logical containers
• used to control access to data and capabilities [1]
• {level} tenants
• settings for Fabric administrators [1]
• {level} domains
• group data that is relevant to a single business area or subject field [1]
• {level} workspaces 
• group Fabric items used by a single team or department [1]
• {type} capacities
• objects that limit compute resource usage for all Fabric workloads [1]
• {capability} metadata scanning
• extracts values from data lakes
• e.g. names, identities, sensitivities, endorsements, etc. 
• can be used to analyze and set governance policies [1]
• {capability} secure and protect data
• assure that data is protected against unauthorized access and destructive attacks [1]
• compliant with data storage regulations applicable in your region [1]
• {tool} data tags
• allows to identity the sensitivity of data and apply data retentions and protection policies [1]
• {tool} workspace roles
• define the users who are authorized to access the data in a workspace [1]
• {tool} data-level controls
• used at the level of Fabric items
• e.g. tables, rows, and columns to impose granular restrictions.
• {tool} certifications
• Fabric is compliant with many data management certifications
• incl. HIPAA BAA, ISO/IEC 27017, ISO/IEC 27018, ISO/IEC 27001, ISO/IEC 27701 [1]
• {feature} OneLake data hub
• allows users to find and explore the data in their estate.
• {feature} endorsement
• allows users to endorse a Fabric item to identity it as of high quality [1]
• help other users to trust the data that the item contains [1]
• {feature} data lineage
• allows users to understand the flow of data between items in a workspace and the impact that a change would have [1]
• {feature} monitoring hub
• allows to monitor activities for the Fabric items for which the user has the permission to view [1]
• {feature} capacity metrics
• app used to monitor usage and consumption
• {feature} allows to automate the identification of sensitive information and provides a centralized repository for metadata [1]
• feature} allows to find, manage, and govern data across various environments
• incl. both on-premises and cloud-based systems [1]
• supports compliance and risk management with features that monitor regulatory adherence and assess data vulnerabilities [1]
• {feature} integrated with other Microsoft services and third-party tools 
• {benefit} enhances its utility
• {benefit} streamlines data access controls
• enforcing policies, and delivering insights into data lineage [1]
• {benefit} helps organizations maintain data integrity, comply with regulations, and use their data effectively for strategic decision-making [1]
• {feature} Data Catalog
• {benefit} allows users to discover, understand, and manage their organization's data assets
• search for and browse datasets
• view metadata
• gain insights into the data’s lineage, classification, and sensitivity labels [1]
• {benefit} promotes collaboration
• users can annotate datasets with tags to improve discoverability and data governance [1]
• targets users and administrator
• {benefit} allows to discover where patient records are held by searching for keywords [1]
• {benefit} allows to label documents and items based on their sensitiveness [1]
• {benefit} allows to use access policies to manage self-service access requests [1]
• {feature} Information Protection
• used to classify, label, and protect sensitive data throughout the organization [1]
• by applying customizable sensitivity labels, users classify records. [1]
• {concept} policies
• define access controls and enforce encryption
• labels follow the data wherever it goes
• helps organizations meet compliance requirements while safeguarding data against accidental exposure or malicious threats [1]
• allows to protect records with policies to encrypt data and impose IRM
• {feature} Data Loss Prevention (DLP)
• the practice of protecting sensitive data to reduce the risk from oversharing [2]
• implemented by defining and applying DLP policies [2]
• {feature} Audit
• user activities are automatically logged and appear in the Purview audit log
• e.g. creating files or accessing Fabric items
• {feature} connect Purview to Fabric in a different tenant
• all functionality is supported, except that 
• {limitation} Purview's live view isn't available for Fabric items [1]
• {limitation} the system can't identify user registration automatically [1]
• {limitation} managed identity can’t be used for authentication in cross-tenant connections [1]
• {workaround} use a service principal or delegated authentication [1]
• {feature} Purview hub
• displays reports and insights about Fabric items [1]
• acts as a centralized location to begin data governance and access more advanced features [1]
• via Settings >> Microsoft Purview hub
• administrators see information about their entire organization's Fabric data estate
• provides information about
• Data Catalog
• Information Protection
• Audit
• the data section displays tables and graphs that analyze the entire organization's items in MF
• users only see information about their own Fabric items and data

Previous Post <<||>> Next Post 

References:

[1] Microsoft Learn (2024) Purview: Govern data in Microsoft Fabric with Purview[link]

[2] Microsoft Learn (2024) Purview: Learn about data loss prevention [link]

[3] Microsoft Learn (2024) [link]

Resources:

Acronyms:

DLP - Data Loss Prevention
M365 - Microsoft 365

MF - Microsoft Fabric
SaaS - Software-as-a-Service

🏭🗒️Microsoft Fabric: Parameters in Dataflows Gen2 [Notes] 🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 26-Apr-2

[Microsoft Fabric] Dataflow Gen2 Parameters

• {def} parameters that allow to dynamically control and customize Dataflows Gen2
• makes them more flexible and reusable by enabling different inputs and scenarios without modifying the dataflow itself [1]
• the dataflow is refreshed by passing parameter values outside of the Power Query editor through either
• Fabric REST API [1]
• native Fabric experiences [1]
• parameter names are case sensitive [1]
• {type} required parameters
• {warning} the refresh fails if no value is passed for it [1]
• {type} optional parameters
• enabled via Parameters >> Enable parameters to be discovered and override for execution [1]
• {limitation} dataflows with parameters can't be
• scheduled for refresh through the Fabric scheduler [1]
• manually triggered through the Fabric Workspace list or lineage view [1]
• {limitation} parameters that affect the resource path of a data source or a destination are not supported [1]
• ⇐ connections are linked to the exact data source path defined in the authored dataflow
• can't be currently override to use other connections or resource paths [1]
• {limitation} can't be leveraged by dataflows with incremental refresh [1]
• {limitation} supports only parameters of the type decimal number, whole number, text and true/false can be passed for override
• any other data types don't produce a refresh request in the refresh history but show in the monitoring hub [1]
• {warning} allow other users who have permissions to the dataflow to refresh the data with other values [1]
• {limitation} refresh history does not display information about the parameters passed during the invocation of the dataflow [1]
• {limitation} monitoring hub doesn't display information about the parameters passed during the invocation of the dataflow [1]
• {limitation} staged queries only keep the last data refresh of a dataflow stored in the Staging Lakehouse [1]
• {limitation} only the first request will be accepted from duplicated requests for the same parameter values [1]
• subsequent requests are rejected until the first request finishes its evaluation [1]

Previous Post <<||>> Next Post

References:
[1] Microsoft Learn (2025) Use public parameters in Dataflow Gen2 (Preview) [link] 

Resources:
[R1] Microsoft Fabric Blog (2025) Passing parameter values to refresh a Dataflow Gen2 (Preview) [link] 

Acronyms:
API - Application Programming Interface

REST - Representational State Transfer

🏭🗒️Microsoft Fabric: Deployment Pipelines [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 26-Apr-2025

[Microsoft Fabric] Deployment Pipelines

• {def} a structured process that enables content creators to manage the lifecycle of their organizational assets [5]
• enable creators to develop and test content in the service before it reaches the users [5]
• can simplify the deployment process to development, test, and production workspaces [5]
• one Premium workspace is assigned to each stage [5]
• each stage can have 
• different configurations [5]
• different databases or different query parameters [5]
• {action} create pipeline
• from the deployment pipelines entry point in Fabric [5]
• creating a pipeline from a workspace automatically assigns it to the pipeline [5]
• {action} define how many stages it should have and what they should be called [5]
• {default} has three stages
• e.g. Development, Test, and Production
• the number of stages can be changed anywhere between 2-10 
• {action} add another stage,
• {action} delete stage
• {action} rename stage 
• by typing a new name in the box
• {action} share a pipeline with others
• users receive access to the pipeline and become pipeline admins [5]
• ⇐ the number of stages are permanent [5]
• can't be changed after the pipeline is created [5]
• {action} add content to the pipeline [5]
• done by assigning a workspace to the pipeline stage [5]
• the workspace can be assigned to any stage [5]
• {action|optional} make a stage public
• {default} the final stage of the pipeline is made public
• a consumer of a public stage without access to the pipeline sees it as a regular workspace [5]
• without the stage name and deployment pipeline icon on the workspace page next to the workspace name [5]
• {action} deploy to an empty stage
• when finishing the work in one pipeline stage, the content can be deployed to the next stage [5] 
• deployment can happen in any direction [5]
• {option} full deployment 
• deploy all content to the target stage [5]
• {option} selective deployment 
• allows select the content to deploy to the target stage [5]
• {option} backward deployment 
• deploy content from a later stage to an earlier stage in the pipeline [5] 
• {restriction} only possible when the target stage is empty [5]
• {action} deploy content between pages [5]
• content can be deployed even if the next stage has content
• paired items are overwritten [5]
• {action|optional} create deployment rules
• when deploying content between pipeline stages, allow changes to content while keeping some settings intact [5] 
• once a rule is defined or changed, the content must be redeployed
• the deployed content inherits the value defined in the deployment rule [5]
• the value always applies as long as the rule is unchanged and valid [5]
• {feature} deployment history 
• allows to see the last time content was deployed to each stage [5]
• allows to to track time between deployments [5]
• {concept} pairing
• {def} the process by which an item in one stage of the deployment pipeline is associated with the same item in the adjacent stage
• applies to reports, dashboards, semantic models
• paired items appear on the same line in the pipeline content list [5]
• ⇐ items that aren't paired, appear on a line by themselves [5]
• the items remain paired even if their name changes
• items added after the workspace is assigned to a pipeline aren't automatically paired [5]
• ⇐ one can have identical items in adjacent workspaces that aren't paired [5]
• [lakehouse]
• can be removed as a dependent object upon deployment [3]
• supports mapping different Lakehouses within the deployment pipeline context [3]
• {default} a new empty Lakehouse object with same name is created in the target workspace [3]
• ⇐ if nothing is specified during deployment pipeline configuration
• notebook and Spark job definitions are remapped to reference the new lakehouse object in the new workspace [3]
• {warning} a new empty Lakehouse object with same name still is created in the target workspace [3]
• SQL Analytics endpoints and semantic models are provisioned
• no object inside the Lakehouse is overwritten [3]
• updates to Lakehouse name can be synchronized across workspaces in a deployment pipeline context [3] 
• [notebook] deployment rules can be used to customize the behavior of notebooks when deployed [4]
• e.g. change notebook's default lakehouse [4]
• {feature} auto-binding
• binds the default lakehouse and attached environment within the same workspace when deploying to next stage [4]
• [environment] custom pool is not supported in deployment pipeline
• the configurations of Compute section in the destination environment are set with default values [6]
• ⇐ subject to change in upcoming releases [6]
• [warehouse]
• [database project] ALTER TABLE to add a constraint or column
• {limitation} the table will be dropped and recreated when deploying, resulting in data loss
• {recommendation} do not create a Dataflow Gen2 with an output destination to the warehouse
• ⇐ deployment would be blocked by a new item named DataflowsStagingWarehouse that appears in the deployment pipeline [10]
• SQL analytics endpoint is not supported
• [Eventhouse]
• {limitation} the connection must be reconfigured in destination that use Direct Ingestion mode [8]
• [EventStream]
• {limitation} limited support for cross-workspace scenarios
• {recommendation} make sure all EventStream destinations within the same workspace [8]
• KQL database
• applies to tables, functions, materialized views [7]
• KQL queryset
• ⇐ tabs, data sources [7]
• [real-time dashboard]
• data sources, parameters, base queries, tiles [7]
• [SQL database]
• includes the specific differences between the individual database objects in the development and test workspaces [9]
• can be also used with
• Copy Job
• dataflows gen2
• data pipeline
• mirrored database
• activator 
• Power BI items

Previous Post <<||>> Next Post 

References:

[1] Microsoft Learn (2024) Get started with deployment pipelines [link]

[2] Microsoft Learn (2024) Implement continuous integration and continuous delivery (CI/CD) in Microsoft Fabric [link]

[3] Microsoft Learn (2024)  Lakehouse deployment pipelines and git integration (Preview) [link]

[4] Microsoft Learn (2024) Notebook source control and deployment [link] 

[5] Microsoft Learn (2024) Introduction to deployment pipelines [link]

[6] Environment Git integration and deployment pipeline [link]

[7] Microsoft Learn (2024) Microsoft Learn (2024) Real-Time Intelligence: Git integration and deployment pipelines (Preview) [link]

[8] Microsoft Learn (2024) Eventstream CI/CD - Git Integration and Deployment Pipeline [link]

[9] Microsoft Learn (2024) Get started with deployment pipelines integration with SQL database in Microsoft Fabric [link]

[10] Microsoft Learn (2025) Source control with Warehouse (preview) [link] 

Resources:

Acronyms:
CLM - Content Lifecycle Management
UAT - User Acceptance Testing