🏭🗒️Microsoft Fabric: Direct Lake [Notes] 🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 13-Oct-2026

[MF] Direct Lake

• {def} Power BI semantic model table storage mode option available in MF [1]
• optimized for large volumes of data to be quickly loaded into memory from Delta tables available in the OneLake [1] 
• leverages VertiPaq query engine [1]
• {benefit} deliver query performance comparable to Import mode 
• without the management overhead of data refresh cycles to load the entire data volume [1]
• similar to Import mode
• {difference}refreshes copies only metadata (aka framing)
•  [Import mode] replicates the data and creates an entire cached copy of the data for the semantic model [1]
•  depends on data preparation being done in the data lake [1]
• {benefit} enables high-performance interactive analysis [1]
•  ideal for semantic models connecting to large data sources
• e.g. lakehouse, warehouse and other data source with Delta tables [1]
• performance depends on well-tuned Delta tables [1]
•  {benefit} performance
• normally outperforms DirectQuery queries when loading and interacting with visuals in reports [1]
• {benefit}maximizes ROI because 
• analyzed data volumes can exceed capacity's max memory limits [1]
• only the needed data to answer a query is loaded into memory [1]  
• {benefit} minimizes data latencies 
•  makes new data available without refresh schedules [1]
• quickly and automatically synchronizing a semantic model with its sources [1]
• {operation} refresh 
• low-cost operation that analyzes the metadata of the latest version of the Delta tables and updates references to the latest files in OneLake
• DL moves data preparation to OneLake
• <-- uses the full breadth of Fabric technologies for data prep
• incl. Spark jobs, T-SQL DML statements, dataflows, pipelines, etc.
• {operation} transcoding
• the process of loading data from OneLake only when a query requests a column for the first time 
• {action} the semantic model receives a DAX (or MDX) query 
• {step} determines what columns are needed to produce the result
• {step} determines which columns are already resident in memory [2]
• {step} loads the missing columns from OneLake [2]
•  typically a fast operation, however it can depend on factors 
•  e.g. the cardinality of data stored in the columns
•  remains resident until there's reason for it to be removed (evicted) from memory
• {reason} the model or table was refreshed after a Delta table update at the source [2]
• {reason} no query used the column for some time [2]
• {reason} other memory management reasons [2]
•  e.g. memory pressure in the capacity due to other, concurrent operations [2]
•  SKU determines the maximum available memory for each Direct Lake semantic model on the capacity [2]
• {operation} framing 
• an operation triggered by a refresh of a semantic model [2]
• it usually takes only a few seconds to complete
• <-- because it's a low-cost operation where the semantic model analyzes the metadata of the latest version of the Delta Lake tables and updates to reference the latest Parquet files in OneLake [2]
• the process might evict resident table column segments and dictionaries from memory if the underlying data changed [2]
• the point in time of the refresh becomes the new baseline for all future transcoding events [2]
• DL queries only consider data in the Delta tables as of the time of the most recent framing operation [2]
• DL tables are queried to return data based on the state of the Delta table at the point of the most recent successful framing operation [2]
• that time isn't necessarily the latest state of the Delta tables [2]
• the semantic model analyzes the Delta log of each Delta table during framing to drop only the affected column segments and to reload newly added data during transcoding [2]
• {optimization} dictionaries usually aren't dropped when incremental framing takes effect, and new values are added to the existing dictionaries [2]
• helps reduce the reload burden and benefits query performance [2]
• when a Delta table receives no updates, no reload is necessary for columns already resident in memory [2]
•  queries show far less performance impact after framing because incremental framing essentially enables the semantic model to update substantial portions of the existing in-memory data in place [2]
• can fail if a Delta table exceeds the Fabric capacity guardrails [2]
•  e.g. when a Delta table has more than 10,000 parquet files [2]
•  {architecture} Direct Lake on OneLake
•  isn't coupled with the SQL analytics endpoint [2]
•  offers tighter integration with OneLake features such as OneLake security [2]
•  offers more efficient DAX query plans because [2]
• <-- isn’t required to check for SQL based security [2]
•  fallback isn't supported [2]
• {architecture} Direct Lake on SQL analytics endpoint
•  a DAX query might use DirectQuery fallback [2]
•  involves seamlessly switching to DirectQuery mode [2]
•  DirectQuery fallback
• retrieves data directly from the SQL analytics endpoint of the lakehouse or the warehouse [2]
•  fallback occurs when SQL based security is detected in the SQL analytics endpoints
•  a DirectQuery operation sends a query to the SQL analytics endpoint. Fallback operations might result in slower query performance [2]
•  {setting} Keep your Direct Lake data up to date
• performs automatic updates of Direct Lake tables [2]
• via semantic model settings >> Refresh 
• {enabled|default} ensures that data changes in OneLake are automatically reflected in the Direct Lake semantic model [2]
•  the semantic model performs a framing operation whenever data modifications in the underlying Delta tables are detected [2]
• {recommendation} leave the setting on, especially when you have a small or medium-sized semantic model
• useful in case of low-latency reporting requirements and Delta tables are modified regularly [2]
•  {disabled} it might be needed to disable automatic updates to allow completion of data preparation jobs or an ETL process before exposing any new data to consumers of the semantic model [2]
•  users can trigger a refresh manually in the Fabric portal [2]
•  via refresh schedule, or programmatically by using the Power BI REST API or TOM [2]
• 
  

References: 
[1] Microsoft Learn (2026) Direct Lake overview [link]
[2] Microsoft Learn (2026) 

Acronyms:
API - Application Programming Interface
DL - Direct Lake 
MDX - Multidimensional Expressions
MF - Microsoft Fabric
REST
SKU - Stock Keeping Unit
TOM - Tabular Object Model
T-SQL - Transact SQL

 

🤖〽️Prompt Engineering: Copilot Unabridged (Part 124: The Impact of Algorithmic Bias on AI: Why It Matters More Than Ever)

Prompt Engineering Series

Prompt: "write a blog post of 600 words on the impact of algorithmic bias on AI"

Introduction

Artificial intelligence (AI) has become woven into the fabric of modern life. It recommends what we watch, filters what we read, evaluates our job applications, approves our loans, and increasingly shapes decisions once made exclusively by humans. Yet beneath the surface of this technological progress lies a persistent and often invisible challenge: algorithmic bias. As AI systems grow more influential, understanding the origins, consequences, and solutions to bias becomes essential - not just for technologists, but for anyone living in a world mediated by algorithms.

What Is Algorithmic Bias?

Algorithmic bias occurs when an AI system produces systematically unfair outcomes - favoring or disadvantaging certain groups - because of patterns embedded in its data, design, or deployment. Contrary to popular belief, bias in AI is rarely the result of malicious intent. Instead, it emerges from the simple fact that AI learns from historical data, and history is full of human imperfections.

If the data reflects societal inequalities, the model will learn those inequalities. If the training set underrepresents certain populations, the model will perform worse for them. And if the objectives or constraints are poorly defined, the system may optimize for the wrong outcomes entirely.

In other words, AI doesn’t just mirror the world - it can magnify its flaws.

Where Bias Creeps In

Bias can enter an AI system at multiple stages:

1. Biased Training Data

AI models learn statistical patterns from examples. If those examples are skewed, incomplete, or unrepresentative, the model inherits those distortions. Classic cases include facial recognition systems that perform poorly on darker skin tones because the training data was overwhelmingly composed of lighter-skinned faces.

2. Problem Framing and Design Choices

Even before data enters the picture, human decisions shape the system. What is the model optimizing for? What counts as a 'successful' prediction? Which variables are included or excluded? These choices embed assumptions that can unintentionally privilege certain outcomes.

3. Feedback Loops in Deployment

Once deployed, AI systems can reinforce their own biases. For example, predictive policing tools may direct more patrols to neighborhoods flagged as 'high risk', generating more recorded incidents and further validating the model’s initial assumptions - even if the underlying crime rates were similar elsewhere.

Why Algorithmic Bias Matters

The consequences of biased AI are not abstract - they affect real people in tangible ways.

1. Inequitable Access and Opportunity

Biased hiring algorithms may filter out qualified candidates. Loan approval models may disproportionately reject applicants from certain demographics. Healthcare algorithms may underestimate risk for specific populations, leading to worse outcomes.

2. Erosion of Trust

When people experience or perceive unfairness in algorithmic decisions, trust in institutions erodes. This is especially damaging in sectors like government, healthcare, and finance, where legitimacy depends on perceived fairness.

3. Amplification of Social Inequalities

AI systems operate at scale. A biased decision made once by a human affects one person; a biased decision made by an algorithm affects thousands or millions. Without intervention, AI can accelerate and entrench existing disparities.

Toward Fairer, More Responsible AI

Addressing algorithmic bias is not a one-time fix - it’s an ongoing process requiring technical, organizational, and societal commitment.

1. Better Data Practices

Diverse, representative datasets reduce the risk of skewed outcomes. Equally important is transparency about data sources and limitations.

2. Rigorous Testing and Auditing

Bias detection tools, fairness metrics, and independent audits help identify issues before they cause harm. Continuous monitoring ensures that systems remain fair as conditions change.

3. Human Oversight and Accountability

AI should augment - not replace - human judgment. Clear accountability structures ensure that organizations remain responsible for the outcomes of their systems.

4. Ethical and Regulatory Frameworks

Governments and industry bodies are increasingly developing guidelines to ensure fairness, transparency, and accountability. These frameworks help align AI development with societal values.

Conclusion

Algorithmic bias is not an inevitable flaw of AI - it’s a challenge we can address with thoughtful design, responsible data practices, and ongoing oversight. As AI continues to shape our world, the choices we make today will determine whether these systems reinforce old inequities or help build a more just and inclusive future.

Disclaimer: The whole text was generated by Copilot (under Windows 10) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post