15 March 2025

💠🛠️🗒️SQL Server: Schemas [Notes]

Disclaimer: This is work in progress based on notes gathered over the years, intended to consolidate information from the various sources. 

Last updated: 15-Mar-2024

[SQL Server 2005] Schemas

  • {def} a collection of database objects that are owned by a single user and form a single namespace
    • a named container for database objects
      • allows to group objects into separate namespaces
      • collection of like objects which provide maintenance and security to those objects as a whole, without affecting objects within other schemas [1]
    • reside within databases
    • fulfilling a common purpose [1]
    • each schema can contain zero or more data structures (aka objects) [1]
    • all objects within a schema share 
      • a common naming context
      • a common security context [10]
    • behavior of schema changed 
      • ⇐ compared to SQL Server 2000
      • schemas are no longer equivalent to database users
        • each schema is a distinct namespace that exists independently of the database user who created it
          • used as a prefix to the object name
          • schema is simply a container of objects [3]
        • code written for earlier releases of SQL Server may return incorrect results, if the code assumes that schemas are equivalent to database users [3]
    • can be owned by any database principal
      • this includes roles and application roles  [3]
      • its ownership is transferable [3]
      • every object is contained by a schema [6]
      • anything contained by it has the same owner [6]
    • separation of ownership [3]
      • ownership of schemas and schema-scoped securables is transferable [3]
      • objects can be moved between schemas [3]
      • a single schema can contain objects owned by multiple database users  [3]
      • multiple database users can share a single default schema  [3]
      • permissions on schemas and schema-contained securables can be managed with greater precision than in earlier releases  [3]
      • each user has a default schema [3]
      • user’s default schema is used for name resolution during object creation or object reference [7]
        •  {warning} a user might not have permission to create objects in the dbo schema, even if that is the user’s default schema [7]
        • when a login in the sysadmin role creates an object with a single part name, the schema is always dbo [7]
        • a database user can be dropped without dropping objects in a corresponding schema  [3]
        • catalog views designed for earlier releases of SQL Server may return incorrect results
          • ⇐ includes sysobjects
          • more than 250 new catalog views were introduced to reflect the changes
        •  when creating a database object, if you specify a valid domain principal (user or group) as the object owner, the domain principal will be added to the database as a schema. The new schema will be owned by that domain principal [3]
    • schema-qualified object name (aka two-part object name)
      • if schema is omitted, a schema resolution is performed (aka implicit resolution)
        • checks whether the object exists in the user's default schema
        • if it doesn't, checks whether it exists in the dbo schema [5]
          • extra costs are involved in resolving the object name (aka name resolution) [5]
            • uses a spinlock [8]
              • in rare occasions a spinlock could not be acquired immediately on such an operation
                • this may occur on a system under significant load [8]
                • the contention appears on the SOS_CACHESTORE spinlock type [8]
                • {resolution} ensure that you always fully qualify your table names [8]
            • if multiple objects with the same name exist in different schemas, the wrong object might be retrieved [5]
        • improves readability
      • {recommendation} always use two-part object names in queries (aka schema-qualify objects) 
      • {poor practice} partition data and objects by using only schemas 
        •  instead of creating multiple databases [1]
      • {poor practice} complex schemas
        • developing a row-based security schema for an entire database using dozens or hundreds of views can create maintenance issues [6]
  • {benefit} simplify database object management
    • groups of tables can be managed from a single point [4]
      • by creation of categories of tables [4]
    • helps navigation through database [4]
    • allow control permissions at schema level 
  • {benefit} provide separation of ownership 
    • allows to manage user permissions at the schema level, and then enhance them or override them at the object level as appropriate [10]
    • {recommendation} manage database object security by using ownership and permissions at the schema level [2]
    • {recommendation} have distinct owners for schemas or use a user without a login as a schema owner [2]
    • {recommendation} not all schemas should be owned by dbo [2]
    • {recommendation} minimize the number of owners for each schema [2]
  • {benefit} enhance security
    • by minimizing the risk of SQL injection
      • by assigning objects to schema it is possible to drop users without rewriting your applications as the name resolution is no longer depend upon the user or principals names 
    • used as an extra hierarchical layer for solution and security management [1]
      • gives architects and developers the ability to choose between the types of logical separation of objects they have created, as well as benefit from having a combination of multiple databases and multiple schemas within them [1]
  • {type} system schemas
    • can't be dropped
    • [default schema] dbo
      • included in each database 
      • if an application needs to  create objects in the under the dbo schema then by granting dbo privileges to the application [12]
        • increases the attack surface of the application [12]
        • increases the severity if the application is vulnerable to SQL Injection attacks [12]
      • can be set and changed by using DEFAULT_SCHEMA option of [3]
        • e.g. CREATE USER <user_name> WITH DEFAULT_SCHEMA = <schema_name>
        • e.g. ALTER USER <user_name> WITH DEFAULT_SCHEMA = <schema_name>
      • if DEFAULT_SCHEMA is left undefined, the database user will have dbo as its default schema [3]
        • [SQL Server 2005] Windows Groups are not allowed to have this property [11]
        • [SQL Server 2012] Windows Groups can also have a defined default schema [1]
          • streamlines the process of creating users
            • if no default schema is specified for a new user, instead is used the default schema of a group where the user is a member [9]
      • {warning} not to be confused with the dbo role [6]
    • INFORMATION_SCHEMA schema
      • an internal, system table-independent view of the SQL Server metadata
      • enable applications to work correctly although significant changes have been made to the underlying system tables
    • guest schema

    • sys schema 
      • provides a way to access all the system tables and views [7]
  • {type} user-defined schemas
    • {best practice} assign objects to user-defined schemas
      • leaving everything in the dbo schema is like putting everything in the root directory of your hard drive [8]
      • it saves the Query Processor a step from having to resolve the schema name out by itself [8]
        • avoid ambiguity
    • {best practice} assign each user a default schema
      •  ensures that if they create an object without specifying a schema, it will automatically go into their assigned container [8]
  • {type} role-based schemas
    • [SQL Server 2012] every fixed database role has a schema of the same name [7]
      • {exception} public role5
  • {action} create objects in schema
    • {prerequisite}
      • schema must exist
      • the user creating the object must have permission to create the object, either directly or through role membership [7]
      • the user creating the object must  either [7]
        • be the owner of the schema 
        • be a member of the role that owns the schema
        • have ALTER rights on the schema 
        • have the ALTER ANY SCHEMA permission in the database
    • {recommendation} group like objects together into the same schema [2]
  • {operation} create schema
    • {recommendation} use two-part names for database object creation and access [2]
  • {operation} change schema (aka modify schema)
    • when applying schema changes to an object and try to manipulate the object data in the same batch, SQL Server may not be aware of the schema changes yet and fail the data manipulation statement with a resolution error [5]
      • the parsing does not check any object names or schemas because a schema may change by the time the statement executes [6]
    • triggers a database lock
    • invalidates existing query plans
      • a new plan will need to be recompiled for the queries as soon as they are run anew
    • not allowed on
      • [SQL Server 2014] [memory-optimized tables]
      • [table variables]
    • {best practice} explicitly list column names in statements in case a schema changes 
  • {operation} schema dropping
  • [windows groups]
    • an exception in the SQL Server security model [11]
    • a secondary identity with additional capabilities that are traditionally reserved only for primary identities [11]
      • require handling not seen in any other security system [11]
    •  can simplify management but due to their hybrid nature, they come with some restrictions [11]
    • {recommendation} for users mapped to Windows groups, try and limit each Windows user to one Windows group that has database access [2]

References:
[1] 40074A: Microsoft SQL Server 2014 for Oracle DBAs, Microsoft, 2015 
[2] Bob Beauchemin et al (2012) SQL Server 2012 Security Best Practices - Operational and Administrative Tasks [whitepaper]
[3] MSDN (2005)User-Schema Separation [link]
[4] Solid Quality Learning (2007) Microsoft SQL Server 2005: Database Essentials Step by Step
[5] Itzik Ben-Gan (2008) Microsoft® SQL Server® 2008 T-SQL Fundamentals
[6] Adam Jorgensen et al (2012) Microsoft® SQL Server® 2012 Bible
[7] Kalen Delaney et al (2013) Microsoft SQL Server 2012 Internals
[8] Buck Woody (2009) SQL Server Best Practices: User-Defined Schemas [link obsolet]
[9] Microsoft (2014) 10977B: Updating Your SQL Server Skills to SQL Server 2014 (Trainer Handbook)
[10] Microsoft (2012) 20465B Designing Database Solutions for SQL Server 2012 (Trainer Handbook
[11] Laurentiu Cristofor (2008) SQL Server: Windows Groups, default schemas, and other properties [link]
[12] Dan Sellers's WebLog (2006) Post Webcast’s Notes: Securing SQL Server 2005 for Developers  [link]
[13] Microsoft Learn (2024) SQL Server 2022: System Information Schema Views (Transact-SQL)
[14] 

Acronyms:
SQL - Structured Query Language

💫🗒️ERP Systems: Microsoft Dynamics 365's Business Performance Analytics (BPA) [notes]

Disclaimer: This is work in progress intended to consolidate information from the various sources and not to provide a complete overview of all the features. Please refer to the documentation for a complete overview!

Last updated: 15-Mar-2025

[Dynamics 365] Business Performance Analytics (BPA)

  • {def} centralized reporting hub within D365 F&O designed to streamline insights and help organizations make faster, data driven decisions [3]
    • solution designed to transform organization's data into actionable insights [1]
    • provides an intuitive out-of-box data model along with familiar tools like Microsoft Excel and Power BI for self-service analytics [4]
      • data extracted from D365 is classified in BPA in the form of value chains
        • ⇐ a group of business processes on top of the value chain [4]
  • {benefit} allows to simplify data insights by providing a unified view of business data across entities in near real time [4]
  • {benefit} allows to streamline financial and operations reporting to reduce the cycle times [4]
  • {benefit} allows users of all technical abilities to quickly access and analyze data to facilitate data driven decisions [4]
  • {benefit} provides auditors with direct access to financial data, making the audit process more efficient
  • {benefit} enables ease of use through familiar apps like Excel and Power BI, in addition to AI driven insights and automation in this platform that can be scalable and extendable [4]
  • {feature} extends into Microsoft Fabric
    • {benefit} provide a scalable, secure environment for handling large data sets and ensuring insights are always powered by the latest technology [3]
  • {feature} ETL process 
    • involves extracting data from finance and operations database, transforming and loading it into Dataverse [4]
      • each of the entities required for the generation of the dimensional model for the value chains that were mentioned earlier, they are backed by the underlying tables in finance and operations database [4]
    • installed in Dataverse, virtual  entities that are created will then pull in the data into the managed data lake [4]
    • the data is then transformed to generate the dimensional  model which is then pushed into the embedded Power BI workspace in the form of analytical tables [4]
    • BPA consumes this data from Power BI workspace to render the power BI reports [4]
    • this data can also be extended to Fabric if there is a need to consolidate data from multiple sources [4]
  • {feature} reports 
    • designed to provide a detailed overview of an organization's financial health [8]
    • further reports will be added to expand the coverage for the value chains [8]
    • out-of-box reports can't be modified
      • ⇐ users cannot rename, delete or edit these type of reports [8]
      • there’s the option to duplicate the base report and edit the version thus created [8]
    • can be shared with other users who have access to BPA 
      • ⇐ they can receive an in-app notification [8]
      • can be shared over email with another user by entering user’s email address [8] 
      • one can configure whether the recipient can edit or view the report [8]
    •   {feature} allows to create a new Power BI or Excel report from scratch [8]
      • {option} start with a blank report or duplicate an existing report [8]
  • {feature} data refresh
    • automatic data refreshes run currently two times a day [4]
      • at 12:00 AM and 12:00 PM UTC
      • the volume of data is also constrained by the storage capacity of the A3 SKU for Power BI Embedded [1]
        • future release, may support additional data reporting capacity [1]
          • ⇐ so that larger data sets can be reported and analyzed [1]
      • the target is to have refreshes every hour or less [3]
    • data volume will be initially for about eight quarters of data [4]
    • extensibility will be supported with bring your own Fabric [4]
  • architecture
    • SaaS solution
      • {capability} immediate deployment 
        • businesses can start to analyze data and generate insights with minimal setup [1]
      • {capability} comprehensive reporting and dashboards
        • provides access to a wide range of preconfigured reports that cover multiple business functions [1]
      • {capability} near-real-time analytics 
        • future releases will offer more frequent data refreshes to enable near-real-time data analysis and reporting
      • {capability} predictive insights 
        • future releases will introduce predictive analytics capabilities that enable businesses to 
          • forecast trends
          • identify risks
          • seize opportunities [1]
      • {capability} user-friendly interface 
        • intuitive design ⇒ minimal training
          • fosters broader adoption 
          • enables a data-driven culture across the organization [1]
      • {capability} cost-effectiveness
        • available as part of D365 license
          • ⇒ provides advanced analytics without requiring significant investments in IT infrastructure [1]
    • DaaS solution
      • {capability} organizations can integrate its data models with their existing data warehousing infrastructure in Microsoft Fabric [1]
        • maximizes the value of existing data solutions [1]
        • positions businesses for future enhancements [1]
      • {capability} unified and scalable data models
        • customers can build custom models on top of a unified framework
          • ensures consistency and scalability across data sets [1]
      • {capability} future-proofing with automatic upgrades
        • data models integrate seamlessly with future D365 updates
          • reduces manual maintenance and ensures access to the latest features [1]
      • {capability} consistency and standardization
        • data models provide consistency and standardization across data sources
          • ensure high data quality and integrity [1]
      • {capability} advanced analytics and AI 
        • by customizing the data models, organizations can take advantage of advanced analytics and AI capabilities [1]
          • deeper insights without having to develop them from scratch [1]
      • {capability} enhanced data governance
        • unified data models support better data governance by providing standardized data definitions, relationships, and hierarchies [1]
          • ensure consistency and quality across the organization [1]
    • requires an integrated Power Platform environment [5]
      • must be integrated with the Microsoft Entra tenant [5]
    • uses shared Dataverse entitlements [1]
      • includes access to the data lake [1]
  • setup
    • dimensions
      • the selection of dimensions might affect the dimension groups that are created using these dimensions and the users who are assigned there [7]
        • e.g. legal entity, business unit
    • dimension groups
      • users can select specific values for the legal entity, or add a range of values [7]
        • selecting an invalid combination of dimension values, the dimension group will filter out all the records on the report [7]
      • {warning} assigning too many dimension groups to a user, slows the load for that user [7]
    • roles
      • determine which reports the user can access [7]
  • security
    • secure data through role-based access control on top of the value chains [7]
    • the first user who signs into the app is assigned the BPA admin role [7]
      • allows a user to access the administrator section of the BPA [7]
        • where the security can be set up [7]
      • has automatically assigned 
        • Microsoft report viewer role 
        • the All Access Dimension group [7]
          • allow the admin to see the data  in all the reports across all the dimensions [7]
    • {feature} dimension-based role-level security
      • ensures that users only see the data relevant to them based on their role
        •  confidently share reports without duplicating them
          • ⇐ data is automatically filtered by organization's security policies [3]
      • simple but powerful way to maintain control while providing access for teams that love working in Excel [3]
  • accessibility
    • can be accessed through either 
      • Power Platform
        • admins can access BPA app through PowerApps' makeup portal [6]
      • Dynamics 365
        • through the BPA preview shortcut in the homepage or the default dashboard [6]
        • for end users, the BPA preview shortcut is provided when they have certain duties associated to their role(s) [6]
  • licensing
    • included in D365 F&O license [4]
  • requirements
    • requires a tier two environment and Dynamics 365 finance version 1.0.38 or later [5]
  • {project} timeline
    • [2025 wave 1] backup and restore custom reports and analytics
      • {benefit} support better lifecycle management and empower customers to develop on sandbox instances before publishing to production [3]
    • 2025: available in all regions where F&O is available [3]
    • Oct-2024: GA

References:
[1] Microsoft Learn (2024) Dynamics 365 Finance: What is Business performance analytics? [link]
[2] Microsoft Learn (2025) Business performance analytics (BPA) with Dynamics 365 Finance [link]
[3] Dynamics 365 Finance - Business Performance Analytics 2025 Release Wave 1 Release Highlights [link]
[4] Dynamics 365 Community (2024) Dynamics 365 Bites: Business Performance Analytics Part 1 [link]
[5] Dynamics 365 Community (2024) Dynamics 365 Bites: Business Performance Analytics Part 2 [link]
[6] Dynamics 365 Community (2024) Dynamics 365 Bites: Business Performance Analytics Part 3 [link]
[7] Dynamics 365 Community (2024) Dynamics 365 Bites: Business Performance Analytics Part 4 [link]   
[8] Dynamics 365 Community (2024) Dynamics 365 Bites: Business Performance Analytics Part 5 [link]
[9] Microsoft Learn (2024) Dynamics 365: Business performance analytics introduction [link

Acronyms:
AI - Artificial Intelligence
BPA - Business Performance Analytics
D365 F&O - Dynamics 365 for Finance and Operations
DaaS - Data-as-a-Service
ETL - Extract, Transfer, Load
GA - General Availability
MF - Microsoft Fabric
PP - Public Preview
SaaS - Software-as-a-Service
SKU - Stock Keeping Unit
UTC - Coordinated Universal Time

14 March 2025

🧩IT: Short Quotes Used in Various Posts

Short quotes used in the various posts:

"A problem well stated is a problem half solved." (Charles F Kettering)Approaching a Query

"An army of principles can penetrate where an army of soldiers cannot." (Thomas Paine)Guiding Principles

"Architecture starts when you carefully put two bricks together." (Ludwig Mies van der Rohe)Guiding Principles

"Data quality requires certain level of sophistication within a company to even understand that it’s a problem." (Colleen Graham): [Who Messed with My Data?]

"Errors, like straws, upon the surface flow;
He who would search for pearls must dive below." (John Dryden)
: [Who Messed with My Data?]

"Everything should be made as simple as possible, but not simpler." (Albert Einstein)Facts, Principles and Practices

"For every complex problem there is an answer that is clear, simple, and wrong." (Henry L Mencken) [Who Messed with My Data?]

"I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail." (Abraham Maslow): [Who Messed with My Data?]

"In preparing for battle I have always found that plans are useless, but planning is indispensable." (Eisenhower quoted by Nixon)Planning Correctly Misunderstood

"It's a bad plan that admits of no modification." (Publilius Syrus)Planning Correctly Misunderstood

"Keep it simple, stupid" (aka KISS): Guiding PrinciplesFacts, Principles and PracticesSimple, but not that Simple

"Management is doing things right […]" (Peter Drucker)Guiding Principles

"No plan ever survived contact with the enemy." (Carl von Clausewitz)Planning Correctly Misunderstood

"Obey the principles without being bound by them." (Bruce Lee)Guiding Principles

"Students are often able to use algorithms to solve numerical problems without completely understanding the underlying scientific concept." (Eric Mazur): [Who Messed with My Data?]

"The ability to simplify means to eliminate the unnecessary so that the necessary may speak." (Hans Hofmann)Facts, Principles and Practices

"The enemy of a good plan is the dream of a perfect plan." (Carl von Clausewitz)Planning Correctly Misunderstood

"The first obligation of Simplicity is that of using the simplest means to secure the fullest effect" (George Lewes, "Style in Literature")Designing for Simplicity

"The weakest spot in a good defense is designed to fail." (Mark Lawrence): [Who Messed with My Data?]

"To err is human; to try to prevent recurrence of error is science." (Anon): [Who Messed with My Data?

13 March 2025

🏭🗒️Microsoft Fabric: Workspaces [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 25-Mar-2025

[Microsoft Fabric] Workspace

  • {def} a collection of items that brings together different functionality in a single environment designed for collaboration
  • {default} created in organization's shared capacity
    • workspaces can be assigned to other capacities
      • includes My Workspaces
      • via Workspace settings >> Premium
    • shared environment that holds live items
      • any changes made directly within the workspace immediately take effect and impact all users
  • components
    • header
      • contains
        • name 
        • brief description of the workspace
        • links to other functionality
    • toolbar 
      • contains 
        • controls for managing items to the workspace 
        • controls for managing files
    • view area
      • enables selecting a view
      • {type} list view
        • {subitem} task flow
          • area in which users can create or view a graphical representation of the data project [3]
            • ⇐ shows the logical flow of the project [3]
              • ⇐ it doesn't show the flow of data [3]
          • can be hided via Show/Hide arrows
        • {subitem} items list
          • area in which the users can see the items and folders in the workspace [3]
          • one can filter the items list by selecting the tasks, if any defined [3]
        • {subitem} resize bar
          • elements that allow to resize the task flow and items list by dragging the resize bar up or down [3]
      • {type} lineage view
        • shows the flow of data between the items in the workspace [3]
  • {feature} workspace settings 
    • allows to manage and update the workspace [3]
  • {feature} contact list 
    • allows to specify which users receive notification about issues occurring in the workspace [3] 
    • {default} contains workspace's creator [3]
  • {feature} SharePoint integration 
    • allows to configure a M365 Group whose SharePoint document library is available to workspace users [3]
      • ⇐ the group is created outside of MF first [3]
      • restrictions may apply to the environment
    • {best practice} give access to the workspace to the same M365 Group whose file storage is configured [3]
      • MF doesn't synchronize permissions between users or groups with workspace access, and users or groups with M365 Group membership [3]
  • {feature} workspace identity
    • an automatically managed service principal that can be associated with a Fabric workspace [6]
      • workspaces with a workspace identity can securely read or write to firewall-enabled ADSL Gen2 accounts through trusted workspace access for OneLake shortcuts [6]
      • Fabric creates a service principal in Microsoft Entra ID to represent the identity [6]
        • ⇐ an accompanying app registration is also created [6]
        • Fabric automatically manages the credentials associated with workspace identities [6]
          • ⇒ prevents credential leaks and downtime due to improper credential handling [6]
    • used to obtain Microsoft Entra tokens without the customer having to manage any credentials [6]
      • Fabric items can use the identity when connecting to resources that support Microsoft Entra authentication [6]
    • can be created in the workspace settings of any workspace except My workspaces
    • automatically assigned to the workspace contributor role and has access to workspace items [6]
  • {feature} workspace roles
    • allows to manage who can do what in a workspace [4]
    • sit on top of OneLake and divide the data lake into separate containers that can be secured independently [4]
    • extend the Power BI workspace roles by associating new MF capabilities 
      • e.g. data integration, data exploration
    • can be assigned to 
      • individual users
      • security groups
      • Microsoft 365 groups
      • distribution lists
    • {role} Admin
    • {role} Member
    • {role} Contributor
    • {role} Viewer
    • user groups
      • members get the role(s) assigned
      • users existing in several group get the highest level of permission that's provided by the roles that they're assigned [4]
      • {concept} [nested group]
  • {concept} current workspace
    • the active open workspace
  • {action} create new workspace
  • {action} pin workspace
  • {action} delete workspace
    • everything contained within the workspace is deleted for all group members [3]
      • the associated app is also removed from AppSource [3]
    • {warning} if the workspace has a workspace identity, that workspace identity will be irretrievably lost [3]
      • this may cause Fabric items relying on the workspace identity for trusted workspace access or authentication to break [3]
    • only admins can perform the operation
  • {action} manage workspace
  • {action} take ownership of Fabric items
    • Fabric items may stop working correctly [5]
      • {scenario} the owner leaves the organization [5]
      • {scenario}the owner don't sign in for more than 90 days [5]
      • in such cases, anyone with read and write permissions on an item can take ownership of the item [5]
        • become the owner of any child items the item might have
        • {limitation} one can't take over ownership of child items directly [5]
          • ⇐ one can take ownership only through the parent item [5]
  • {limitation} workspace retention
    • personal workspace
      • {default} 30 days [8]
    • collaborative workspace
      • {default} 7 days [8]
      • configurable between 7 to 90 days [8]
        • via Define workspace retention period setting [8]
    •  during the retention period, Fabric administrators can restore, respectively delete permanently the workspace [8]
      • after that, the workspace is deleted permanently and it and its contents are irretrievably lost [8]
  • {limitation} can contain a maximum of 1000 items [8]
    • includes both parent and child items [8]
      • refers to Fabric and Power BI items [8]
    • workspaces with more items have 180-day extension period applied automatically as of 10-Apr-2025 [8]
  • {limitation} certain special characters aren't supported in workspace names when using an XMLA endpoint [3]
  • {limitation} a user or a service principal can be a member of up to 1000 workspaces [3]
  • {feature} auditing
    • several activities are audited for workspaces [3]
      • CreateFolder
      • DeleteFolder
      • UpdateFolder
      • UpdateFolderAccess
  • {feature} workspace monitoring 
    • Eventhouse secure read-only database that collects and organizes logs and metrics from a range of Fabric items in the workspace [1]
      • accessible only to workspace users with at least a contributor role [1]
      • users can access and analyze logs and metrics [1]
      • the data is aggregated or detailed [1]
      • can be queried via KQL or SQL [1]
      • supports both historical log analysis and real-time data streaming [1]
      • accessible from the workspace [1]
        • one can build and save query sets and dashboards to simplify data exploration [1]
      • use the workspace settings to delete the database [1]
        •  wait about 15 minutes before recreating a deleted database [1]
    • {action} share the database
      • users need workspace member or admin role [1]
    • {limitation} one can enable either 
      • workspace monitoring 
      • log analytics
        • if enabled, the log analytics configuration must be deleted first before enabling workspace monitoring [1]
          • one should wait for a few hours before enabling workspace monitoring [1]
    • {limitation} retention period for monitoring data: 30 days [1]
    • {limitation}the ingestion can't be configured to filter for specific log type or category [1]
      • e.g. error or workload type.
    • {limitation} user data operation logs aren't available even though the table is available in the monitoring database [1]
    • {prerequisite} Power BI Premium or Fabric capacity [1]
    • {prerequisite} workspace admins can turn on monitoring for their workspaces tenant setting is enabled [1]
      • enabling the setting, requires Fabric administrator rights [1]
    • {prerequisite} admin role in the workspace [1]
  • workspace permissions
    •  the first security boundary for data within OneLake [7]
      • each workspace represents a single domain or project area where teams can collaborate on data [7]
      • security is managed through Fabric workspace roles [7]
    • items can have permissions configured separately from the workspace roles [7]
      • permissions can be configured either by [7]
        • sharing an item 
        • managing the permissions of an item

References:
[1] Microsoft Learn (2024) Fabric: What is workspace monitoring (preview)? [link]
[2] Microsoft Fabric Update Blog (2024) Announcing preview of Workspace Monitoring? [link]
[3] Microsoft Learn (2024) Fabric: Workspaces in Microsoft Fabric and Power BI [link]
[4] Microsoft Learn (2024) Fabric: Roles in workspaces in Microsoft Fabric [link]
[5] Microsoft Learn (2024) Fabric: Take ownership of Fabric items [link]
[6] Microsoft Learn (2024) Fabric: Workspace identity [link]
[7] Microsoft Learn (2024) Fabric: Role-based access control (RBAC) [link]
[8] Microsoft Learn (2024) Fabric: Manage workspaces [link]

Resources:
[R1] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]

Acronyms:
ADSL Gen2 - Azure Data Lake Storage Gen2
KQL - Kusto Query Language
M365 - Microsoft 365
MF - Microsoft Fabric
SQL - Structured Query Language

12 March 2025

🏭🎗️🗒️Microsoft Fabric: Query Acceleration [Notes] 🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 12-Mar-2025

Query Acceleration
Query Acceleration [2]

[Microsoft Fabric] Query Acceleration

  • {def} 
    • indexes and caches on the fly data landing in OneLake [2]
      • {benefit} allows to 
        • analyze real-time streams coming directly into Eventhouse and combine it with data landing in OneLake 
        • ⇐ either coming from mirrored databases, Warehouses, Lakehouses or Spark [2] 
        • ⇒ accelerate data landing in OneLake
          • ⇐ including existing data and any new updates, and expect similar performance [1]
          • eliminates the need to 
            • manage ingestion pipelines [1]
            • maintain duplicate copies of data [1]
          • ensures that data remains in sync without additional effort [4]
          • the initial process is dependent on the size of the external table [4]
      • ⇐ provides significant performance comparable to ingesting data in Eventhouse [1]
        • in some cases up to 50x and beyond [2]
      • ⇐ supported in Eventhouse over delta tables from OneLake shortcuts, etc. [4]
        • when creating a shortcut from an Eventhouse to a OneLake delta table, users can choose if they want to accelerate the shortcut [2]
        • accelerating the shortcut means equivalent ingestion into the Eventhouse
        • ⇐  optimizations that deliver the same level of performance for accelerated shortcuts as native Eventhouse tables [2]
        • e.g. indexing, caching, etc. 
      • all data management is done by the data writer and in the Eventhouse the accelerated table shortcut [2]  
      • behave like external tables, with the same limitations and capabilities [4]
        • {limitation} materialized view aren't supported [1]
        • {limitation} update policies aren't supported [1]
    • allows specifying a policy on top of external delta tables that defines the number of days to cache data for high-performance queries [1]
      • ⇐ queries run over OneLake shortcuts can be less performant than on data that is ingested directly to Eventhouses [1]
        • ⇐ due to network calls to fetch data from storage, the absence of indexes, etc. [1]
    • {costs} charged under OneLake Premium cache meter [2]
      • ⇐ similar to native Eventhouse tables [2]
      • one can control the amount of data to accelerate by configuring number of days to cache [2]
      • indexing activity may also count towards CU consumption [2]
    • {limitation} the number of columns in the external table can't exceed 900 [1]
    • {limitation} query performance over accelerated external delta tables which have partitions may not be optimal during preview [1]
    • {limitation} the feature assumes delta tables with static advanced features
      • e.g. column mapping doesn't change, partitions don't change, etc
      • {recommendation} to change advanced features, first disable the policy, and once the change is made, re-enable the policy [1]
    • {limitation} schema changes on the delta table must also be followed with the respective .alter external delta table schema [1]
      • might result in acceleration starting from scratch if there was breaking schema change [1]
    • {limitation} index-based pruning isn't supported for partitions [1]
    • {limitation} parquet files with a compressed size higher than 6 GB won't be cached [1]

References:
[1] Microsoft Learn (2024) Fabric: Query acceleration for OneLake shortcuts - overview (preview) [link]
[2] Microsoft Fabric Updates Blog (2024) Announcing Eventhouse Query Acceleration for OneLake Shortcuts (Preview) [link]
[3] Microsoft Learn (2024) Fabric: Query acceleration over OneLake shortcuts (preview) [link]
[4] Microsoft Fabric Updates Blog (2025) Eventhouse Accelerated OneLake Table Shortcuts – Generally Available [link

Resources:
[R1] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]

🏭🎗️🗒️Microsoft Fabric: Activator [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 12-Mar-2025

Microsoft Fabric Activator
Microsoft Fabric Activator [1]

[Microsoft Fabric] Activator

  • [def] no-code experience for automatically taking actions when patterns or conditions are detected in changing data [1]
  • monitors data in Power BI reports and eventstreams
    • when the data hits certain thresholds or matches other patterns [1]
      • it automatically takes appropriate action 
        • e.g. alerting users, kicking off Power Automate workflows [1]
    • [eventstream] takes actions on patterns or conditions that are detected in eventstream data [2]
  • allows building a digital nervous system that acts across all the data estate, at scale, and in a timely manner [1]
    • business users can describe business conditions in a no-code experience to launch actions and call into third party action systems [1]
      • actions
        • e.g. email, Teams notifications, Power Automate flows,
    • business users can self-serve their needs and reduce their reliance on internal IT and developer teams [1]
      • ⇒ no developers needed to manage and maintain custom in-house monitoring or alerting solutions [1]
  • {concept} activator 
    • the entity created by using Activator [2]
      •  holds all the information necessary to connect to data, monitor for conditions, and act [2] 
      • typically created for each business process or area monitored [2]
  • {concept} event 
    • an observation about the state of an object
      • some identifier for the object itself
      • a timestamp
      • the values for fields monitored
      • [Power BI] data observed from Power BI is treated as an eventstream [1]
        • events are observations made of the data on a regular schedule that typically matches the refresh frequency of the semantic model [1]
  • {concept} eventstream
    • an instance of the Eventstream item in Fabric
    • lets users bring real-time events into Fabric, transform them, and then route them to various destinations without writing any code (no-code) [2]
  • {concept} objects
    • the business objects that are monitored
      • can be physical objects or less tangible concepts 
      • the object is modeled by 
        • connecting one or more eventstreams
        • choosing a column for the object ID
        • specifying the fields intended to make properties of the object
    • created from streaming data
    • identified by unique columns in one or more stream
  • {concept} rules
    • define the conditions to detect on objects, and the actions to take when those conditions are met [1]
    • rules built on attributes are used to monitor how an attribute on an object ID changes over time [2]
      • some limits are defined in terms of the number of distinct object IDs being tracked by Activator within the lookback period. 
      • ⇐ Activator tracks active object IDs [2]
        • an active object ID is an object where events are arriving within the stored period [2]
    • {type} rules on events
      • allows to get an activation for every event that comes in on an eventstream
      • can track the state of something over time
    • {type} rules on events that are added to an object
    • one picks specific columns and the unique column to bundle into an object, and create rules that monitor events and report on either the arrival of that object, or the arrival of an object that meets a defined condition
    • the rule activates every time a new event comes in on the eventstream object
    • ⇐ one can identify which instance it came in for
    • {type} rules on an object's properties
    • allows to monitor a property on objects over time
  • {concept} properties
    • useful when reusing logic across multiple activators [1]
  • {concept} lookback period
    • the amount of historical data to be queried [2]
      •  ⇐ Activator needs to track historical data to ensure that correct actions can be computed [2]
      • depends on how a rule is defined and the data volume (events per second) of the data that is needed to evaluate the rule [2]

References:
[1] Microsoft Learn (2024) Fabric: What is Activator? [link
[2] Microsoft Learn (2024) Fabric: Activator basic concepts [link
[3] Microsoft Learn (2025) Fabric: Real Time Intelligence L200 Pitch Deck [link]

Resources:
[R1] Microsoft Learn (2024) Microsoft Fabric exercises [link]
[R2] Microsoft Fabric Updates Blog (2024) Enhance fraud detection with Activator in Microsoft Fabric [link]
[R3] Microsoft Fabric Updates Blog (2024) Announcing the General Availability of Activator [link]
[R4] Evolved Activator Destination in Eventstream – Process & Route Events to Activator (Preview) [link]
[R5] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]

Acronyms:
KQL  - Kusto Query Language
MF - Microsoft Fabric
RT - Real-Time

11 March 2025

🏭🎗️🗒️Microsoft Fabric: Real-Time Dashboards (RTD) [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 10-Mar-2025

Real-Time Intelligence architecture
Real-Time Intelligence architecture [5]

[Microsoft Fabric] Real-Time Dashboard

  • [def]
    • a collection of tiles
      • optionally organized in pages
        • act as containers of tiles
        • organize tiles into logical groups
          • e.g. by data source or by subject area
        • used to create a dashboard with multiple views
          • e.g. dashboard with a drillthrough from a summary page to a details page [1]
      • each tile has 
        • an underlying query
        • a visual representation
    • exists within the context of a workspace [1]
      • always associated with the workspace used to create it [1]
  • {concept} tile
    • uses  KQL snippets to retrieve data and render visuals [1]
    • can be added directly from queries written in a KQL queryset [1]
  • {concept} data source
    • reusable reference to a specific database in the same workspace as the dashboard [1]
  • {concept} parameters 
    • significantly improve dashboard rendering performance [1]
    • enable to use filter values as early as possible in the query
      • filtering is enabled when the parameter is included in the query associated with the tiles [1]
  • {concept} cloud connection
    • uses dashboard owner's identity to give access to the underlying data source to other users [2]
    • when not used for 90 days, it will expire [2]
      • ⇒ a new gateway connection must be set up [2]
      • via Manage connections >> Gateways page >> Edit credentials and verify the user again
    • a separate connection is needed for each data source [2]
  • {feature} natively export KQL queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed [1]
    • the fully integrated dashboard experience provides improved query and visualization performance [1]
  • {feature} encrypted at rest
    • dashboards and dashboard-related metadata about users are encrypted at rest using Microsoft-managed keys [1]
  • {feature} auto refresh
    • allows to automatically update the data on a dashboard without manually reloading the page or clicking a refresh button [1]
    • can be set by a database editor
      • both editors and viewers can change the actual rate of auto refresh while viewing a dashboard [1]
    • database editors can limit the minimum refresh rate that any viewer can set
      • ⇐ reduces the cluster load
      • when set, database users can't set a refresh rate lower than the minimum [1]
  • {feature} explore data 
    • enables users to extend the exploration of dashboards beyond the data displayed in the tiles [3]
      • begins with viewing the data and its corresponding visualization as they appear on the tile [3]
      • users can add or removing filters and aggregations, and use further visualizations [3]
        • ⇐ no knowledge of KQL is needed [3]
  • {feature} conditional formatting
    • allows users to format data points based on their values, utilizing 
      • colors
        • {rule} color by condition
          • allows to set one or more logical conditions that must be met for a value to be colored [4]
          • available for table, stat, and multi stat visuals [4]
        • {rule} color by value
          • allows to visualize values on a color gradient [4]
          • available for table visuals [4]
      • tags
      • icons
    • can be applied either 
      • to a specific set of cells within a designated column [4]
      • to entire rows [4]
    • one or more conditional formatting rules can be applied for each visual [4]
      • when multiple rules conflict, the last rule defined takes precedence over any previous ones [4]
  • {action} export dashboard
    • dashboards can be exported to a JSON file
    • can be useful in several scenarios 
      • {scenario} version control
        • the file can be used to restore the dashboard to a previous version [1]
      • {scenario} dashboard template
        • the file can be used as template for creating new dashboards [1]
      • {scenario} manual editing
      • edit the file to modify the dashboard and imported the file back to the dashboard [1]
    • ADX dashboards can be exported and imported as RT dashboards [6]
  • {action} share dashboard
    • one can specify if the user can view, edit, or share [2]
    • ⇐ the permissions are not for the underlying data [2]
      • permissions are set by defining the identity that the dashboard uses for accessing data from each data sources[2]
      • {type|default} pass-through identity
        • used when authenticating to access the underlying data source [2]
        • the user is only able to view the data in the tiles [2]
      • {type} dashboard editor’s identity:
        • allows the user to use editor’s identity, and thus permissions[2]
          • the editor defines a cloud connection that the dashboard uses to connect to the relevant data source [2]
          • only editors can define cloud connections and permissions for a specific real-time dashboard [2]
            • each editor that modifies the real-time dashboard needs to set up own cloud connection [2]
            • if a valid connection doesn't exist, the user is able to view the real-time dashboard but will only see data if they themselves have access to it [2]
  • {action} revoke a user’s access permissions
    • remove access from the dashboard [2]
    • remove the cloud connection.
      • via Settings >> Manage connections and gateways >> Options >> Remove
    • remove the user from the cloud connection.
      • via Settings >> Manage connections and gateways >> Options >> Manage users >> {select User} >> Delete
    • edit the Data source access permissions.
      • via Data source >> New data source >> edit >> Data source access >> Pass-through identity
      • ⇐ the user uses own identity to access the data source [2]
  • {prerequisite} a workspace with a Microsoft Fabric-enabled capacity [1]
  • {prerequisite} a KQL database with data [1]
  • {setting} Users can create real-time dashboards [1]
    References:
    [1] Microsoft Learn (2024) Fabric: Create a Real-Time Dashboard [link
    [2] Microsoft Learn (2024) Fabric: Real-Time Dashboard permissions (preview) [link
    [3] Microsoft Learn (2024) Fabric: Explore data in Real-Time Dashboard tiles [link
    [4] Microsoft Learn (2024) Fabric: Apply conditional formatting in Real-Time Dashboard visuals [link]
    [5] Microsoft Learn (2025) Fabric: Real Time Intelligence L200 Pitch Deck [link]
    [6] Microsoft Fabric Updates Blog (2024) Easily recreate your ADX dashboards as Real-Time Dashboards in Fabric, by Michal Bar [link]
    [7] Microsoft Learn (2025) Create Real-Time Dashboards with Microsoft Fabric [link]

    Resources:
    [R1] Microsoft Learn (2024) Microsoft Fabric exercises [link]
    [R2] Microsoft Fabric Updates Blog (2024) Announcing Real-Time Dashboards generally available [link]
    [R3] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]

    Acronyms:
    ADX - Azure Data Explorer
    KQL  - Kusto Query Language
    MF - Microsoft Fabric
    RT - Real-Time

    10 March 2025

    🧭Business Intelligence: Perspectives (Part 28: Cutting through Complexity)

    Business Intelligence Series
    Business Intelligence Series

    Independently of the complexity of the problems, one should start by framing the problem(s) correctly and this might take several steps and iterations until a foundation is achieved, upon which further steps can be based. Ideally, the framed problem should reflect reality and should provide a basis on which one can build something stable, durable and sustainable. Conversely, people want quick low-cost fixes and probably the easiest way to achieve this is by focusing on appearances, which are often confused with more.

    In many data-related contexts, there’s the tendency to start with the "solution" in mind, typically one or more reports or visualizations which should serve as basis for further exploration. Often, the information exchange between the parties involved (requestor(s), respectively developer(s)) is kept to a minimum, though the formalized requirements barely qualify for the minimum required. The whole process starts with a gap that can create further changes as the development process progresses, with all the consequences deriving from this: the report doesn’t satisfy the needs, more iterations are needed - requirements’ reevaluation, redesign, redevelopment, retesting, etc.

    The poor results are understandable, all parties start with a perspective based on facts which provide suboptimal views when compared with the minimum basis for making the right steps in the right direction. That’s not only valid for reports’ development but also for more complex endeavors – data models, data marts and warehouses, and other software products. Data professionals attempt to bridge the gaps by formalizing and validating the requirements, building mock-ups and prototypes, testing, though that’s more than many organizations can handle!

    There are simple reports or data visualizations for which not having prior knowledge of the needed data sources, processes and the business rules has a minimal impact on the further steps of the processes involved in building the final product(s). However, "all generalizations are false" to some degree, and there’s a critical point after which minimal practices tend to create more waste than companies can afford. Consequently, applying the full extent of the processes can lead to waste when the steps aren’t imperative for the final product.

    Even if one is aware of all the implications, one’s experience and the application of best practices doesn’t guarantee the quality of the results as long as some kind of novelty, unknown, fuzziness or complexity is involved. Novelty can appear in different ways – process, business rules, data or problem formulations, particularities that aren’t easily perceived or correctly understood. Each important minor piece of information can have an exponential impact under the wrong circumstances.

    The unknown can encompass novelty, though can be also associated with the multitude of facts not explicitly and/or directly considered. "The devil is in details" and it’s so easy for important or minor facts to remain hidden under the veil of suppositions, expectations, respectively under the complex and fuzzy texture of logical aspects. Many processes operate under strict rules, though there are various consequences, facts or unnecessary information that tend to increase the overall complexity and fuzziness.

    Predefined processes, procedures and practices can help cut and illuminate through this complex structure associated with the various requirements and aspects of problems. Plunging headfirst can be occasionally associated with the need to evaluate what is known and unknown from facts and data’s perspective, to identify the gaps and various factors that can weigh in the final solution. Unfortunately, too often it’s nothing of this!  

    Besides the multitude of good/best practices and problem-solving approaches, all one has is his experience and intuition to cut through the overall complexity. False steps are inevitable for finding the approachable path(s) from the requirements to the solution.

    09 March 2025

    🏭🎗️🗒️Microsoft Fabric: Real-Time Hub (RTH) [Notes]

    Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

    Last updated: 9-Mar-2025

    Real-Time Intelligence architecture
    Real-Time Intelligence architecture [2]

    [Microsoft Fabric] Real-Time Hub [RTH]

    • [def]  single, tenant-wide, unified, logical place for streaming data-in-motion [1]
      • enables to easily discover, ingest, manage, and consume data-in-motion from a wide variety of sources [1]
        • each user in the tenant can view and edit all the events or streams that they have access to
      • makes it easy to collaborate and develop streaming applications within one place [1]
        • allows to blend data in motion and data at rest for accelerated insight discovery [2]
        • provides out-of-box connectors that make it easy to ingest data into MF from a wide variety of sources [1]
        • allows to create streams for the supported sources
          • after the streams are created, users can 
            • process them by 
              • opening the parent eventstream in an editor [1]
              • adding transformations to transform or process the data that's streaming into MF [1]
                • e.g. aggregate, expand, filter, group by, manage fields, and union, 
              • sending the output data from transformations into supported destinations [1]
            • analyze them
            • set alerts on them [1]
      • {goal} provide a large catalog of data in motion 
        • ⇐ brought in with just a few clicks in Get Events [2]
        • {feature} subscribe to internal and external discrete events [2] 
        • {feature} quickly connect experiences for Microsoft streaming sources [2]
          • e.g. IoT Hub
        • {feature} access system events emitted by MF and Azure storage [2]
        • {feature} ingest data streams from all clouds 
          • e.g. AWS, Kinesis, Google Pub Sub, etc.
      • {goal} work across full event life cycle
        • allows to analyze, build lightweight models, and trigger actions over all data in motion, working across the life cycle of events to create derived and enriched streams [2]
        • {feature} create triggers through simple, embedded experiences [2]
        • {feature} pen event streams to process and route events without writing any code [2]
        • {feature} land in Eventhouse for further analysis [2]
        • {feature} analyze and build lightweight models [2]
        • {feature} create derived and enriched streams [2]
      • {goal} consume data from anywhere
        • allows to view all time-oriented data in motion, readily available for cross workload use in OneLake [2]
        • {feature} discover events and seamlessly consume them from across organization [2]
        • {feature} expose and use well-established open source APIs, standards, protocols and connectors [2]
        • {feature} maintain data ownership data is not trapped in Microsoft’s proprietary formats [2]
        • {feature} integrate seamlessly with other experiences in MF [2]
        • {goal} simplify integration of stream processing frameworks [2]
    References:
    [1] Microsoft Learn (2025) Microsoft Fabric: Introduction to Fabric Real-Time hub [link
    [2] Microsoft Learn (2025) Real Time Intelligence L200 Pitch Deck [link]

    Resources:
    [R1] Microsoft Learn (2024) Microsoft Fabric exercises [link]
    [R2] Microsoft Fabric Update Blog (2024) Announcing new event categories in Fabric Real-Time Hub [link]
    [R3] Microsoft Fabric Update Blog (2024) Accelerating Insights with Fabric Real-Time Hub (generally available) [link]
    [R4] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]

    Acronyms:
    API - Application Programming Interface
    AWS - Amazon Web Services
    IoT - Internet of Things
    KQL  - Kusto Query Language
    MF - Microsoft Fabric
    RT - Real-Time
    RTH - Real-Time Hub
    Related Posts Plugin for WordPress, Blogger...

    About Me

    My photo
    Koeln, NRW, Germany
    IT Professional with more than 25 years experience in IT in the area of full life-cycle of Web/Desktop/Database Applications Development, Software Engineering, Consultancy, Data Management, Data Quality, Data Migrations, Reporting, ERP implementations & support, Team/Project/IT Management, etc.