🏭🗒️Microsoft Fabric: Shortcuts [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 8-Dec-2024

[Microsoft Fabric] Shortcut

• {def} object that points to other internal or external storage location (aka shortcut) [1] and that can be used for data access
• {goal} unifies existing data without copying or moving it [2]
• ⇒ data can be used multiple times without being duplicated [2]
• {benefit} helps to eliminate edge copies of data [1]
• {benefit} reduces process latency associated with data copies and staging [1]
• is a mechanism that allows to unify data across domains, clouds, and accounts through a namespace [1]
• ⇒ allows creating a single virtual data lake for the entire enterprise [1]
• ⇐ available in all Fabric experiences [1]
• ⇐ behave like symbolic links [1]
• independent object from the target [1]
• appear as folder [1]
• can be used by workloads or services that have access to OneLake [1]
• transparent to any service accessing data through the OneLake API [1]
• can point to 
• OneLake locations
• ADLS Gen2 storage accounts
• Amazon S3 storage accounts
• Dataverse
• on-premises or network-restricted locations via PDF 
• {capability} create shortcut to consolidate data across artifacts or workspaces, without changing data's ownership [2]
• {capability} data can be compose throughout OneLake without any data movement [2]
• {capability} allow instant linking of data already existing in Azure and in other clouds, without any data duplication and movement [2]
• ⇐ makes OneLake the first multi-cloud data lake [2]
• {capability} provides support for industry standard APIs
• ⇒ OneLake data can be directly accessed via shortcuts by any application or service [2]
• {operation} creating a shortcut
• can be created in 
• lakehouses
• KQL databases
• ⇐ shortcuts are recognized as external tables [1]
• can be created via 
• Fabric UI 
• REST API
• can be created across items [1]
• the item types don't need to match [1]
• e.g. create a shortcut in a lakehouse that points to data in a data warehouse [1]
• [lakehouse] tables folder
• represents the managed portion of the lakehouse 
• shortcuts can be created only at the top level [1]
• ⇒ shortcuts aren't supported in other subdirectories [1]
• if shortcut's target contains data in the Delta\Parquet format, the lakehouse automatically synchronizes the metadata and recognizes the folder as a table [1]
• [lakehouse] files folder
• represents the unmanaged portion of the lakehouse [1]
• there are no restrictions on where shortcuts can be created [1]
• ⇒ can be created at any level of the folder hierarchy [1]
• ⇐table discovery doesn't happen in the Files folder [1]
• {operation} renaming a shortcut
• {operation} moving a shortcut
• {operation} deleting a shortcut 
• doesn't affect the target [1]
• ⇐ only the shortcut object is deleted [1]
  
• ⇐ the shortcut target remains unchanged [1]
• shortcuts don't perform cascading deletes [1]
• moving, renaming, or deleting a target path can break the shortcut [1]
• {operation} delete file/folder
• file or folder within a shortcut can be deleted when the permissions in the shortcut target allows it [1]
• {permissions} users must have permissions in the target location to read the data [1]
• when a user accesses data through a shortcut to another OneLake location, the identity of the calling user is used to authorize access to the data in the target path of the shortcut [1]
• when accessing shortcuts through Power BI semantic models or T-SQL, the calling user’s identity is not passed through to the shortcut target [1]
•  the calling item owner’s identity is passed instead, delegating access to the calling user [1]
• OneLake manages all permissions and credentials
• {feature} shortcut caching 
• {def} mechanism used to reduce egress costs associated with cross-cloud data access [1]
• when files are read through an external shortcut, the files are stored in a cache for the Fabric workspace [1]
• subsequent read requests are served from cache rather than the remote storage provider [1]
• cached files have a retention period of 24 hours
• each time the file is accessed the retention period is reset [1]
• if the file in remote storage provider is more recent than the file in the cache, the request is served from remote storage provider and the updated file will be stored in cache [1]
• if a file hasn’t been accessed for more than 24hrs it is purged from the cache [1]
• {restriction} individual files greater than 1GB in size are not cached [1]
• {restriction} only GCS, S3 and S3 compatible shortcuts are supported [1]
• {limitation} maximum number of shortcuts [1] 
• per Fabric item: 100,000
• in a single OneLake path: 10
• direct shortcuts to shortcut links: 5
• {limitation} ADLS and S3 shortcut target paths can't contain any reserved characters from RFC 3986 section 2.2 [1]
• {limitation} shortcut names, parent paths, and target paths can't contain "%" or "+" characters [1]
• {limitation} shortcuts don't support non-Latin characters[1]
• {limitation} Copy Blob API not supported for ADLS or S3 shortcuts[1]
• {limitation} copy function doesn't work on shortcuts that directly point to ADLS containers
• {recommended} create ADLS shortcuts to a directory that is at least one level below a container [1]
• {limitation} additional shortcuts can't be created inside ADLS or S3 shortcuts [1]
• {limitation} lineage for shortcuts to Data Warehouses and Semantic Models is not currently available[1]
• {limitation} it may take up to a minute for the Table API to recognize new shortcuts [1]

Previous Post <<||>>  Next Post

References:
[1] Microsoft Fabric (2024) OneLake shortcuts [link]
[2] Microsoft Fabric (2024) Fabric Analyst in a Day [course notes]

Acronyms:

ADLS - Azure Data Lake Storage

AWS - Amazon Web Services

GCS - Google Cloud Storage

KQL - Kusto Query Language

OPDG - on-premises data gateway