🏭🗒️Microsoft Fabric: OneLake [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 12-Mar-2024

Microsoft Fabric & OneLake

[Microsoft Fabric] OneLake

• a single, unified, logical data lake for the whole organization [2]
• designed to be the single place for all an organization's analytics data [2]
• provides a single, integrated environment for data professionals and the business to collaborate on data projects [1]
• stores all data in a single open format [1]
• its data is governed by default
• combines storage locations across different regions and clouds into a single logical lake, without moving or duplicating data
• similar to how Office applications are prewired to use OneDrive
• saves time by eliminating the need to move and copy data 
• comes automatically with every Microsoft Fabric tenant [2]
• automatically provisions with no extra resources to set up or manage [2]
• used as native store without needing any extra configuration [1
• accessible by all analytics engines in the platform [1]
• all the compute workloads in Fabric are preconfigured to work with OneLake
• compute engines have their own security models (aka compute-specific security) 
• always enforced when accessing data using that engine [3]
• the conditions may not apply to users in certain Fabric roles when they access OneLake directly [3]
• built on top of ADLS  [1]
• supports the same ADLS Gen2 APIs and SDKs to be compatible with existing ADLS Gen2 applications [2]
• inherits its hierarchical structure
• provides a single-pane-of-glass file-system namespace that spans across users, regions and even clouds
• data can be stored in any format
• incl. Delta, Parquet, CSV, JSON
• data can be addressed in OneLake as if it's one big ADLS storage account for the entire organization [2]
• uses a layered security model built around the organizational structure of experiences within MF [3]
• derived from Microsoft Entra authentication [3]
• compatible with user identities, service principals, and managed identities [3]
• using Microsoft Entra ID and Fabric components, one can build out robust security mechanisms across OneLake, ensuring that you keep your data safe while also reducing copies and minimizing complexity [3]
• hierarchical in nature 
• {benefit} simplifies management across the organization
• its data is divided into manageable containers for easy handling
• can have one or more capacities associated with it
• different items consume different capacity at a certain time
• offered through Fabric SKU and Trials
• {component} OneCopy
• allows to read data from a single copy, without moving or duplicating data [1]
• {concept} Fabric tenant
• a dedicated space for organizations to create, store, and manage Fabric items.
• there's often a single instance of Fabric for an organization, and it's aligned with Microsoft Entra ID [1]
• ⇒ one OneLake per tenant
• maps to the root of OneLake and is at the top level of the hierarchy [1]
• can contain any number of workspaces [2]
• {concept} capacity
• a dedicated set of resources that is available at a given time to be used [1]
• defines the ability of a resource to perform an activity or to produce output [1]
• {concept} domain
• a way of logically grouping together workspaces in an organization that is relevant to a particular area or field [1]
• can have multiple [subdomains]
• {concept} subdomain
• a way for fine tuning the logical grouping of the data
• {concept} workspace 
• a collection of Fabric items that brings together different functionality in a single tenant [1]
• different data items appear as folders within those containers [2]
• always lives directly under the OneLake namespace [4]
• {concept} data item
• a subtype of item that allows data to be stored within it using OneLake [4]
• all Fabric data items store their data automatically in OneLake in Delta Parquet format [2]
• {concept} Fabric item
• a set of capabilities bundled together into a single component [4] 
• can have permissions configured separately from the workspace roles [3]
• permissions can be set by sharing an item or by managing the permissions of an item [3]
• acts as a container that leverages capacity for the work that is executed [1]
• provides controls for who can access the items in it [1]
• security can be managed through Fabric workspace roles
• enable different parts of the organization to distribute ownership and access policies [2]
• part of a capacity that is tied to a specific region and is billed separately [2]
• the primary security boundary for data within OneLake [3]
• represents a single domain or project area where teams can collaborate on data [3]
• [encryption] encrypted at rest by default using Microsoft-managed key [3]
• the keys are rotated appropriately per compliance requirements [3]
• data is encrypted and decrypted transparently using 256-bit AES encryption, one of the strongest block ciphers available, and it is FIPS 140-2 compliant [3]
• {limitation} encryption at rest using customer-managed key is currently not supported [3]
• {general guidance} write access
• users must be part of a workspace role that grants write access [4] 
• rule applies to all data items, so scope workspaces to a single team of data engineers [4] 
• {general guidance}Lake access: 
• users must be part of the Admin, Member, or Contributor workspace roles, or share the item with ReadAll access [4] 
• {general guidance} general data access 
• any user with Viewer permissions can access data through the warehouses, semantic models, or the SQL analytics endpoint for the Lakehouse [4] 
• {general guidance} object level security:
• give users access to a warehouse or lakehouse SQL analytics endpoint through the Viewer role and use SQL DENY statements to restrict access to certain tables [4]
• {feature|preview} trusted workspace access
• allows to securely access firewall-enabled Storage accounts by creating OneLake shortcuts to Storage accounts, and then use the shortcuts in the Fabric items [5]
• based on [workspace identity]
• {benefit} provides secure seamless access to firewall-enabled Storage accounts from OneLake shortcuts in Fabric workspaces, without the need to open the Storage account to public access [5]
• {limitation} available for workspaces in Fabric capacities F64 or higher
• {concept} workspace identity
• a unique identity that can be associated with workspaces that are in Fabric capacities
• enables OneLake shortcuts in Fabric to access Storage accounts that have [resource instance rules] configured
• {operation} creating a workspace identity
• Fabric creates a service principal in Microsoft Entra ID to represent the identity [5]
• {concept} resource instance rules
• a way to grant access to specific resources based on the workspace identity or managed identity [5] 
• {operation} create resource instance rules 
• created by deploying an ARM template with the resource instance rule details [5]

https://sql-troubles.blogspot.com/2024/03/microsoft-fabric-medallion-architecture.html

Previous Post <<||>>  Next Post

Acronyms:

ADLS - Azure Data Lake Storage
AES - Advanced Encryption Standard 

ARM - Azure Resource Manager

FIPS - Federal Information Processing Standard
SKU - Stock Keeping Units

References:
[1] Microsoft Learn (2023) Administer Microsoft Fabric (link)
[2] Microsoft Learn (2023) OneLake, the OneDrive for data (link)
[3] Microsoft Learn (2023) OneLake security (link)
[4] Microsoft Learn (2023) Get started securing your data in OneLake (link}
[5] Microsoft Fabric Updates Blog (2024) Introducing Trusted Workspace Access for OneLake Shortcuts, by Meenal Srivastva (link)

Resources:
[1] 

🛡️Information Security: Asymmetric Encryption (Definitions)

"Requires a pair of keys to encode and decode information. One key is used to encrypt the plain text, and a second key, which is part of the matched pair, is used to decrypt that cipher text." (Marilyn Miller-White et al, "MCITP Administrator: Microsoft® SQL Server™ 2005 Optimization and Maintenance 70-444", 2007)

"Asymmetric encryption is an encryption model in which the encryption and decryption processes use different keys. Modern asymmetric encryption algorithms are based on the public key/private key pairs, in which the encryption and decryption keys are different but nontrivially related. The public key is widely known and distributed for encryption while the private key is kept secret and used for decryption. Although the keys are related, it is considered infeasible to try to derive the private key from the public key." (Michael Coles & Rodney Landrum, , "Expert SQL Server 2008 Encryption", 2008)

"Asymmetric encryption is encryption that requires two different keys: one to encrypt data and another to decrypt it. The most common form of asymmetric encryption is public key encryption, in which the two keys are mathematically related." (Michael Coles, "Pro T-SQL 2008 Programmer's Guide", 2008)

"Asymmetric encryption, also known as public-key encryption, is a form of data encryption where the encryption key (also called the public key) and the corresponding decryption key (also called the private key) are different. A message encrypted with the public key can be decrypted only with the corresponding private key. The public key and the private key are related mathematically, but it is computationally infeasible to derive the private key from the public key. Therefore, a recipient could distribute the public key widely." (Ninghui Li, "Asymmetric Encryption", 2009)

"Encryption that requires two different keys: one to encrypt data and another to decrypt it. The most common form of asymmetric encryption is public key encryption, in which the two keys are mathematically related." (Miguel Cebollero et al, "Pro T-SQL Programmer’s Guide" 4th Ed., 2015)

"Public-key cryptography refers to a cryptographic system requiring two separate keys, one of which is secret and one of which is public. Although different, the two parts of the key pair are mathematically linked. One key locks or encrypts the plaintext, and the other unlocks or decrypts the cipher text. Neither key can perform both functions by itself. The public key may be published without compromising security, while the private key must not be revealed to anyone not authorized to read the messages." (Addepalli V N Krishna & M Balamurugan, "Security Mechanisms in Cloud Computing-Based Big Data", 2019)

💻IT: Blockchain (Definitions)

"A block chain is a perfect place to store value, identities, agreements, property rights, credentials, etc. Once you put something like a Bit coin into it, it will stay there forever. It is decentralized, disinter mediated, cheap, and censorship-resistant." (Kirti R Bhatele et al, "The Role of Artificial Intelligence in Cyber Security", 2019)

"A system made-up of blocks that are used to record transactions in a peer-to-peer cryptocurrency network such as bitcoins." (Murad Al Shibli, "Hybrid Artificially Intelligent Multi-Layer Blockchain and Bitcoin Cryptology", 2020)

"A chain of blocks containing data that is bundled together. This database is shared across a network of computers (so-called distributed ledger network). Each data block links to the previous block in the blockchain through a cryptographic hash of the previous block, a timestamp, and transaction data. The blockchain only allows data to be written, and once that data has been accepted by the network, it cannot be changed." (Jurij Urbančič et al, "Expansion of Technology Utilization Through Tourism 4.0 in Slovenia", 2020)

"A system in which a record of transactions made in Bitcoin or another cryptocurrency is maintained across several computers that are linked in a peer-to-peer network. Amany M Alshawi, "Decentralized Cryptocurrency Security and Financial Implications: The Bitcoin Paradigm", 2020)

"An encrypted ledger that protects transaction data from modification." (David T A Wesley, "Regulating the Internet, Encyclopedia of Criminal Activities and the Deep Web", 2020)

"Blockchain is a decentralized, immutable, secure data repository or digital ledger where the data is chronologically recorded. The initial block named as Genesis. It is a chain of immutable data blocks what has anonymous individuals as nodes who can transact securely using cryptology. Blockchain technology is subset of distributed ledger technology." (Umit Cali & Claudio Lima, "Energy Informatics Using the Distributed Ledger Technology and Advanced Data Analytics", 2020)

"Blockchain is a meta-technology interconnected with other technologies and consists of several architectural layers: a database, a software application, a number of computers connected to each other, peoples’ access to the system and a software ecosystem that enables development. The blockchain runs on the existing stack of Internet protocols, adding an entire new tier to the Internet to ensure economic transactions, both instant digital currency payments and complicated financial contracts." (Aslı Taşbaşı et al, "An Analysis of Risk Transfer and Trust Nexus in International Trade With Reference to Turkish Data", 2020) 

"Is a growing list of records, called blocks, which are linked using cryptography. Each block contains a cryptographic hash of the previous block a timestamp, and transaction data. (Vardan Mkrttchian, "Perspective Tools to Improve Machine Learning Applications for Cyber Security", 2020)

"This is viewed as a mechanism to provide further protection and enhance the security of data by using its properties of immutability, auditability and encryption whilst providing transparency amongst parties who may not know each other, so operating in a trustless environment." (Hamid Jahankhani & Ionuț O Popescu, "Millennials vs. Cyborgs and Blockchain Role in Trust and Privacy", 2020)

"A blockchain is a data structure that represents the record of each accounting move. Each account transaction is signed digitally to protect its authenticity, and no one can intervene in this transaction." (Ebru E Saygili & Tuncay Ercan, "An Overview of International Fintech Instruments Using Innovation Diffusion Theory Adoption Strategies", 2021)

"A system in which a record of transactions made in bitcoin or another cryptocurrency are maintained across several computers that are linked in a peer-to-peer network." (Silvije Orsag et al, "Finance in the World of Artificial Intelligence and Digitalization", 2021)

"It is a decentralized computation and information sharing platform that enables multiple authoritative domains, who don’t trust each other, to cooperate, coordinate and collaborate in a rational decision-making process." (Vinod Kumar & Gotam Singh Lalotra, "Blockchain-Enabled Secure Internet of Things", 2021)

"A concept consisting of the methods, technologies, and tool sets to support a distributed, tamper-evident, and reliable way to ensure transaction integrity, irrefutability, and non-repudiation. Blockchains are write-once, append-only data stores that include validation, consensus, storage, replication, and security for transactions or other records." (Forrester)

[hybrid blockchain:] "A network with a combination of characteristics of public and private blockchains where a blockchain may incorporate select privacy, security and auditability elements required by the implementation." (AICPA)

[private blockchain:] "A restricted access network controlled by an entity or group which is similar to a traditional centralized network." (AICPA)

"A technology that records a list of records, referred to as blocks, that are linked using cryptography. Each block contains a cryptographic hash of the previous block, a timestamp and transaction data." (AICPA)

[public blockchain:] "An open network where participants can view, read and write data, and no one participant has control (e.g., Bitcoin, Ethereum)." (AICPA)

⛏️Data Management: Encryption (Definitions)

"A method for keeping sensitive information confidential by changing data into an unreadable form." (Microsoft Corporation, "SQL Server 7.0 System Administration Training Kit", 1999)

"The encoding of data so that the plain text is transformed into something unintelligible, called cipher text." (Tom Petrocelli, "Data Protection and Information Lifecycle Management", 2005)

"Reordering of bits of data to make it unintelligible (and therefore useless) to an unauthorized third party, while still enabling the authorized user to use the data after the reverse process of decryption." (David G Hill, "Data Protection: Governance, Risk Management, and Compliance", 2009)

"To transform information from readable plain text to unreadable cipher text to prevent unintended recipients from reading the data." (Janice M Roehl-Anderson, "IT Best Practices for Financial Managers", 2010)

"The process of transforming data using an algorithm (called a cipher) to make it unreadable to anyone except those possessing special knowledge, usually referred to as a key." (Craig S Mullins, "Database Administration", 2012)

"The process of converting readable data (plaintext) into a coded form (ciphertext) to prevent it from being read by an unauthorized party." (Microsoft, "SQL Server 2012 Glossary", 2012)

"The cryptographic transformation of data to produce ciphertext." (Manish Agrawal, "Information Security and IT Risk Management", 2014)

"The process of scrambling data in such a way that it is unreadable by unauthorized users but can be unscrambled by authorized users to be readable again." (Weiss, "Auditing IT Infrastructures for Compliance, 2nd Ed", 2015)

"The transformation of plaintext into unreadable ciphertext." (Shon Harris & Fernando Maymi, "CISSP All-in-One Exam Guide, 8th Ed", 2018)

"In computer security, the process of transforming data into an unintelligible form in such a way that the original data either cannot be obtained or can be obtained only by using a decryption process." (Sybase, "Open Server Server-Library/C Reference Manual", 2019)

"Encryption is about translating the data into complex codes that cannot be interpreted (decrypted) without the use of a decryption key. These keys are typically distributed and stored separately. There are two types of encryption: symmetric key encryption and public key encryption. In symmetric key encryption, the key to both encrypt and decrypt is exactly the same. Public key encryption has two different keys. One key is used to encrypt the values (the public key), and one key is used to decrypt the data (the private key)." (Piethein Strengholt, "Data Management at Scale", 2020)

"The process of encoding data in such a way to prevent unauthorized access." (AICPA)