💠🛠️🗒️SQL Server: Columnstore Indexes [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources. It considers only on-premise SQL Server, for other platforms please refer to the documentation.

Last updated: 15-Feb-2024

[SQL Server] columnstore indexes (CI)

• {def} a technology for storing, retrieving and managing data by using a columnar data format (aka columnstore) 
• store compressed data on a per-column rather than a per-row basis [5]
• {benefit} designed for analytics and data warehousing workloads
• data warehousing
• {scenario} store fact tables and large dimension tables
• ⇐ tend to require full table scans rather than table seeks
• analytics workloads
• {scenario} [SQL Server 2016 SP1] can be used for real-time analytics on operational databases
• ⇐ an updatable nonclustered columnstore index can be created on a rowstore table
• {benefit} performance increase 
• can achieve up to 100x better performance [4]
• offers an order of magnitude better performance than a rowstore index
• {feature} uses batch mode execution
• improves query performance typically by two to four times
• have high performance gains for analytic queries that scan large amounts of data, especially on large tables (>1 million rows) 
• {benefit} reduces significantly the data warehouse storage costs
• {feature} data compression
• ⇒ provides high compression rates, typically by 10 times
• ⇒ reduces total I/O from the physical media
• ⇐ queries often select only a few columns from a table
• minimizes or eliminates system I/O bottlenecks
• reduces significantly the memory footprint
• ⇒ query performance can improve 
• because SQL Server can perform more query and data operations in memory
• {benefit} built in memory
• ⇒ sufficient memory must be available 
• {benefit} part of the database engine
• no special hardware is needed
• {concept} columnstore 
• {def} data structure logically organized as a table with rows and columns, and physically stored in a column-wise data format
• stores values from the same domain which commonly have similar values
• when a query references a column, then only that column is fetched from disk [3]
• ⇐ the columns not requested are skipped 
• ⇒ they are not loaded into memory 
• when a query is executed, the rows must be reconstructed
• ⇒ row reconstruction takes some time and uses some CPU and memory resources [3]
• [SQL Server 2016] columnstore index on rowstore tables
• columnstore is updated when data changes in the rowstore table
• both indexes work against the same data
• {concept}rowstore
• {def} data that's logically organized as a table with rows and columns, and physically stored in a row-wise data format
• ⇐ the traditional way to store relational table data
• refers to a table where the underlying data storage format is either
• a heap
• a clustered index
• a memory-optimized table
• {concept} rowstore index
• performs best on queries that seek into the data, when searching for a particular value, or for queries on a small range of values
• ⇒ appropriate for transactional workloads 
• because they tend to require mostly table seeks instead of table scans
• {concept} rowgroup
• {def} a group of rows that are compressed into columnstore format at the same time
• {constraint} has a maximum number of rows per rowgroup, which is 1,048,576 =2^20 rows
• contains one column segment for every column in the table
• can have more than one delta rowgroup that form the deltastore
• e.g. when multiple threads create columnstore indexes using parallel execution plans [5]
• ⇐ each thread will work with its own subset of data, creating separate rowgroups [5]
• [partitions] each table partition has its own set of row groups [5]
• ⇐  too many partitions may prevent workloads from benefiting from a CCI [11]
• ⇐ data aren’t pushed into a compressed columnstore segment until the rowgroup limit is reached
• {event} rowgroup is compressed
• marked as read-only [16]
• a compressed rowgroup is considered as fragmented when either 
• row number < rowgroup limit but dictionary size reached the maximum
• nothing can be done to increase the number of rows [15]
• the trim_reason is other than DICTIONARY_SIZE
• it has nonzero deleted rows that exceeds a minimum threshold [15]
• {event} all data from rowgroup deleted 
• transitions from COMPRESSED into TOMBSTONE state
• later removed by the tuple-mover background process
• {event} rows in the columnstore indexes can be moved to different locations
• row-id in the nonclustered indexes aren’t updated 
• ⇐ the mappings between old and new row locations are stored in an internal structure (aka mapping index) 
• {event} rowgroup build
• all column data are combined on a per-row group basis, encoded and compressed [5]
• the rows within a row group can be rearranged if that helps to achieve a better compression rate [5]
• {feature} data compression
• the table is sliced into rowgroups, and each rowgroup is compresses in a column-wise manner
• the number of rows in the rowgroup must be 
• large enough to improve compression rates
• small enough to benefit from in-memory operations
• having too many small rowgroups decreases columnstore index’s quality
• uses its own compression mechanism 
• ⇒ row or page compression cannot be used on it [3]
• [SQL Server 2016] page compression has been removed
• ⇐ in some cases, page compression disallowed the creation of columnstore indexes with a very large number of columns [5]
• {feature} compression delay
• computed when a delta rowgroup is closed [7]
• keeps the ‘active’ rows in delta rowgroup and only transition these rows to compressed rowgroup after a specified delay [7]
• ⇐ reduces the overall maintenance overhead of NCCI [7]
• ⇒ leads to a larger number of delta rowgroups [7]
• {best practice} if the workload is primarily inserting data and querying it, the default COMPRESSION_DELAY of 0 is the recommended option [7]
• {best practice} [OLTP workload] if > 10% rows are marked deleted in recently compressed rowgroups, then consider a value that accommodates the behavior [7]
• via: create nonclustered columnstore index with (compression_delay= 150)
• {feature} data encoding
• all values in the data are replaced with 64-bit integers using one of two encoding algorithms
• {concept} dictionary encoding
• stores distinct values from the data in a separate structure (aka dictionary} 
• every value in a dictionary has a unique ID assigned [5]
• the ID is used for replacement
• {concept} global dictionary
• shared across all segments that belong to the same index partition [5]
• {concept} local dictionary
• created for individual segments using values that are not present in the global dictionary
• {concept} value-based encoding
• mainly used for numeric and integer data types that do not have enough duplicated values [5]
• dictionary encoding would be inefficient [5]
• converts integer and numeric values to a smaller range of 64-bit integers in 2 steps
• {step} [numeric data types] are converted to integers using the minimum positive exponent (aka magnitude that allows this conversion) [5]
• {goal} convert all numeric values to integers [5]
• [integer data types] the smallest negative exponent is chosen that can be applied to all values without losing their precision [5]
• {goal} reduce the interval between the minimum and maximum values stored in the segment [5]
• {step} the minimum value (aka base value) in the segment is identified and subtracted it from all other values [5]
• ⇒ makes the minimum value in the segment number 0 [5]
• after encoding the data are compressed and stored as a LOB allocation unit
• {concept} column segment 
• {def} a column of data from within the rowgroup
• is compressed together and stored on physical media
• SQL Server loads an entire segment to memory when it needs to access its data
• {concept} segment metadata 
• store metadata about each segment 
• e.g. minimum and maximum values
• ⇐ segments that do not have the required data are skipped [5]
• {concept} deltastore
• {def} all of the delta rowgroups of a columnstore index
• its operations are handled behind the scenes
• can be in either states
• {state} open (aka open delta store) 
• accepts new rows and allow modifications and deletions of data
• {state} closed (aka closed data store)
• a delta store is closed when it reaches its rowgroup limit
• {concept} delta rowgroup 
• {def} a clustered B-tree index that's used only with columnstore indexes
• improves columnstore compression and performance by storing rows until the number of rows reaches the rowgroup limit and are then moved into the columnstore
• {event} reaches the maximum number of rows
• it transitions from an ‘open’ to ‘closed’ state
• a closed rowgroup is compressed by the tuple-mover and stored into the columnstore as COMPRESSED rowgroup
• {event} compressed
• the existing delta rowgroup transitions into TOMBSTONE state to be removed later by the tuple-mover when there is no reference to it
• {concept} tuple-mover 
• background process that checks for closed row group
• if it finds a closed rowgroup, it compresses the delta rowgroup and stores it into the columnstore as a COMPRESSED rowgroup
• {concept} clustered columnstore index (CCI) 
• is the primary storage for the entire table
• {characteristic) updatable
• has two structures that support data modifications
• ⇐ both use the B-Tree format to store data [5]
• ⇐ created on demand [5]
• delete bitmap
• indicates which rows were deleted from a table
• upon deletion the row continues to be stored into the rowgroup
• during query execution SQL Server checks the delete bitmap and excludes deleted rows from the processing [5]
• delta store
• includes newly inserted rows
• updating a row triggers the deletion of the existing row and insertion of a new version of a row to a delta store
• ⇒ the update does not change the row data
• ⇒ the updated row is inserted to a delete bitmap
• [partitions] each partition can have a single delete bitmap and multiple delta stores
• ⇐ this makes each partition self-contained and independent from other partitions
• ⇒ allows performing a partition switch on tables that have clustered columnstore indexes defined [5]
• {feature} supports minimal logging for batch sizes >= rowgroup’s limit [12]
• [SQL Server 2017] supports non-persisted computed columns in clustered columnstore indexes [2]
• store some data temporarily into a clustered index (aka deltastore) and a btree list of IDs for deleted rows
• ⇐ {benefit} reduces fragmentation of the column segments and improves performance
• combines query results from both the columnstore and the deltastore to return the correct query results
• [partitions] too many partitions can hurt the performance of a clustered columnstore index [11]
• {concept} nonclustered columnstore index (NCCI)
• {def} a secondary index that's created on a rowstore table
• is defined as one or more columns of the table and has an optional condition that filters the rows
• designed to be used for workloads involving a mix of transactional and analytics workload*
• functions the same as a clustered columnstore index
• ⇐ has same performance optimizations (incl. batchmode operators)
• {exception} doesn’t supports persisted computed columns
• can’t be created on a columnstore index that has a computed column [2]
• however behave differently between the various versions of SQL Server
• [SQL Server 2012|2014] {restriction} readonly
• contains a copy of part or all of the rows and columns in the underlying table
• include a row-id , which is either the address of
• a row in a heap table 
• a clustered index key value
• includes all columns from the clustered index even when not explicitly defined in the CREATE statement
• the not specified columns will not be available in the sys.index_columns view
• [SQL Server 2016] multiple nonclustered rowstore indexes can be created on a columnstore index and perform efficient table seeks on the underlying columnstore
• ⇒ once created, makes it possible to drop one or more btree nonclustered indexes
• enables real-time operational analytics where the OLTP workload uses the underlying clustered index while analytics run concurrently on the columnstore index
• {concept} batch mode execution (aka vector-based execution, vectorized execution) 
• {def} query processing method used to process multiple rows together in groups of rows, or batches, rather than one row at a time
• SQL Server can push a predicate to the columnstore index scan operator, preventing unnecessary rows from being loaded into the batch [5]
• queries can process up to 900 rows together
• enables efficient query execution (by a 3-4x factor) [4]
• ⇐ the size of the batches varies to fit into the CPU cache
• ⇒ reduces the number of times that the CPU needs to request external data from memory or other components [5]
• improves the performance of aggregations, which can be calculated on a per-batch rather than a per-row basis [5]
• tries to minimize the copy of data between operators by creating and maintaining a special bitmap that indicates if a row is still valid in the batch [5]
• ⇐ subsequent operators will ignore the non-valid rows
• every operator has a queue of work items (batches) to process [5]
• worker threads from a shared pool pick items from queues and process them while migrating from operator to operator [5]
• is closely integrated with, and optimized around, the columnstore storage format.
• columnstore indexes use batch mode execution
• ⇐ improves query performance typically by two to four times
• {concept} tuple mover
• single-threaded process that works in the background, preserving system resources
• runs every five minutes
• converts closed delta stores to row groups that store data in a column-based storage format [5]
• can be disabled via trace flag T-634 
• ⇐ the conversion of closed delta stores to row groups can be forced by reorganizing an index [5]
• runs in parallel using multiple threads
• decreases significantly conversion time at a cost of extra CPU load and memory usage [5]
• via: ALTER INDEX REORGANIZE command
• it doesn’t prevent other sessions from inserting new data into a table [5]
• deletions and data modifications would be blocked for the duration of the operation [5]
• {recommendation} consider forcing index reorganization manually to reduce execution, and therefore locking, time [5]
• considered fragmented if it has
• multiple delta rowgroups
• deleted rows
• require maintenance like that of regular B-Tree indexes [5]
• {issue] partially populated row groups
• {issue} overhead of delta store and delete bitmap scans during query execution
• rebuilding the columnstore index addresses the issues
• the strategy depends on the volatility of the data and the ETL processes implemented in the system [5]
• {recommendation} rebuild indexes when a table has a considerable volme of deleted rows and/or a large number of partially populated rowgroups [5]
• {recommendation} rebuild partition(s) that still have a large number of rows in open delta stores after the ETL process has completed, especially if the ETL process does not use a bulk insert API [5]
• creating/dropping/disabling/rebuilding functions like any other index
• columnstore statistics 
• a statistics object is created at the time of columnstore index creation; however, it is neither populated nor updated afterward [5]
• ⇐ SQL Server relies on segment information, B-Tree indexes (when available), and column-level statistics when deciding if a columnstore index needs to be used [5]
• it is beneficial to create missing column-level statistics on the columns that participate in a columnstore index and are used in query predicates and as join keys [5]
• ⇐ statistics rarely update automatically on very large tables [5]
• ⇒ statistics must be updated ‘manually’
• [SQL Server 2019] included into the schema-only clone of a database functionality [8]
• enable performance troubleshooting without the need to manual capture the statistics information
• columnstore indexes has been added to sp_estimate_data_compression_savings. In SQL Server 2019 both 
• COLUMNSTORE and COLUMNSTORE_ARCHIVE have been added to allow you to estimate the space savings if 
• either of these indexes are used on a table.
• via DBCC CLONEDATABASE
• [in-memory tables] 
• {limitation} a columnstore index must include all the columns and can’t have a filtered condition [2]
• {limitation} queries on columnstore indexes run only in InterOP mode, and not in the in-memory native mode [2]
• {operation} designing columnstore indexes
• {best practice} understand as much as possible data’s characteristics
• {best practice} identify workload’s characteristics
• {operation} create a clustered columnstore index
• via CREATE CLUSTERED COLUMNSTORE INDEX command
• not needed to specify any columns in the statement
• ⇐ the index will include all table columns
• {operation} index rebuilding 
• forces SQL Server to remove deleted rows physically from the index and to merge the delta stores’ and row groups’ data [5]
• all column segments are recreated with row groups fully populated [5]
• [<SQL Server 2019] offline operation
• [SQL Server 2019 Enterprise] online operation
• ⇒ higher availability 
• ⇐ pausing and resuming create and rebuild operations are not supported [11]
• very resource intensive process
• holds a schema modification (Sch-M) lock on the table
• ⇒ prevents other sessions from accessing it [5]
• ⇐ the overhead can be mitigated by using table/index partitioning
• ⇒ indexes will be rebuild on a partition basis for those partition with volatile data [5]
• {operation} index reorganization 
• [<SQL Server 2019] a reorganize operation is required to merge smaller COMPRESSED rowgroups, following an internal threshold policy that determines how to remove deleted rows and combine the compressed rowgroups
• [SQL Server 2019] a background merge task also works to merge COMPRESSED rowgroups from where a large number of rows has been deleted
• ⇐ after merging smaller rowgroups, the index quality should be improved.
• the tuple-mover is helped by a background merge task that automatically compresses smaller OPEN delta rowgroups that have existed for some time as determined by an internal threshold, or merges COMPRESSED rowgroups from where a large number of rows has been deleted
• via: ALTER INDEX REORGANIZE command
• [SQL Server 2016] performs additional defragmentation
• removes deleted rows from row groups that have 10 or more percent of the rows logically deleted [5]
• merges closed row groups together, keeping the total number of rows less than or equal than rowgroup’s limit [5]
• ⇐ both processes can be done together [5]
• [SQL Server 2014] the only action performed is compressing and moving the data from closed delta stores to rowgroups [5] 
• ⇐ delete bitmap and open delta stores stay intact [5]
• via: ALTER INDEX REORGANIZE
• uses all available system resources while it is running [5]
• ⇒ speeds up the execution process 
• reduce the time during which other sessions cannot modify or delete data in a table [5]
• close and compress all open row groups
• via: ALTER INDEX REORGANIZE WITH (COMPRESS_ALL_ROW_GROUPS = ON)
• row groups aren’t merged during this operation [5]
• {operation} estimate compression savings
• [SQL Server 2019] COLUMNSTORE and COLUMNSTORE_ARCHIVE added
• allows estimating the space savings if either of these indexes are used on a table [8]
• {limitation} not available in all editions 
• via: sp_estimate_data_compression_savings 
• {operation} [bulk loads] when the number of rows is less than deltastore’s limit, all the rows go directly to the deltastore
• [large bulk load] most of the rows go directly to the columnstore without passing through the deltastore
• some rows at the end of the bulk load might be too few in number to meet the minimum size of a rowgroup
• ⇒ the final rows go to the deltastore instead of the columnstore
• bulk insert operations provide the number of rows in the batch as part of the API call [5]
• best results are achieved by choosing a batch size that is divisible by rowgroup’s limit [5]
• ⇐ guarantees that every batch produces one or several fully populated row groups [5]
• ⇒ reduce the total number of row groups in a table [5]
• ⇒ improves query performance
• ⇐ the batch size shouldn’t exceed rowgroup’s limit [5]
• row groups can be still created on the fly in a manner to similar a bulk insert when the size of the insert batch is close to or exceeds [5]
• {operation} [non-bulk operations] trickle inserts go directly to a delta store
• {feature} parallel inserts
• [SQL Server 2016] requires following conditions for parallel insert on CCI [6]
• must specify TABLOCK
• no NCI on the clustered columnstore index
• no identity column
• database compatibility is set to 130
• {recommendation} minimize the use of string columns in facts tables [5]
• string data use more space
• their encoding involves additional overhead during batch mode execution [5]
• queries with predicates on string columns may have less efficient execution plans that also require significantly larger memory grants as compared to their non-string counterparts [5]
• {recommendation} [SQL Server 2012|2014] do not push string predicates down toward the lowest operators in execution plans.
• {recommendation} add another dimension table and replace the string value in the facts table with a synthetic, integer-based ID key that references a new table [5]
• {operation} upgrading to SQL Server 2016
• make sure that queries against the tables with columnstore indexes can utilize parallelism in case if database compatibility level less than 130 [5]
• {feature} [SQL Server 2019] automated columnstore index maintenance [8]
• {improvement} [SQL Server 2019] better columnstore metadata memory management
• {improvement} [SQL Server 2019] low-memory load path for columnstore tables
• {improvement} [SQL Server 2019] improved performance for bulk loading to columnstore indexes
• {improvement} [SQL Server 2019] server startup process has been made faster for databases that use in-memory columnstore tables for HTAP
• {feature} DMVs
• sys.column_store_segments 
• returns one row for each column per segment
• sys.column_store_dictionaries 
• provides information about the dictionaries used by a columnstore index
• sys.dm_db_column_store_row_group_physical_stats
• rowgroup statuses
• sys.column_store_row_groups
• provides clustered columnstore index information on a per-segment basis

Previous Post <<||>> Next Post

References:
[1] SQL Docs (2020) Columnstore indexes: Overview [link]

[2] Microsoft Learn (2024) SQL: What's new in columnstore indexes  [link]
[3] Dejan Sarka et al (2012) Exam 70-463: Implementing a Data Warehouse with Microsoft SQL Server 2012 (Training Kit)

[4] SQL Docs (2019) Columnstore indexes - Query performance [link]

[5] Dmitri Korotkevitch (2016) Pro SQL Server Internals 2nd Ed.

[6] Microsoft Learn (2016) Columnstore Index: Parallel load into clustered columnstore index from staging table [link]

[7] Microsoft Learn (2016) Columnstore Index Defragmentation using REORGANIZE Command [link]

[8] Microsoft (2018) Microsoft SQL Server 2019: Technical white paper [link]

Acronyms:
CCI - clustered columnstore index
CI - columnstore index
DBCC - Database Console Commands
DMV - Dynamic Management View
ETL - Extract, Transform, Load
HTAP - Hybrid Transactional/Analytical Processing 
LOB - Line of Business
NCCI - nonclustered columnstore index
OLTP - On-Line Transaction Processing
SP - Service Pack

🏭🗒️Microsoft Fabric: Parquet Format [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 31-Jan-2024

[Microsoft Fabric] Parquet format

• {definition} open source, column-oriented data file format designed for efficient data storage and retrieval [1]
• provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk [1]
• designed to be a common interchange format for both batch and interactive workloads [1]
• {characteristic} open source file format
• similar to other columnar-storage file formats available in Hadoop [1]
• e.g. RCFile, ORC
• became an industry standard 
•  {benefit} provides interoperability across multiple tools
• {characteristic} language agnostic [1]
• different programming languages can be used to manipulate the data
• {characteristic} column-based format [1]
• files are organized by column
• ⇐ rather than by row
• ⇒ saves storage space and speeds up analytics queries [1]
•  reads only the needed columns 
• ⇐ non-relevant data are skipped
• ⇒ greatly minimizes the IO [1]
• aggregation queries are less time-consuming compared to row-oriented databases [1]
• {benefit} increased data throughput and performance [1]
• ⇒ recommended for analytical workloads
• {characteristic} highly efficient data compression/decompression [1]
• supports flexible compression options and efficient encoding schemes [1]
• data can be compressed by using one of the several codecs available [1]
• ⇒ different data files can be compressed differently [1]
•  reduced storage requirements [1]
• by at least one-third on large datasets
• ⇒ {benefit} saves on cloud storage space
•  greatly improves scan and deserialization time [1]
• ⇒ {benefit} reduces the processing costs
• {downside} can be slower to write than row-based file formats
• primarily because they contain metadata about the file contents 
• though have fast read times
• {characteristic} supports complex data types and advanced nested data structures [1]
• implemented using the record-shredding and assembly algorithm
• accommodates complex data structures that can be used to store the data [1]
• optimized to work with complex data in bulk and features different ways for efficient data compression and encoding types [1]
• the approach is best especially for those queries that need to read certain columns from a large table [1]
• {characteristic} cloud-ready
• works best with interactive and serverless technologies [1]
• {characteristic} immutable
• a file can't be update to modify the column name, reorder or drop columns [2]
• ⇐ requires rewriting the whole file [2]
• {characteristic} binary-based file
• ⇒ not easily readable (by humans)
• {characteristic} self-describing 
•  contains metadata about schema and structure
• {concept} row groups (aka segments) 
• contains data from the same columns
• {constraint} column names are case sensitive
• {concept} file footer 
• stores metadata statistics for each row group [2]
• min/max statistics 
• the number of rows
• can be leveraged by data processing engines to run queries more efficiently [2]
• ⇐ depending on the query, entire row group can be skipped [2]
• {concept} file header
•  large datasets can be split across multiple parquet files
• ⇐ the structure can be flat or hierarchical 
• managing multiple files has several challenges
• the files can be used to define a table (aka parquet table)
• ⇐ {constraint} the files must have the same definition
• ⇐ schema enforcement must be coded manually [2]
• {limitation} [Data Lake] no support for ACID transactions [2]
• ⇒ easy to corrupt [2]
• partially written files will break any subsequent read operations
• the compute engine will try to read in the corrupt files and error out [2]
• corrupted files must be manually identified and deleted manually to fix the issues [2]
• {limitation} it's not easy to delete rows from it [2]
• requires reading all the data, filtering out the data not needed, and then rewriting the entire table [2]
• {limitation} doesn't support DML transactions [2]
• {limitation} there is no change data feed [2]
• {limitation} slow file listing [2]
• small files require excessive I/O overhead
• ideally the files should be between 64 MB and 1 GB
• ideally the files should be compacted into larger files (aka small file compaction, bin-packing)
• {limitation} expensive footer reads to gather statistics for file skipping [2]
• fetching all the footers and building the file-level metadata for the entire table is slow [2]
• ⇐ it requires a file-listing operation [2]
• the effectiveness of data skipping depends on how many files can be can skipped when performing a query [2]
• {limitation} doesn't support schema enforcement [2]
• {limitation} doesn't support check constraints [2]
• {limitation} doesn't support data versioning [2]
• {concept} table partitioning
• {definition} common optimization approach used to store the data of the same table in different directories, with partitioning column values encoded in the path of each partition directory [6]
• {recommendation} avoid partitioning by columns with very high cardinality
• {concept} bin-packing (aka compaction, bin-compaction)
• aims to produce evenly-balanced data files with respect to their size on disk, 
• ⇐ but not necessarily in respect to the number of tuples per file [7]
• requires an algorithm that efficiently organizes the files into equal size containers [6]
• {characteristic} idempotent
•  if it is run twice on the same dataset, the second run has no effect [7]
• {feature} [Microsoft Fabric] V-order
• {definition} write time optimization to the parquet file format that enables lightning-fast reads under the MF compute engines [3]
• applies special sorting, row group distribution, dictionary encoding and compression on parquet files [3]
• requires less compute engines resources in to read it [3]
• provides further cost efficiency and performance
• has a 15% impact on average write times but provides up to 50% more compression [3]
• {characteristic} open-source parquet format compliant
• all parquet engines can read it as a regular parquet file [3]
• ⇐ table properties and optimization commands can be used on control V-Order on its partitions [3]
• compatible with other features [3]
• applied at parquet file level [3]
• enabled by default
• {command} OPTIMIZE
• merges all changes into bigger, consolidated parquet files (aka bin-compaction) [3]
• [Spark] dynamically optimizes partitions while generating files with a default 128 MB size [5]
• the target file size may be changed per workload requirements using configurations [5]
• properly designing the table physical structure based on the ingestion frequency and expected read patterns is likely more important than running the optimization command [3]
• running the compaction operation brings the data lake in an unusable state for readers [7]
• {warning} manually compacting the files is inefficient and error prone [7]
• no way to differentiate files that contain new data from files that contain existing data that was just compacted into new files [7]
• [Delta Lake] when ZORDER and VORDER are used together, Apache Spark performs bin-compaction, ZORDER, VORDER sequentially [3]

Previous Post <<||>>  Next Post

Resources:
[1] Databricks (2023) What is Parquet? (link)
[2] Data Lake (2023) Delta Lake vs. Parquet Comparison (link)
[3] Data Mozart (2023) Parquet file format – everything you need to know! (link)
[4] Microsoft Learn (2023) Query Parquet files using serverless SQL pool in Azure Synapse Analytics (link)
[5] Microsoft Learn (2023) Lakehouse tutorial: Prepare and transform data in the lakehouse (link)
[6] Apache Spark (2023) Spark SQL Guide (link)
[7] Delta Lake (2023) Delta Lake Small File Compaction with OPTIMIZE (link)
[8] Delta Lake (2023) Optimizations (link)

Acronyms:
ACID - atomicity, consistency, isolation, durability
IO - Input/Output
MF - Microsoft Fabric

ORC - Optimized Row Columnar
RCFile - Record Columnar File

⛏️Data Management: Hash (Definitions)

"A number (often a 32-bit integer) that is derived from column values using a lossy compression algorithm. DBMSs occasionally use hashing to speed up access, but indexes are a more common mechanism." (Peter Gulutzan & Trudy Pelzer, "SQL Performance Tuning", 2002)

"A set of characters generated by running text data through certain algorithms. Often used to create digital signatures and compare changes in content." (Tom Petrocelli, "Data Protection and Information Lifecycle Management", 2005)

"Hash, a mathematical method for creating a numeric signature based on content; these days, often unique and based on public key encryption technology." (Bo Leuf, "The Semantic Web: Crafting infrastructure for agency", 2006)

[hash code:] "An integer calculated from an object. Identical objects have the same hash code. Generated by a hash method." (Michael Fitzgerald, "Learning Ruby", 2007)

"An unordered collection of data where keys and values are mapped. Compare with array." (Michael Fitzgerald, "Learning Ruby", 2007)

"A cryptographic hash is a fixed-size bit string that is generated by applying a hash function to a block of data. Secure cryptographic hash functions are collision-free, meaning there is a very small possibility of generating the same hash for two different blocks of data. A secure cryptographic hash function should also be one-way, meaning it is infeasible to retrieve the original text from the hash." (Michael Coles & Rodney Landrum, "Expert SQL Server 2008 Encryption", 2008)

"A hash is the result of applying a mathematical function or transformation on data to generate a smaller 'fingerprint' of the data. Generally, the most useful hash functions are one-way collision-free hashes that guarantee a high level of uniqueness in their results." (Michael Coles, "Pro T-SQL 2008 Programmer's Guide", 2008)

"The output of a hash function." (Mark S Merkow & Lakshmikanth Raghavan, "Secure and Resilient Software Development", 2010)

"A number based on the hash value of a string." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"1.Data allocated in an algorithmically randomized fashion in an attempt to evenly distribute data and smooth access patterns. 2.Verb. To calculate a hash key for data." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"A hash is the result of applying a mathematical function or transformation on data to generate a smaller 'fingerprint' of the data. Generally, the most useful hash functions are one-way collision-free hashes that guarantee a high level of uniqueness in their results." (Jay Natarajan et al, "Pro T-SQL 2012 Programmer's Guide" 3rd Ed., 2012)

"An unordered association of key/value pairs, stored such that you can easily use a string key to look up its associated data value. This glossary is like a hash, where the word to be defined is the key and the definition is the value. A hash is also sometimes septisyllabically called an “associative array”, which is a pretty good reason for simply calling it a 'hash' instead." (Jon Orwant et al, "Programming Perl" 4th Ed., 2012)

"In a hash cluster, a unique numeric ID that identifies a bucket. Oracle Database uses a hash function that accepts an infinite number of hash key values as input and sorts them into a finite number of buckets. Each hash value maps to the database block address for the block that stores the rows corresponding to the hash key value (department 10, 20, 30, and so on)." (Oracle, "Database SQL Tuning Guide Glossary", 2013)

"The result of applying a mathematical function or transformation to data to generate a smaller 'fingerprint' of the data. Generally, the most useful hash functions are one-way, collision-free hashes that guarantee a high level of uniqueness in their results." (Miguel Cebollero et al, "Pro T-SQL Programmer’s Guide" 4th Ed., 2015)

[hash code:] "The output of the hash function that is associated with the input object" (Nell Dale et al, "Object-Oriented Data Structures Using Java" 4th Ed., 2016)

"A numerical value produced by a mathematical function, which generates a fixed-length value typically much smaller than the input to the function. The function is many to one, but generally, for all practical purposes, each file or other data block input to a hash function yields a unique hash value." (William Stallings, "Effective Cybersecurity: A Guide to Using Best Practices and Standards", 2018)

"The number generated by a hash function to indicate the position of a given item in a hash table." (IEEE 610.5-1990)

⛏️Data Management: Data Compression (Definitions)

"any kind of data reduction method that preserves the application-specific information." (Teuvo Kohonen, "Self-Organizing Maps 3rd Ed.", 2001)

"The process of reducing the size of data by use of mathematical algorithms." (Tom Petrocelli, "Data Protection and Information Lifecycle Management", 2005)

"1.Algorithms or techniques that change data to a smaller physical size that contains the same information. 2.The process of changing data to be stored in a smaller physical or logical space." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"Encoding information in such a way that its representation consumes less space in memory" (Hasso Plattner, "A Course in In-Memory Data Management: The Inner Mechanics of In-Memory Databases 2nd Ed.", 2014)

"Compression is a data management technique that uses repeating patterns in data to reduce the storage needed to hold the data. A compression algorithm for databases should perform compression and decompression operations as fast as possible. This often entails a trade-off between the speed of compression/decompression and the size of the compressed data. Faster compression algorithms can lead to larger compressed data than other, slower algorithms." (Dan Sullivan, "NoSQL for Mere Mortals®", 2015)

"Reducing the amount of space needed to store a piece of data" (Nell Dale & John Lewis, "Computer Science Illuminated" 6th Ed., 2015)

"The process of reducing the size of a data file by encoding information using fewer bits than the original file." (Faithe Wempen, "Computing Fundamentals: Introduction to Computers", 2015)

"A method that reduces the amount of space needed for storing data. See also client compression and hardware compression." (CommVault, "Documentation 11.20", 2018)

"Any technique used to reduce the amount of storage required to store data." (IEEE 610.5-1990)