Showing posts with label operations. Show all posts
Showing posts with label operations. Show all posts

26 March 2025

💠🏭🗒️Microsoft Fabric: Polaris SQL Pool [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources and may deviate from them. Please consult the sources for the exact content!

Unfortunately, besides the references papers, there's almost no material that could be used to enhance the understanding of the concepts presented. 

Last updated: 26-Mar-2025

Read and Write Operations in Polaris [2]

[Microsoft Fabric] Polaris SQL Pool

  • {def} distributed SQL query engine that powers Microsoft Fabric's data warehousing capabilities
    • designed to unify data warehousing and big data workloads while separating compute and state for seamless cloud-native operations
    • based on a robust DCP 
      • designed to execute read-only queries in a scalable, dynamic and fault-tolerant way [1]
      • a highly-available micro-service architecture with well-defined responsibilities [2]
        • data and query processing is packaged into units (aka tasks) 
          • can be readily moved across compute nodes and re-started at the task level
        • widely-partitioned data with a flexible distribution model [2]
        • a task-level "workflow-DAG" that is novel in spanning multiple queries [2]
        • a framework for fine-grained monitoring and flexible scheduling of tasks [2]
  • {component} SQL Server Front End (SQL-FE)
    • responsible for 
      • compilation
      • authorization
      • authentication
      • metadata
        • used by the compiler to 
          • {operation} generate the search space (aka MEMO) for incoming queries
          • {operation} bind metadata to data cells
          • leveraged to ensure the durability of the transaction manifests at commit [2]
            • only transactions that successfully commit need to be actively tracked to ensure consistency [2]
            • any manifests and data associated with aborted transactions are systematically garbage-collected from OneLake through specialized system tasks [2]
  • {component} SQL Server Backend (SQL-BE)
    • used to perform write operations on the LST [2]
      • inserting data into a LST creates a set of Parquet files that are then recorded in the transaction manifest [2]
      • a transaction is represented by a single manifest file that is modified concurrently by (one or more) SQL BEs [2]
        • SQL BE leverages the Block Blob API provided by ADLS to coordinate the concurrent writes  [2]
        • each SQL BE instance serializes the information about the actions it performed, either adding a Parquet file or removing it [2]
          • the serialized information is then uploaded as a block to the manifest file
          • uploading the block does not yet make any visible changes to the file [2]
            • each block is identified by a unique ID generated on the writing SQL BE [2]
        • after completion, each SQL BE returns the ID of the block(s) it wrote to the Polaris DCP [2]
          • the block IDs are then aggregated by the Polaris DCP and returned to the SQL FE as the result of the query [2]
      • the SQL FE further aggregates the block IDs and issues a Commit Block operation against storage with the aggregated block IDs [2]
        • at this point, the changes to the file on storage will become effective [2]
      • changes to the manifest file are not visible until the Commit operation on the SQL FE
        • the Polaris DCP can freely restart any part of the operation in case there is a failure in the node topology [2]
      • the IDs of any blocks written by previous attempts are not included in the final list of block IDs and are discarded by storage [2]
    • [read operations] SQL BE is responsible for reconstructing the table snapshot based on the set of manifest files managed in the SQL FE
      • the result is the set of Parquet data files and deletion vectors that represent the snapshot of the table [2]
        • queries over these are processed by the SQL Server query execution engine [2]
        • the reconstructed state is cached in memory and organized in such a way that the table state can be efficiently reconstructed as of any point in time [2]
          • enables the cache to be used by different operations operating on different snapshots of the table [2]
          • enables the cache to be incrementally updated as new transactions commit [2]
  • {feature} supports explicit user transactions
    • can execute multiple statements within the same transaction in a consistent way
      • the manifest file associated with the current transaction captures all the (reconciled) changes performed by the transaction [2]
        • changes performed by prior statements in the current transaction need to be visible to any subsequent statement inside the transaction (but not outside of the transaction) [2]
    • [multi-statement transactions] in addition to the committed set of manifest files, the SQL BE reads the manifest file of the current transaction and then overlays these changes on the committed manifests [1]
    • {write operations} the behavior of the SQL BE depends on the type of the operation.
      • insert operations 
        • only add new data and have no dependency on previous changes [2]
        • the SQL BE can serialize the metadata blocks holding information about the newly created data files just like before [2]
        • the SQL FE, instead of committing only the IDs of the blocks written by the current operation, will instead append them to the list of previously committed blocks
          • ⇐ effectively appends the data to the manifest file [2]
    • {update|delete operations} 
      • handled differently 
        • ⇐ since they can potentially further modify data already modified by a prior statement in the same transaction [2]
          • e.g. an update operation can be followed by another update operation touching the same rows
        • the final transaction manifest should not contain any information about the parts from the first update that were made obsolete by the second update [2]
      • SQL BE leverages the partition assignment from the Polaris DCP to perform a distributed rewrite of the transaction manifest to reconcile the actions of the current operation with the actions recorded by the previous operation [2]
        • the resulting block IDs are sent again to the SQL FE where the manifest file is committed using the (rewritten) block IDs [2]
  • {concept} Distributed Query Processor (DQP)
    • responsible for 
      • distributed query optimization
      • distributed query execution
      • query execution topology management
  • {concept} Workload Management (WLM)
    •  consists of a set of compute servers that are, simply, an abstraction of a host provided by the compute fabric, each with a dedicated set of resources (disk, CPU and memory) [2]
      • each compute server runs two micro-services
        • {service} Execution Service (ES) 
          • responsible for tracking the life span of tasks assigned to a compute container by the DQP [2]
        • {service} SQL Server instance
          • used as the back-bone for execution of the template query for a given task  [2]
            • ⇐ holds a cache on top of local SSDs 
              • in addition to in-memory caching of hot data
            • data can be transferred from one compute server to another
              • via dedicated data channels
              • the data channel is also used by the compute servers to send results to the SQL FE that returns the results to the user [2]
              • the life cycle of a query is tracked via control flow channels from the SQL FE to the DQP, and the DQP to the ES [2]
  • {concept} cell data abstraction
    • the key building block that enables to abstract data stores
      • abstracts DQP from the underlying store [1]
      • any dataset can be mapped to a collection of cells [1]
      • allows distributing query processing over data in diverse formats [1]
      • tailored for vectorized processing when the data is stored in columnar formats [1] 
      • further improves relational query performance
    • 2-dimenstional
      • distributions (data alignment)
      • partitions (data pruning)
    • each cell is self-contained with its own statistics [1]
      • used for both global and local QO [1]
      • cells can be grouped physically in storage [1]
      • queries can selectively reference either cell dimension or even individual cells depending on predicates and type of operations present in the query [1]
    • {concept} distributed query processing (DQP) framework
      • operates at the cell level 
      • agnostic to the details of the data within a cell
        • data extraction from a cell is the responsibility of the (single node) query execution engine, which is primarily SQL Server, and is extensible for new data types [1], [2]
  • {concept} dataset
    • logically abstracted as a collection of cells [1] 
    • can be arbitrarily assigned to compute nodes to achieve parallelism [1]
    • uniformly distributed across a large number of cells 
      • [scale-out processing] each dataset must be distributed across thousands of buckets or subsets of data objects,
      •  such that they can be processed in parallel across nodes
  • {concept} session
    • supports a spectrum of consumption models, ranging from serverless ad-hoc queries to long-standing pools or clusters [1]
    • all data are accessible from any session [1]
      • multiple sessions can access all underlying data concurrently  [1]
  • {concept} Physical Metadata layer
    • new layer introduced in the SQL Server storage engine [2]
See also: Polaris

References:
[1] Josep Aguilar-Saborit et al (2020) POLARIS: The Distributed SQL Engine in Azure Synapse, Proceedings of the VLDB Endowment PVLDB 13(12) [link]
[2] Josep Aguilar-Saborit et al (2024), Extending Polaris to Support Transactions [link]
[3] Gjnana P Duvvuri (2024) Microsoft Fabric Warehouse Deep Dive into Polaris Analytic Engine [link]

Resources:
[R1] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]
[R2] Patrick Pichler (2023) Data Warehouse (Polaris) vs. Data Lakehouse (Spark) in Microsoft Fabric [link]
[R3] Tiago Balabuch (2023) Microsoft Fabric Data Warehouse - The Polaris engine [link]

Acronyms:
CPU - Central Processing Unit
DAG - Directed Acyclic Graph
DB - Database
DCP - Distributed Computation Platform 
DQP - Distributed Query Processing 
DWH - Data Warehouses 
ES - Execution Service
LST - Log-Structured Table
SQL BE - SQL Backend
SQL FE - SQL Frontend
SSD - Solid State Disk
WAL - Write-Ahead Log
WLM - Workload Management

🏭🗒️Microsoft Fabric: External Data Sharing [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 26-Mar-2025

External data sharing
External data sharing [1]

[Microsoft Fabric] External data sharing 

  • {def} feature that enables Fabric users to share data from their tenant with users in another Fabric tenant (aka cross-tenant sharing) [1]
    • the data is shared in-place from OneLake storage locations in the sharer's tenant [1]
      • ⇒ no data is actually copied to the other tenant [1]
      • creates a OneLake shortcut in the other tenant that points back to the original data in the sharer's tenant [1]
      • data is exposed as read-only [1]
      • data can be consumed by any OneLake compatible Fabric workload in that tenant [1]
    • {benefit} allows for efficient and secure data sharing without duplicating data
      • the shared data remains read-only for the consumer, ensuring data integrity and consistency [2]
      • multiple tables and folders can be shared at once [2]
  • {prerequisite} Fabric admins must turn on external data sharing both in the sharer's tenant and in the external tenant
    • by specifying who can create and accept external data shares [1]
    • users can share data residing in tables or files within supported Fabric items [1]
      • require standard Fabric read and reshare permissions for the item being shared [1]
      • the user creating the share invites a user from another tenant with a link to accept the external data share [1]
        • upon accepting the share, the recipient chooses a lakehouse where a shortcut to the shared data will be created [1]
        • the links work only for users in external tenants
          • for sharing data within the same OneLake storage accounts with users in the same tenant, use OneLake shortcuts [1]
        • {limitation} shortcuts contained in folders that are shared via external data sharing won't resolve in the consumer tenant [1]
    • access is enabled via a dedicated Fabric-to-Fabric authentication mechanism 
    • ⇐ doesn’t require Entra B2B guest user access [1]
  • {operation} create an external data share in the provider tenant)
    • external data shares can be created for tables or files in lakehouses and warehouses, and in KQL, SQL, mirrored databases [1]
    • {limitation} the sharer can't control who has access to the data in the consumer's tenant [1]
  • {operation} accept an external data share in consuming tenant)
    • only lakehouses can be chosen for the operation
    • the consumer can grant access to the data to anyone [1]
      • incl. guest users from outside the consumer's organization [1]
    • data can be transferred across geographic boundaries when it's accessed within the consumer's tenant [1]
  • {operation} revoke extern data shares
    • any user in the sharing tenant with read and reshare permissions on an externally shared item can revoke the external data share at any time [1]
      • via Manage permissions >> External data shares tab
      • can be performed of any item the user has read and reshare permissions on [3]
      • {warning}a revoked external data share can't be restored [3] 
        • irreversibly severs all access from the receiving tenant to the shared data [3]
        • a new external data share can be created instead [3]
  • applies to
    • lakehouse
      • an entire lakehouse schema can be shared [2]
        • shares all the tables in the schema [2]
        • any changes to the schema are immediately reflected in the consumer’s lakehouse [2]
    • mirrored database
    • KQL database
    • OneLake catalog
  • can be consumed via 
    • Spark workloads
      • notebooks or Spark
    • lakehouse SQL Analytics Endpoint
    • semantic models
    • ⇐ data can be shared from a provider and consumed in-place via SQL queries or in a Power BI report [2]
  • {feature} external data sharing APIs
    • support service principals for admin and user operations
      • can be used to automate the creation or management of shares [2]
        • supports service principals or managed identity [2]
    • {planned} data warehouse support
      • share schemas or tables from a data warehouse [2]
    • {planned} shortcut sharing
      • share OneLake shortcuts using external data sharing [2]
      • ⇒ data residing outside of Fabric will be externally shareable
        • S3, ADLS Gen2, or any other supported shortcut location [2]
    • {planned} consumer APIs
      • consumer activities, including viewing share details and accepting shares, will soon be available via API [2]

References:
[1] Microsoft Learn (2024) External data sharing in Microsoft Fabric [link]
[2] Microsoft Fabric Updates Blog (2025) External data sharing enhancements out now [link]
[3] Microsoft Learn (2024) Fabric: Manage external data shares [link]

Resources:
[R1] Analytics on Azure Blog (2024) External Data Sharing With Microsoft Fabric [link]
[R2] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]

Acronyms:
ADLS - Azure Data Lake Storage
API - Application Programming Interface
B2B - business-to-business
KQL - Kusto Query Language

09 February 2025

🏭🗒️Microsoft Fabric: Sharding [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 9-Feb-2024

 [Microsoft Fabric] Data Partitioning (aka Sharding)

  • {definition} "a process where small chunks of the database are isolated and can be updated independently of other shards" [2]
  • allows a logical database to be partitioned across multiple physical servers [1]
    • each partition is referred to as a shard
    • the largest tables are partitioned across multiple database servers [1]
      • when operating on a record, the application must determine which shard will contain the data and then send the SQL to the appropriate server [1]
    • partitioning is based on a Key Value
      • e.g. such as a user ID
  • proven technique for achieving data processing on a massive scale [1]
    • solution used at the largest websites 
      • e.g. Facebook, Twitter
      • usually associated with rapid growth
        •  ⇒ the approach needs to be dynamic [1]
    • the only way to scale a relational database to massive web use [1]
      • together with caching and replication [1]
  • {drawback} involves significant operational complexities and compromises [1]
    • the application must contain logic that understands the location of any particular piece of data and the logic to route requests to the correct shard [1]
    • requests that can only be satisfied by accessing more than one shard thus need complex coding as well, whereas on a nonsharded database a single SQL statement might suffice.
  • {drawback} high operational costs [1]
  • {drawback} application complexity
    • it’s up to the application code to route SQL requests to the correct shard [1]
      • ⇒ a dynamic routing layer must be implemented
        • ⇐ most massive websites are adding shards as they grow [1]
        • layer required to maintain Memcached object copies and to differentiate between the master database and read-only replicas [1]
  • {drawback} crippled SQL
    • [sharded database] it is not possible to issue a SQL statement that operates across shards [1]
      • ⇒ usually SQL statements are limited to row-level access [1]
      • ⇒ only programmers can query the database as a whole [1]
      • joins across shards cannot be implemented, nor can aggregate GROUP BY operations [1]
  • {drawback} loss of transactional integrity
    • ACID transactions against multiple shards are not possible and/or not practical [1]
      • ⇐ {exception} there are database systems that support 2PC
        • involves considerable troubleshooting as conflicts and bottlenecks can occur [1]
  • {drawback} operational complexity. 
    • load balancing across shards becomes extremely problematic
      • adding new shards requires a complex rebalancing of data [1]
      • changing the database schema requires a rolling operation across all the shards [1]
        • ⇒ can lead to transitory inconsistencies in the schema [1]
    • a sharded database entails a huge amount of operational effort and administrator skill [1]
  • {concept} CAP (Consistency, Availability, and Partition) theorem 
    • in a distributed database system, one can have at most only two of CAP tolerance [1]
    • consistency
      • every user of the database has an identical view of the data at any given instant [1]
    • availability
      • in the event of a failure, the database remains operational [1]
    • partition tolerance
      • the database can maintain operations in the event of the network’s failing between two segments of the distributed system [1]
  • {concept} partitioning
    • {def} core pattern of building scalable services by dividing state (data) and compute into smaller accessible units to improve scalability and performance [5]
      • ⇐ determines that a particular service partition is responsible for a portion of the complete state of the service.
        • a partition is a set of replicas)
    • {type} [stateless services] a logical unit that contains one or more instances of a service [5]
      • partitioning a stateless service is a very rare scenario
      • scalability and availability are normally achieved by adding more instances
      • {subtype} externally persisted state
        • persists its state externally [5]
          • e.g. databases in Azure SQL Database
      • {subtype} computation-only services
        • service that do not manage any persistent state e.g. calculator or image thumbnailing [5]
    • {type} scalable stateful services
      • partition state (data)
      • a partition of a stateful service as a scale unit that is highly reliable through replicas that are distributed and balanced across the nodes in a cluster
      • the state must be accessed and stored
        • ⇒ bound by
          • network bandwidth limits
          • system memory limits
          • disk storage limits
        • {scenario} run into resource constraints in a running cluster
          • {recommendation} scale out the cluster to accommodate the new requirements [4]
  • {concept}distributed systems platform used to build hyper-scalable, reliable and easily managed applications for the cloud [6]
    • ⇐ addresses the significant challenges in developing and managing cloud applications
    • places the partitions on different nodes [5]
      • allows partitions to grow to a node's resource limit
        • ⇐ partitions are rebalances across nodes [5]
      • {benefit} ensures the continued efficient use of hardware resources [5]
    • {default} makes sure that there is about the same number of primary and secondary replicas on each node
      • ⇒ nodes that hold replicas can serve more traffic and others that serve less traffic [5]
      • hot and cold spots may appear in a cluster
          • ⇐ it should be preferably avoided
        • {recommendation} partition the state so is evenly distributed across all partitions [5]
        • {recommendation} report load from each of the replicas for the service [5]
    • provides the capability to report load consumed by services [5]
      • e.g.  amount of memory, number of records
      • detects which partitions server higher loads than others [5]
        • ⇐ based on the metrics reported
      • rebalances the cluster by moving replicas to more suitable nodes, so that overall no node is overloaded  [5]
      • ⇐ it's not always possible to know how much data will be in a given partition
        • {recommendation} adopt a partitioning strategy that spreads the data evenly across the partitions [5]
          • {benefit} prevents situations described in the voting example [5]
        • {recommendation} report load
          • {benefit} helps smooth out temporary differences in access or load over time [5]
    • {recommendation} choose an optimal number of partitions to begin with
      • ⇐ there's nothing that prevents from starting out with a higher number of partitions than anticipated [5]
        • ⇐ assuming the maximum number of partitions is a valid approach [5]
      • ⇒ one may end up needing more partitions than initially considered [5]
        • ⇐ {constraint} the partition count can't be changed after the fact [5]
          •  ⇒ apply more advanced partition approaches
            • e.g. creating a new service instance of the same service type
            • e.g. implement client-side logic that routes the requests to the correct service instance

References:
[1] Guy Harrison (2015) Next Generation Databases: NoSQL, NewSQL, and Big Data
[2] DAMA International (2017) "The DAMA Guide to the Data Management Body of Knowledge" 2nd Ed
[3] Microsoft Fabric (2024) External data sharing in Microsoft Fabric [link]
[4] Microsoft Fabric (2024) Data sharding policy [link]
[5] Microsoft Fabric (2024) Partition Service Fabric reliable services [link]
[6] MSDN (2015) Microsoft Azure - Azure Service Fabric and the Microservices Architecture [link]

Resources:
[R1] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]

Acronyms:
ACID - atomicity, consistency, isolation, durability
2PC - Two Phase Commit
CAP - Consistency, Availability, Partition

20 April 2024

⚡️🗒️Power BI: Visual Calculations [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 2-Nov-2026

[feature] Visual Calculations (aka Visual Calcs)

  • {def} a type of DAX calculation that's defined and executed in the scope of a visual [1] [5]
    • represent columns added to a virtual table of filtered and aggregated data [5]
      • not directly connected to the rest of the semantic model [5]
    • {characteristic} defined and evaluated only on the filtered and aggregated data that is visible in the visual 
      • the visual matrix in which the visual calculation is created [5] 
        • aka calculations table that’s defined in the DAX query [5]
      •  the filter context dictates what the measures and fields on the visual return, and the visual calculation takes those values as input for its evaluation [5]
        • ⇐ a visual calculation is only indirectly affected by filter context, not directly, the way a measure or field reference is [5]
    • {characteristic} evaluate at runtime [5]
      • like measures do [5] 
      • ⇒ they can be fully dynamic in the context of the visual [5]
    • {characteristic} a single query is sent to the source [5]
      • ⇐ no such guarantee exists for analysis that relies on measures [5] 
      • ⇐ how pronounced the effects are depends on the size of the data and the complexity of the DAX used [5]
    • {characteristic} executed on a row-by-row basis
      • like a calculated column [5]
    • {characteristic} calculated on the fly [4]
      • like a measure [5]
    • {characteristic} stored on the visual
      • ⇒ aren’t part of the semantic model [5]
      • ⇒ can refer to any data in the visual
        •  incl. columns, measures, or other visual calculations [1]
      • ⇒ anything in the model must be added to the visual before the visual calculation can refer to it [1]
      • ⇒ no need to worry about filter context [5]
      • ⇒ they can refer to the visual structure
      • ⇒ leads to more flexibility
    • {characteristic} operate on aggregated data instead of the detail level [1]
      • ⇒ leads to performance benefits [1]
  • {benefit} combine the simplicity of context from calculated columns with the on-demand calculation flexibility from measures [1]
    • ⇐ the context is 'visible
    • ⇐ share behaviors with calculated columns and measures [5]
      • ⇐ have important differences
        • ⇐ in how they can be used, where they are stored, when they are computed [5]
  • {benefit} make it easier to create calculations (that were previously hard to create) 
    • leads to simpler DAX, easier maintenance, better performance [1]
    • reuse the results from its components [2]
    • simpler than measures, more trustworthy than Excel [5]
  • {benefit} the visual context in which they operate not only describes what data is on the visual (which is then iterated over in a row context) but also the structure of the visual 
    • ⇒ a visual calculation can refer to the axes of a visual, such as the x-axis or the y-axis instead of the actual field [5]
    • ⇐ enables visual calculations to be highly flexible [5]
      • ⇐ a visual calculation can continue to work even if the field that is on the x-axis changes by simply referring to the axis instead of the actual field [5]
  • {benefit} highly visual [5] always show the data the calculation works on, and tools are provided to easily validate the results [5]
  • {benefit} should perform better than measures 
    • ⇐ because they’re executed as part of the DAX query to fetch the results instead of independently [5]
  • {limitation} unaware of the semantic model [5]
    • ⇐ as they are not part of the semantic model) [5]
    • ⇒ DAX functions that perform actions on that model don’t work (return an error) [5]
      •  e. g. RELATED, USERELATIONSHIP
  • {limitation} can’t be reused across multiple reports [5] 
    • ⇒ leads to code duplication [5]
    • ⇐ only live on the visual, which will avoid cluttering of the semantic model [5]
    • they’re saved in the metadata of the report, inside the visual definition. [5]
    • can only be used inside one report and that definitions cannot be shared across multiple reports that share the same semantic model [5]
  • (limitation) do not work together with the 'show items with no data' option, because it would result in performance issues.[5]
  • {recommendation} create measures for any key calculations that should be reused across multiple reports [5]
    •  o    allows to establish a SVoT, better reusability, maintenance, design, etc.
  • introduce a set of functions specific to visual calculations [1]
    • {category} medium-level functions
      • {function} COLLAPSE
        • the calculation is evaluated at a higher level of the axis [1]
        • navigate to a higher level in the lattice formed by the fields on the axes of the visual matrix [5] 
        • most often used for percentage of parent, grandparent, and total calculations [5]
      • {function} COLLAPSEALL
        • the calculation is evaluated at the total level of the axis [1]
        • does not take extra parameters because it always moves to the highest level on the axis [5] navigate to the highest level of the lattice on the axis specified [5]
      • {function} EXPAND
        • the calculation is evaluated at a lower level of the axis [1]
        • navigate to a lower level in the lattice formed by the fields on the axes of the visual matrix [5]
        • often used for aggregated descendant averages [5]
        • the reverse of COLLAPSE
      • {function}EXPANDALL
        • the calculation is evaluated at the leaf level of the axis [1]
        • always moves down to the lowest level (leaf level) on the axis and doesn’t take these parameters [5]
        • the reverse of COLLAPSEALL [5]
      • {function} FIRST
        • retrieves a value from the first element on a specified axis [5]
        • often used to compare against a base period or entity [5]
        • retrieves the value of the column in the first element on the specified axis since the last time the calculation was reset [5]
        • an easier-to-use shortcut to the INDEX function with the position parameter set to 1 [5]
      • {function} ISATLEVEL
        • checks whether specified columns are on the current level of the axis [5]
        • returns a Boolean (True/False) value
        • guaranteed to work correctly in visual calculations with functions that navigate the levels of the lattice [5]
        • unlike ISINSCOPE, ISFILTERED inspection functions [5]
      • {function} LAST
        • refers to the last row of an axis [1]
        • retrieves a value from the last element on a specified axis [5]
        • the reverse of FIRST
        • often used to compare against the most recent entry [5]
      • {function} LOOKUP
        • evaluates an expression with a value from a cell in the provided visual matrix using filters [5]
        • anything that is not specified is inferred from the context [5]
        • often used to compare against a specific value in the visual matrix [5]
      • {function} LOOKUPWITHTOTALS
        • uses the total for any filter that is not specified [5]
        • returns the value of the expression provided at the specified coordinates after filters have been applied [5]
          • any filter that is not specified is treated as referring to the total
          • if no single value can be determined, an error is returned [5]
          • ⇐ instead of using the context to infer any filter that is not specified like LOOKUP does [5]
        • relies on absolute navigation in the visual matrix [5]
          • ⇐ COLLAPSE and COLLAPSEALL rely on relative navigation on the lattice [5]
            • ⇐ upon case can be used to retrieve same result [5]
      • {function} NEXT
        • retrieves a value from a next element on a specified axis [5]
        • retrieves the value of the column from a next element on the specified axis since the last time the calculation was reset [5]
        • provides a shortcut to the OFFSET function with a positive value provided for the delta parameter [5]
      • {function} PREVIOUS
        • retrieves a value from a previous element on a specified axis [5]
        • retrieves the value of the column from an earlier element on the specified axis since the last time the calculation was reset [5]
        • provides a shortcut to the OFFSET function with a negative value provided for the delta parameter [5]
      • {function} RANGE
        • provides a range of rows relative from the current position on the axis [5]
        • shortcut to WINDOW 
        • returns a context 
          • ⇒ it must be used with other functions, such as CALCULATE, to actually perform a calculation [5]
        • often employed to calculate a moving sum [5]
    • {category} high-level functions
      • {function} MOVINGAVERAGE
        • adds a moving average on an axis [1]
        • involves selecting a slice of the values on an axis and returning the average over that slice [5]
        • most often used to calculate averages across periods [5]
        • provides an easier-to-use shortcut to the WINDOW function [5]
      • {function} RUNNINGSUM
        • returns the sum of all values in a column on the axis since the last time the calculation was reset, up to and including the current value [5]
          • if no reset is defined, RUNNINGSUM starts at the top of the visual matrix and continues to the end, following the sort order [5]
        • created specifically for visual calculations [5]
          • often used for Pareto analysis [5]
        • provides an easier-to-use shortcut to the WINDOW function [5]
        • it’s possible to write a running sum in a measure, though the code becomes more complex to write [5] 
          • ⇐ include explicit references to columns on which the calculation works [5]
            • ⇒ if the user changes the columns on the visual, the measure will return unexpected results and will have to be updated to reflect the changes [5]
    • {category} low-level functions
      • ⇐ {exception} are available in standard DAX
      • these functions are easier to use, though they are less flexible than their foundational counterparts [5]
      • {recommendation} rewrite visual calculation using DAX for more flexibility [5]
        • {recommendation} start with the easier-to-use visual calculations exclusive functions and resort to other functions only when needed [5]
    • {category}foundational functions
      • {parameter} relation 
        • table expression that defines from which output a value is returned, namely a table expression [5]
        • name of a table or a DAX statement that returns a table, such as ADDCOLUMNS or SUMMARIZECOLUMNS
        • any columns specified in the partitionBy parameter must come from the relation parameter or from a related table [5]
      • {parameter} orderBy
        • specifies how each partition on the relation or axis is sorted [5]
        • accepts only the ORDERBY function
        • partitions are defined by using either partitionBy or reset [5]
        • {default} ordering by every column that is in the relation or on the axis that is not specified in partitionBy or reset [5]
      • {parameter} blanks
        • specify how blank values on the axis should be ordered while the calculation traverses the axis (in a visual calculation) or the relation [5]
        • does not sort anything in the values of the visual matrix (e.g. visual calculations or measures) [5] it sorts the values of the fields on the axis used [5]
        • {value} DEFAULT
          • indicates that blank numerical values are ordered between zero and negative values. For blank textual values, the blank values are ordered before all text values, including empty text values [5]
        • {value} FIRST 
          • blank values are always ordered at the beginning, regardless of ascending or descending sorting order [5]
        • {value] LAST 
          • blank values are always ordered at the end, regardless of ascending or descending sorting order [5]
      • {parameter} partitionBy
        • specifies how the relation or axis is partitioned [5]
        • accepts the PARTITIONBY function [5]
        • {recommendation} use reset instead of partitionBy  
          • ⇐ it’s the easiest way of achieving the same result [5]
      • {parameter} matchBy
        • defines how to match data to identify unique rows [5]
        • use this parameter if you do not have anything that can uniquely identify the rows in your relation (aka composite key) [5]
        • when using axis in visual calculations, there's not need to use matchBy in visual calculations because the axis always has unique identifiers [5]
          • if no matchBy is specified and the columns in orderBy and partitionBy cannot uniquely identify every row in the relation, the foundational function will try to find the least number of additional columns required to uniquely identify the rows and append these to the orderBy value (even if you did not specify orderBy) [5] 
      • {parameter} resets
        • accepts the MATCHBY function
        • available only for visual calculations
        • the calculation is reset by dividing the data on the axis into slices, the same way partitionBy does [5]
        • {def} reset parameter
          •  defines when a calculation should restart as it traverses the visual matrix
        • {limitation} one cannot specify both reset and partitionBy
          • one can think of the reset parameter as being mapped to the partitionBy parameter, however reset automatically includes parent levels and partitionBy does not [5]
      • {function} INDEX
        • returns a row in an absolute position
        • position parameter defines the absolute position on the relation or axis from which to obtain the data.
      • {function} OFFSET
        • returns a row in a relative position
        • {parameter} delta
          • specifies the relative position on the relation or axis from which to obtain the data [5]
          • when specifying a delta that causes a relative movement that does not exist on the partition, when specifying  0 or BLANK(), then OFFSET will not perform a relative movement, and the context is set to the current row [5]
          • any DAX expression that returns a scalar value is valid [5]
      • {function} RANK
        • provides a ranking of each row within a partition, sorted by the specified sort order [5]
        • returns a blank value for total rows when used in measures but returns a value on those rows when used in visual calculations [5]
        • expects a table and applies an expression to rank the rows in the table [5]
        • {warning} it's not the column-based version of RANKX [5]
        • {parameter} ties parameter
          • {value|default} SKIP
            • if two rows end up with the same rank, they will both be assigned the same rank and the next rank number will be skipped [5]
          • {value}DENSE
            • the next rank number will not be skipped [5]
      • {function} ROWNUMBER
        • returns a unique ranking of each row within a partition, sorted by the specified sort order [5]
        • returns an error if it cannot uniquely identify each row [5]
          • ⇒ guarantees that the same number will never be assigned twice [5]
      • {function}WINDOW
        • returns multiple rows, which are positioned at a selectable absolute or relative interval [5]
        • {parameter} from
          • indicate where the window start
        • {parameter} to
          • indicate where the window ends
        • {parameter} from_type
          • specifies whether the window starts at either 
          • {default} relative position (REL)
            • then a negative value provided for this parameter specifies the number of rows to go back from the current position to get the first (or last) row in the window [5]
          • absolute position (ABS) [4]
            • indicate the 1-based absolute position in the current partition of the start and end of the window [4]
            • 1= first row, -1 = last row
        • {parameter} to_type
          • see from_type
      • {category} supportive functions
        • used as inputs to specific foundational function parameters with the same name [5]
        • on their own, these functions provide no value [5]
        • {recommendation) specify an axis value for the relation parameter and use reset as needed when using foundational functions in visual calculations [5]
          • the Axis parameter 
            •  defines the path or direction through the visual matrix that a calculation follows, controlling how functions move through the data displayed in a visual
              • tells PBI how to traverse the visual's data when performing the calculation, giving precise control on how the calculations are performed
      • {function} ORDERBY
        • define the sorting order within each partition of the relation or axis on which the function operates [5]
      • {function} PARTITIONBY
        • indicates if and how to slice up the data in the relation or axis on which the foundational function operates [5]
        • it can be skept if no slices are defined [5]
      • {function} MATCHBY
        • instruct DAX on how to determine the current row [5]
  • {category} shared functions
    • DAX functions which are shared across all experiences [5]
  • {category} exclusive functions 
    • functions introduced in DAX solely for visual calculations [5]
  • {category} blocked functions 
    • functions that reach out to the model [5]
  • {default} most of them are evaluated row-by-row [1]
    • ⇐ like a calculated column
    • there's no need to add an aggregation function [1]
      •  it's better not to add such aggregates when they're not necessary [1]
  • {operation} create calculation
    • adds the visual calculation to the visual
    • it's possible to create visual calculations directly in the service [5]
    • allows to create very complex visual calculations in steps and hide any irrelevant intermediate results [5]
    • creation is not traced as activity in audit logs but is covered in a generic activity named Update Report Content [5]
  • {operation} hide calculation
    • calculations that aren't needed in the visual can be hidden [2]
    • hidden fields enable users to hide elements from the visual [5]
    • {def} hidden field
      • a column in a visual intended solely to support a visual calculation, but is not displayed to the report consumer
  • {operation} copy calculation
    • copies the calculation between visuals and if intermediary steps are not there, they will be copied as well [planned] [2]
  • {operation} formatting 
    • ⇐ do not take on the format of any measures used to create them [5]
  • {operation} view a visual as a visual matrix 
    • enables users to add more calculations, which can be seen as new columns in the visual matrix [5]
  • {feature} templates 
    • ready available calculation constructs 
    • {benefit} make it easier to write common calculations [1]
  • {feature|planned} support for Scanner API [2]
  • {feature} explore
    • new experience that allows to explore data in a focused way [4]
    • allows adding visual calculations to visuals [4]
  • {feature} parameter pickers
    • allows to create visual calculations faster by picking parameters [3]
    • {limitation} only available for required parameters on functions that are exclusive to visual calculations (and select other functions) that have a defined list of options [3]
      •  required parameters that can take any text, or numerical value will not get a parameter picker, and neither will many DAX functions [3]
  • {feature} visual preview 
    • shows users what the visual will look like when leaving the visual calculations edit mode and returning to the report [5]
    • allows users to see what impact newly added visual calculations have on the visual, and how the visual will look like if certain measures or visual calculations are hidden [5]
  • {feature} visual matrix
    • {def} the internal tabular representation of the data shown in the visual that PBI uses to evaluate the calculation
    • the data representation of your visual 
      • ⇐ shows you the outcomes of all newly added calculations [5] 
      • ⇐ the simplest representation of the data used to create the visual [5]
    • offers a way to structure data dynamically based on rows and columns in a WYSIWYG fashion [5]
    • doesn't display any formatting that may be applied to the visual itself [5]
    • every value in a visual calculation must exist in the visual matrix
      • ⇒ both the original value residing in the model and the visual calculation will be shown in the resulting visual [5]
  • {feature} formula bar 
  • allows to write and edit visual calculations [5]
  • {parameter} axis 
    • influences how the visual calculation traverses the visual matrix [1] 
      • ⇐ defines the direction in which the running sum should be calculate [5]
    • can be seen as the axis of a chart, which has an x-axis and a y-axis [5]
    • not available in measures, calculated columns, or calculated tables [5]
    • defines the direction in which the running sum should be calculated: over rows, columns, or a combination [5]
    • {default} set to the first axis in the visual
    • {value} ROWS
      • the visual calculation is evaluated row-by-row in the visual matrix, from top to bottom. [1]
    • {value} COLUMNS
      • the visual calculation is evaluated row-by-row in the visual matrix, from left to right [1]
    • {value} ROWS COLUMNS
      • calculates vertically across rows from top to bottom, continuing column by column from left to right [1]
    • {value} COLUMNS ROWS
      • calculates horizontally across columns from left to right, continuing row by row from top to bottom [1]
    • {warning} not all visuals provide all axes, and some visuals provide no axes [1]
      • references to a non-existent or invalid axis is permissible and will be ignored [3]
  • {parameter} reset 
    • influences if and when the function resets its value to 0 or switches to a different scope while traversing the visual matrix [1]
    • expects there to be multiple levels on the axis [1]
      • ⇐ use PARTITIONBY if there's only one level on the axis [1]
    • {value|default} NONE
      • means the visual calculation is never restarted [1]
    • {value} HIGHESTPARENT 
      • resets the calculation when the value of the highest parent on the axis changes [1]
    • {value} LOWESTPARENT 
      • resets the calculations when the value of the lowest parent on the axis changes [1]
    • {value} numerical value
      • refers to the fields on the axis, with the highest field being one [1]
  • {concept} lattice
    • formed by all the fields on all the axes [5]
    • visual calculations calculate results on various levels on the lattice [5]
    • lattice navigation functions allow users to explicitly move around in the lattice [5]
      • e.g. COLLAPSE, COLLAPSEALL, EXPAND, EXPANDALL
    • {default} the data type of a visual calculation is decimal number [5]
  • {concept} format strings
    • allow for a more fine-grained level of formatting
  • {feature} traceability 
    • prevents redundancy and conflicting definitions by identifying inconsistent calculations [5] 
    • using audit trails and versioning systems further enhances accountability and enables swift corrections when needed [5]
  • {limitation} functions that rely on model relationships  aren't available
    • e.g. USERELATIONSHIP, RELATED or RELATEDTABLE
  • {limitation} not all visual types are supported [1]
    • ⇐ for the full list of limitations see [1]
  • {limitation} one can't filter on visual calculations [1]
  • {limitation} underlying data can't be exported [1]
  • {limitation} don't support conditional formatting
  • {concept} skippable parameters
    • introduced with visual calculations [5]
    • allow for cleaner code because users can simply omit any unnecessary optional parameters [5]
      • ⇐ unlike with DAX functions that do not support skippable parameters [5]
  • {concept} telemetry 
    • the collection and analysis of data to monitor, measure, and optimize system performance[5]
    • can provide valuable insights into the usage patterns, dependencies, and performance of visual calculations [5]
    • allows administrators can identify discrepancies and ensure that visual calculations align with organizational standards [5]
  •  

References: 
[1] Microsoft Learn (2024) Power BI: Using visual calculations [preview] [link]
[2] SSBI Central (2024) Visual Calculations - Making DAX easier, with Jeroen ter Heerdt [link]
[3] Microsoft Power BI Updates (2025) Power BI June 2025 Feature Summary [link]
[4] Microsoft Learn (2025) Power BI: Use Explore (preview) in the Power BI service [link]
[5] Jeroen ter Heerdt et al (2026) Microsoft Power BI Visual Calculations: Simplifying DAX 

05 March 2024

🧭Business Intelligence: Data Culture (Part I: Generative AI - No Silver Bullet)

Business Intelligence
Business Intelligence Series

Talking about holy grails in Data Analytics, another topic of major importance for an organization’s "infrastructure" is data culture, that can be defined as the collective beliefs, values, behaviors, and practices of an organization’s employees in harnessing the value of data for decision-making, operations, or insight. Rooted in data literacy, data culture is an extension of an organization’s culture in respect to data that acts as enabler in harnessing the value of data. It’s about thinking critically about data and how data is used to create value. 

The current topic was suggested by PowerBI.tips’s webcast from today [3] and is based on Brent Dykes’ article from Forbes ‘Why AI Isn’t Going to Solve All Your Data Culture Problems’ [1]. Dykes’ starting point for the discussion is Wavestone's annual data executive survey based on which the number of companies that reported they had "created a data-driven organization" rose sharply from 23.9 percent in 2023 to 48.1 percent in 2024 [2]. The report’s authors concluded that the result is driven by the adoption of Generative AI, the capabilities of OpenAI-like tools to generate context-dependent meaningful text, images, and other content in response to prompts. 

I agree with Dykes that AI technologies can’t be a silver bullet for an organization data culture given that AI either replaces people’s behaviors or augments existing ones, being thus a substitute and not a cure [1]. Even for a disruptive technology like Generative AI, it’s impossible to change so much employees’ mindset in a so short period of time. Typically, a data culture matures over years with sustained effort. Therefore, the argument that the increase is due to respondent’s false perception is more than plausible. There’s indeed a big difference between thinking about an organization as being data-driven and being data-driven. 

The three questions-based evaluation considered in the article addresses this difference, thinking vs. being. Changes in data culture don’t occur just because some people or metrics say so, but when people change their mental models based on data, when the interpersonal relations change, when the whole dynamics within the organization changes (positively). If people continue the same behavior and practices, then there are high chances that no change occurred besides the Brownian movement in a confined space of employees, that’s just chaotic motion.  

Indeed, a data culture should encourage the discovery, exploration, collaboration, discussions [1] respectively knowledge sharing and make people more receptive and responsive about environmental or circumstance changes. However, just involving leadership and having things prioritized and funded is not enough, no matter how powerful the drive. These can act as enablers, though more important is to awaken and guide people’s interest, working on people’s motivation and supporting the learning process through mentoring. No amount of brute force can make a mind move and evolve freely unless the mind is driven by an inborn curiosity!

Driving a self-driving car doesn’t make one a better driver. Technology should challenge people and expand their understanding of how data can be used in different contexts rather than give solutions based on a mass of texts available as input. This is how people grow meaningfully and how an organization’s culture expands. Readily available answers make people become dull and dependent on technology, which in the long-term can create more problems. Technology can solve problems when used creatively, when problems and their context are properly understood, and the solutions customized accordingly.

Unfortunately, for many organizations data culture will be just a topic to philosophy about. Data culture implies a change of mindset, perception, mental models, behavior, and practices based on data and not only consulting the data to confirm one’s biases on how the business operates!

Previous Post <<||>> Next Post

Resources:
[1] Forbes (2024) Why AI Isn’t Going To Solve All Your Data Culture Problems, by Brent Dykes (link)
[2] Wavestone (2024) 2024 Data and AI Leadership Executive Survey (link)
[3] Power BI tips (2024) Ep.299: AI & Data Culture Problems (link)

21 December 2016

♟️Strategic Management: Operations (Just the Quotes)

"Strategy, or the art of properly directing masses upon the theater of war, either for defense or for invasion. […] Strategy is the art of making war upon the map, and comprehends the whole theater of operations. Grand Tactics is the art of posting troops upon the battle-field according to the accidents of the ground, of bringing them into action, and the art of fighting upon the ground, in contradistinction to planning upon a map. Its operations may extend over a field of ten or twelve miles in extent. Logistics comprises the means and arrangements which work out the plans of strategy and tactics. Strategy decides where to act; logistics brings the troops to this point; grand tactics decides the manner of execution and the employment of the troops." (Antoine-Henri Jomini, "The Art of War", 1838)

"A policy therefore might be likened to strategy, the broad, overall, long term conception which gives direction and purpose to the tactics of immediately daily operations and decisions." (Lawrence K. Frank, "National Policy for the Family", 1948)

"Essential to organization planning, then, is the search for an ideal form of organization to reflect the basic goals of the enterprise. This entails not only charting the main lines of organization and reflecting the organizational philosophy of the enterprise leaders (e.g., shall authority be as centralized as possible, or should the company try to break its operations down into semiautonomous product or territorial divisions?), but also a sketching out of authority relationships throughout the structure." (Harold Koontz & Cyril O Donnell, "Principles of Management", 1955)

"The concern of OR with finding an optimum decision, policy, or design is one of its essential characteristics. It does not seek merely to define a better solution to a problem than the one in use; it seeks the best solution... [It] can be characterized as the application of scientific methods, techniques, and tools to problems involving the operations of systems so as to provide those in control of the operations with optimum solutions to the problems." (C West Churchman et al, "Introduction to Operations Research", 1957)

"The essential task of management is to arrange organizational conditions and methods of operations so that people can achieve their own goals best by directing their own efforts toward organizational objectives." (Douglas McGregor, "The Human Side of Enterprise", 1960)

"Now we are looking for another basic outlook on the world - the world as organization. Such a conception - if it can be substantiated - would indeed change the basic categories upon which scientific thought rests, and profoundly influence practical attitudes. This trend is marked by the emergence of a bundle of new disciplines such as cybernetics, information theory, general system theory, theories of games, of decisions, of queuing and others; in practical applications, systems analysis, systems engineering, operations research, etc. They are different in basic assumptions, mathematical techniques and aims, and they are often unsatisfactory and sometimes contradictory. They agree, however, in being concerned, in one way or another, with ‘systems’, ‘wholes’ or ‘organizations’; and in their totality, they herald a new approach." (Ludwig von Bertalanffy, "General System Theory", 1968)

"The dogma of delegation is simple - the Sixth Truth of Management again: either the delegatee is capable of running the operation successfully by himself or he isn't. This handy formula relieves the top executive of any responsibility except that of finding, supervising, and (at the appropriate time) moving the men who are doing all the work. He Can then truly manage by exception: he does not get worked up over operations that are going well, but concentrates on the plague spots, where everything, including the management, is going badly." (Robert Heller, "The Naked Manager: Games Executives Play", 1972)

"Perhaps the fault [for the poor implementation record for models] lies in the origins of managerial model-making - the translation of methods and principles of the physical sciences into wartime operations research. [...] If hypothesis, data, and analysis lead to proof and new knowledge in science, shouldn’t similar processes lead to change in organizations? The answer is obvious-NO! Organizational changes (or decisions or policies) do not instantly pow from evidence, deductive logic, and mathematical optimization." (Edward B Roberts, "Interface", 1977)

"Management science [operations research] also involves the philosophy of approaching a problem in a logical manner (i.e., a scientific approach). The logical, consistent, and systematic approach to problem solving can be as useful (and valuable) as the knowledge of the mechanics of the mathematical techniques themselves." (Bernard W. Taylor III, "Introduction to Management Science, 1986)

"Strategy means abstract thinking and planning, as opposed to tactics, which are the individual operations used to implement strategy. Tactics are specific; strategy is general. Tactics tend to be immediate, strategy long-term." (Bruce Pandolfini, "Weapons of Chess: An omnibus of chess strategy", 1989)

"At the heart of reengineering is the notion of discontinuous thinking - of recognizing and breaking away from the outdated rules and fundamental assumptions that underlie operations. Unless we change these rules, we are merely rearranging the deck chairs on the Titanic. We cannot achieve breakthroughs in performance by cutting fat or automating existing processes. Rather, we must challenge old assumptions and shed the old rules that made the business underperform in the first place." (Michael M Hammer, "Reengineering Work: Don't Automate, Obliterate", Magazine, 1990)

"Even with simple and usable models, most organizations will need to upgrade their analytical skills and literacy. Managers must come to view analytics as central to solving problems and identifying opportunities - to make it part of the fabric of daily operations." (Dominic Barton & David Court, "Making Advanced Analytics Work for You", 2012)

"DevOps recognizes the importance of culture. The acronym CAMS (culture, automation, measurement, and sharing) is used to encapsulate its key themes. Culture is acknowledged as all important in making development and IT operations work together effectively. But what is culture in this context? It is not so much about an informal dress code, flexible hours, or a free in-house cafeteria as it is about how decisions are taken, norms of behavior, protocols of communication, and the ways of navigating hierarchy and bureaucracy to get things done." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

11 June 2016

♜Strategic Management: Resilience (Definitions)

"The ability to recover from challenges or to overcome obstacles. In a social-ecological context this refers to the innovation capacity of the organization to successfully address societal and environmental challenges." (Rick Edgeman & Jacob Eskildsen, "Social-Ecological Innovation", 2014)

"The quality of being able to absorb systemic 'shocks' without being destroyed even if recovery produces an altered state to that of the status quo ante." (Philip Cooke, "Regional Innovation Systems in Centralised States: Challenges, Chances, and Crossovers", 2015)

"The ability of an organization to quickly adapt to disruptions while maintaining continuous business operations and safeguarding people, assets, and overall brand equity. Business resilience goes a step beyond disaster recovery, by offering post-disaster strategies to avoid costly downtime, shore up vulnerabilities, and maintain business operations in the face of additional, unexpected breaches." (William Stallings, "Effective Cybersecurity: A Guide to Using Best Practices and Standards", 2018)

"A capability to anticipate, prepare for, respond to, and recover from significant multi-hazard threats with minimum damage to social well-being, the economy, and the environment." (Carolyn N Stevenson, "Addressing the Sustainable Development Goals Through Environmental Education", 2019)

"The ability of a project to readily resume from unexpected events, threats or actions." (Phil Crosby, "Shaping Mega-Science Projects and Practical Steps for Success", 2019)

"The ability of an infrastructure to resist, respond and overcome adverse events" (Konstantinos Apostolou et al, "Business Continuity of Critical Infrastructures for Safety and Security Incidents", 2020)

"The capacity to respond to, adapt and learn from stressors and changing conditions." (Naomi Borg & Nader Naderpajouh, "Strategies for Business Sustainability in a Collaborative Economy", 2020)

"The word resilience refers to the ability to overcome critical moments and adapt after experiencing some unusual and unexpected situation. It also indicates return to normal." (José G Vargas-Hernández, "Urban Socio-Ecosystems Green Resilience", 2021)

"Operational resilience is a set of techniques that allow people, processes and informational systems to adapt to changing patterns. It is the ability to alter operations in the face of changing business conditions. Operationally resilient enterprises have the organizational competencies to ramp up or slow down operations in a way that provides a competitive edge and enables quick and local process modification." (Gartner)

[Operational resilience:] "The ability of an organization to absorb the impact of any unexpected event without failing to deliver on its brand promise." (Forrester)

[Business resilience:] "The ability to thrive in the face of unpredictable events and circumstances without deteriorating customer experience or sacrificing the long-term viability of the company." (Forrester)

29 March 2016

♜Strategic Management: Decision-Making (Definitions)

[decision-making:] "The process of making choices in a project team environment. Several types of decision-making are useful in projects: consensus, leader-imposed, delegated, voting, and scoring models." (Timothy J  Kloppenborg et al, "Project Leadership", 2003)

[semistructured decisions:] "Decisions in which only some of the phases are structured; require a combination of standard solution procedures and individual judgment." (Linda Volonino & Efraim Turban, "Information Technology for Management 8th Ed", 2011)

[strategic decision:] "refers to a decision that exhibits the following characteristics: it is made in a situation of uncertainty, of incomplete information, in a complex environment, variable/mutating environment (as opposed to 'all things being otherwise equal'); it is not recurrent, therefore the decision maker is relatively deprived; it may have far-reaching (favorable or adverse) consequences that could jeopardize the survivability of the enterprise; it is systemic (many elements with many relationships among them); the decision maker does not have experience-proven models (we cannot resort to 'turnkey' mechanisms). " (Humbert Lesca & Nicolas Lesca, "Weak Signals for Strategic Intelligence: Anticipation Tool for Managers", 2011)

[strategic decisions:] "Decisions for sustained enterprise success and business growth." (Linda Volonino & Efraim Turban, "Information Technology for Management 8th Ed", 2011)

[tactical decisions:] "Decisions ensuring that existing operations and processes are in alignment with business objectives and strategies." (Linda Volonino & Efraim Turban, "Information Technology for Management 8th Ed", 2011)

[decision-making processes:] "Management processes that define objectives, study alternatives, analyze available data, and reflect on intuitive beliefs. They interpret findings and compare alternates to form a conclusion or make a choice upon which the organization may act." (Carl F Lehmann, "Strategy and Business Process Management", 2012)

[microdecision:] "A small decision made many times by many workers at the front line of the organization. They usually have a significant impact on organizational performance due to their sheer volume." (Evan Stubbs, "Delivering Business Analytics: Practical Guidelines for Best Practice", 2013)

[decision-making:] "How decisions are made, based on what types of resources, information, and specific processes are available." (Jim Davis & Aiman Zeid, "Business Transformation: A Roadmap for Maximizing Organizational Insights", 2014)

[decision-making] "the process of making choices or reaching conclusions, especially on important political or business matters." (Ken Sylvester, "Negotiating in the Leadership Zone", 2015)

[tactical decisions:] "broader decision questions than operational ­decisions, semistructured in nature, some but not all information ­necessary to make the decision is available, primarily internally focused and made by middle-level managers." (Daniel J. Power & Ciara Heavin, "Data-Based Decision Making and Digital Transformation", 2018)

[operating or function-specific decisions:] "day-to-day, routine ­decisions with a concise decision question and a clear, well-defined, and structured algorithm to make a choice among alternatives." (Daniel J. Power & Ciara Heavin, "Data-Based Decision Making and Digital Transformation", 2018)

[strategic decisions:] "complex, nonroutine, unstructured decisions involving many different and connected parts. Some variables may not be well understood, often information required to make the decision may be unavailable, incomplete, and in some situations information may be known to be flawed or inaccurate. These decisions usually involve a high degree of uncertainty about outcomes. If implemented, strategic ­decisions often result in major changes in an organization." (Daniel J. Power & Ciara Heavin, "Data-Based Decision Making and Digital Transformation", 2018)

03 October 2015

🪙Business Intelligence: Operations (Just the Quotes)

"Unfortunately, just collecting the data in one place and making it easily available isn’t enough. When operational data from transactions is loaded into the data warehouse, it often contains missing or inaccurate data. How good or bad the data is a function of the amount of input checking done in the application that generates the transaction. Unfortunately, many deployed applications are less than stellar when it comes to validating the inputs. To overcome this problem, the operational data must go through a 'cleansing' process, which takes care of missing or out-of-range values. If this cleansing step is not done before the data is loaded into the data warehouse, it will have to be performed repeatedly whenever that data is used in a data mining operation." (Joseph P Bigus,"Data Mining with Neural Networks: Solving business problems from application development to decision support", 1996)

"[…] dirt and stains are more noticeable on white or light-colored clothing. In the same way, dirty data and data quality issues have existed for a long time. But due to the inherent nature of operational data these issues have not been as visible or immense enough to affect the bottom line. Just as dark clothing hides spills and stains, dirty data have been hidden or ignored in operational data for decades." (Margaret Y Chu, "Blissful Data", 2004)

"Gauging the quality of the operational data becomes an important first step in predicting potential dirty data issues for an organization. But many organizations are reluctant to commit the time and expense to assess their data. Some organizations wait until dirty data issues blow up in their faces. The greater the pain being experienced, the bigger the commitment to improving data quality." (Margaret Y Chu, "Blissful Data", 2004)

"Processes must be implemented to prevent bad data from entering the system as well as propagating to other systems. That is, dirty data must be intercepted at its source. The operational systems are often the source of informational data; thus dirty data must be fixed at the operational data level. Implementing the right processes to cleanse data is, however, not easy." (Margaret Y Chu, "Blissful Data", 2004)

"Variance establishes the comparison benchmark for each KPI. It has two requirements: (1) the basis for change and (2) change calculation. The most commonly applied references for the basis are relative periodic comparisons: year ago, quarter ago, and month ago. Other types of change basis are forecast, operational plan, quota, and so on. The most commonly applied values for change calculations are Difference, Percentage Change, and Percent Point Change." (Shadan Malik, "Enterprise Dashboards: Design and best practices for IT", 2005)

"There are four levels of data in the architected environment - the operational level, the atomic (or the data warehouse) level, the departmental (or the data mart) level, and the individual level. These different levels of data are the basis of a larger architecture called the corporate information factory (CIF). The operational level of data holds application-oriented primitive data only and primarily serves the high-performance transaction-processing community. The data-warehouse level of data holds integrated, historical primitive data that cannot be updated. In addition, some derived data is found there. The departmental or data mart level of data contains derived data almost exclusively. The departmental or data mart level of data is shaped by end-user requirements into a form specifically suited to the needs of the department. And the individual level of data is where much heuristic analysis is done." (William H Inmon, "Building the Data Warehouse" 4th Ed., 2005)

"Metrics can serve two purposes: identifying problems and measuring performance. When the goal is to identify problems and pinpoint areas of operational inefficiency and ineffectiveness, defining the right metric requires a bit of detective work. It requires you to uncover the data residue of a problem and to determine what evidence can be found and how exactly it shows up. When the goal is to measure performance, the right success metrics focus on measures that can be controlled and where improvement in the metric is an unambiguously good thing." (Zach Gemignani et al, "Data Fluency", 2014)

"Whereas a data warehouse combines databases across an entire enterprise, a data mart is usually smaller and focuses on a particular subject or department. A data mart is a subset of a data warehouse, typically consisting of a single subject area (e.g., marketing, operations). A data mart can be either dependent or independent. A dependent data mart is a subset that is created directly from the data warehouse. It has the advantages of using a consistent data model and providing quality data. [...] An independent data mart is a small warehouse designed for a strategic business unit (SBU) or a department, but its source is not an EDW." (Ramesh Sharda et al, "Business Intelligence: A Managerial Perspective on Analytics" 3rd Ed., 2014)

"Data strategy is even less understood [thank business strategy], so the chances of success can be further decreased, simply because you need organisation-wide commitment and buy-in to succeed. Data does not exist in a bubble; it is not the preserve of a function that can fix it for all, detached from touching everyone else. It is core to how you run the organisation, and without a focus on where you are heading, it is going to trip the organisation up at every turn - regulatory compliance; operational effectiveness; financial performance; customer and employee experience; essentially, the efficiency in managing virtually every activity in the organisation." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"A data architecture needs to have the robustness and ability to support multiple data management and operational models to provide the necessary business value and agility to support an enterprise’s business strategy and capabilities." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"A data mesh is inherently multimodal, and data products can be provided via a variety of means. Event streams remain the best option for the majority of data products, as it is far easier to power both operational and analytical use cases through a stream than a batch of files at rest." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"Data Fabric is a distributed data architecture that connects scattered data across tools and systems with the objective of providing governed access to fit-for-purpose data at speed. Data Fabric focuses on Data Governance, Data Integration, and Self-Service data sharing. It leverages a sophisticated active metadata layer that captures knowledge derived from data and its operations, data relationships, and business context. Data Fabric continuously analyzes data management activities to recommend value-driven improvements. Data Fabric works with both centralized and decentralized data systems and supports diverse operational models." (Sonia Mezzetta, "Principles of Data Fabric: Become a data-driven organization by implementing Data Fabric solutions efficiently", 2023)

"Data products should remain stable and be decoupled from the operational/transactional applications. This requires a mechanism for detecting schema drift, and avoiding disruptive changes. It also requires versioning and, in some cases, independent pipelines to run in parallel, giving your data consumers time to migrate from one version to another." (Piethein Strengholt, "Data Management at Scale: Modern Data Architecture with Data Mesh and Data Fabric" 2nd Ed., 2023)

"Over 80% of models are never operationalized because the efforts involved in deploying them are enormous and the models are deployed and found to produce drift or fairness issues that outweigh the benefits."  (Eberhard Hechler et al, "Data Fabric and Data Mesh Approaches with AI", 2023)

"Choosing the right data ingestion strategy is a significant business decision that partially determines how well your organization can leverage its data for business decision making and operations. The stakes are high; the wrong strategy can lead to poor data quality, performance issues, increased costs, and even regulatory compliance breaches." (James Serra, "Deciphering Data Architectures", 2024)

Related Posts Plugin for WordPress, Blogger...

About Me

My photo
Koeln, NRW, Germany
IT Professional with more than 25 years experience in IT in the area of full life-cycle of Web/Desktop/Database Applications Development, Software Engineering, Consultancy, Data Management, Data Quality, Data Migrations, Reporting, ERP implementations & support, Team/Project/IT Management, etc.