21 November 2026

🪙Business Intelligence: ETL/ELT (Just the Quotes)

"The desire to collect as much data as possible must be balanced with an approximation of which data sources are useful to address a business issue. It is worth mentioning that often the value of internal data is high. Most internal data has been cleansed and transformed to suit the mission. It should not be overlooked simply because of the excitement of so much other available data." (Mike Fleckenstein & Lorraine Fellows, "Modern Data Strategy", 2018)

"A data pipeline is a series of transformation steps (functions) executed as the data flows from one step to another. Data mesh refrains from using pipelines as a top-level architectural paradigm and in between data products. The challenge with pipelines as currently used is that they don’t create clear interfaces, contracts, and abstractions that can be maintained easily as the pipeline complexity complexity grows. Due to lack of abstractions, single failure in the pipeline causes cascading failures." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"A data product’s primary job is to consume data from upstream sources using its input data ports, transform it, and serve the result as permanently accessible data via its output data ports." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"In the case of data mesh, a data product is an architectural quantum. It is the smallest unit of architecture that can be independently deployed and managed. It has high functional cohesion, i.e., performing a specific analytical transformation and securely sharing the result as domain-oriented analytical data. It has all the structural components that it requires to do its function: the transformation code, the data, the metadata, the policies that govern the data, and its dependencies to infrastructure." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"A data pipeline is an artifact of a data engineering process. It transforms raw data into data ready for analytics. These in turn help solve problems, aid support decisions, and make our lives more convenient. In some ways, it can be thought of as the stitch between the OLTP and OLAP systems. Data pipelines are sometimes referred to as ETL, which stands for extract, transform, load, and it has a variation called extract, load, transform (ELT). The main difference between the two is whether the incoming data is first saved to disk and then transformed (data wrangling) or vice versa. The processing is loosely referred to as ETL. Although, it is fair to say ELT is relevant in the context of Data Lakes and unstructured data, whereas ETL is used for Data Warehouses." (Anindita Mahapatra, "Simplifying Data Engineering and Analytics with Delta", 2022)

"Any change in the reporting requirement had to go through a long-winded process of data model changes, ETL code changes, and respective changes to the reporting system. Often, the ETL process was a specialized skill and became a bottleneck for reducing data to insight turnover time. The nature of analytics is unique. The more you see the output, the more you demand. Many EDW projects were deemed a failure. The failure was not from a technical perspective, but from a business perspective. Operationally, the design changes required to cater to these fast-evolving requirements were too difficult to handle." (Pradeep Menon, "Data Lakehouse in Action", 2022)

"Historically, for their analytics needs, enterprises relied upon a set of tightly coupled tools, typically provided by a single vendor. Nowadays, nearly all of the components of a traditional data warehouse are independent and interchangeable. Those independent tools can be flexibly combined to provide a modern data stack. It is common for current enterprises to have separate tools for data ingestion, data pipelines, data storage and querying, data visualization and business intelligence, and data quality. Furthermore, data can flow in the opposite direction out of the data warehouse in what is referred to as reverse extract, transform, and load (ETL)." (Fadi Maali & Jason Lim, "Implementing a Modern Data Catalog to Power Data Intelligence: Make Trustworthy Data Central to Your Organization", 2022)

"Traditional data stores used for analytics, such as data marts and data ware-houses, followed an ETL process, extract first by making a copy from the source, then transformations are made upon this copy, and then the data is loaded into the target system. ETL tools require processing engines for running transformations prior to loading data into a destination. Running these engines performing transformations before the load phase results in a more complex data replication process." (Scott Burk et al, It’s All Analytics - Part II: Designing an Integrated AI, Analytics, and Data Science Architecture for Your Organization, 2022)

"An enterprise data warehouse is a central repository of integrated and transformed, structured data from disparate sources and used for reporting and data analysis." (Eberhard Hechler et al, "Data Fabric and Data Mesh Approaches with AI", 2023)

"A data lake is a distributed repository of raw and unprocessed data stored in its original format, with-out a predefined schema or structure. A data lake is designed to support a wide range of data types, sources, and use cases, such as exploration, discovery, and data experimentation. A data lake follows a 'schema on read' approach. Data is structured and processed only when it is accessed or consumed by a user or application (Extract, Load, Transform (ELT)). A data lake also enables data democratiza-tion, meaning data is accessible and available to anyone who needs it, without barriers or restrictions." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"A data warehouse is a centralized repository of structured, cleaned, and verified data that has been extracted, transformed, and loaded from various sources. These steps are commonly called ETL, which stands for Extract, Transform, Load. This data processing methodology involves extracting data from multiple sources, transforming it to meet business needs, and loading it into a destination for analysis and consultation." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Data transformation sits at the heart of every successful data platform, serving as the critical bridge between data ingestion and data consumption. While basic transformations might involve simple cleaning and formatting, advanced transformation techniques encompass complex operations such as data enrichment, sophisticated deduplication, machine learning-based predictions, and the creation of derived metrics that weren’t present in the original data sources. These processes are essential for organizations looking to extract maximum value from their data investments." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"It should be noted that, unlike Dataflow Gen2, in pipelines, it is not mandatory to enable staging to load data into a warehouse. Indeed, pipelines are designed for more general orchestration scenarios where you can combine various activities such as transformations, API calls, and so on to create complex workflows. They are not specifically focused on data preparation but rather on end-to-end process automation. Pipelines are more flexible and used for a variety of orchestration tasks, whereas Dataflow Gen2 is specifically designed for data preparation and transformation, hence the requirement for staging in that case." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"The transformation phase represents the most resource-intensive stage of most data projects, often consuming 60-80% of total project time and effort. This significant investment stems from the inherent complexity of converting raw, inconsistent data into clean, structured, and enriched information ready for business use. Every data quality issue must be identified and resolved, every business rule must be correctly implemented, and every integration point must be properly validated. This meticulous work serves as the essential bridge between raw data ingestion and meaningful business insights." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)

"Data may indeed be the new oil. But just like crude oil, data needs refining. It must be transformed into information. This is why we clean, combine, model, and visualize data. The output of all this work - whether you do it on your own, get some help, or use a (semi-)automatic process - includes reports and dashboards that provide insights into various aspects of the organization’s dealings, which decision-makers can then consume to make critical business decisions." (Jeroen ter Heerdt et al, "Microsoft Power BI Visual Calculations: Simplifying DAX", 2026)

No comments:

Related Posts Plugin for WordPress, Blogger...

About Me

My photo
Koeln, NRW, Germany
IT Professional with more than 25 years experience in IT in the area of full life-cycle of Web/Desktop/Database Applications Development, Software Engineering, Consultancy, Data Management, Data Quality, Data Migrations, Reporting, ERP implementations & support, Team/Project/IT Management, etc.