📊Business Intelligence: Data Ingestion (Defintions)

"Data ingestion is the first step in the data engineering lifecycle. It involves gathering data from diverse sources such as databases, SaaS applications, file sources, APIs and IoT devices into a centralized repository like a data lake, data warehouse or lakehouse. This enables organizations to clean and unify the data to leverage analytics and AI for data-driven decision-making." (Databricks) [link]

"Data ingestion is the import and collection of data from databases, APIs, sensors, logs, files, or other sources into a centralized storage or computing system. Data ingestion and transformation renders massive collections of data accessible and usable for analysis, processing, and visualization. It’s a fundamental step in data management and analytics workflows, enabling organizations to glean insights from their data." (ScyllaDB) [link] 

"Data ingestion is the process of collecting data from one or more sources and loading it into a staging area or object store for further processing and analysis. Ingestion is the first step of analytics-related data pipelines, where data is collected, loaded and transformed for insights." (Fivetran) [link] 

"Data ingestion is the process of collecting and importing data files from various sources into a database for storage, processing and analysis." (IBM) [link]

"Data ingestion is the process of transporting data from one or more sources to a target site for further processing and analysis. This data can originate from a range of sources, including data lakes, IoT devices, on-premises databases, and SaaS apps, and end up in different target environments, such as cloud data warehouses or data marts." (Striim) [link] 

"Data ingestion is the process of importing large, assorted data files from multiple sources into a single, cloud-based storage medium - a data warehouse, data mart or database - where it can be accessed and analyzed." (Cognizant) [link] 

"Data ingestion is the process of moving and replicating data from data sources to destination such as a cloud data lake or cloud data warehouse." (Informatica) [link] 

"Data ingestion refers to the tools & processes used to collect data from various sources and move it to a target site, either in batches or in real-time." (Qlik) [link]

"Data ingestion refers to collecting and importing data from multiple sources and moving it to a destination to be stored, processed, and analyzed." (Teradata) [link] 

"The process of obtaining, importing, and processing data for later use or storage in a database. This process often involves altering individual files by editing their content and/or formatting them to fit into a larger document. An effective data ingestion methodology begins by validating the individual files, then prioritizes the sources for optimum processing, and finally validates the results. When numerous data sources exist in diverse formats (the sources may number in the hundreds and the formats in the dozens), maintaining reasonable speed and efficiency can become a major challenge. To that end, several vendors offer programs tailored to the task of data ingestion in specific applications or environments.' (CODATA)

🧭Business Intelligence: Perspectives (Part 33: Data Lifecycle for Analytics)

Business Intelligence Series

In the context of BI, Analytics and other data-related topics, the various parties usually talk about data ingestion, preparation, storage, analysis and visualization, often ignoring processes like data generation, collection, and interpretation. It’s also true that a broader discussion may shift the attention unnecessarily, though it’s important to increase people’s awareness in respect to data’s full lifecycle. Otherwise, many of the data solutions become a mix of castles built into the air, respectively structures of cards waiting for the next flurry to be blown away. 

Data is generated continuously by organizations, their customers, vendors, and third parties, as part of a complex network of processes, systems and integrations that extend beyond their intended boundaries. Independently of their type, scope and various other characteristics, all processes consume and generate data at a rapid pace that steadily exceeds organizations’ capabilities to make good use of it.

There are also scenarios in which the data must be collected via surveys, interviews, forms, measurements or direct observations, and whatever processes are used to elicit some aspect of importance. The volume and other characteristics of data generated in this way may depend on the goals and objectives in scope, respectively the methods, procedures and even the methodologies used. 

Data ingestion is the process of importing data from the various sources into a central or intermediary repository for storage, processing, analysis and visualization. The repository can be a data mart, warehouse, lakehouse, data lake or any other destination intended for the intermediary or the final intended destination of data. Moreover, data can have different levels of quality in respect to its intended usage.

Data storage refers to the systems and approaches used to securely retain, organize, and access data throughout its journey within the various layers of the infrastructure. It focuses on where and how data is stored, independently on whether that’s done on-premises, in the cloud or across hybrid environments.

Data preparation is the process of transforming the data into a form close to what is intended for analysis and visualization. It may involve data aggregation, enrichment, transposition and other operations that facilitate further steps. It’s probably the most important step in a data project given that the final outcome can have an important impact on data analysis and visualization, facilitating or impeding the respective processes. 

Data analysis consists of a multitude of processes that attempt to harness value from data in its various forms of aggregation. The ultimate purpose is to infer meaningful information, respectively knowledge from the data augmented as insights. The road from raw data to these targeted outcomes is a tedious one, where recipes can help and imped altogether. Expecting value from any pile of data can easily become a costly illusion when data, processes and their usage is poorly understood and harnessed. 

Data visualization is the means of presenting data and its characteristics in the form of figures, diagrams and other forms of representation that facilitate data’s navigation, perception and understanding for various purposes. Usually, the final purpose is fact-checking, decision-making, problem-solving, etc., though there is a multitude of steps in between. Especially in these areas there are mixed good and poor practices altogether.  

Data interpretation is the attempt of drawing meaningful conclusions from the data, information and knowledge gained mainly from data analysis and visualization. It is often a subjective interpretation as it’s usually regarded from people’s understanding of the various facts as they are considered. The inferences made in the process can be a matter of gut feeling, respectively of mature analysis. It’s about sense-making, contextualization, critical thinking, pattern recognition, internalization and externalization, and other similar cognitive processes.

Previous Post <<||>> Next Post