🔢Saurabh Gupta - Collected Quotes

"A data warehouse follows a pre-built static structure to model source data. Any changes at the structural and configuration level must go through a stringent business review process and impact analysis. Data lakes are very agile. Consumption or analytical layer can be modified to fit in the model requirements. Consumers of a data lake are not constant; therefore, schema and modeling lies at the liberty of analysts and scientists." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Data in the data lake should never get disposed. Data driven strategy must define steps to version the data and handle deletes and updates from the source systems." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Data governance policies must not enforce constraints on data - Data governance intends to control the level of democracy within the data lake. Its sole purpose of existence is to maintain the quality level through audits, compliance, and timely checks. Data flow, either by its size or quality, must not be constrained through governance norms. [...] Effective data governance elevates confidence in data lake quality and stability, which is a critical factor to data lake success story. Data compliance, data sharing, risk and privacy evaluation, access management, and data security are all factors that impact regulation." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Data Lake induces accessibility and catalyzes availability. It warrants data discovery platforms to soak the data trends at a horizontal scale and produce visual insights. It largely cuts down the time that goes into data preparation and exhaustive data analysis." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Data Lake is a single window snapshot of all enterprise data in its raw format, be it structured, semi-structured, or unstructured. Starting from curating the data ingestion pipeline to the transformation layer for analytical consumption, every aspect of data gets addressed in a data lake ecosystem. It is supposed to hold enormous volumes of data of varied structures." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Data lake is an ecosystem for the realization of big data analytics. What makes data lake a huge success is its ability to contain raw data in its native format on a commodity machine and enable a variety of data analytics models to consume data through a unified analytical layer. While the data lake remains highly agile and data-centric, the data governance council governs the data privacy norms, data exchange policies, and the ensures quality and reliability of data lake." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Data swamp, on the other hand, presents the devil side of a lake. A data lake in a state of anarchy is nothing but turns into a data swamp. It lacks stable data governance practices, lacks metadata management, and plays weak on ingestion framework. Uncontrolled and untracked access to source data may produce duplicate copies of data and impose pressure on storage systems." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Data warehousing, as we are aware, is the traditional approach of consolidating data from multiple source systems and combining into one store that would serve as the source for analytical and business intelligence reporting. The concept of data warehousing resolved the problems of data heterogeneity and low-level integration. In terms of objectives, a data lake is no different from a data warehouse. Both are primary advocates of terms like 'single source of truth' and 'central data repository'." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"Metadata is the key to effective data governance. Metadata in this context is the data that defines the structure and attributes of data. This could mean data types, data privacy attributes, scale, and precision. In general, quality of data is directly proportional to the amount and depth of metadata provided. Without metadata, consumers will have to depend on other sources and mechanisms." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

"The quality of data that flows within a data pipeline is as important as the functionality of the pipeline. If the data that flows within the pipeline is not a valid representation of the source data set(s), the pipeline doesn’t serve any real purpose. It’s very important to incorporate data quality checks within different phases of the pipeline. These checks should verify the correctness of data at every phase of the pipeline. There should be clear isolation between checks at different parts of the pipeline. The checks include checks like row count, structure, and data type validation." (Saurabh Gupta et al, "Practical Enterprise Data Lake Insights", 2018)

🔢Aniruddha Deswandikar - Collected Quotes

"A data contract is a definition of how two parties (the producer and the consumer) must exchange data. It defines the structure, format, service level agreement, sensitivity, and any other information that could be important for the producer or the consumer of the data." (Aniruddha Deswandikar, "Engineering Data Mesh in Azure Cloud", 2024)

"A data mesh is an architectural pattern implemented on top of a standardized enterprise cloud infrastructure." (Aniruddha Deswandikar,"Engineering Data Mesh in Azure Cloud", 2024)

"A data mesh splits the boundaries of the exchange of data into multiple data products. This provides a unique opportunity to partially distribute the responsibility of data security. Each data product team can be made responsible for how their data should be accessed and what privacy policies should be applied." (Aniruddha Deswandikar,"Engineering Data Mesh in Azure Cloud", 2024)

"A data quality solution cannot be taken up as one large project. It needs to be built brick by brick. Implement the easy checks first and then tackle the complex quality requirements." (Aniruddha Deswandikar, "Engineering Data Mesh in Azure Cloud", 2024)

"Accuracy questions whether the value of the data is as it should be. [...] Completeness indicates whether all the necessary data has been included. [...] Consistency means that the data is consistent across systems. [...] Timeliness is about how recent the data is. All data has an original source. [...] Validity refers to whether the data is valid. Data must adhere to some business rules.  [...] Uniqueness means that the data should only appear once in a dataset. [...] Reliability ensures that data is being collected from the source of truth or from another verified data source." (Aniruddha Deswandikar, "Engineering Data Mesh in Azure Cloud", 2024)

"Authentication means validating a user by using credentials to ensure that they are a valid user on the enterprise system. Authorization validates their rights to access a particular resource or perform certain operations on it. [...] Authorization is the process of granting or denying a set of actions that can be performed on a resource based on a set of permissions." (Aniruddha Deswandikar, "Engineering Data Mesh in Azure Cloud", 2024)

"Data security is about preventing unauthorized access to data and the policies and methods surrounding this access. It also protects the system from hackers and malicious users who could steal data. Data privacy, on the other hand, is about collecting, retaining, and recycling personal and sensitive data. There could be a few overlaps between security and privacy." (Aniruddha Deswandikar, "Engineering Data Mesh in Azure Cloud", 2024)

"Federation is about providing autonomy to each data product owner to make their own decisions about the storage, computing, and sharing of data. However, this autonomy cannot come at a risk to the security and compliance standards of the company." (Aniruddha Deswandikar, "Engineering Data Mesh in Azure Cloud", 2024)

"To explain a data mesh in one sentence, a data mesh is a centrally managed network of decentralized data products. The data mesh breaks the central data lake into decentralized islands of data that are owned by the teams that generate the data. The data mesh architecture proposes that data be treated like a product, with each team producing its own data/output using its own choice of tools arranged in an architecture that works for them. This team completely owns the data/output they produce and exposes it for others to consume in a way they deem fit for their data." (Aniruddha Deswandikar, "Engineering Data Mesh in Azure Cloud", 2024)

🎯C S V Murthy - Collected Quotes

"[a scatter diagram] is a graph in which the values of two variables are plotted along two axes, the pattern of the resulting points revealing any correlation present. It graphs pairs of numerical data, with one variable on each axis, to look for a relationship between them. If the variables are correlated, the points will fall along a line or curve. The better the correlation, the tighter the points will have the line." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Decision tree is a graphical representation of a decision situation in which decision situation points (nodes) are connected together by arcs (one for each alternative on a decision) and terminate in ovals (the action that is the result of all the decisions made on the path leading to that oval). [...] A tree is made up of multilevel group of elements called nodes. A node is nothing more than a point at which subsidiary data originate. This particular logical data structure is called a tree simply because it looks like a tree, usually turned upside down. Genealogists use a schema called a tree to show ancestral descent of a person, family or group. Data associated by a tree schema are hierarchical. They branch from a point or node without forming loops or polygons. Data presented in a tree structure make two conditions. First, the tree must have a single root node. Second, all nodes other than the root node must be related to one and only one higher level node." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Every interaction includes both presentation and dialogue. Presentation provides the layout of information on a computer screen. Dialogue provides an interaction sequence between a user and computer. Interfaces and dialogue will help users to solve their problems. Presentation must include objects that the user can readily understand in terms of their daily work. The dialogue must correspond to user’s normal work and to their mental model of the system (Mental model is the way a user sees a problem). Both presentation/dialogue depend on what users are doing." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Information relevance refers to the extent to which information is appropriate for the decision-making situation facing the manager. Extraneous or extra information distracts the decision-maker from the assigned task and information overload frustrates the decision-maker and impairs the decision-making process. Relevant information must pertain to the problems, decisions and responsibilities of the recipient." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Information that is complete means information that covers key issues and is sufficient to support the decision-making situation at hand without critical omissions. The more complete a body of information, is obviously, the more expensive it is to develop and maintain. Care must also be taken not to provide extra information than needed, due to its expense, and not to provide so much information that the recipient will suffer from information overload (information indigestion)." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Ridge Regression is a technique for analysing multiple regression data that suffer from multicollinearity. When multicollinearity occurs, least squares estimates are unbiased, but their variances are large. So, they may be far from the true value. By adding a degree of bias to the regression estimates, principal components regression reduces the standard errors." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Spectral methods are a class of techniques used in applied mathematics and scientific computing to numerically solving certain differential equations, potentially involving the use of the fast Fourier transform. This is an algorithm that samples a signal over a period of time and divides it into its frequency components. These components are single sinusoidal oscillations at distant frequencies each with their own amplitude and phase." (C S V Murthy, "Data and Businesss Analytics", 2020)

"The concept of programmed decisions is important because the ultimate (and unachievable) goal of information systems is to provide purely programmed decisions. Because this is not possible, we seek to provide the optimum type of information to the human decision-maker, who then makes non-programmable decisions. Decisions lend themselves to programming techniques if they are repetitive and routine, and if a procedurs can be worked out for handling them so that each is neither an ad hoc decision nor one to be treated as a new situation each time it arises." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Timeliness means that information is available when it is needed. Most managers function in a dynamic environment of change, demands updated and current information. Computerised information systems have the ability to gather, sort, analyse, store, retrieve, and transmit large amounts of information in a very short period of time. Completeness of information is the extent to which information is all there." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Understanding complex information systems begins with a clear understanding of information and its general characteristics. Information can be considered as the very blood of an organisation, but it must be properly understood and appropriately distinguished from data. Too many times, the terms ‘data’ and ‘information’ are used interchangeably, but the terms most clearly mean entirely different things. Data should be defined as raw, unsummarised and unanalysed facts. Information is data that has been presented in such a way as to alter the receiver’s understanding. Data are the raw materials from which information is derived. This is a necessary distinction for the manager to make, because loads of data can be generated, without producing even one iota of useful information." (C S V Murthy, "Data and Businesss Analytics", 2020)

"Visualisation is any technique for creating images, diagrams or animations to communicate a message; techniques used to communicate data or information by encoding it as visual objects, e.g., points, lines or bars contained in graphics. One of the most important benefits of visualisation is that it allows us visual access to huge amounts of data in easily digestible visuals. Well designed data graphics are usually the simplest, and at the same time, the most powerful." (C S V Murthy, "Data and Businesss Analytics", 2020) 

🎯William H Inmon - Collected Quotes

"There are four levels of data in the architected environment - the operational level, the atomic (or the data warehouse) level, the departmental (or the data mart) level, and the individual level. These different levels of data are the basis of a larger architecture called the corporate information factory (CIF). The operational level of data holds application-oriented primitive data only and primarily serves the high-performance transaction-processing community. The data-warehouse level of data holds integrated, historical primitive data that cannot be updated. In addition, some derived data is found there. The departmental or data mart level of data contains derived data almost exclusively. The departmental or data mart level of data is shaped by end-user requirements into a form specifically suited to the needs of the department. And the individual level of data is where much heuristic analysis is done." (William H Inmon, "Building the Data Warehouse" 4th Ed., 2005)

"To interpret and understand information over time, a whole new dimension of context is required. While content of information remains important, the comparison and understanding of information over time mandates that context be an equal partner to content. And in years past, context has been an undiscovered, unexplored dimension of information." (William H Inmon, "Building the Data Warehouse" 4th Ed., 2005)

"When management receives the conflicting reports, it is forced to make decisions based on politics and personalities because neither source is more or less credible. This is an example of the crisis of data credibility in the naturally evolving architecture." (William H Inmon, "Building the Data Warehouse" 4th Ed., 2005)

"An interesting aspect of KPIs are that they change over time. At one moment in time the organization is interested in profitability. There will be one set of KPIs that measure profitability. At another moment in time the organization is interested in market share. There will be another set of KPIs that measure market share. As the focus of the corporation changes over time, so do the KPIs that measure that focus." (William H Inmon & Daniel Linstedt, "Data Architecture: A Primer for the Data Scientist: Big Data, Data Warehouse and Data Vault", 2015)

"Both the ODS and a data warehouse contain subject-oriented, integrated information. In that regard they are similar. But an ODS contains data that can be individually updated, deleted, or added. And a data warehouse contains nonvolatile data. A data warehouse contains snapshots of data. Once the snapshot is taken, the data in the data warehouse does not change. So when it comes to volatility, a data warehouse and an ODS are very different." (William H Inmon & Daniel Linstedt, "Data Architecture: A Primer for the Data Scientist: Big Data, Data Warehouse and Data Vault", 2015)

"In general, analytic processing is known as 'heuristic' processing. In heuristic processing the requirements for analysis are discovered by the results of the current iteration of processing. […] In heuristic processing you start with some requirements. You build a system to analyze those requirements. Then, after you have results, you sit back and rethink your requirements after you have had time to reflect on the results that have been achieved. You then restate the requirements and redevelop and reanalyze again. Each time you go through the redevelopment exercise is called an 'iteration'. You continue the process of building different iterations of processing until such time as you achieve the results that satisfy the organization that is sponsoring the exercise." (William H Inmon & Daniel Linstedt, "Data Architecture: A Primer for the Data Scientist: Big Data, Data Warehouse and Data Vault", 2015)

"There are, however, many problems with independent data marts. Independent data marts: (1) Do not have data that can be reconciled with other data marts (2) Require their own independent integration of raw data (3) Do not provide a foundation that can be built on whenever there are future analytical needs." (William H Inmon & Daniel Linstedt, "Data Architecture: A Primer for the Data Scientist: Big Data, Data Warehouse and Data Vault", 2015)

"There is then a real mismatch between the volume of data and the business value of data. For people who are examining repetitive data and hoping to find massive business value there, there is most likely disappointment in their future. But for people looking for business value in nonrepetitive data, there is a lot to look forward to." (William H Inmon & Daniel Linstedt, "Data Architecture: A Primer for the Data Scientist: Big Data, Data Warehouse and Data Vault", 2015)

"A defining characteristic of the data lakehouse architecture is allowing direct access to data as files while retaining the valuable properties of a data warehouse. Just do both!" (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"At first, we threw all of this data into a pit called the 'data lake'. But we soon discovered that merely throwing data into a pit was a pointless exercise. To be useful - to be analyzed - data needed to (1) be related to each other and (2) have its analytical infrastructure carefully arranged and made available to the end user. Unless we meet these two conditions, the data lake turns into a swamp, and swamps start to smell after a while. [...] In a data swamp, data just sits there are no one uses it. In the data swamp, data just rots over time." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"Data privacy, data confidentiality, and data protection are sometimes incorrectly diluted with security. For example, data privacy is related to, but not the same as, data security. Data security is concerned with assuring the confidentiality, integrity, and availability of data. Data privacy focuses on how and to what extent businesses may collect and process information about individuals." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"Data visualization adds credibility to any message. [...] Data visualizations are incredibly cold mediums because they require a lot of interpretation and participation from the audience. While boring numbers are authoritative, data visualization is inclusive. [...] Data visualizations absorb the viewer in the chart and communicate the author’s credibility through active participation. Like a good teacher, they walk the reader through the thought process and convince him/her effortlessly."

"Data visualization‘s key responsibilities and challenges include the obligation to earn your audience’s attention - do not take it for granted." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"In general, a data or data set contains its sensitivity or controversial nature only if it is linked or related to an individual’s personal information. Else an isolated, abandoned, or unrelated sensitive or controversial attribute has no significance." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"It is dangerous to do an analysis and merge data with very different quality profiles. As a general rule, the veracity of merged data is only as good as the worst data that has been merged. [...] Not knowing the quality of the data being analyzed jeopardizes the entire analysis." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"Once you combine the data lake along with analytical infrastructure, the entire infrastructure can be called a data lakehouse. [...] The data lake without the analytical infrastructure simply becomes a data swamp. And a data swamp does no one any good." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"The data lakehouse architecture presents an opportunity comparable to the one seen during the early years of the data warehouse market. The unique ability of the lakehouse to manage data in an open environment, blend all varieties of data from all parts of the enterprise, and combine the data science focus of the data lake with the end user analytics of the data warehouse will unlock incredible value for organizations. [...] "The lakehouse architecture equally makes it natural to manage and apply models where the data lives." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"Raw data without appropriate visualization is like dumped construction raw materials at a building construction site. The finished house is the actual visuals created from those data like raw materials." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

"With the data lakehouse, it is possible to achieve a level of analytics and machine learning that is not feasible or possible any other way. But like all architectural structures, the data lakehouse requires an understanding of architecture and an ability to plan and create a blueprint." (Bill Inmon et al, "Building the Data Lakehouse", 2021)

🔢Angelika Klidas - Collected Quotes

"Also, remember that data literacy is not just a set of technical skills. There is an equal need and weight for soft skills and business skills. This can be misleading for some technical resources within an organization, as those technical resources may believe they are data literate by default as they are data architects or data analysts. They have the existing technical skills, but maybe they do not have any deep proficiencies in other skills such as communicating with data, challenging assumptions, and mitigating bias, or perhaps they do not have an open mindset to be open to different perspectives." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"Critical thinking is part of data literacy; it is the ability to question the logic of arguments or assumptions and examine evidence in order to determine whether a claim is true, false, or uncertain." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"Current decision-making in business suffers from insight gaps. Organizations invest in data and analytics, hoping that will provide them with insights that they can use to make decisions, but in reality, there are many challenges and obstacles that get in the way of that process. One of the biggest challenges is that these organizations tend to focus on technology and hard skills only. They are definitely important, but you will not automatically get insights and better decisions with hard skills alone. Using data to make better data-informed decisions requires not only hard skills but also soft skills as well as mindsets." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"Data literacy is not achieved by mastering a uniform set of competencies that applies to everyone. Those that are relevant to each individual can vary significantly depending on how they interact with data and which part of the data process they are involved in." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"Decision-makers are constantly provided data in the form of numbers or insights, or similar. The challenge is that we tend to believe every number or piece of data we hear, especially when it comes from a trusted source. However, even if the source is trusted and the data is correct, insights from the data are created when we put it in context and apply meaning to it. This means that we may have put incorrect meaning to the data and then made decisions based on that, which is not ideal. This is why anyone involved in the process needs to have the skills to think critically about the data, to try to understand the context, and to understand the complexity of the situation where the answer is not limited to just one specific thing. Critical thinking allows individuals to assess limitations of what was presented, as well as mitigate any cognitive bias that they may have." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"Data literacy is something that affects everyone and every organization. The more people who can debate, analyze, work with, and use data in their daily roles, the better data-informed decision-making will be." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"It is also important to note that data literacy is not about expecting to or becoming an expert; rather, it is a journey that must begin somewhere." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"Organizations must have a plan and vision for data literacy, which they then communicate to all employees. They will need to develop and foster a culture that embraces data literacy and data-informed decisions. They will need to provide employees with access to various learning content specific to data literacy. Along their journey, they will need to make sure they benchmark and measure progress toward their vision and celebrate successes along the way." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"People can get confused or experience anxiety when they have to work with data and analytics. As we data literacy geeks say, people shouldn’t be pushed to work with data and analytics - they should do this because they want to. [...] Visualizations that are not understood present another risk. To be successful with data and analytics, we need visualizations that are presented in a clear meaningful way. If we do not take care of the data literacy levels within an organization, we might lose our public. Therefore, it is necessary to think of the risk of overwhelming our readers/viewers." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

🔢Zach Gemignani - Collected Quotes

"A culture of data fluency needs to be built on a shared understanding of the data sources, data analysis, key metrics, and data products. It requires employees to be on the same page about how data is used and why it is important." (Zach Gemignani et al, "Data Fluency", 2014)

"Any presentation of data, whether a simple calculated metric or a complex predictive model, is going to have a set of assumptions and choices that the producer has made to get to the output. The more that these can be made explicit, the more the audience of the data will be open to accepting the message offered by the presenter." (Zach Gemignani et al, "Data Fluency", 2014)

"Broadly defined, data means events that are captured and made available for analysis. A data source is a consistent record of these events. And a data product translates this record of events into something that can easily be understood. [...] Data products can be organized and characterized by a series of continuums that describe the nature of the data and how it is presented." (Zach Gemignani et al, "Data Fluency", 2014)

"[...] communicating with data is less often about telling a specific story and more like starting a guided conversation. It is a dialogue with the audience rather than a monologue. While some data presentations may share the linear approach of a traditional story, other data products (analytical tools, in particular) give audiences the flexibility for exploration. In our experience, the best data products combine a little of both: a clear sense of direction defined by the author with the ability for audiences to focus on the information that is most relevant to them. The attributes of the traditional story approach combined with the self-exploration approach leads to the guided safari analogy." (Zach Gemignani et al, "Data Fluency", 2014)

"Creating a data fluent organization doesn’t just happen. It starts with people who love using data as a tool to improve their job performance - people who have learned to converse with others in the language of data. It needs people who expect and demand better, more useful data products from themselves and others. It starts with you." (Zach Gemignani et al, "Data Fluency", 2014)

"Data alone isn’t valuable. In fact, it can be expensive in time and resources to manage and maintain. The analysis of this data is closer to something that is valuable. A clearly communicated analysis starts to transform a reflection of the world into knowledge in the minds of people. Even so, knowledge alone does not make your organization better. It is the decisions and actions of people - based on this data-sourced knowledge - that is the goal. But these decisions are seldom made in a vacuum. In most organizations, decisions are a collaborative, social experience. People come together to discuss options, review their knowledge of the situation, and arrive at a path to go down. Herein is one of the great powers of effective data products: They can shape and guide these discussions. Conclusions are seldom clear-cut, even when there is data to support a direction." (Zach Gemignani et al, "Data Fluency", 2014)

"Data captures actions and characteristics of the real world and transforms them into something that can be examined and explored after the fact." (Zach Gemignani et al, "Data Fluency", 2014)

"Data visualizations are designed to emphasize patterns and deviations in data. In fact, each specific chart type is well suited to highlighting particular forms of insight. A skilled author of data products will choose the right visualization to emphasize a message. The data, chart, and supporting descriptions should work in harmony to point out what is interesting. The reader simply goes along for the ride." (Zach Gemignani et al, "Data Fluency", 2014)

"Goals associated with a few, well-understood key metrics is a powerful combination. For both internal and external stakeholders, there is a strong alignment between organization mission, vision, goals, and tracking of progress. The efforts of everyone can be directed at these measurable goals, and people will focus on the processes that can impact these metrics." (Zach Gemignani et al, "Data Fluency", 2014)

"In fact, the analogy to storytelling is limited when applied to communicating with data. Data visualization has fundamental characteristics missing from traditional storytelling. For example, interactive data visualizations let audiences explore information to find insights that resonate with them. Visualizations take shape based to a large extent on the underlying data. And as this data changes, the emphasis and message of the visualization is likely to change." (Zach Gemignani et al, "Data Fluency", 2014)

"Metrics can serve two purposes: identifying problems and measuring performance. When the goal is to identify problems and pinpoint areas of operational inefficiency and ineffectiveness, defining the right metric requires a bit of detective work. It requires you to uncover the data residue of a problem and to determine what evidence can be found and how exactly it shows up. When the goal is to measure performance, the right success metrics focus on measures that can be controlled and where improvement in the metric is an unambiguously good thing." (Zach Gemignani et al, "Data Fluency", 2014)

"Most discussions of decision making assume that only senior executives make decisions or that only senior executives' decisions matter. This is a dangerous mistake. Decisions are made at every level of the organization, beginning with individual professional contributors and frontline supervisors. These apparently low-level decisions are extremely important in a knowledge-based organization." (Zach Gemignani et al, "Data Fluency", 2014)

"The most common mistake in ineffective data products is an inability to make difficult decisions about what information is most important. [...] Often information gets included in data products for reasons that are superfluous to the purpose, audience, and message - reasons that cater the product to someone influential or use information that has been included historically. The bar should be higher." (Zach Gemignani et al, "Data Fluency", 2014)

"We have an inbuilt ability to manipulate visual metaphors in ways we cannot do with the things and concepts they stand for—to use them as malleable, conceptual Tetris blocks or modeling clay that we can more easily squeeze, stack, and reorder. And then - whammo! - a pattern emerges, and we’ve arrived someplace we would never have gotten by any other means." (Zach Gemignani et al, "Data Fluency", 2014)

🎯🗒️Harvinder Atwal - Collected Quotes

"Agile practices can bring discipline to data science through support for the values and principles of DataOps. However, just as there is no perfect agile framework or set of practices for software development, there is no single set of best agile practices for data science. The right practices to use are context and organization specific and help data analytics teams become more adaptable and collaborative and tighten feedback loops to produce faster (and better) results. The successful application of agile and Lean thinking to data analytics requires observation, constant experimentation, and adjustment." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020)

"DataOps does not prescribe a particular agile framework, set of practices, artifacts, or roles. Nonetheless, instead of inventing new ways of working, it is better to adapt existing proven practices and combining them as needed. The following are useful agile practices aligned with DataOps principles. However, none are compulsory, and if you find they do not add value, do not use them. Agile frameworks are not mutually exclusive, and practices are situation and context-dependent. You can explore alternatives as long as you stay true to DataOps values and principles." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020)

"DataOps is not a project. The first iteration of DataOps is not the last. With minimum viable DataOps in place and benefits publicized, the next stage is to expand practices to more domains and epics. The second objective is to get to a tipping point where it becomes more compelling to continue the journey of implementing DataOps practices, principles, and values than to resist them. Results speak louder than words. It remains essential to avoid diversions into time-consuming political battles with hard-to-change departments or to waste time developing a large-scale change management program." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020) 

"DataOps methodology is the best way to eliminate barriers, collaborate, and maximize the chances of success. DataOps turns data science and analytics from the craft industry it is today in most organizations into a slick manufacturing operation. DataOps enables rapid data product development and creates an assembly line that converts raw data from multiple sources into production data products with a minimum of waste. (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020) 

"DataOps requires that data scientists, data analysts, and data engineers have quick access to data, tools, and infrastructure to eliminate bottlenecks. That is, they need to be able to access, add, or modify data quickly by themselves. We term this availability to data self-service. Through self-service, data analytics professionals can create data products in far less time than with traditional approaches." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020)

"DataOps should be part of a well-thought-out data strategy that lays the foundation for a transformation. Actually, all organizations that want to use data for data-sharing or analytical purposes need a data strategy. The only variation will be the depth of strategy and complexity of use cases. A start-up’s data strategy might not need the same detail and span as a multinational corporation, but it should still define a means to prepare for the future." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020)

"DevOps effectiveness increases when there is less heterogeneity in the technology stack. Complexity increases the probability of errors and slows down the flow of deployment because teams find it hard to scale their expertise and apply consistent patterns across data pipelines. The focus of data analytics teams after adopting version control should be to standardize and simplify the set of technologies they use from languages, libraries, and databases to data engineering tools." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020)

"The final stage of DevOps for DataOps is to automate the build of pipeline environments and give data pipeline developers self-serve ability to create, test, and deploy changes." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020) 

"Unless dealing with a greenfield situation, it is not possible to jump straight to the end state of all analytical work following the DataOps methodology. Respecting the principles of agile and DevOps, the movement to the end goal must be in iterative, small, and frequent steps." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020) 

"While there are undoubtedly success stories, there is also plenty of evidence that substantial investment in data science is not generating the returns expected for a majority of organizations." (Harvinder Atwal, "Practical DataOps: Delivering Agile Data Science at Scale", 2020) 

🎯Stephen Few - Collected Quotes

"An effective dashboard is the product not of cute gauges, meters, and traffic lights, but rather of informed design: more science than art, more simplicity than dazzle. It is, above all else, about communication." (Stephen Few, "Information Dashboard Design", 2006)

"Most dashboards fail to communicate efficiently and effectively, not because of inadequate technology (at least not primarily), but because of poorly designed implementations. No matter how great the technology, a dashboard's success as a medium of communication is a product of design, a result of a display that speaks clearly and immediately. Dashboards can tap into the tremendous power of visual perception to communicate, but only if those who implement them understand visual perception and apply that understanding through design principles and practices that are aligned with the way people see and think." (Stephen Few, "Information Dashboard Design", 2006)

"A signal is a useful message that resides in data. Data that isn’t useful is noise. […] When data is expressed visually, noise can exist not only as data that doesn’t inform but also as meaningless non-data elements of the display (e.g. irrelevant attributes, such as a third dimension of depth in bars, color variation that has no significance, and artificial light and shadow effects)." (Stephen Few, "Signal: Understanding What Matters in a World of Noise", 2015)

"Apart from the secondary benefits of digital data, which are many, such as faster and cheaper information collection and distribution, the primary benefit is better decision making based on evidence. Despite our intellectual powers, when we allow our minds to become disconnected from reliable information about the world, we tend to screw up and make bad decisions." (Stephen Few, "Signal: Understanding What Matters in a World of Noise", 2015)

"Data contain descriptions. Some are true, some are not. Some are useful, most are not. Skillful use of data requires that we learn to pick out the pieces that are true and useful. [...] To find signals in data, we must learn to reduce the noise - not just the noise that resides in the data, but also the noise that resides in us. It is nearly impossible for noisy minds to perceive anything but noise in data." (Stephen Few, "Signal: Understanding What Matters in a World of Noise", 2015)

"Signals always point to something. In this sense, a signal is not a thing but a relationship. Data becomes useful knowledge of something that matters when it builds a bridge between a question and an answer. This connection is the signal." (Stephen Few, "Signal: Understanding What Matters in a World of Noise", 2015)

"The term data, unlike the related terms facts and evidence, does not connote truth. Data is descriptive, but data can be erroneous. We tend to distinguish data from information. Data is a primitive or atomic state (as in ‘raw data’). It becomes information only when it is presented in context, in a way that informs. This progression from data to information is not the only direction in which the relationship flows, however; information can also be broken down into pieces, stripped of context, and stored as data. This is the case with most of the data that’s stored in computer systems. Data that’s collected and stored directly by machines, such as sensors, becomes information only when it’s reconnected to its context."  (Stephen Few, "Signal: Understanding What Matters in a World of Noise", 2015)

"Everything that informs us of something useful that we didn't already know is a potential signal. If it matters and deserves a response, its potential is actualized." (Stephen Few)

"One of the great purposes of education today is to help us filter the data, to reduce it to what's true and useful." (Stephen Few)

🎯Jean-Georges Perrin - Collected Quotes

"Accessibility is another key principle. It’s not enough for data to be findable; In fact, once found, a data product must also be easily accessible. Accessibility includes providing comprehensive documentation that explains how to use the data, as well as ensuring that the data can be easily integrated into various applications and workflows. A good data product should be as straightforward to use as a well-designed software application, with clear instructions and support."  (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Consider data silos. Data silos hinder data accessibility and collaboration, making it difficult to gain a holistic view and leverage the full potential of the available data. They present a real, present, and formidable challenge that almost all data practitioners experience in modern enterprises. Data silos, much like isolated islands in an immense ocean, are repositories of data that are confined within specific departments or systems, disconnected from the broader organizational data landscape. This segregation results in a fragmented data ecosystem, where valuable insights remain untapped, and the collective intelligence of the enterprise is underutilized." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Data Mesh addresses data governance challenges by advocating for a federated governance model, which positions accountability for governance with the data owners who are most knowledgeable about the data. In this model, governance is decentralized, with each domain team responsible for the governance of its data products. This approach ensures that governance decisions are made by those who have the deepest understanding of the data’s context, use, and risks. It leads to more relevant, efficient, and effective governance practices that are closely aligned with the specific needs of each domain." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Data Mesh advocates for domain-driven ownership of data, enabling individual teams to manage and share their data effectively while aligning with the overall organizational objectives. By embracing this paradigm, enterprises can gradually dismantle the barriers of data silos, paving the way for a more integrated, agile, and data-centric organizational culture." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"[...] Data Mesh introduces agility into the data landscape, emphasizing decentralized ownership, responsive data management, and collaborative cross-functional teams. Just as Agile promotes self-organizing teams, Data Mesh advocates for domain-oriented decentralized ownership, putting the power of data in the hands of individual domain teams. In an Agile context, customer collaboration involves continuous engagement with stakeholders to understand their evolving needs. Likewise, Data Mesh encourages domain teams to engage with data consumers within their organization, gathering feedback, and iterating on their data products to meet their specific requirements." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"[...] Data Mesh is, then, at its foundation, a conceptual framework in the realm of data architecture, which emphasizes decentralized data ownership and architecture. It recognizes that in large organizations, data is vast and varied, where each business domain has autonomy over its own data. By decentralizing control, it empowers individual domains to manage and make decisions about their data while maintaining a cohesive overall structure. And presumably, with this autonomy comes better, more localized, and faster decisions, which in-turn, leads to speed and agility." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Federated computational governance is essential for maintaining consistency and compatibility across the Data Mesh. It ensures that despite the decentralized nature of data ownership, there is a unified framework governing how data is managed, used, and shared. This unified approach is crucial in preventing data silos, ensuring data interoperability, and maintaining the overall integrity of the data ecosystem." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Implementing federated computational governance requires a delicate balance. It involves creating governance structures that are robust enough to ensure consistency and compliance, yet flexible enough to accommodate the unique needs and contexts of different data products. This balance is key to fostering an environment where innovation can thrive without compromising the standards and protocols essential for a cohesive data ecosystem." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"In the context of a Data Mesh, a data product is a package of data that is self-contained, self-descriptive, and oriented towards a specific business purpose or function. But, it is not just a mere collection of data, rather it is a coherent unit that provides value to its consumers, akin to a product in any other industry." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Observability extends the concept of reliability. It’s about having the ability to monitor the health and performance of the data product. By using tools to track various metrics like response times and error rates, organizations can proactively manage the data product’s health. This proactive management plays a crucial role in maintaining the product’s reliability, as it allows for the early identification and resolution of potential issues before they escalate." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"[...] scalability and maintainability are key aspects of a valuable data product. It should be capable of handling increasing volumes of data or user demands without necessitating significant redesign or rework. Alongside scalability, maintainability - the ease with which a data product can be updated, modified, or repaired - is critical for its long-term utility. This also includes the product’s ability to evolve based on user feedback and changing business needs, ensuring that it remains relevant and valuable over time." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Self-serve capability in a Data Mesh not only empowers users but also fosters a culture of innovation and agility. It enables individuals to leverage data for their specific needs, encouraging experimentation and personalized analysis. This capability reduces bottlenecks typically associated with centralized data systems, where requests for data access and analysis can slow down decision-making processes." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The fourth principle, Reuse, focuses on the ability to apply data in multiple contexts. This principle is particularly important in maximizing the value of data. By designing data products to be modular and reusable, they can be used across different projects and applications. For instance, a data product containing customer demographic information can be used by marketing teams for campaign planning, by sales teams for sales strategy development, and by product development teams for market analysis."  (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The principle of a clear boundary in a Data Mesh ensures that every data product is a well-defined entity within the larger ecosystem. This clarity prevents overlap and confusion, establishing a clear understanding of the data product’s purpose and scope. It aids in managing expectations and directs efforts and resources appropriately, ensuring that each data product can effectively fulfill its intended role." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The ramifications of data silos extend beyond mere inefficiencies; they actively hinder collaboration and innovation within an organization. When data is trapped in silos, it becomes difficult for teams to access the information they need to collaborate effectively. This lack of accessibility and visibility leads to duplicated efforts, inconsistent data practices, and a general sense of organizational disjointedness." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The role of an empowered owner is multifaceted. They are tasked with ensuring that the data product aligns with both specific business requirements and the overarching governance framework. This alignment is crucial for maintaining the integrity and usefulness of the data product, ensuring it remains a valuable asset within the organization’s data landscape." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"The structure of a data product is self-contained, meaning that it includes everything necessary for its effective utilization. It adheres to strict standards of quality and governance, thereby ensuring reliability, security, and compliance with relevant regulations. This comprehensive approach makes data products a trusted and dependable resource within the organization. They are designed with user accessibility in mind, offering interfaces and documentation that are easily navigable by a wide range of users, from data experts to those with minimal technical expertise." (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

"Usability is a key determinant of a data product’s value - if it is complex or unintuitive, its potential utility diminishes irrespective of the underlying data quality. Therefore, the design and interface of a data product should facilitate ease of use to ensure that it can be effectively employed by its target users. Somewhat related to this is interoperability - in other words it is also usable from an operations perspective. A valuable data product should not only function in isolation but also integrate seamlessly with other data products. This interoperability is vital for comprehensive analytics and insight generation, as it allows for the combination and analysis of data across various domains. Additionally, compliance with regulatory requirements and security standards is non-negotiable."  (Jean-Georges Perrin & Eric Broda, "Implementing Data Mesh: Principles and Practice to Design, Build, and Implement Data Mesh", 2024)

🎯Shonna D Watters - Collected Quotes

"Analytics provides a way to demonstrate the linkage between people and business outcomes. HR analytics (also called people analytics or talent analytics) use measurement and analysis techniques to understand, improve, and optimize the people side of business. Data are the raw numbers you track. [...] Metrics focus on counting, tracking, and presenting past data. Analytics uses statistics to help you see patterns in the data." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Data analytics is a powerful tool to increase the likelihood that you have the right problem. Both quantitative and qualitative data serve a purpose in supporting a hypothesis. They allow you to objectively measure and identify patterns and relationships." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Data mining is a common way of handling big data. It’s used to analyze big data and overcome some of the limitations of human information processing and traditional analytic techniques. This approach applies machine learning algorithms to find patterns of relationships between elements in large, messy data sets. The objective of data mining is to use the detected patterns to predict future outcomes and make better decisions." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Data scientists have extensive backgrounds in computer science, coding, machine learning, and statistics. Such an extensive background takes a long time to develop and is usually highly compensated. This has posed a tremendous barrier to many organizations. But this isn’t the only problem with relying solely on data experts. As organizations become more advanced, they are becoming more interested in using data to predict future outcomes. Merely relying on data from the past won’t suffice. Businesses must be forward thinking in how they collect their data to best serve predictive analytics. This means the employees who collect the data (e.g., those rating performance or creating and administering an engagement survey) need to understand how those data will later be analyzed." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Halo error is another common cognitive bias in employee ratings. This refers to a “halo” or aura that surrounds all ratings of an individual. If a manager has to rate an employee on several different dimensions, he or she would assign the same rating to that person on every dimension. Sometimes this may be attributable to laziness. But this also can reflect an underlying perception that performance is a singular dimension: people are either good or poor performers." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Hypotheses build the foundation for data analytics. Develop alternative hypotheses to explain the issue at hand. These hypotheses will guide your data collection." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"It’s important to look at multiple hypotheses. If your initial hypothesis isn’t supported, you’ll have little direction for moving forward. […] Thinking through viable alternative hypotheses will help you get a sense of where to find data in your organization. […] start by brainstorming. Think about the information that may be available. Is there a way it might be relevant? What sources are available to you?" (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Lagging indicators are metrics describing what happened in the past. 'Lagging' refers to the time lapse between an action and a specific outcome. […] Leading indicators are metrics that provide early indications of your progress toward an objective. […] Lagging and leading indicators work together, allowing you to make a more comprehensive evaluation." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Leniency error occurs when raters are unusually easy in their ratings, while severity error refers to the tendency to be unusually harsh in one’s ratings. Sometimes individuals commit these errors because of the language used in a rating scale. Terms like “average” and “outstanding” are relative and may lead a manager to use a personal average rather than an average that other managers may be using." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Measures of central tendency and variability work together to give you a concise summary of your data. When you don’t have any outliers, the mean is the most common indicator of central tendency. But on its own, the mean doesn’t tell you much. It isn’t until you also take the standard deviation into account that you really have a sense of your data." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Missing data can pose more or less of a problem depending on how random they are. Missing values scattered haphazardly throughout your data set may be inconvenient, but they don’t necessarily pose a threat to your analyses. When missing data follow a clearer pattern, there may be a problem. In those cases, you might need to revisit how you collect your data." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"People aren’t as good at making decisions as they think. We like to think of ourselves as rational actors, but our informational-processing limitations, emotions, and biases get in our way. The world is complex and humans have developed ways to help simplify it. So-called cognitive biases are ways our brains help us take shortcuts to deal with four primary problems: informati"n overload, lack of meaning, the need to act fast, and knowing what needs to be remembered for later." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Percentiles offer another way to understand how a data point fits into the bigger picture. […] A percentile score tells you what percentage of people fall below an individual on a given metric, so where an individual falls relative to everyone else."  (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"There are a few cognitive biases that commonly distort performance ratings. These inaccuracies may be intentional or due purely to human error in rating. Central tendency bias is the inclination to choose a rating somewhere in the middle of a scale, even when a more extreme score (for better or worse) is a better description. Raters often fall into this pattern when more extreme ratings require a written justification." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

"Understanding data is really about two types of expertise: data expertise and analytics expertise. Data expertise involves working directly with data - data extraction, cleaning, transformation, and management. Analytics expertise involves data analysis, data visualization, and validation." (Shonna D Watters et al, "The Practical Guide for HR Analytics: Using data to inform, transform, and empower HR decisions", 2019)

🔢Adam Bellemare - Collected Quotes

"A data mesh is inherently multimodal, and data products can be provided via a variety of means. Event streams remain the best option for the majority of data products, as it is far easier to power both operational and analytical use cases through a stream than a batch of files at rest." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"Bad data is costly to fix, and it’s more costly the more widespread it is. Everyone who has accessed, used, copied, or processed the data may be affected and may require mitigating action on their part. The complexity is further increased by the fact that not every consumer will “fix” it in the same way. This can lead to divergent results that are divergent with others and can be a nightmare to detect, track down, and rectify." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"Creating data products requires that domain owners have a degree of autonomy in modeling, building, and delivering data to their consumers. However, by empowering them with autonomy and independence, you run the risk of a significant technological sprawl across data product implementations, making it more difficult for consumers to use the data products for their own ends. Federated governance focuses on finding an equilibrium between the needs of the consumers, the autonomy of the data product owners, the business compliance and security requirements, and global data product requirements." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"Data has historically been treated as a second-class citizen, as a form of exhaust or by-product emitted by business applications. This application-first thinking remains the major source of problems in today’s computing environments, leading to ad hoc data pipelines, cobbled together data access mechanisms, and inconsistent sources of similar-yet-different truths. Data mesh addresses these shortcomings head-on, by fundamentally altering the relationships we have with our data. Instead of a secondary by-product, data, and the access to it, is promoted to a first-class citizen on par with any other business service." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"Data mesh architectures are inherently decentralized, and significant responsibility is delegated to the data product owners. A data mesh also benefits from a degree of centralization in the form of data product compatibility and common self-service tooling. Differing opinions, preferences, business requirements, legal constraints, technologies, and technical debt are just a few of the many factors that influence how we work together." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"Data mesh promotes data to a product with the same rigor, ownership, and feature management of any other product in your business. The free-for-all, 'figure it out yourself' data access is replaced with purpose-built, maintained, and supported modes. It is as much a social shift as it is a technological shift and requires both top-down and bottom-up buy-in." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"Enforcing a schema at read time, instead of at write time, leads to a proliferation of what we call 'bad data'. The lack of write-time checks means that data written into HDFS may not adhere to the schemas that the readers are using in their existing work […]. Some bad data will cause consumers to halt processing, while other bad data may go silently undetected. While both of these are problematic, silent failures can be deadly and difficult to detect." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"Federated governance can be roughly broken down into two main tasks. The first is establishing cross-organization policies, including data product standards and datah andling requirements, that apply to all users of the data mesh. The second is providing guidance on creating and using data products with self-service tools to make it easy to participate in the data mesh." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"The premise of the data mesh solution is simple. Publish important business data sets to dedicated, durable, and easily accessible data structures known as data products. The original creators of the data are responsible for modeling, evolution, quality, and support of the data, treating it with the same first-class care given to any other product in the organization. Prospective consumers can explore, discover, and subscribe to the data products they need for their business use cases. The data products should be well-described, easy to interpret, and form the basis for a set of self-updating data primitives for powering both business services and analytics." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

"The problem of bad data has existed for a very long time. Data copies diverge as their original source changes. Copies get stale. Errors detected in one data set are not fixed in duplicate ones. Domain knowledge related to interpreting and understanding data remains incomplete, as does support from the owners of the original data." (Adam Bellemare, "Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures", 2023)

🎯Zhamak Dehghani - Collected Quotes

"A data pipeline is a series of transformation steps (functions) executed as the data flows from one step to another. Data mesh refrains from using pipelines as a top-level architectural paradigm and in between data products. The challenge with pipelines as currently used is that they don’t create clear interfaces, contracts, and abstractions that can be maintained easily as the pipeline complexity complexity grows. Due to lack of abstractions, single failure in the pipeline causes cascading failures." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"A data product encapsulates more than just the data. It needs to contain all the structural components needed to manifest its baseline usability characteristics - discoverable, understandable, addressable, etc. - in an autonomous fashion, while continuing to share data in a compliant and secure manner." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"A data product’s primary job is to consume data from upstream sources using its input data ports, transform it, and serve the result as permanently accessible data via its output data ports." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Another myth is that we shall have a single source of truth for each concept or entity. […] This is a wonderful idea, and is placed to prevent multiple copies of out-of-date and untrustworthy data. But in reality it’s proved costly, an impediment to scale and speed, or simply unachievable. Data Mesh does not enforce the idea of one source of truth. However, it places multiple practices in place that reduces the likelihood of multiple copies of out-of-date data." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data lake architecture suffers from complexity and deterioration. It creates complex and unwieldy pipelines of batch or streaming jobs operated by a central team of hyper-specialized data engineers. It deteriorates over time. Its unmanaged datasets, which are often untrusted and inaccessible, provide little value. The data lineage and dependencies are obscured and hard to track." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data is a collection of facts put together according to a model. The data model is an approximation of reality, good enough for the (analytical) tasks at hand." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"In addition to limitations of scale, other challenges of data centralization are data quality and resilience to change. This is because business domains and teams that are most familiar with the data are not responsible for data quality." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"[...] the governance function is accountable to define what constitutes data quality and how each data product communicates that in a standard way. It’s no longer accountable for the quality of each data product. The platform team is accountable to build capabilities to validate the quality of the data and communicate its quality metrics, and each domain (data product owner) is accountable to adhere to the quality standards and provide quality data products." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data management of the future must build in embracing change, by default. Rigid data modeling and querying languages that expect to put the system in a straitjacket of a never-changing schema can only result in a fragile and unusable analytics system. [...] The data management of the future must support managing and accessing data across multiple hosting platforms, by default." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data Mesh attempts to strike a balance between team autonomy and inter-term interoperability and collaboration, with a few complementary techniques. It gives domain teams autonomy to have control of their local decision making, such as choosing the best data model for their data products. While it uses the computational governance policies to impose a consistent experience across all data products; for example, standardizing on the data modeling language that all domains utilize." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data mesh focuses on the impact of the data and not its volumes. It values data usability, data satisfaction, data availability, and data quality over the volume of the data." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"[...] data mesh introduces a fundamental shift that the owners of the data products must communicate and guarantee an acceptable level of quality and trustworthiness - specific to their domain - as an intrinsic characteristic of their data product. This means cleansing and running automated data integrity tests at the point of the creation of a data product." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data Mesh is a sociotechnical approach to share, access and manage analytical data in complex and large-scale environments - within or across organizations." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data mesh is a solution for organizations that experience scale and complexity, where existing data warehouse or lake solutions have become blockers in their ability to get value from data at scale and across many functions of their business, in a timely fashion and with less friction." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data mesh is an element of a data strategy that fosters a data-driven organization to get value from data at scale." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data Mesh must allow for data models to change continuously without fatal impact to downstream data consumers, or slowing down access to data as a result of synchronizing change of a shared global canonical model. Data Mesh achieves this by localizing change to domains by providing autonomy to domains to model their data based on their most intimate understanding of the business without the need for central coordinations of change to a single shared canonical model." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Data mesh [...] reduces points of centralization that act as coordination bottlenecks. It finds a new way of decomposing the data architecture without slowing the organization down with synchronizations. It removes the gap between where the data originates and where it gets used and removes the accidental complexities - aka pipelines - that happen in between the two planes of data. Data mesh departs from data myths such as a single source of truth, or one tightly controlled canonical data model." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"In short, a monolithic architecture, technology, and organizational structure are not suitable for analytical data management of large-scale and complex organizations." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"In the case of data mesh, a data product is an architectural quantum. It is the smallest unit of architecture that can be independently deployed and managed. It has high functional cohesion, i.e., performing a specific analytical transformation and securely sharing the result as domain-oriented analytical data. It has all the structural components that it requires to do its function: the transformation code, the data, the metadata, the policies that govern the data, and its dependencies to infrastructure." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"One of the limitations of data management solutions today is how we have attempted to manage its unwieldy complexity, how we have decomposed an ever-growing monolithic data platform and team to smaller partitions. We have chosen the path of least resistance, a technical partitioning." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"The distributed nature of data mesh demands immutability to give confidence to data users that (1) there is consistency between multiple data products for a point-in-time piece of data and (2) once they read data at a point in time, that data doesn’t change and they can reliably repeat the reads and processing." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"There are a set of characteristics that can be grouped together as quality. These attributes aren’t intended to define whether a data product is good or bad. They just communicate the threshold of guarantees the data product expects to meet, which may be well within an acceptable range for certain use cases." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Unlike other analytical data management paradigms, data mesh does not embrace the concept of the mythical single source of truth. Every data product provides a truthful portion of the reality - for a particular domain - to the best of its ability, a single slice of truth." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Ultimately, Data Mesh’s goal is to enable organizations to thrive in the face of the growth of data sources, growth of data users and use cases, and the increasing change in cadence and complexity. Adopting Data Mesh, organizations must thrive in agility, creating data-driven value while embracing change." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

🔢Sid Adelman - Collected Quotes

"Data archeology (finding bad data), data cleansing (correcting bad data), and data quality enforcement (preventing data defects at the source) should be business objectives. Therefore, data quality initiatives are business initiatives and require the involvement of business people, such as information consumers and data originators." (Sid Adelman et al, "Data Strategy", 2005)

"Data strategy is one of the most ubiquitous and misunderstood topics in the information technology (IT) industry. Most corporations' data strategy and IT infrastructure were not planned, but grew out of "stovepipe" applications over time with little to no regard for the goals and objectives of the enterprise. This stovepipe approach has produced the highly convoluted and inflexible IT architectures so prevalent in corporations today." (Sid Adelman et al, "Data Strategy", 2005)

"Dealing with [...] resistance is where social sensitivity, leadership, and power come into play. Social sensitivity is the ability to read the players and respond appropriately to their concerns. Leadership and power can quickly overcome most resistance to change and allow you to establish an environment and convince management to properly support the data strategy." (Sid Adelman et al, "Data Strategy", 2005)

"It is important to remember that the 'single version of the truth' - or enterprise logical data model - is not and should not be built all at once (that would take too long), but that it evolves over time as the project-specific logical data models are merged, one-by-one, a project at a time." (Sid Adelman et al, "Data Strategy", 2005)

"The chaos without a data strategy is not as obvious, but the indicators abound: dirty data, redundant data, inconsistent data, the inability to integrate, poor performance, terrible availability, little accountability, users who are increasingly dissatisfied with the performance of IT, and the general feeling that things are out of control." (Sid Adelman et al, "Data Strategy", 2005)

"The data strategist is responsible for creating and maintaining the data strategy. This includes fully understanding the strategic goals of the organization. [...] The data strategist must know (or learn) the existing environment including the important internal databases, the external data that will be integrated, and the data quality characteristics. The data strategist must be aware of the data volumes expected in the next five years. [...] The data strategist must be aware of changes in the business that will require more complex transactions and queries. He or she must also be aware of governmental factors including regulations and governmental reporting requirements. The data strategist must know about the requirements of service level agreements (SLAs) for both performance and availability and be sure that the data strategy supports those SLAs (it's also likely that the data strategist would have input into creating those SLAs.) And finally, the data strategist must be wired into the politics of the organization so that his or her proposals will be pragmatic and accepted by management and staff." (Sid Adelman et al, "Data Strategy", 2005)

"The folks in IT don't like change if they believe it will diminish the power of the IT group. This is particularly true for managers. Managers put forward countless reasons why the organization should stay as is, especially if a change can decrease the number of employees they control because managers often equate headcount to power in the organization." (Sid Adelman et al, "Data Strategy", 2005) [?!]

"The vision of a data strategy that fits your organization has to conform to the overall strategy of IT, which in turn must conform to the strategy of the business. Therefore, the vision should conform to and support where the organization wants to be in 5 years." (Sid Adelman et al, "Data Strategy", 2005)

"Working without a data strategy is analogous to a company allowing each department and each person within each department to develop its own financial chart of accounts. This empowerment allows each person in the organization to choose his own numbering scheme. Existing charts of accounts would be ignored as each person exercises his or her own creativity." (Sid Adelman et al, "Data Strategy" 1st Ed., 2005)

"You cannot boil the ocean; you have to prioritize your data integration deliverables. An enterprise-wide data integration effort must be carved up into small iterative projects, starting with the most critical data and working down to the less significant data. The business people working with the data integration team must determine which data is most appropriate for integration. Some data might not be suitable for integration at all, such as department-specific data, highly secured data, and data that is too risky to integrate. The team also needs to look at historical data and decide how much of it to include in the data integration process." (Sid Adelman et al, "Data Strategy" 1st Ed., 2005)

🔢Ian Wallis - Collected Quotes

"A data strategy is the opportunity to bring data, one of the most important assets your organisation has, to the fore and to drive the future direction of the organisation." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"A data strategy which no longer reflects the priorities of the organisation as a whole is doomed to fail, and likely to struggle to keep any momentum beyond the immediate term." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"A KPI is a performance measure that demonstrates how effectively an organisation is achieving its critical objectives. They are used to track performance over a period of time to ensure the organisation is heading in the desired direction, and are quantifiable to guide whether activities need to be dialled up or down, resources adjusted or management resource focused on understanding what is in play that may be holding back the organisation." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"Culture is not something that can be read in a corporate document (though many organisations will claim to have values, beliefs and other concepts that articulate the culture as the corporate centre wants it to be seen). It is intangible and can be challenging to comprehend to those on the outside looking in. Much of it is unspoken, a series of behavioural norms which are engrained in the fabric of the organisation and drive attitudes of employees to one another, management, change programmes and any external (to the group, as well as the organisation) effort to drive change that may be resisted simply because it ‘isn’t the way we do things around here’." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"Data has a value, without which an organisation is largely a shell, worthless and of limited appeal other than as a means of sweeping up fixed assets at a knock-down price. It is the lifeblood of an organisation, so whether you regard it as the water that is essential to life or the blood circulating around the body, without it our organisations are not functional." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"Data strategy is even less understood [thank business strategy], so the chances of success can be further decreased, simply because you need organisation-wide commitment and buy-in to succeed. Data does not exist in a bubble; it is not the preserve of a function that can fix it for all, detached from touching everyone else. It is core to how you run the organisation, and without a focus on where you are heading, it is going to trip the organisation up at every turn - regulatory compliance; operational effectiveness; financial performance; customer and employee experience; essentially, the efficiency in managing virtually every activity in the organisation." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"I am using ‘data strategy’ as an overarching term to describe a far broader set of capabilities from which sub-strategies can be developed to focus on particular facets of the strategy, such as management information (MI) and reporting; analytics, machine learning and AI; insight; and, of course, data management." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"If there is one all too common a failing in data strategies, it is the temptation to make them too detailed through either straying into implementation activities or overplaying the content by providing too much information. The key is to recognise the level of information that needs to be imparted to make the data strategy coherent and likely to be endorsed, with as little information as is necessary to be able to make the point cogently. Brevity, and associated clarity in what needs to be achieved and why, is a winning formula in gaining senior executive sponsorship." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"It is also important to regard the data strategy as a living document. Do not regard it as a masterpiece, never to be reviewed, amended or critiqued within the time frame it covers, but instead see it as a strategy that can flex to the changing demands of an organisation." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"[...] it is always useful to learn from past mistakes, but evidence shows that most strategies fail due to an inability to follow through into execution." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"In the same vein, data strategy is often a misnomer for a much wider scope of coverage, but the lack of coherence in how we use the language has led to data strategy being perceived to cover data management activities all the way through to exploitation of data in the broadest sense. The occasional use of information strategy, intelligence strategy or even data exploitation strategy may differentiate, but the lack of a common definition on what we mean tends to lead to data strategy being used as a catch-all for the more widespread coverage such a document would typically include. Much of this is due to the generic use of the term ‘data’ to cover everything from its capture, management, governance through to reporting, analytics and insight." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"Many organisations start a data strategy from a need to get data into some sort of organised state in which it is feasible to demonstrate compliance. In my opinion, compliance should be a component of a data strategy, not the data strategy in itself." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"The challenge with using OKRs is to focus on just three to five objectives - sounds simple enough, but so many organisations follow the ‘if it moves, track it’ philosophy such that they can’t see the wood for the trees." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"The key for a successful data strategy is to align it clearly with the corporate strategy. The data strategy is a crucial enabler of the corporate strategy, and the data strategy should clearly call out those components that have a clear line of sight to delivering, or enabling, the corporate goals. If the data strategy does not align to the corporate goals it will be a much more challenging task to get the wider organisation to buy into it, not least because it will fail to have any resonance with the objectives of the organisational leaders and be regarded as optional at best." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"The KPI juggernaut has been misused and abused in too many organisations to the extent it has devalued the concept of KPIs. KPIs used well - the ten things that really matter to an organisation - can, in my experience, be a real galvanising force to get focus and attention put in those areas which really can make a difference. The rest is a distraction, there through some misplaced view that more adds value when actually it detracts through losing the focus from where it needs to be." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"The nature of the change that the data strategy is to drive will be determined by the appetite and commitment of the organisation to change. It will also be shaped by the maturity of the organisation, with the maturity assessment process having identified and demonstrated where the gaps lie, and the resolve of the organisation to set its own pace and objectives to be achieved by the time of the next assessment." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

"The premise of OKRs is to keep objectives and results simple and flexible, ensuring they align with business goals and enterprise initiatives guided by regular reviews to assess progress during the quarter. The intent is to keep OKRs clear and accountable, as well as measurable, with between three and five objectives recommended at a high level that can each be tracked by three to five key measures. They should be ambitious goals, even uncomfortable, in challenging aspirations, making them stretch targets." (Ian Wallis, "Data Strategy: From definition to execution", 2021)

🎯Bennie Haelen - Collected Quotes

"A data lake is a cost-effective central repository to store structured, semi-structured, or unstructured data at any scale, in the form of files and blobs. The term 'data lake' came from the analogy of a real river or lake, holding the water, or in this case data, with several tributaries that are flowing the water (aka “data”) into the lake in real time." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data lakes have some very strong benefits. A data lake architecture enables the consolidation of an organization’s data assets into one central location. Data lakes are format agnostic and rely on open source formats, such as Parquet and Avro. These formats are well understood by a variety of tools, drivers, and libraries, enabling smooth interoperability. Data lakes are deployed on mature cloud storage subsystems, allowing them to benefit from the scalability, monitoring, ease of deployment, and low storage costs associated with these systems. Automated DevOps tools, such as Terraform, have well-established drivers, enabling automated deployments and maintenance." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data lakes require very large, scalable storage systems, like the ones typically offered in cloud environments. The storage needs to be durable and scalable and should offer interoperability with a variety of third-party tools, libraries, and drivers. Note that data lakes separate the concepts of storage and compute, allowing both to scale independently. Independent scaling of storage and compute allows for on-demand, elastic fine-tuning of resources, allowing our solution architectures to be more flexible. The ingress and egress channels to the storage systems should support high bandwidths, enabling the ingestion or consumption of large batch volumes, or the continuous flow of large volumes of streaming data, such as IoT and streaming media." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Data silos often start to develop as the gap between data engineering activities and data science activities begins to grow. Data scientists frequently spend the majority of their time creating separate ETL and data pipelines that clean and transform data and prepare it into features for their models. These silos usually develop because the tools and technologies used for data engineering don’t support the same activities for data scientists." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Delta Lake brings capabilities such as transactional reliability and support for UPSERTs and MERGEs to data lakes while maintaining the dynamic horizontal scalability and separation of storage and compute of data lakes. Delta Lake is one solution for building data lakehouses, an open data architecture combining the best of data warehouses and data lakes." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Like data lakes, the lakehouse architecture leverages low-cost cloud storage systems with the inherent flexibility and horizontal scalability of those systems. The goal of a lakehouse is to use existing high-performance data formats, such as Parquet, while also enabling ACID transactions (and other features). To add these capabilities, lakehouses use an open-table format, which adds features like ACID transactions, record-level operations, indexing, and key metadata to those existing data formats. This enables data assets stored on low-cost storage systems to have the same reliability that used to be exclusive to the domain of an RDBMS. Delta Lake is an example of an open-table format that supports these types of capabilities." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"MLOps best practices include the need to reproduce and validate every stage of the ML workflow. The ability to reproduce a model reduces the risk of errors, and ensures the correctness and robustness of the ML solution. Consistent data is the most difficult challenge faced in reproducibility, and an ML model will only reproduce the exact same result if the exact same data is used. And since data is constantly changing over time, this can introduce significant challenges to ML reproducibility and MLOps." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Modern data platforms leveraging ETL for analytics will always be consumers of data as they ingest data from various data sources. And as organizations continue to collect, process, and analyze data from a growing number of data sources, the ability to swiftly handle schema evolution and data validation is a critical aspect of any data platform. [...] Delta Lake gives you flexibility to evolve a table’s schema through dynamic and explicit schema updates, while also enforcing schema validation." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Querying by timestamp makes it easy to perform time-series analysis because we can compare the data of the same table to itself at two different points in time. And while there are other ETL patterns we can follow to capture historical data and enable time-series analysis (e.g., slowly changing dimensions and change data feeds), time travel provides a quick and easy way to perform ad hoc analysis for tables that may not have these ETL patterns in place." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Schema evolution in Delta Lake refers to the ability to evolve the schema of a Delta table over time, while preserving the existing data in the table. In other words, schema evolution allows us to add, remove, or modify columns in an existing Delta table without losing any data or breaking any downstream jobs that depend on the table. This is important as your data and business needs change over time and you may need to add new columns to your table or modify the existing columns to support new use cases." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"The main goal of the transaction log is to enable multiple readers and writers to operate on a given version of a dataset file simultaneously and to provide additional information, like data skipping indexes to the execution engine for more performant operations. The Delta Lake transaction log always shows the user a consistent view of the data and serves as a single source of truth. It is the central repository that tracks all changes the user makes to a Delta table." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"The lakehouse is centered around the idea of unification and combining the best elements of different technologies in a single place. This means it is also important that the data flow within the lakehouse itself supports this unification of data. In order to support all use cases, this data flow requires merging batch and streaming data into a single data flow to support scenarios across the entire data lifecycle."(Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Traditionally, data lakes have always operated under the principle of schema on read, but have always had challenges enforcing schema on write. This means there is no predefined schema when data is written to storage, and a schema is only adapted when the data is processed. It is imperative for the case of analytics and data platforms that your table formats enforce the schema on write to prevent introducing change-breaking processes, and to maintain proper data quality and integrity." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Unlike data warehouses, data lakes support all data types, including semi-structured and unstructured data, enabling workloads such as media processing. Because of their high throughput ingress channels, they are very well suited for streaming use cases, such as ingesting IoT sensor data, media streaming, or web clickstreams. However, as data lakes become more popular and widely used, organizations started recognizing some challenges with traditional data lakes. While the underlying cloud storage is relatively inexpensive, building and maintaining an effective data lake requires expert skills, resulting in high-end staffing or increased consulting services costs." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"Unstructured and semi-structured data are often critical for AI and machine learning use cases, whereas structured and semi-structured data are critical for BI use cases. Because it natively supports all three types of data classifications, you can create a unified system that supports these diverse workloads in a data lake. These workloads can complement each other in a well-designed processing architecture [...]. A data lake helps solve many of the challenges related to data volumes, types, and cost, and while Delta Lake runs on top of a data lake, it is optimized to run best on a cloud data lake." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"When changing clustered columns, liquid clustering does not require the entire table to be rewritten. This clustering evolution is due to the dynamic data layout feature of liquid clustering and offers a significant advantage over partition features mentioned earlier in the chapter. Traditional partitioning is a fixed data layout and does not support changing how a table is partitioned without having to rewrite the entire table. This clustering evolution can be essential as query patterns for a table can often change over time, and this allows you to dynamically adapt to new query patterns without any significant overhead or challenges." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"When you leverage Delta Lake with Structured Streaming, you get both the transactional guarantees of Delta Lake and the powerful programming model of Apache Spark Structured Streaming. With Delta Lake, you can now use Delta tables as both streaming sources and sinks, enabling a continuous processing model that processes your data through the Raw, Bronze, Silver, and Gold data lake layers in a streaming fashion, eliminating the need for batch jobs, resulting in a simplified solution architecture."(Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)

"With cloud data lakes you typically pay for what you use, so your costs always align with your data volumes. Since there is only a single storage layer, less data movement across different systems, availability settings, and decoupled storage versus compute, you have isolated and minimized costs for just data storage. For greater cost allocation, most cloud data lakes offer buckets, or containers (filesystems, not to be confused with application containers), to store different layers of the data (e.g., raw versus transformed data). These containers allow you to have finer-grained cost allocation for different areas of your organization. Since data sources and volumes are growing exponentially, it is extremely important to allocate and optimize costs without limiting the volume or variety of data that can be stored." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)