"When you leverage Delta Lake with Structured Streaming, you get both the transactional guarantees of Delta Lake and the powerful programming model of Apache Spark Structured Streaming. With Delta Lake, you can now use Delta tables as both streaming sources and sinks, enabling a continuous processing model that processes your data through the Raw, Bronze, Silver, and Gold data lake layers in a streaming fashion, eliminating the need for batch jobs, resulting in a simplified solution architecture." (Bennie Haelen & Dan Davis, "Delta Lake: Up and Running - Modern Data Lakehouse Architectures with Delta Lake", 2023)
"A lake based on the medallion architecture combines the best of lakes and data warehouses. By breaking down silos and eliminating data duplication, it becomes a standard for building data platform architecture." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)
"A Medallion architecture is a data design pattern used to logically organize data, most often in a lakehouse, using three layers for the data platform, with the goal of incrementally and progressively improving the structure and quality of data as it flows through each layer of the data architecture (from Bronze ⇒ Silver ⇒ Gold layer)." (Piethein Strengholt, "Building Medallion Architectures: Designing with Delta Lake and Spark", 2025)
"But what makes the Medallion architecture so crucial for your organization’s data strategy? And what compelling conclusions can be drawn from this journey? The answer lies in its flexible, modular approach that allows organizations to tailor their data processes to specific needs. While the concept of three distinct layers offers a structured approach, it’s not a one-size-fits-all solution. The key is understanding the strengths and limitations of each layer, which can be adapted to better align with operational realities and strategic goals." (Piethein Strengholt, "Building Medallion Architectures: Designing with Delta Lake and Spark", 2025)
"Implementing a Medallion architecture isn’t about following a checklist set of instructions. It’s not about taking the exact, fixed process and trying to fit your unique organization around it. Instead, focus on providing your data consumers with context - organize your data so they understand when it has been cleaned, when it is ready for consumption. Help them find the data to empower their work as fast and easily as possible. Once you have that, you can fit the processes to your design." (Piethein Strengholt, "Building Medallion Architectures: Designing with Delta Lake and Spark", 2025)
"[...] Medallion architectures are a fantastic concept that is widely misunderstood. Too often, they’re treated as a rigid, step-by-step framework, when they are actually a flexible approach to making sense of an evolving landscape. They are an attempt to simplify a decade of organic evolution and technical innovation into concepts that can be presented to nontechnical users. But simplification comes at a cost: it leaves huge gaps for debate, misinterpretation, and frustration." (Piethein Strengholt, "Building Medallion Architectures: Designing with Delta Lake and Spark", 2025)
"Medallion architectures emerged as the bridge to guide organizations through the lakehouse era. Instead of being the domain of niche companies with unique technical challenges, data lakes have become the de facto technology for data platforms; the doors have been thrown wide open - come on in, the water’s lovely." (Piethein Strengholt, "Building Medallion Architectures: Designing with Delta Lake and Spark", 2025)
"Traditionally, data engineers are responsible for the first steps of data transformation, commonly referred to as the transition from the 'bronze' stage to the 'silver' stage. This phase includes the normalization of raw data to clean and organize it into a structured and accessible format. Data Engineers ensure that data is properly ingested, stored, and prepared for subsequent steps. Their work focuses on building robust data pipelines and applying basic transformations that make the data usable. Next, responsibility may be handed over to an analytics engineer, who takes charge of the transition from the 'silver' stage to the 'gold' stage. This step involves more complex transformations aimed at refining, enriching, and modeling the data to meet specific analytical needs. The analytics engineer ensures that the data is ready to be used in reports, dashboards, and advanced analyses. The transition to the 'gold' stage means that the data is fully prepared for analytic use, providing strategic insights from consolidated data sources." (Christopher Maneu et al, "The Definitive Guide to Microsoft Fabric From discovery to building a unified, secure, and scalable data platform", 2025)
No comments:
Post a Comment