tag:blogger.com,1999:blog-187307432024-03-19T03:05:17.328+01:00SQL TroublesA blog on SQL, data, databases, programming, Project Management and IT related topicsAdrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.comBlogger1913125tag:blogger.com,1999:blog-18730743.post-80227336582348352462024-03-18T18:53:00.008+01:002024-03-18T19:17:08.495+01:00Strategic Management: Strategy (Notes)Disclaimer: This is work in progress intended to consolidate information from various sources. Last updated: 18-Mar-2024Strategy{definition} "the determination of the long-term goals and objectives of an enterprise, and the adoption of courses of action and the allocation of resources necessary for carrying out these goals" [4]{goal} bring all tools and insights together to create an Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-53334271884839951102024-03-17T21:27:00.003+01:002024-03-18T23:50:27.604+01:00Business Intelligence: Data Products (Part II: The Complexity Challenge) Business Intelligence SeriesCreating data products within a data mesh resumes in "partitioning" a given set of inputs, outputs and transformations to create something that looks like a Lego structure, in which each Lego piece represents a data product. The word partition is improperly used as there can be overlapping in terms of inputs, outputs and transformations, though in an ideal solution theAdrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-74135989757446971232024-03-17T18:33:00.008+01:002024-03-18T22:09:55.121+01:00Business Intelligence: Data Products (Part I: A Lego Exercise) Business Intelligence SeriesOne can define a data product as the smallest unit of data-driven architecture that can be independently deployed and managed (aka product quantum) [1]. In other terms one can think of a data product like a box (or Lego piece) which takes data as inputs, performs several transformations on the data from which result several output data (or even data visualizations or aAdrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-31547452527687954342024-03-16T06:11:00.003+01:002024-03-19T00:55:43.802+01:00Business Intelligence: A Software Engineer's Perspective VII (Think for Yourself!)Business Intelligence SeriesAfter almost a quarter-century of professional experience
the best advice I could give to younger professionals is to "gather
information and think for themselves", and with this the reader can close
the page and move forward! Anyway, everybody seems to be looking for sudden
enlightenment with minimal effort, as if the effort has no meaning in the
process!
In whateverAdrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-9709258052599403642024-03-15T18:16:00.017+01:002024-03-18T21:22:56.481+01:00Data Warehousing: Data Mesh (Notes) Disclaimer: This is work in progress intended to consolidate information from various sources. Last updated: 17-Mar-2024Data Products with a Data MeshData Mesh{definition} "a sociotechnical approach to share, access and manage analytical data in complex and large-scale environments - within or across organizations" [1]⇐ there is no default standard or reference implementation of data Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-4318131077618485992024-03-14T04:15:00.017+01:002024-03-19T01:47:15.335+01:00Business Intelligence: Zhamak Dehghani's Data Mesh - Monolithic Warehouses and Lakes (Debunked) Business Intelligence SeriesIn [1] the author categorizes data warehouses (DWHs) and lakes as monolithic architectures, as opposed to data mesh's distributed architecture, which makes me circumspect about term's use. There are two general definitions of what monolithic means: (1) formed of a single large block (2) large, indivisible, and slow to change.In software architecture one can Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-10127398864000979852024-03-13T20:01:00.017+01:002024-03-19T00:17:44.382+01:00Book Review: Zhamak Dehghani's Data Mesh: Delivering Data-Driven Value at Scale (2021)
Zhamak Dehghani's "Data Mesh: Delivering Data-Driven Value at Scale" (2021) is a must read book for the data professional. So, here I am, finally managing to read it and give it some thought, even if it will probably take more time and a few more reads for the ideas to grow. Working in the fields of Business Intelligence and Software Engineering for almost a Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-32652609163595397352024-03-12T06:43:00.023+01:002024-03-18T23:49:32.038+01:00Systems Engineering: A Play of Problems (Much Ado about Nothing) Disclaimer: This post was created just for fun. No problem was hurt or solved in the process! Updated: 16-Mar-2024On ProblemsEverybody has at least a problem. If somebody doesn’t have a problem, he’ll make one. If somebody can't make a problem, he can always find a problem. One doesn't need to search long for finding a problem. Looking for a problem one sees problems. Not having a Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-16265859066580389982024-03-12T03:08:00.002+01:002024-03-12T03:14:28.301+01:00Microsoft Fabric: OneLake (Notes) Disclaimer: This is work in progress intended to consolidate information from various sources. Last updated: 12-Mar-2024Microsoft Fabric & OneLakeOneLakea single, unified, logical data lake for the whole organization [2]designed to be the single place for all an organization's analytics data [2]provides a single, integrated environment for data professionals and the business to Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-39014773373650089972024-03-11T21:56:00.002+01:002024-03-11T22:16:17.094+01:00Business Intelligence: Key Performance Indicators (Between Certainty and Uncertainty)Business Intelligence SeriesDespite the huge collection of documented Key Performance Indicators (KPIs) and best practices on which KPIs to choose, choosing a reliable set of KPIs that reflect how the organization performs in achieving its objectives continues to be a challenge for many organizations. Ideally, for each objective there should be only one KPIs that reflects the target and the Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-13506230684234626542024-03-10T22:15:00.003+01:002024-03-12T03:11:26.903+01:00Microsoft Fabric: Medallion Architecture (Notes)Disclaimer: This is work in progress intended to consolidate information from various sources. Last updated: 10-Mar-2024Medallion Architecture in Microsoft Fabric [1]Medallion architecturea recommended data design pattern used to organize data in a lakehouse logically [2]compatible with the concept of data mesh{goal} incrementally and progressively improve the structure and quality of data Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-34008331716380745452024-03-10T20:17:00.003+01:002024-03-10T22:16:57.703+01:00Microsoft Fabric: Lakehouse (Notes) Disclaimer: This is work in progress intended to consolidate information from various sources. Last updated: 10-Mar-2024Lakehousea unified platform that combines the capabilities of data lakebuilt on top of the OneLake scalable storage layer using and Delta format tables [1]support ACID transactions through Delta Lake formatted tables for data consistency and integrity&Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-43949582266103070742024-03-10T03:51:00.014+01:002024-03-10T19:56:49.957+01:00Power BI: Dataflows Gen 1 (Notes) Disclaimer: This is work in progress intended to consolidate information from various sources. Last updated: 10-Mar-2024Dataflows Architecture [3]Dataflow (Gen1)a type of cloud-based ETL tool for building and executing scalable data transformation processes [1]a collection of tables created and managed in workspaces in the Power BI service [4]acts as building blocks on top of one Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-53121452215699543452024-03-10T01:38:00.018+01:002024-03-15T18:07:34.048+01:00Microsoft Fabric: Dataflows Gen2 (Notes)Disclaimer: This is work in progress intended to consolidate information from various sources. Last updated: 10-Mar-2024Dataflow (Gen2) Architecture [4]Dataflow (Gen2) new generation of dataflows that resides alongside the Power BI Dataflow (Gen1) [2]brings new features and improved experience [2]similar to Dataflow Gen1 in Power BI [2]allows to extract data from various Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-18093267791121097432024-03-07T18:14:00.000+01:002024-03-07T18:14:13.790+01:00Data Migrations (DM): The SQL Server Perspective (Licensing Costs and Edition Choices)Data Migration SeriesA Data Migration (DM) moves all or a subset of the data available from one or more system(s) into other system(s). For this purpose, especially in ERP Implementations, one can use a SQL Server as intermediate layer, where SSIS can be used for the data extraction and exporting, SSRS for reporting the errors, while the database engine for the heavy processing. Master Data and Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-32423599151978100732024-03-06T15:47:00.003+01:002024-03-06T18:25:25.518+01:00Business Intelligence: Data Culture and Leadership (Necessary but not Sufficient) Business Intelligence SeriesContinuing the idea from the previous post on Brent Dykes’
article on data culture and Generative AI [1], it’s worth discussing about the
relationship between data culture and leadership. Leadership belongs to a list
of select words everybody knows about but fails to define them precisely, especially
when many traits are associated with leadership, respectively when Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-3006472529688410072024-03-05T18:11:00.011+01:002024-03-06T18:26:03.150+01:00Business Intelligence: Data Culture and Generative AI (No Silver Bullet)Business Intelligence SeriesTalking about holy grails in Data Analytics, another topic of major importance for an organization’s "infrastructure" is data culture, that can be defined as the collective beliefs, values, behaviors, and practices of an organization’s employees in harnessing the value of data for decision-making, operations, or insight. Rooted in data literacy, data culture is an Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-47076597705454871572024-03-04T04:33:00.007+01:002024-03-15T18:02:21.476+01:00Business Intelligence: Microsoft Fabric's Domains and the Data Mesh I (The Challenge of Structure Matching)Business Intelligence SeriesThe holy grail of building a Data Analytics infrastructure seems to be nowadays the creation of a data mesh, a decentralized data architecture that organizes data by specific business domains. This endeavor proves to be difficult to achieve given the various challenges faced – data integration, data ownership, data product creation and ownership, enablement of Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-7292776950865596732024-03-04T00:38:00.012+01:002024-03-19T01:22:37.993+01:00Business Intelligence: A Software Engineer's Perspective VI (The Data Citizen) Business Intelligence SeriesMore than a century ago, Jerbert G Wells wrote on mathematical literacy: "[...] the time may not be very remote when it will be understood that for complete initiation as an efficient citizen of one of the new great complex world-wide States that are now developing, it is as necessary to be able to compute, to think in averages and maxima and minima, as it is now to beAdrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-47882425891966636992024-03-02T01:12:00.006+01:002024-03-04T17:35:44.706+01:00Business Intelligence: Microsoft Releases for the BI Technology Stack (Timeline)Business Intelligence SeriesI started some years back to put together a timeline for the most important events happening in the BI technology stack (work in progress):2023: Microsoft announces Microsoft Fabric (>>)Synapse Data Warehouse is the next generation of data warehousing in Microsoft Fabric with native support for the delta lake.Data Engineering & Data Science workloads with Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-61290984723319238612024-02-29T23:39:00.015+01:002024-03-18T23:09:43.417+01:00R Language: Visualizing the Iris DatasetWhen working with a dataset that has several numeric features, it's useful to visualize it to understand the shapes of each feature, usually by category or in the case of the iris dataset by species. For this purpose one can use a combination between a boxplot and a stripchart to obtain a visualization like the one below (click on the image for a better resolution):Iris features by species (box &Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-57472306533277027972024-02-28T05:15:00.006+01:002024-03-19T00:56:15.491+01:00Business Intelligence: A Software Engineer's Perspective V (From Process Management to Mental Models in Knowledge Gaps)Business Intelligence Series An organization's business processes are probably one of its most important assets because they reflect the business model, philosophy and culture, respectively link the material, financial, decisional, informational and communicational flows across the whole organization with implication in efficiency, productivity, consistency, quality, adaptability, agility, Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-28713237499331523652024-02-27T22:58:00.010+01:002024-03-18T23:12:24.126+01:00Book Review: Rolf Hichert & Jürgen Faisst's International Business Communication Standards (IBCS Version 1.2)Over the last months I found several references to Rolf Hichert & Jürgen Faisst's booklet on business communication standards [1]. It draw my attention especially because it attempts to provide a standard for reports and data visualizations, which frankly it seems like a tremendous endeavor if done right. The two authors founded the IBCS institute 20 years ago, which is a host, training Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-88286016668930242852024-02-26T23:54:00.006+01:002024-02-27T01:51:18.701+01:00R Language: Data Summaries without Using a DataFrameComing back to the R language after several years and trying to remember some basic functions proved to be a bit challenging, even if the syntax is quite simple. Therefore, I considered putting together a few calls as refresher based on Youden-Beale data. To run the below code you'll need to install the R language and RStudio.In case you don't have the package installed, run the next two lines:
Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0tag:blogger.com,1999:blog-18730743.post-66081328907962880992024-02-21T07:44:00.006+01:002024-03-19T00:56:29.333+01:00Business Intelligence: A Software Engineer's Perspective IV (The Loom of Interactions) Business Intelligence Series The process of developing or creating a report is quite simple - there's a demand for data, usually a business problem, the user (aka requestor) defines a set of requirements, the data professional writes one or more queries to address the requirements, which are then used to build one or more reports. The report(s) is/are reviewed by the requestor and with this Adrianhttp://www.blogger.com/profile/07612735744573114626noreply@blogger.com0