Business Intelligence: Mea Culpa (Part III: Problem Solving)

Business Intelligence Series

I've been working for more than 20 years in BI and Data Analytics area, in combination with Software Engineering, ERP implementations, Project Management, IT services and several other areas, which allowed me to look at many recurring problems from different perspectives. One of the things I learnt is that problems are more complex and more dynamic than they seem, respectively that they may require tailored dynamic solutions. Unfortunately, people usually focus on one or two immediate perspectives, ignoring the dynamics and the multilayered character of the problems!

Sometimes, a quick fix and limited perspective is what we need to get started and fix the symptoms, and problem-solvers usually stop there. When left unsupervised, the problems tend to kick back, build up momentum and appear under more complex forms in various places. Moreover, the symptoms can remain hidden until is too late. To this also adds the political agendas and the further limitations existing in organizations (people, money, know-how, etc.).

It seems much easier to involve external people (individual experts, consultancy companies) to solve the problem(s), though unless they get a deep understanding of the business and the issues existing in it, the chances are high that they solve the wrong problems and/or implement the wrong solutions. Therefore, it's more advisable to have internal experts, when feasible, and that's the point where business people with technical expertise and/or IT people with business expertise can help. Ideally, one should have a good mix and the so called competency centers can do a great job in handling the challenges of organizations. 

Between business and IT people there's a gap that can be higher or lower depending on resources know-how or the effort made by organizations to reduce it. To this adds the nature of the issues existing in organizations, which can vary considerable across departments, organizations or any other form of establishment. Conversely, the specific skillset can be transmuted where needed, which might happen naturally, though upon case also considerable effort needs to be involved in the process.

Being involved in similar tasks, one may get the impression that one can do whatever the others can do. This can happen in IT as well on the business side. There can be activities that can be done by parties from the other group, though there are also many exceptions in both directions, especially when one considers that one can’t generalize the applicability and/or transmutation of skillset. 

A more concrete example is the know-how needed by a businessperson to use the BI infrastructure for answering business questions, and ideally for doing all or at least most of the activities a BI professional can do. Ideally, as part of the learning path, it would be helpful to have a pursuable path in between the two points. The mastery of tools helps in the process though there are different mindsets involved.

Unfortunately, the data-related fields are full of overconfident people who get the problem-solving process wrong. Data-based problem-solving resumes in gathering the right facts and data, building the right conceptual model, identifying the right questions to ask, collecting more data, refining methods and solutions, etc. There’s aways an easy wrong way to solve a problem!

The mastery of tools doesn’t imply the mastery of business domains! What people from the business side can bring is deeper insight in the business problems, though getting from there to implementing solutions can prove a long way, especially when problems require different approaches, different levels of approximations, etc. No tool alone can bridge such gaps yet! Frankly, this is the most difficult to learn and unfortunately many data professionals seem to get this wrong!

Previous Post <<||>> Next Post

Business Intelligence: One Person Can’t Learn or Do Everything in Microsoft Fabric

Business Intelligence Series

Today’s Explicit Measures webcast [1] considered an article written by Kurt Buhler (The Data Goblins): [Microsoft] "Fabric is a Team Sport: One Person Can’t Learn or Do Everything" [2]. It’s a well-written article that deserves some thought as there are several important points made. I can’t say I agree with the full extent of some statements, even if some disagreements are probably just a matter of semantics.

My main disagreement starts with the title “One Person Can’t Learn or Do Everything”. As clarified in webcast's chat, the author defines “everything" as an umbrella for “all the capabilities and experiences that comprise Fabric including both technical (like Power BI) or non-technical (like adoption data literacy) and everything in between” [1].

For me “everything” is relative and considers a domain's core set of knowledge, while "expertise" (≠ "mastery") refers to the degree to which a person can use the respective knowledge to build back-to-back solutions for a given area. I’d say that it becomes more and more challenging for beginners or average data professionals to cover the core features. Moreover, I’d separate the non-technical skills because then one will also need to consider topics like Data, Project, Information or Knowledge Management.

There are different levels of expertise, and they can vary in depth (specialization) or breadth (covering multiple areas), respectively depend on previous experience (whether one worked with similar technologies). Usually, there’s a minimum of requirements that need to be covered for being considered as expert (e.g. certification, building a solution from beginning to the end, troubleshooting, performance optimization, etc.). It’s also challenging to roughly define when one’s expertise starts (or ends), as there are different perspectives on the topics. 

Conversely, the term expert is in general misused extensively, sometimes even with a mischievous intent. As “expert” is usually considered an external consultant or a person who got certified in an area, even if the person may not be able to build solutions that address a customer’s needs. 

Even data professionals with many years of experience can be overwhelmed by the volume of knowledge, especially when one considers the different experiences available in MF, respectively the volume of new features released monthly. Conversely, expertise can be considered in respect to only one or more MF experiences or for one area within a certain layer. Lot of the knowledge can be transported from other areas – writing SQL and complex database objects, modelling (enterprise) semantic layers, programming in Python, R or Power Query, building data pipelines, managing SQL databases, etc. 

Besides the standard documentation, training sessions, and some reference architectures, Microsoft made available also some labs and other material, which helps discovering the features available, though it doesn’t teach people how to build complete solutions. I find more important than declaring explicitly the role-based audience, the creation of learning paths for the various roles.

During the past 6-7 months I've spent on average 2 days per week learning MF topics. My problem is not the documentation but the lack of maturity of some features, the gaps in functionality, identifying the respective gaps, knowing what and when new features will be made available. The fact that features are made available or changed while learning makes the process more challenging. 

My goal is to be able to provide back-to-back solutions and I believe that’s possible, even if I might not consider all the experiences available. During the past 22 years, at least until MF, I could build complete BI solutions starting from requirements elicitation, data extraction, modeling and processing for data consumption, respectively data consumption for the various purposes. At least this was the journey of a Software Engineer into the world of data. 

References:
[1] Explicit Measures (2024) Power BI tips Ep.328: Microsoft Fabric is a Team Sport (link)
[2] Data Goblins (2024) Fabric is a Team Sport: One Person Can’t Learn or Do Everything (link)

Data Science: Generalists vs Specialists in the Field of Data Science

Division of labor favorizes the tasks done repeatedly, where knowledge of the broader processes is not needed, where aspects as creativity are needed only at a small scale. Division invaded the IT domains as tools, methodologies and demands increased in complexity, and therefore Data Science and BI/Analytics make no exception from this.

The scale of this development gains sometimes humorous expectations or misbelieves when one hears headhunters asking potential candidates whether they are upfront or backend experts when a good understanding of both aspects is needed for providing adequate results. The development gains tragicomical implications when one is limited in action only to a given area despite the extended expertise, or when a generalist seems to step on the feet of specialists, sometimes from the right entitled reasons. 

Headhunters’ behavior is rooted maybe in the poor understanding of the domain of expertise and implications of the job descriptions. It’s hard to understand how people sustain of having knowledge about a domain just because they heard the words flying around and got some glimpse of the connotations associated with the words. Unfortunately, this is extended to management and further in the business environment, with all the implications deriving from it. 

As Data Science finds itself at the intersection between Artificial Intelligence, Data Mining, Machine Learning, Neurocomputing, Pattern Recognition, Statistics and Data Processing, the center of gravity is hard to determine. One way of dealing with the unknown is requiring candidates to have a few years of trackable experience in the respective fields or in the use of a few tools considered as important in the respective domains. Of course, the usage of tools and techniques is important, though it’s a big difference between using a tool and understanding the how, when, why, where, in which ways and by what means a tool can be used effectively to create value. This can be gained only when one’s exposed to different business scenarios across industries and is a tough thing to demand from a profession found in its baby steps. 

Moreover, being a good data scientist involves having a deep insight into the businesses, being able to understand data and the demands associated with data – the various qualitative and quantitative aspects. Seeing the big picture is important in defining, approaching and solving problems. The more one is exposed to different techniques and business scenarios, with right understanding and some problem-solving skillset one can transpose and solve problems across domains. However, the generalist will find his limitations as soon a certain depth is reached, and the collaboration with a specialist is then required. A good collaboration between generalists and specialists is important in complex projects which overreach the boundaries of one person’s knowledge and skillset. 

Complexity is addressed when one can focus on the important characteristic of the problem, respectively when the models built can reflect the demands. The most important skillset besides the use of technical tools is the ability to model problems and root the respective problems into data, to elaborate theories and check them against reality. 

Complex problems can require specialization in certain fields, though seldom one problem is dependent only on one aspect of the business, as problems occur in overreaching contexts that span sometimes the borders of an organization. In addition, the ability to solve problems seem to be impacted by the diversity of the people involved into the task, sometimes even with backgrounds not directly related to organization’s activity. As in evolution, a team’s diversity is an important factor in achievement and learning, most gain being obtained when knowledge gets shared and harnessed beyond the borders of teams.

Note:
Written as answer to a Medium post on Data Science generalists vs specialists.

Performance Management: Competency (Definitions)

"An ability to perform business processes, which are supported by necessary available resources, practices, and activities, allowing the organization to offer products/services." (Jiri Hodík et al, "e-Cat for Partner Profiling and Competency Management Tool", 2008)

"Present or target capacity of a group or an individual to perform a cognitive, affective, social or psychomotor skill with regard to certain area of knowledge and in a specific context. The context consists in defining whether the skill can be attributed to the knowledge in a guided or autonomous way, in simple or complex, familiar or new situations, in a global or partial, persistent or sporadic manner." (Gilbert Paquette et al, "Principled Construction and Reuse of Learning Designs", Handbook of Research on Learning Design and Learning Objects: Issues, Applications, and Technologies, 2009)

"The ability to do something successfully or efficiently, often broken down into skills, knowledge, and attitude." (Alfonso Urquiza, "Competency Management Information Systems", 2009)

"The underlying characteristics of an individual (a motive, trait, skill, aspect of one’s self image or social role, or a body of knowledge) which underlie performance or behavior at work." (Jorge Valdés-Conca & Lourdes Canós-Darós, "B2E Relationships, Intranets, and Competency Management", 2009)

"A cluster of knowledges, understandings, skills, attitudes, values, and interests that are required for the performance of a function. In this case the function would be to be competent in counseling adult learners." (John A Henschke, "Counseling in an Andragogical Approach", 2012)

"A specific, identifiable, definable, and measurable knowledge, skill, ability, and/or other deployment-related characteristic (e.g., attitude, behavior, physical ability) which a human resource may possess and which is necessary for, or material to, the performance of an activity within a specific business context." Nancy B Hastings & Karen L Rasmussen, "Designing and Developing Competency-Based Education Courses Using Standards", 2017)

"Competency is the ability to demonstrate a specified level of knowledge or skill." (Christine K S Irvine & Jonathan M Kevan, "Competency-Based Education in Higher Education", 2017)

"Expected capacity the learner should build to be successful in his/her career. Competency is written in broader terms and are not directly measurable." (Devrim Ozdemir & Carla Stebbins, "A Framework for the Evaluation of Competency-Based Curriculum", 2017)

"Competencies are specific knowledge-based skills, abilities, or expertise in a subject area. When these skillsets are shared across a profession, they are said to have core competencies." (Valerie A Storey et al, "Developing a Clinical Leadership Pipeline: Planning, Operation, and Sustainability", 2019)

"Multidimensional construct which represents what a person is capable of doing. It includes knowledge, skills, experience, abilities, values, attitudes, personality traits, among others." (Geraldina Silveyra et al, "Proposal of a Comprehensive Model of Teachable Entrepreneurship Competencies (M-TEC): Literature Review and Theoretical Foundations", 2019)

"The ability to act successfully on the basis of practical experience, skill, and knowledge in solving professional problems. Is understood as a formal system characteristic, which is described as a set of requirements for the knowledge, skills and qualities of the employee for a function, position or role in the organization." (Vitaly V Martynov et al, "CSRP: System Design Technology of Training Information Support of Competent Professionals", 2019)

"A guiding tool including knowledge, abilities, distinguished personal attributes, and behaviours for higher performance contributing to achieving strategic goals of the company." (Mustafa K Topcu, "Competency Framework for the Fourth Industrial Revolution", 2020)

"Proficiency or mastery of identified knowledge, skills or abilities." (Ernst Jan van Weperen et al, "Sustainable Entrepreneurial Thinking: Developing Pro-Active, Globally Aware Citizens", 2020)

"Capacity to perform something in an effective manner. Involves individual attitudes, knowledge, and skills necessary, and behaviors." (Christiane Molina, "Management Education for a Sustainable World: Aiming for More Than Business as Usual", 2021)

"Competency refers to observable and measurable skills that integrate the knowledge, skills, values, and attitudes required of a professional in the practice of his specialty." (Maria M P Calimag, "The ePortfolio: Technology-Enhanced Authentic Assessment in the Continuum of Medical Education", 2021)

"The sum of knowledge, skills, values, attitudes, and individual characteristics that enable a person to perform actions successfully." (Almudena Eizaguirre et al, "A Methodological Proposal to Analyse the Process for Implementing Competency-Based Learning (CBL) in a Business School", 2021)

Software Engineering: Experts (Just the Quotes)

"Many people imagine that graphic charts cannot be understood except by expert mathematicians who have devoted years of study to the subject. This is a mistaken idea, and if instead of passing over charts as if they were something beyond their comprehension more people would make an effort to read them, much valuable time would be saved. It is true that some charts covering technical data are difficult even for an expert mathematician to understand, but this is more often the fault of the person preparing the charts than of the system." (Allan C Haskell, "How to Make and Use Graphic Charts", 1919)

"Today's scientific investigations are so complicated that even experts in related fields may not understand them well. But there is a logic in the planning of experiments and in the analysis of their results that all intelligent people can grasp, and this logic is a great help in determining when to believe what we hear and read and when to be skeptical. This logic has a great deal to do with statistics, which is why statisticians have a unique interest in the scientific method, and why some knowledge of statistics can so often be brought to bear in distinguishing good arguments from bad ones." (Robert Hooke, "How to Tell the Liars from the Statisticians", 1983)

"All things which are proved to be impossible must obviously rest on some assumptions, and when one or more of these assumptions are not true then the impossibility proof fails - but the expert seldom remembers to carefully inspect the assumptions before making their 'impossible' statements." (Richard Hamming, "The Art of Doing Science and Engineering: Learning to Learn", 1997)

"In an argument between a specialist and a generalist the expert usually wins by simply: (1) using unintelligible jargon, and (2) citing their specialist results which are often completely irrelevant to the discussion. The expert is, therefore, a potent factor to be reckoned with in our society. Since experts are both necessary, and also at times do great harm in blocking significant progress, they need to be examined closely. All too often the expert misunderstands the problem at hand, but the generalist cannot carry though their side to completion. The person who thinks they understand the problem and does not is usually more of a curse (blockage) than the person who knows they do not understand the problem." (Richard Hamming, "The Art of Doing Science and Engineering: Learning to Learn", 1997)

"Know the subject matter, learn it fast, or get a trustworthy expert. To identify the unknown, you must know the known. But don't be afraid to challenge experts on the basis of your logical reasoning. Sometimes a knowledge of the subject matter can blind the expert to the novel or unexpected." (Herbert F Spirer et al, "Misused Statistics" 2nd Ed, 1998)

"One reason we tend to accept statistics uncritically is that we assume that numbers come from experts who know what they're doing. [...] There is a natural tendency to treat these figures as straightforward facts that cannot be questioned." (Joel Best, "Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists", 2001)

"The fact that cognitive diversity matters does not mean that if you assemble a group of diverse but thoroughly uninformed people, their collective wisdom will be smarter than an expert's. But if you can assemble a diverse group of people who possess varying degrees of knowledge and insight, you're better off entrusting it with major decisions rather than leaving them in the hands of one or two people, no matter how smart those people are." (James Surowiecki, "The Wisdom of Crowds", 2005)

"Abstractions matter to users too. Novice users want programs whose abstractions are simple and easy to understand; experts want abstractions that are robust and general enough to be combined in new ways. When good abstractions are missing from the design, or erode as the system evolves, the resulting program grows barnacles of complexity. The user is then forced to master a mass of spurious details, to develop workarounds, and to accept frequent, inexplicable failures." (Daniel Jackson, "Software Abstractions", 2006)

"Much data in databases has a long history. It might have come from old 'legacy' systems or have been changed several times in the past. The usage of data fields and value codes changes over time. The same value in the same field will mean totally different thing in different records. Knowledge or these facts allows experts to use the data properly. Without this knowledge, the data may bc used literally and with sad consequences. The same is about data quality. Data users in the trenches usually know good data from bad and can still use it efficiently. They know where to look and what to check. Without these experts, incorrect data quality assumptions are often made and poor data quality becomes exposed." (Arkady Maydanchik, "Data Quality Assessment", 2007)

"Asking experts to do boring and repetitive, and yet technically demanding tasks is the most certain way of ensuring human error that we can think of, short of sleep deprivation, or inebriation." (David Farley & Jez Humble, "Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation", 2010)

"Experts in the 'Problem' area proceed to elaborate its complexity. They design complex Systems to attack it. This approach guarantees failure, at least for all but the most pedestrian tasks. The problem is a Problem precisely because it is incorrectly conceptualized in the first place, and a large System for studying and attacking the Problem merely locks in the erroneous conceptualization into the minds of everyone concerned. What is required is not a large System, but a different approach. Trying to design a System in the hope that the System will somehow solve the Problem, rather than simply solving the Problem in the first place, is to present oneself with two problems in place of one." (John Gall, "The Systems Bible: The Beginner's Guide to Systems Large and Small"[Systematics 3rd Ed.], 2011)

confusing, steeped in mystery and only truly understood by a few highly technical experts." (Alan Pennington, "The Customer Experience Book", 2016)

"Data from the customer interactions is the lifeblood for any organisation to view, understand and optimise the customer experience both remotely and on the front line! In the same way that customer experience experts understand that it’s the little things that count, it’s the small data that can make all the difference." (Alan Pennington, "The Customer Experience Book", 2016)

"Feature extraction is also the most creative part of data science and the one most closely tied to domain expertise. Typically, a really good feature will correspond to some real‐world phenomenon. Data scientists should work closely with domain experts and understand what these phenomena mean and how to distill them into numbers." (Field Cady, "The Data Science Handbook", 2017)

"The greatest leaders possess a combination of divergent traits: they are both experts and naïve, creative and efficient, serious and playful, social and reclusive - or at the very least, they surround themselves with this dynamic." (Beau Lotto, "Deviate: The Science of Seeing Differently", 2017)

"We, newbies and young programmers, don't like chaos because it makes us dependent on experts. We have to beg for information and feel bad." (Yegor Bugayenko, "Code Ahead", 2018)

"Data-intensive projects generally involve at least one person who understands all the nuances of the application, process, and source and target data. These are the people who also know about all the abnormalities in the data and the workarounds to deal with them, and are the experts. This is especially true in the case of legacy systems that store and use data in a manner it should not be used. The knowledge is not documented anywhere and is usually inside the minds of the people. When the experts leave, with no one having a true understanding of the data, the data are not used properly and everything goes haywire." (Rupa Mahanti, "Data Quality: Dimensions, Measurement, Strategy, Management, and Governance", 2019)

"I believe that the backlash against statistics is due to four primary reasons. The first, and easiest for most people to relate to, is that even the most basic concepts of descriptive and inferential statistics can be difficult to grasp and even harder to explain. […] The second cause for vitriol is that even well-intentioned experts misapply the tools and techniques of statistics far too often, myself included. Statistical pitfalls are numerous and tough to avoid. When we can't trust the experts to get it right, there's a temptation to throw the baby out with the bathwater. The third reason behind all the hate is that those with an agenda can easily craft statistics to lie when they communicate with us  […] And finally, the fourth cause is that often statistics can be perceived as cold and detached, and they can fail to communicate the human element of an issue." (Ben Jones, "Avoiding Data Pitfalls: How to Steer Clear of Common Blunders When Working with Data and Presenting Analysis and Visualizations", 2020)

"It is also important to note that data literacy is not about expecting to or becoming an expert; rather, it is a journey that must begin somewhere." (Angelika Klidas & Kevin Hanegan, "Data Literacy in Practice", 2022)

"Expert knowledge is a term covering various types of knowledge that can help define or disambiguate causal relations between two or more variables. Depending on the context, expert knowledge might refer to knowledge from randomized controlled trials, laws of physics, a broad scope of experiences in a given area, and more." (Aleksander Molak, "Causal Inference and Discovery in Python", 2023)