🧭Business Intelligence: Perspectives (Part XXV: Grounding the Roots)

Business Intelligence Series

When building something that is supposed to last, one needs a solid foundation on which the artifact can be built upon. That’s valid for castles, houses, IT architectures, and probably most important, for BI infrastructures. There are so many tools out there that allow building a dashboard, report or other types of BI artifacts with a few drag-and-drops, moving things around, adding formatting and shiny things. In many cases all these steps are followed to create a prototype for a set of ideas or more formalized requirements keeping the overall process to a minimum. 

Rapid prototyping, the process of building a proof-of-concept by focusing at high level on the most important design and functional aspects, is helpful and sometimes a mandatory step in eliciting and addressing the requirements properly. It provides a fast road from an idea to the actual concept, however the prototype, still in its early stages, can rapidly become the actual solution that unfortunately continues to haunt the dreams of its creator(s). 

Especially in the BI area, there are many solutions that started as a prototype and gained mass until they start to disturb many things around them with implications for security, performance, data quality, and many other aspects. Moreover, the mass becomes in time critical, to the degree that it pulled more attention and effort than intended, with positive and negative impact altogether. It’s like building an artificial sun that suddenly becomes a danger for the nearby planet(s) and other celestial bodies. 

When building such artifacts, it’s important to define what goals the end-result must or would be nice to have, differentiating clearly between them, respectively when is the time to stop and properly address the aspects mandatory in transitioning from the prototype to an actual solution that addresses the best practices in scope. It’s also the point when one should decide upon solution’s feasibility, needed quality acceptance criteria, and broader aspects like supporting processes, human resources, data, and the various aspects that have impact. Unfortunately, many solutions gain inertia without the proper foundation and in extremis succumb under the various forces.

Developing software artifacts of any type is a balancing act between all these aspects, often under suboptimal circumstances. Therefore, one must be able to set priorities right, react and change direction (and gear) according to the changing context. Many wish all this to be a straight sequential road, when in reality it looks more like mountain climbing, with many peaks, valleys and change of scenery. The more exploration is needed, the slower the progress.

All these aspects require additional time, effort, resources and planning, which can easily increase the overall complexity of projects to the degree that it leads to (exponential) effort and more important - waste. Moreover, the complexity pushes back, leading to more effort, and with it to higher costs. On top of this one has the iteration character of BI topics, multiple iterations being needed from the initial concept to the final solution(s), sometimes many steps being discarded in the process, corners are cut, with all the further implications following from this. 

Somewhere in the middle, between minimum and the broad overextending complexity, is the sweet spot that drives the most impact with a minimum of effort. For some organizations, respectively professionals, reaching and remaining in the zone will be quite a challenge, though that’s not impossible. It’s important to be aware of all the aspects that drive and sustain the quality of artefacts, data and processes. There’s a lot to learn from successful as well from failed endeavors, and the various aspects should be reflected in the lessons learned. 

Previous Post  <<||>>   Next Post

🏗️Software Engineering: Solutions (Just the Quotes)

"Design problems - generating or discovering alternatives - are complex largely because they involve two spaces, an action space and a state space, that generally have completely different structures. To find a design requires mapping the former of these on the latter. For many, if not most, design problems in the real world systematic algorithms are not known that guarantee solutions with reasonable amounts of computing effort. Design uses a wide range of heuristic devices - like means-end analysis, satisficing, and the other procedures that have been outlined - that have been found by experience to enhance the efficiency of search. Much remains to be learned about the nature and effectiveness of these devices." (Herbert A Simon, "The Logic of Heuristic Decision Making", [in "The Logic of Decision and Action"], 1966)

"Of course the technological base on which one builds is always advancing. As soon as one freezes a design, it becomes obsolete in terms of its concepts. But implementation of real products demands phasing and quantizing. The obsolescence of an implementation must be measured against other existing implementations, not against unrealized concepts. The challenge and the mission are to find real solutions to real problems on actual schedules with available resources." (Fred P Brooks, "The Mythical Man-Month: Essays", 1975)

"If solutions could be offered within the existing system, there would be no need to design. Thus designers have to transcend the existing system. Their task is to create a different system or devise a new one. That is why designers say they can truly define the problem only in light of the solution. The solution informs them as to what the real problem is." (Béla H Bánáthy, "Systems Design of Education", 1991)

"The methods of science include controlled experiments, classification, pattern recognition, analysis, and deduction. In the humanities we apply analogy, metaphor, criticism, and (e)valuation. In design we devise alternatives, form patterns, synthesize, use conjecture, and model solutions." (Béla H Bánáthy, "Designing Social Systems in a Changing World", 1996)

"The purpose of a conceptual model is to provide a vocabulary of terms and concepts that can be used to describe problems and/or solutions of design. It is not the purpose of a model to address specific problems, and even less to propose solutions for them. Drawing an analogy with linguistics, a conceptual model is analogous to a language, while design patterns are analogous to rhetorical figures, which are predefined templates of language usages, suited particularly to specific problems." (Peter P Chen [Ed.], "Advances in Conceptual Modeling", 1999)

"Design patterns are standard solutions to recurring problems, named to help people discuss them easily and to think about design. [...] Patterns are abstract, core solutions to problems that recur in different contexts but encounter the same 'forces' each time." (Ian Graham & Alan O'Callaghan, "An introduction to patterns", 2003)

"Design patterns provide the cores of ready-made solutions that can be used to solve many of software's most common problems. Some software problems require solutions that are derived from first principles. But most problems are similar to past problems, and those can be solved using similar solutions, or patterns." (Steve C McConnell, "Code Complete: A Practical Handbook of Software Construction" 2nd Ed., 2004)

"A heuristic is a rule applied to an existing solution represented in a perspective that generates a new" (and hopefully better) solution or a new set of possible solutions." (Scott E Page, "The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools and Societies", 2008)

"Anti-patterns are the dark side of patterns - common mistakes we fall into repeatedly. Sometimes they seem to be good solutions that experience has demonstrated don' work in practice. On other occasions, we know that they're not a good idea, but we fall into them anyway." (Paul Butcher, "Debug It! Find, Repair, and Prevent Bugs in Your Code", 2009)

"Interim solutions, however, acquire inertia (or momentum, depending on your point of view). Because they are there, ultimately useful and widely accepted, there is no immediate need to do anything else. Whenever a stakeholder has to decide what action adds the most value, there will be many that are ranked higher than proper integration of an interim solution. Why? Because it is there, it works, and it is accepted. The only perceived downside is that it does not follow the chosen standards and guidelines - except for a few niche markets, this is not considered to be a significant force." (Klaus Marquardt, [in Kevlin Henney’s "97 Things Every Programmer Should Know", 2010])

"Design patterns are high-level abstractions that document successful design solutions. They are fundamental to design reuse in object-oriented development." (Ian Sommerville, "Software Engineering" 9th Ed., 2011)

"A software design pattern is a reusable solution to a reoccurring software design problem. […] A design pattern is not a concrete design or implementation, such as an algorithm that can be used as-is, but rather a gene." (Eddie Burris, "Programming in the Large with Design Patterns", 2012)

"Knowledge of design patterns simplifies software design by reducing the number of design problems that have to be solved from first principles. Design problems that match documented design patterns have ready-made solutions. The remaining. problems that don't match documented design patterns must be solved from first principles. Even here, knowledge of design patterns can potentially help with original design. Design patterns are paragons of good design. Studying design patterns helps to develop the intellectual concepts and principles needed to solve unique design problems from first principles." (Eddie Burris, "Programming in the Large with Design Patterns", 2012)

"Not all problem-solution pairs are patterns. If the concept of a pattern is applied too loosely, it could dilute what it means to be a pattern. Although there is no formal criteria or litmus test for what is and isn’t a pattern, there are a few problem-solution pairs that are not generally thought of as patterns." (Eddie Burris, "Programming in the Large with Design Patterns", 2012)

"What makes a design pattern unique is its intent. The intent of a pattern is the problem solved or reason for using it. […] What distinguishes one pattern from another is the problem solved. You can infer the solution structure and problem solved from the pattern name but you can’t infer the pattern name from the solution structure alone." (Eddie Burris, "Programming in the Large with Design Patterns", 2012) 

"Automation is an essential backbone of DevOps. Automation is the use of solutions to reduce the need for human work. Automation can ensure that the software is built the same way each time, that the team sees every change made to the software, and that the software is tested and reviewed in the same way every day so that no defects slip through or are introduced through human error." (Michael Hüttermann et al, "DevOps for Developers", 2013)

"In analytics, it's more important for individuals to be able to formulate problems well, to prototype solutions quickly, to make reasonable assumptions in the face of ill-structured problems, to design experiments that represent good investments, and to analyze results." (Foster Provost & Tom Fawcett, "Data Science for Business", 2013)

"A design pattern usually suggests a scheme for structuring the classes in a design solution and defines the required interactions among those classes. In other words, a design pattern describes some commonly recurring structure of communicating classes that can be used to solve some general design problems. Design pattern solutions are typically described in terms of classes, their instances, their roles and collaborations." (Rajib Mall, "Fundamentals of Software Engineering" 4th Ed., 2014)

"Sometimes you can't fit everything in. Remember that the sprint is great for testing risky solutions that might have a huge payoff. So you'll have to reverse the way you would normally prioritize. If a small fix is so good and low-risk that you're already planning to build it next week, then seeing it in a prototype won't teach you much. Skip those easy wins in favor of big, bold bets." (Jake Knapp et al, "Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days", 2016)

"The art of questioning is to ignite innovative thinking; the science of questioning is to frame system thinking, with the progressive pursuit of better solutions." (Pearl Zhu, "Leadership Master: Five Digital Trends to Leap Leadership Maturity", 2016)

"Programming is the process of taking an algorithm and encoding it into a notation that the computer can execute. These notation systems are referred to as programming languages. [...] Programming is an important part of what a computer scientist does. It is through programming that we create a representation of our solution. However, the solutions that we achieve are often affected by the process and language that we choose. [...] Programming languages must then provide a way to represent both the process and the data required by the solution." (Bradley N Miller et al, "Python Programming in Context", 2019)

"One of the limitations of data management solutions today is how we have attempted to manage its unwieldy complexity, how we have decomposed an ever-growing monolithic data platform and team to smaller partitions. We have chosen the path of least resistance, a technical partitioning." (Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale", 2021)

"Finding the right answer is important, of course. But more important is developing the ability to see that problems have multiple solutions, that getting from X to Y demands basic skills and mental agility, imagination, persistence, patience." (Mary H Futrell)

"Some problems are just too complicated for rational logical solutions. They admit of insights, not answers." (Jerome B Wiesner)