Showing posts with label understanding. Show all posts
Showing posts with label understanding. Show all posts

15 February 2025

🧭Business Intelligence: Perspectives (Part XXVII: A Tale of Two Cities II)

Business Intelligence Series
Business Intelligence Series
There’s a saying that applies to many contexts ranging from software engineering to data analysis and visualization related solutions: "fools rush in where angels fear to tread" [1]. Much earlier, an adage attributed to Confucius provides a similar perspective: "do not try to rush things; ignore matters of minor advantage". Ignoring these advices, there's the drive in rapid prototyping to jump in with both feet forward without checking first how solid the ground is, often even without having adequate experience in the field. That’s understandable to some degree – people want to see progress and value fast, without building a foundation or getting an understanding of what’s happening, respectively possible, often ignoring the full extent of the problems.

A prototype helps to bring the requirements closer to what’s intended to achieve, though, as the practice often shows, the gap between the initial steps and the final solutions require many iterations, sometimes even too many for making a solution cost-effective. There’s almost always a tradeoff between costs and quality, respectively time and scope. Sooner or later, one must compromise somewhere in between even if the solution is not optimal. The fuzzier the requirements and what’s achievable with a set of data, the harder it gets to find the sweet spot.

Even if people understand the steps, constraints and further aspects of a process relatively easily, making sense of the data generated by it, respectively using the respective data to optimize the process can take a considerable effort. There’s a chain of tradeoffs and constraints that apply to a certain situation in each context, that makes it challenging to always find optimal solutions. Moreover, optimal local solutions don’t necessarily provide the optimum effect when one looks at the broader context of the problems. Further on, even if one brought a process under control, it doesn’t necessarily mean that the process works efficiently.

This is the broader context in which data analysis and visualization topics need to be placed to build useful solutions, to make a sensible difference in one’s job. Especially when the data and processes look numb, one needs to find the perspectives that lead to useful information, respectively knowledge. It’s not realistic to expect to find new insight in any set of data. As experience often proves, insight is rarer than finding gold nuggets. Probably, the most important aspect in gold mining is to know where to look, though it also requires luck, research, the proper use of tools, effort, and probably much more.

One of the problems in working with data is that usually data is analyzed and visualized in aggregates at different levels, often without identifying and depicting the factors that determine why data take certain shapes. Even if a well-suited set of dimensions is defined for data analysis, data are usually still considered in aggregate. Having the possibility to change between aggregates and details is quintessential for data’s understanding, or at least for getting an understanding of what's happening in the various processes. 

There is one aspect of data modeling, respectively analysis and visualization that’s typically ignored in BI initiatives – process-wise there is usually data which is not available and approximating the respective values to some degree is often far from the optimal solution. Of course, there’s often a tradeoff between effort and value, though the actual value can be quantified only when gathering enough data for a thorough first analysis. It may also happen that the only benefit is getting a deeper understanding of certain aspects of the processes, respectively business. Occasionally, this price may look high, though searching for cost-effective solutions is part of the job!

Previous Post  <<||>> Next Post

References:
[1] Alexander Pope (cca. 1711) An Essay on Criticism

11 September 2024

🗄️Data Management: Data Culture (Part IV: Quo vadis? [Where are you going?])

Data Management Series

The people working for many years in the fields of BI/Data Analytics, Data and Process Management probably met many reactions that at the first sight seem funny, though they reflect bigger issues existing in organizations: people don’t always understand the data they work with, how data are brought together as part of the processes they support, respectively how data can be used to manage and optimize the respective processes. Moreover, occasionally people torture the data until it confesses something that doesn’t necessarily reflect the reality. It’s even more deplorable when the conclusions are used for decision-making, managing or optimizing the process. In extremis, the result is an iterative process that creates more and bigger issues than whose it was supposed to solve!

Behind each blunder there are probably bigger understanding issues that need to be addressed. Many of the issues revolve around understanding how data are created, how are brought together, how the processes work and what data they need, use and generate. Moreover, few business and IT people look at the full lifecycle of data and try to optimize it, or they optimize it in the wrong direction. Data Management is supposed to help, and it does this occasionally, though a methodology, its processes and practices are as good as people’s understanding about data and its use! No matter how good a data methodology is, it’s as weak as the weakest link in its use, and typically the issues revolving around data and data understanding are the weakest link. 

Besides technical people, few businesspeople understand the full extent of managing data and its lifecycle. Unfortunately, even if some of the topics are treated in the books, they are too dry, need hands on experience and some thought in corroborating practices with theories. Without this, people will do things mechanically, processes being as good as the people using them, their value becoming suboptimal and hinder the business. That’s why training on Data Management is not enough without some hands-on experience!

The most important impact is however in BI/Data Analytics areas - how the various artifacts are created and used as support in decision-making, process optimization and other activities rooted in data. Ideally, some KPIs and other metrics should be enough for managing and directing a business, however just basing the decisions on a set of KPIs without understanding the bigger picture, without having a feeling of the data and their quality, the whole architecture, no matter how splendid, can breakdown as sandcastle on a shore meeting the first powerful wave!

Sometimes it feels like organizations do things from inertia, driven by the forces of the moment, initiatives and business issues for which temporary and later permanent solutions are needed. The best chance for solving many of the issues would have been a long time ago, when the issues were still small to create any powerful waves within the organizations. Therefore, a lot of effort is sometimes spent in solving the consequences of decisions not made at the right time, and that can be painful and costly!

For building a good business one needs also a solid foundation. In the past it was enough to have a good set of products that are profitable. However, during the past decade(s) the rules of the game changed driven by the acerb competition across geographies, inefficiencies, especially in the data and process areas, costing organizations on the short and long term. Data Management in general and Data Quality in particular, even if they’re challenging to quantify, have the power to address by design many of the issues existing in organizations, if given the right chance!

Previous Post <<||>> Next Post

28 December 2007

🏗️Software Engineering: Understanding (Just the Quotes)

"I consider computer science to be the art and science of exploiting automatic digital computers, and of creating the technology necessary to understand their use. It deals with such related problems as the design of better machines using known components:, the design and implementation of adequate software systems for communication between man and machine, and the design and analysis of methods of representing information by abstract symbols and of processes for manipulating these symbols." (George E Forsythe, "Stanford University's Program in Computer Science", 1965) 

"Most programs are too big to be comprehended as a single chunk. They must be divided into smaller pieces that can be conquered separately. That is the only way to write them reliably; it is the only way to read and understand them. [...] When a program is not broken up into small enough pieces, the larger modules often fail to deliver on these promises. They try to do too much, or too many different things, and hence are difficult to maintain and are too specialized for general use." (Brian W Kernighan & Phillip J Plauger, "The Elements of Programming Style", 1974)

"Recursion represents no saving of time or storage. Somewhere in the computer must be maintained a list of all the places a recursive routine is called, so the program can eventually find its way back. But the storage for that list is shared among many different uses. More important, it is managed automatically; many of the burdens of storage management and control flow are placed on the compiler, not on the programmer. And since bookkeeping details are hidden, the program can be much easier to understand. Learning to think recursively takes some effort, but it is repaid with smaller and simpler programs." (Brian W Kernighan & Phillip J Plauger, "The Elements of Programming Style", 1974)

"The beginning of wisdom for a programmer is to recognize the difference between getting his program to work and getting it right. A program which does not work is undoubtedly wrong; but a program which does work is not necessarily right. It may still be wrong because it is hard to understand; or because it is hard to maintain as the problem requirements change; or because its structure is different from the structure of the problem; or because we cannot be sure that it does indeed work." (Michael A Jackson, "Principles of Program Design", 1975)

"The aim of the model is of course not to reproduce reality in all its complexity. It is rather to capture in a vivid, often formal, way what is essential to understanding some aspect of its structure or behavior." (Joseph Weizenbaum, "Computer power and human reason: From judgment to calculation" , 1976)

"Make no mistake about it: Computers process numbers - not symbols. We measure our understanding (and control) by the extent to which we can arithmetize an activity." (Alan J Perlis, "Epigrams on Programming", 1982)

"Wherever there is modularity there is the potential for misunderstanding: Hiding information implies a need to check communication." (Alan J Perlis, "Epigrams on Programming", 1982)

"Whether you call it a 'team' or an 'ensemble' or a 'harmonious work group' is not what matters; what matters is helping all parties understand that the success of the individual is tied irrevocably to the success of the whole." (Tom DeMarco & Timothy Lister, "Peopleware: Productive Projects and Teams", 1987)

"[Object-oriented analysis is] the challenge of understanding the problem domain and then the system's responsibilities in that light." (Edward Yourdon, "Object-Oriented Design", 1991) 

"An important symptom of an emerging understanding is the capacity to represent a problem in a number of different ways and to approach its solution from varied vantage points; a single, rigid representation is unlikely to suffice." (Howard Gardner, "The Unschooled Mind", 1991)

"The fundamentals of language are not understood to this day. [...] Until we understand languages of communication involving humans as they are then it is unlikely many of our software problems will vanish." (Richard W Hamming, "The Art of Probability for Scientists and Engineers", 1991)

"A problem with this 'waterfall' approach is that there will then be no user interface to test with real users until this last possible moment, since the 'intermediate work products' do not explicitly separate out the user interface in a prototype with which users can interact. Experience also shows that it is not possible to involve the users in the design process by showing them abstract specifications documents, since they will not understand them nearly as well as concrete prototypes." (Jakob Nielsen, "Usability Engineering", 1993)

"One should not start full-scale implementation efforts based on early user interface designs. Instead, early usability evaluation can be based on prototypes of the final systems that can be developed much faster and much more cheaply, and which can thus be changed many times until a better understanding of the user interface design has been achieved." (Jakob Nielsen, "Usability Engineering", 1993)

"Users are not designers, so it is not reasonable to expect them to come up with design ideas from scratch. However, they are very good at reacting to concrete designs they do not like or that will not work in practice. To get full benefits from user involvement, it is necessary to present these suggested system designs in a form the users can understand." (Jakob Nielsen, "Usability Engineering", 1993)

"Crude complexity is the length of the shortest message that will describe a system, at a given level of coarse graining, to someone at a distance, employing language, knowledge, and understanding that both parties share (and know they share) beforehand." (Murray Gell-Mann, "What is Complexity?" Complexity Vol. 1" (1), 1995)

"Good design protects you from the need for too many highly accurate components in the system. But such design principles are still, to this date, ill-understood and need to be researched extensively. Not that good designers do not understand this intuitively, merely it is not easily incorporated into the design methods you were taught in school. Good minds are still needed in spite of all the computing tools we have developed." (Richard Hamming, "The Art of Doing Science and Engineering: Learning to Learn", 1997)

"In an argument between a specialist and a generalist the expert usually wins by simply:" (1) using unintelligible jargon, and" (2) citing their specialist results which are often completely irrelevant to the discussion. The expert is, therefore, a potent factor to be reckoned with in our society. Since experts are both necessary, and also at times do great harm in blocking significant progress, they need to be examined closely. All too often the expert misunderstands the problem at hand, but the generalist cannot carry though their side to completion. The person who thinks they understand the problem and does not is usually more of a curse" (blockage) than the person who knows they do not understand the problem." (Richard Hamming, "The Art of Doing Science and Engineering: Learning to Learn", 1997)

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand." (Martin Fowler, "Refactoring: Improving the Design of Existing Code", 1999)

"Computer programs are complex by nature. Even if you could invent a programming language that operated exactly at the level of the problem domain, programming would be complicated because you would still need to precisely define relationships between real-world entities, identify exception cases, anticipate all possible state transitions, and so on. Strip away the accidental work involved in representing these factors in a specific programming language and in a specific computing environment, and you still have the essential difficulty of defining the underlying real-world concepts and debugging your understanding of them." (Steve C McConnell," After the Gold Rush : Creating a True Profession of Software Engineering", 1999)

"We plan because: We need to ensure that we are always working on the most important thing we need to do. We need to coordinate with other people. When unexpected events occur we need to understand the consequences for the first two." (Kent Beck & Martin Fowler, "Planning Extreme Programming", 2000)

"Note that a project always begins as a concept, and a concept is usually a bit fuzzy. Our job as a team is to clarify the concept, to turn it into a shared understanding that the entire team will accept. It is failure to do this that causes many project failures." (James P Lewis, "Project Planning, Scheduling, and Control" 3rd Ed., 2001)

"As the least conscious layer of the user experience, the conceptual model has the paradoxical quality of also having the most impact on usability. If an appropriate conceptual model is faithfully represented throughout the interface, after users recognize and internalize the model, they will have a fundamental understanding of what the application does and how to operate it." (Bob Baxley, "Making the Web Work: Designing Effective Web Applications", 2002)

"We build models to increase productivity, under the justified assumption that it's cheaper to manipulate the model than the real thing. Models then enable cheaper exploration and reasoning about some universe of discourse. One important application of models is to understand a real, abstract, or hypothetical problem domain that a computer system will reflect. This is done by abstraction, classification, and generalization of subject-matter entities into an appropriate set of classes and their behavior." (Stephen J Mellor, "Executable UML: A Foundation for Model-Driven Architecture", 2002)

"If the design, or some central part of it, does not map to the domain model, that model is of little value, and the correctness of the software is suspect. At the same time, complex mappings between models and design functions are difficult to understand and, in practice, impossible to maintain as the design changes. A deadly divide opens between analysis and design so that insight gained in each of those activities does not feed into the other." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Many things can put a project off course: bureaucracy, unclear objectives, and lack of resources, to name a few. But it is the approach to design that largely determines how complex software can become. When complexity gets out of hand, developers can no longer understand the software well enough to change or extend it easily and safely. On the other hand, a good design can create opportunities to exploit those complex features." (Eric Evans, "Domain-Driven Design: Tackling complexity in the heart of software", 2003)

"Design patterns give names to practical knowledge; they define a high-level vocabulary for understanding and solving business statements graphically. Design patterns are presented in a standard format; they're like recipes in a cookbook or dress patterns in a catalog. Above all, they are practical, first as instructional materials and then as development tools." (Alan Chmura & J Mark Heumann, "Logical Data Modeling: What it is and How to do it", 2005)

"Abstractions matter to users too. Novice users want programs whose abstractions are simple and easy to understand; experts want abstractions that are robust and general enough to be combined in new ways. When good abstractions are missing from the design, or erode as the system evolves, the resulting program grows barnacles of complexity. The user is then forced to master a mass of spurious details, to develop workarounds, and to accept frequent, inexplicable failures." (Daniel Jackson, "Software Abstractions", 2006)

"Programming is the ability to talk to the computer in a language it can understand and using grammar and syntax that it can follow to get it to perform useful tasks for you." (Adrian Kingsley-Hughes & Kathie Kingsley-Hughes, "Beginning Programming", 2007)

"We tend to form mental models that are simpler than reality; so if we create represented models that are simpler than the actual implementation model, we help the user achieve a better understanding. [...] Understanding how software actually works always helps someone to use it, but this understanding usually comes at a significant cost. One of the most significant ways in which computers can assist human beings is by putting a simple face on complex processes and situations. As a result, user interfaces that are consistent with users' mental models are vastly superior to those that are merely reflections of the implementation model." (Alan Cooper et al,  "About Face 3: The Essentials of Interaction Design", 2007)

"I find OOP methodologically wrong. It starts with classes. It is as if mathematicians would start with axioms. You do not start with axioms - you start with proofs. Only when you have found a bunch of related proofs, can you come up with axioms. You end with axioms. The same thing is true in programming: you have to start with interesting algorithms. Only when you understand them well, can you come up with an interface that will let them work." (Alexander Stepanov, [Interview with A. Stepanov] 2008)

"The majority of the cost of a software project is in long-term maintenance. In order to minimize the potential for defects as we introduce change, it's critical for us to be able to understand what a system does. As systems become more complex, they take more and more time for a developer to understand, and there is an ever greater opportunity for a misunderstanding. Therefore, code should clearly express the intent of its author. The clearer the author can make the code, the less time others will have to spend understanding it. This will reduce defects and shrink the cost of maintenance." (Robert C Martin, "Clean Code: A Handbook of Agile Software Craftsmanship", 2008)

"We see a lot of feature-driven product design in which the cost of features is not properly accounted. Features can have a negative value to customers because they make the products more difficult to understand and use. We are finding that people like products that just work. It turns out that designs that just work are much harder to produce that designs that assemble long lists of features." (Douglas Crockford, "JavaScript: The Good Parts", 2008)

"Design has the power to enrich our lives by engaging our emotions through image, form, texture, color, sound, and smell. The intrinsically human-centered nature of design thinking points to the next step: we can use our empathy and understanding of people to design experiences that create opportunities for active engagement and participation." (Tim Brown, "Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation", 2009)

"Prototypes should command only as much time, effort, and investment as is necessary to generate useful feedback and drive an idea forward. The greater the complexity and expense, the more 'finished' it is likely to seem and the less likely its creators will be to profit from constructive feedback - or even to listen to it. The goal of prototyping is not to create a working model. It is to give form to an idea to learn about its strengths and weaknesses and to identify new directions for the next generation of more detailed, more refined prototypes. A prototype's scope should be limited. The purpose of early prototypes might be to understand whether an idea has functional value." (Tim Brown, "Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation", 2009)

"We developers can easily develop blind spots. We necessarily have a different perspective from our users, and that can mean we miss important information that would be obvious to someone who understands things from their point of view. Furthermore, our focus tends to be on working out how to make the software work, not proving that it's broken." (Paul Butcher, "Debug It! Find, Repair, and Prevent Bugs in Your Code", 2009)

"Although it is focused on the code, refactoring has a large impact on the design of a system. It is vital for senior designers and architects to understand the principles of refactoring and to use them in their projects." (Jay Fields et al, "Refactoring: Ruby Edition", 2010)

"Understanding the causes of system failures may help organizations avoid them, although there are no guarantees." (Phil Simon, "Why New Systems Fail: An Insider's Guide to Successful IT Projects", 2010)

"What can you do to actually make your code tell the truth as clearly as possible? Strive for good names. Structure your code with respect to cohesive functionality, which also eases naming. Decouple your code to achieve orthogonality. Write automated tests explaining the intended behavior and check the interfaces. Refactor mercilessly when you learn how to code a simpler, better solution. Make your code as simple as possible to read and understand." (Peter Sommerlad, [in Kevlin Henney’s "97 Things Every Programmer Should Know", 2010])

"Complexity carries with it a lack of predictability different to that of chaotic systems, i.e. sensitivity to initial conditions. In the case of complexity, the lack of predictability is due to relevant interactions and novel information created by them." (Carlos Gershenson, "Understanding Complex Systems", 2011)

"Few would deny the importance of writing quality code. High quality code contains less bugs, and is easier to understand and easier to maintain. However, the precise definitions of code quality can be more subjective, varying between organizations, teams, and even individuals within a team." (John F Smart, "Jenkins: The Definitive Guide", 2011)

"Programming is a personal activity and there is no general process that is usually followed. Some programmers start with components that they understand, develop these, and then move on to less-understood components. Others take the opposite approach, leaving familiar components till last because they know how to develop them. Some developers like to define data early in the process then use this to drive the program development; others leave data unspecified for as long as possible." (Ian Sommerville, "Software Engineering" 9th Ed., 2011)

"The conceptual model is not the users' mental model of the application. [...] users of an application form mental models of it to allow them to predict its behavior. A mental model is the user's high-level understanding of how the application works; it allows the user to predict what the application will do in response to various user-actions. Ideally, a user's mental model of an application should be similar to the designers' conceptual model, but in practice the two models may differ signicantly. Even if a user-s mental model is the same as the designer's conceptual model, they are distinct models." (Jeff Johnson & Austin Henderson, "Conceptual Models", 2011)

"If the user can't understand it, the design and the designer have failed." (Joel Katz, "Designing Information: Human factors and common sense in information design", 2012)

"Programming is a science dressed up as art, because most of us don't understand the physics of software and it's rarely, if ever, taught. The physics of software is not algorithms, data structures, languages, and abstractions. These are just tools we make, use, and throw away. The real physics of software is the physics of people. Specifically, it's about our limitations when it comes to complexity and our desire to work together to solve large problems in pieces. This is the science of programming: make building blocks that people can understand and use easily, and people will work together to solve the very largest problems." (Pieter Hintjens, "ZeroMQ: Messaging for Many Applications", 2012)

"Development is a design process. Design processes are generally evaluated by the value they deliver rather than a conformance to plan. Therefore, it makes sense to move away from plan-driven projects and toward value-driven projects. [...] The realization that the source code is part of the design, not the product, fundamentally rewires our understanding of software." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"Feature extraction is also the most creative part of data science and the one most closely tied to domain expertise. Typically, a really good feature will correspond to some real-world phenomenon. Data scientists should work closely with domain experts and understand what these phenomena mean and how to distill them into numbers." (Field Cady, "The Data Science Handbook", 2017)

"Once we understand our user's mental model, we can capture it in a conceptual model. The conceptual model is a representation of the mental model using elements, relationships, and conditions. Our design and final system will be the tangible result of this conceptual model." (Pau Giner & Pablo Perea, "UX Design for Mobile, 2017)

"There aren't enough programmers in the world to do the amount of programming involved in making computers do everything we want or need." (Brian W Kernighan, "Understanding the Digital World", 2017)

"A key contribution of DevOps was to raise awareness of the problems lingering in how teams interacted" (or not) across the delivery chain, causing delays, rework, failures, and a lack of understanding and empathy toward other teams. It also became clear that such issues were not only happening between application development and operations teams but in interactions with many other teams involved in software delivery, like QA, InfoSec, networking, and more." (Matthew Skelton & Manuel Pais, "Team Topologies: Organizing Business and Technology Teams for Fast Flow", 2019)

"Data-intensive projects generally involve at least one person who understands all the nuances of the application, process, and source and target data. These are the people who also know about all the abnormalities in the data and the workarounds to deal with them, and are the experts. This is especially true in the case of legacy systems that store and use data in a manner it should not be used. The knowledge is not documented anywhere and is usually inside the minds of the people. When the experts leave, with no one having a true understanding of the data, the data are not used properly and everything goes haywire." (Rupa Mahanti, "Data Quality: Dimensions, Measurement, Strategy, Management, and Governance", 2019)

"Teams are always works in progress, but they are also your best shot at delivering value continuously and sustainably by aligning them with the business. Ideally, teams should be long lived and autonomous, with engaged team members. However, teams don't live in isolation. They need to understand how and when to interact with each other. And these team interactions need to evolve over time to support the distinct phases of discovery and execution that products and technology go through during their lifetimes." (Matthew Skelton & Manuel Pais, "Team Topologies: Organizing Business and Technology Teams for Fast Flow", 2019)

"People are inherently imperfect - we like to say that humans are mostly a collection of intermittent bugs. But before you can understand the bugs in your coworkers, you need to understand the bugs in yourself. We're going to ask you to think about your own reactions, behaviors, and attitudes - and in return, we hope you gain some real insight into how to become a more efficient and successful software engineer who spends less energy dealing with people-related problems and more time writing great code." (Titus Winters, "Software Engineering at Google: Lessons Learned from Programming Over Time", 2020)

"Great engineering managers find ways to give work meaning and make that meaning broadly understood. They align the realities of the engineering work they are tasked with to the aspirations and beliefs of their team members. [...] For your engineers, translating the why in a way they can understand and accept is a powerful tool for alignment and guiding decisions in the direction you want. [...] Translating outside of your team and upward to leadership" (managing up) is oftentimes the most impactful translation of all." (Morgan Evans, "Engineering Manager's Handbook", 2023)

"The thing that makes software design difficult is that we must express thoughts about a problem and a solution we typically do not understand fully, using a language that does not contain many of our accustomed features of expression, to a system that is unforgiving of mistakes." (Alistair Cockburn)

"Documentation is not understanding, process is not discipline, formality is not skill." (Jim Highsmith)

"One measure of our understanding is the number of independent ways we are able to get to the same result." (Richard P Feynman) 

"The guy who knows about computers is the last person you want to have creating documentation for people who don't understand computers." (Adam Osborne)

Related Posts Plugin for WordPress, Blogger...

About Me

My photo
Koeln, NRW, Germany
IT Professional with more than 24 years experience in IT in the area of full life-cycle of Web/Desktop/Database Applications Development, Software Engineering, Consultancy, Data Management, Data Quality, Data Migrations, Reporting, ERP implementations & support, Team/Project/IT Management, etc.