🔭Data Science: Heuristics (Just the Quotes)

"Heuristic reasoning is reasoning not regarded as final and strict but as provisional and plausible only, whose purpose is to discover the solution of the present problem. We are often obliged to use heuristic reasoning. We shall attain complete certainty when we shall have obtained the complete solution, but before obtaining certainty we must often be satisfied with a more or less plausible guess. We may need the provisional before we attain the final. We need heuristic reasoning when we construct a strict proof as we need scaffolding when we erect a building." (George Pólya,How to Solve It", 1945)

"The attempt to characterize exactly models of an empirical theory almost inevitably yields a more precise and clearer understanding of the exact character of a theory. The emptiness and shallowness of many classical theories in the social sciences is well brought out by the attempt to formulate in any exact fashion what constitutes a model of the theory. The kind of theory which mainly consists of insightful remarks and heuristic slogans will not be amenable to this treatment. The effort to make it exact will at the same time reveal the weakness of the theory." (Patrick Suppes," A Comparison of the Meaning and Uses of Models in Mathematics and the Empirical Sciences", Synthese  Vol. 12" (2/3), 1960)

"Design problems - generating or discovering alternatives - are complex largely because they involve two spaces, an action space and a state space, that generally have completely different structures. To find a design requires mapping the former of these on the latter. For many, if not most, design problems in the real world systematic algorithms are not known that guarantee solutions with reasonable amounts of computing effort. Design uses a wide range of heuristic devices - like means-end analysis, satisficing, and the other procedures that have been outlined - that have been found by experience to enhance the efficiency of search. Much remains to be learned about the nature and effectiveness of these devices." (Herbert A Simon,The Logic of Heuristic Decision Making", [inThe Logic of Decision and Action"], 1966)

"Intelligence has two parts, which we shall call the epistemological and the heuristic. The epistemological part is the representation of the world in such a form that the solution of problems follows from the facts expressed in the representation. The heuristic part is the mechanism that on the basis of the information solves the problem and decides what to do." (John McCarthy & Patrick J Hayes,Some Philosophical Problems from the Standpoint of Artificial Intelligence", Machine Intelligence 4, 1969)

"Consider any of the heuristics that people have come up with for supervised learning: avoid overfitting, prefer simpler to more complex models, boost your algorithm, bag it, etc. The no free lunch theorems say that all such heuristics fail as often" (appropriately weighted) as they succeed. This is true despite formal arguments some have offered trying to prove the validity of some of these heuristics." (David H Wolpert,The lack of a priori distinctions between learning algorithms", Neural Computation Vol. 8(7), 1996)

"Heuristic (it is of Greek origin) means discovery. Heuristic methods are based on experience, rational ideas, and rules of thumb. Heuristics are based more on common sense than on mathematics. Heuristics are useful, for example, when the optimal solution needs an exhaustive search that is not realistic in terms of time. In principle, a heuristic does not guarantee the best solution, but a heuristic solution can provide a tremendous shortcut in cost and time." (Nikola K Kasabov,Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"Theories of choice are at best approximate and incomplete. One reason for this pessimistic assessment is that choice is a constructive and contingent process. When faced with a complex problem, people employ a variety of heuristic procedures in order to simplify the representation and the evaluation of prospects. These procedures include computational shortcuts and editing operations, such as eliminating common components and discarding nonessential differences. The heuristics of choice do not readily lend themselves to formal analysis because their application depends on the formulation of the problem, the method of elicitation, and the context of choice." (Amos Tversky & Daniel Kahneman,Advances in Prospect Theory: Cumulative Representation of Uncertainty" [inChoices, Values, and Frames"], 2000)

"Behavioural research shows that we tend to use simplifying heuristics when making judgements about uncertain events. These are prone to biases and systematic errors, such as stereotyping, disregard of sample size, disregard for regression to the mean, deriving estimates based on the ease of retrieving instances of the event, anchoring to the initial frame, the gambler’s fallacy, and wishful thinking, which are all affected by our inability to consider more than a few aspects or dimensions of any phenomenon or situation at the same time." (Hans G Daellenbach & Donald C McNickle,Management Science: Decision making through systems thinking", 2005)

"A decision theory that rests on the assumptions that human cognitive capabilities are limited and that these limitations are adaptive with respect to the decision environments humans frequently encounter. Decision are thought to be made usually without elaborate calculations, but instead by using fast and frugal heuristics. These heuristics certainly have the advantage of speed and simplicity, but if they are well matched to a decision environment, they can even outperform maximizing calculations with respect to accuracy. The reason for this is that many decision environments are characterized by incomplete information and noise. The information we do have is usually structured in a specific way that clever heuristics can exploit." (E Ebenhoh,Agent-Based Modelnig with Boundedly Rational Agents", 2007)

"Optimization systems (or optimizers, as they are often referred to) aim to optimize in a systematic way, oftentimes using a heuristics-based approach. Such an approach enables the AI system to use a macro level concept as part of its low-level calculations, accelerating the whole process and making it more light-weight. After all, most of these systems are designed with scalability in mind, so the heuristic approach is most practical." (Yunus E Bulut & Zacharias Voulgaris,AI for Data Science: Artificial Intelligence Frameworks and Functionality for Deep Learning, Optimization, and Beyond", 2018)

"The social world that humans have made for themselves is so complex that the mind simplifies the world by using heuristics, customs, and habits, and by making models or assumptions about how things generally work (the ‘causal structure of the world’). And because people rely upon" (and are invested in) these mental models, they usually prefer that they remain uncontested." (Dr James Brennan,Psychological  Adjustment to Illness and Injury", West of England Medical Journal Vol. 117 (2), 2018)

"Many AI systems employ heuristic decision making, which uses a strategy to find the most likely correct decision to avoid the high cost" (time) of processing lots of information. We can think of those heuristics as shortcuts or rules of thumb that we would use to make fast decisions." (Jesús Barrasa et al,Knowledge Graphs: Data in Context for Responsive Businesses", 2021)

"Once we know something is fat-tailed, we can use heuristics to see how an exposure there reacts to random events: how much is a given unit harmed by them. It is vastly more effective to focus on being insulated from the harm of random events than try to figure them out in the required details" (as we saw the inferential errors under thick tails are huge). So it is more solid, much wiser, more ethical, and more effective to focus on detection heuristics and policies rather than fabricate statistical properties." (Nassim N Taleb,Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applications" 2nd Ed., 2022)

🤖〽️Prompt Engineering: Copilot Unabridged (Part 176: How AI Can Detect Rare Patterns That Humans Might Overlook)

Prompt Engineering Series

Prompt: "write a post of 600 words on how AI can detect rare patterns that humans might overlook"

Introduction

One of the most transformative capabilities of modern Artificial Intelligence (AI) is its ability to detect patterns that are too subtle, too rare, or too complex for humans to notice. While humans excel at intuitive reasoning, contextual understanding, and creative interpretation, we are limited by cognitive bandwidth, memory constraints, and perceptual biases. AI, by contrast, thrives in environments where scale, precision, and exhaustive analysis are essential. Understanding how AI uncovers rare patterns reveals not only its technical power but also its potential to reshape scientific discovery, business strategy, and our understanding of complex systems.

The first reason AI can detect rare patterns is its ability to process massive datasets without fatigue or bias. Humans can analyze only a small number of variables at once, and our attention is easily overwhelmed by noise. AI systems, especially those built on deep learning or advanced statistical models, can examine millions of data points simultaneously. They can identify correlations that occur only once in a million cases - signals so faint that they disappear into the background for human observers. This ability is particularly valuable in fields like fraud detection, where unusual behavior is intentionally hidden, or in medical diagnostics, where early signs of disease may be nearly invisible.

A second advantage lies in AI’s capacity to operate beyond human intuition. Humans rely heavily on heuristics - mental shortcuts that help us navigate the world efficiently but can blind us to unexpected relationships. AI does not share these cognitive shortcuts. It does not assume which variables matter or which patterns are plausible. Instead, it evaluates all possibilities, including those that defy conventional wisdom. This openness allows AI to uncover patterns that humans would never think to look for. In scientific research, for example, AI has identified previously unknown relationships between genetic markers and diseases, not because it 'understood' biology, but because it was not constrained by human assumptions about what should or should not be related.

Another key factor is AI’s ability to detect patterns across multiple scales simultaneously. Humans tend to focus on either the big picture or the fine details, but rarely both at once. AI can analyze micro‑patterns—minute fluctuations, rare anomalies, subtle deviations - while also tracking macro‑patterns that unfold across long time horizons. This multi‑scale analysis is essential in fields like climate modeling, financial forecasting, and cybersecurity. A human analyst might notice a sudden spike in activity, but an AI system can detect the faint precursors that occurred months earlier, revealing a pattern that only becomes meaningful when viewed across scales.

AI also excels at identifying nonlinear relationships, which are notoriously difficult for humans to detect. Many real‑world systems - ecosystems, markets, neural networks - do not behave in simple, linear ways. Small changes can produce disproportionate effects, and interactions between variables can create emergent behavior. AI models, especially neural networks, are designed to capture these nonlinearities. They can map complex relationships that would be invisible to traditional statistical methods or human intuition. This capability allows AI to detect rare patterns that emerge only when multiple variables interact in specific, unusual ways.

Finally, AI’s ability to detect rare patterns is amplified by continuous learning. Humans learn slowly and forget quickly. AI systems can update their models in real time, incorporating new data as it arrives. This allows them to detect emerging patterns before they become obvious. In cybersecurity, for example, AI can identify a new type of attack based on a handful of early signals. In healthcare, AI can detect subtle shifts in patient data that indicate a rare complication long before symptoms appear.

The ability of AI to detect rare patterns is not a replacement for human judgment. Instead, it is a complement - a way to extend our perceptual reach and reveal structures hidden beneath the surface of complexity. When humans and AI work together, combining intuition with computation, we gain a deeper, more nuanced understanding of the world.

Disclaimer: The whole text was generated by Copilot (under Windows 11) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🤖〽️Prompt Engineering: Copilot Unabridged (Part 171: How AI Can Give the Illusion of Knowing)

Prompt Engineering Series

Prompt: "write a post of 600 words on how AI can give the illusion of knowing"

Introduction

Artificial Intelligence (AI) has become remarkably good at sounding knowledgeable. It can answer questions fluently, generate explanations on command, and produce text that feels coherent, confident, and even insightful. Yet beneath this surface competence lies a paradox: AI systems do not 'know' in any human sense. What they offer is an illusion of knowing - a convincing simulation of understanding built from patterns, probabilities, and linguistic mimicry. Exploring how this illusion is created reveals both the power of modern AI and the limits that still define it.

At the heart of the illusion is statistical prediction. Large language models are trained on vast amounts of text, learning the patterns that govern how words, ideas, and arguments tend to appear together. When prompted, the model generates the most statistically plausible continuation. This process is mechanical, but the output often feels intentional. A well‑structured paragraph or a precise explanation triggers our instinct to attribute understanding to the speaker. Humans are wired to interpret language as evidence of thought, so when AI produces fluent language, we naturally assume it reflects comprehension.

Another contributor to the illusion is contextual coherence. Modern AI systems can maintain a thread of conversation, refer back to earlier points, and adapt their tone or style. This creates the impression of a stable internal model of the world. But the coherence is local, not global. The system does not hold beliefs or maintain a unified worldview; it simply stitches together contextually appropriate responses. It can sound authoritative even when it is improvising. The illusion arises because humans equate coherence with cognition.

AI also benefits from the authority effect. When a system responds instantly, confidently, and without hesitation, it mirrors the behavior of an expert. Confidence is persuasive, even when it is unwarranted. This is why AI‑generated errors - often called hallucinations - can be so misleading. The model does not distinguish between truth and falsehood; it distinguishes only between likely and unlikely sequences of text. A fabricated citation or an invented fact can be delivered with the same fluency as a verified one. The illusion of knowing persists because the style of the answer feels right, even when the substance is wrong.

A subtler mechanism behind the illusion is pattern overfitting to human expectations. AI systems learn not only the content of human writing but also its rhetorical structures: how arguments are framed, how explanations unfold, how uncertainty is expressed. When the model mirrors these structures, it feels like it is reasoning. But it is not reasoning; it is reproducing the shape of reasoning. The distinction is crucial. Humans infer meaning from structure, so when AI imitates the structure, we project meaning onto it.

Finally, the illusion is strengthened by our own cognitive shortcuts. Humans rely on heuristics - processing fluency, familiarity, and narrative coherence - to judge whether something 'makes sense'. AI outputs are optimized for these very qualities. When a response is easy to read, aligns with familiar patterns, and fits a coherent narrative, we interpret it as knowledgeable. The illusion is not only produced by the AI; it is co‑constructed by our minds.

Understanding these mechanisms does not diminish the value of AI. On the contrary, it clarifies what AI is genuinely good at: synthesizing information, generating possibilities, and supporting human thinking. But it also highlights the importance of maintaining a critical stance. AI can simulate knowledge, but it cannot replace the human capacity for judgment, interpretation, and meaning‑making.

The illusion of knowing is powerful, but it is still an illusion. Recognizing it allows us to use AI more wisely - appreciating its strengths while remaining aware of its boundaries.

Disclaimer: The whole text was generated by Copilot (under Windows 11) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🤖〽️Prompt Engineering: Copilot Unabridged (Part 111: The Availability Heuristic in AI - When Familiarity Skews Intelligence)

 

Prompt Engineering Series

Prompt: "write a post of 600 words on what is needed for creating a foundation for the further development of AI"

Introduction

In the age of Artificial Intelligence (AI), we often assume that machines are immune to human biases. But the truth is more nuanced. AI systems, while built on data and algorithms, are deeply influenced by human psychology - especially during their design, training, and interpretation. One cognitive bias that plays a significant role is the availability heuristic.

Coined by psychologists Amos Tversky and Daniel Kahneman, the availability heuristic is a mental shortcut where people estimate the probability of an event based on how easily examples come to mind. For instance, if you recently heard about a plane crash, you might overestimate the risk of flying - even though statistically, it's safer than driving. This bias helps us make quick decisions, but it often leads to errors in judgment.

How It Shows Up in AI Systems

AI models are trained on data - lots of it. But the availability of certain data types can skew the model’s understanding of reality. If a dataset contains more examples of one type of event (say, fraudulent transactions from a specific region), the AI may overestimate the likelihood of fraud in that region, even if the real-world distribution is different. This is a direct reflection of the availability heuristic: the model 'sees' more of something and assumes it’s more common.

Moreover, developers and data scientists are not immune to this bias. When selecting training data or designing algorithms, they may rely on datasets that are readily available or familiar, rather than those that are representative. This can lead to biased outcomes, especially in sensitive domains like healthcare, hiring, or criminal justice. 

Human Interpretation of AI Outputs

The availability heuristic doesn’t just affect AI systems - it also affects how humans interpret them. When users interact with AI tools like ChatGPT or recommendation engines, they often accept the first answer or suggestion without questioning its accuracy. Why? Because it’s available, and our brains are wired to trust what’s easy to access.

This is particularly dangerous in high-stakes environments. For example, a doctor using an AI diagnostic tool might favor a diagnosis that the system presents prominently, even if it’s not the most accurate. If the AI has been trained on a dataset where a certain condition appears frequently, it might over-represent that condition in its suggestions. The human, influenced by availability bias, might accept it without deeper scrutiny.

The Role of Information Overload

In today’s digital world, we’re bombarded with information. AI systems help us filter and prioritize, but they also reinforce the availability heuristic. Search engines, social media algorithms, and news aggregators show us what’s popular or trending - not necessarily what’s accurate. As a result, we form opinions and make decisions based on what we see most often, not what’s most valid.

This creates echo chambers and reinforces stereotypes. For instance, if an AI-powered news feed frequently shows stories about crime in urban areas, users may develop a skewed perception of urban safety - even if crime rates are declining.

Mitigating the Bias

To combat the availability heuristic in AI, both developers and users must be proactive:

• Diversify training data to ensure models reflect reality, not just what’s easy to collect.
• Design transparent systems that explain how decisions are made.
• Educate users about cognitive biases and encourage critical thinking.
• Audit AI outputs regularly to identify patterns of overrepresentation or omission.

Conclusion

The availability heuristic is a powerful psychological bias that influences both the design and interpretation of AI systems. As we rely more on AI to guide decisions, understanding and mitigating this bias becomes essential - not just for accuracy, but for fairness and trust.

Just try the prompt on Copilot or your favorite AI-powered assistant! Have you got a different/similar result? How big or important is the difference? Any other thoughts?
Just share the link to the post with me and I'll add it to this post as a resource!

Disclaimer: The whole text was generated by Copilot (under Windows 11) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

📊Graphical Representation: Sense-making in Data Visualizations (Part 3: Heuristics)

Graphical Representation Series

 

Consider the following general heuristics in data visualizations (work in progress):

• plan design
• plan page composition
• text
• title, subtitles
• dates 
• refresh, filters applied
• parameters applied
• guidelines/tooltips
• annotation 
• navigation
• main page(s)
• additional views
• drill-through
• zoom in/out
• next/previous page
• landing page
• slicers/selections
• date-related
• date range
• date granularity
• functional
• metric
• comparisons
• categorical
• structural relations
• icons/images
• company logo
• button icons
• background
• pick a theme
• choose a layout and color schema
• use a color palette generator
• use a focused color schema or restricted palette
• use consistent and limited color scheme
• use suggestive icons
• use one source (with similar design)
• use formatting standards
• create a visual hierarchy 
• use placement, size and color for emphasis
• organize content around eye movement pattern
• minimize formatting changes
• 1 font, 2 weights, 4 sizes
• plan the design
• build/use predictable and consistent templates
• e.g. using Figma
• use layered design
• aim for design unity
• define & use formatting standards
• check changes
• GRACEFUL
• group visuals with white space 
• right chart type
• avoid clutter
• consistent & limited color schema
• enhanced readability 
• formatting standard
• unity of design
• layered design
• keep it simple 
• be predictable and consistent 
• focus on the message
• identify the core insights and design around them
• pick suggestive titles/subtitles
• use dynamics subtitles
• align content with the message
• avoid unnecessary complexity
• minimize visual clutter
• remove the unnecessary elements
• round numbers
• limit colors and fonts
• use a restrained color palette (<5 colors)
• stick to 1-2 fonts 
• ensure text is legible without zooming
• aggregate values
• group similar data points to reduce noise
• use statistical methods
• averages, medians, min/max
• categories when detailed granularity isn’t necessary
• highlight what matters 
• e.g. actionable items
• guide attention to key areas
• via annotations, arrows, contrasting colors 
• use conditional formatting
• do not show only the metrics
• give context 
• show trends
• via sparklines and similar visuals
• use familiar visuals
• avoid questionable visuals 
• e.g. pie charts, gauges
• avoid distortions
• preserve proportions
• scale accurately to reflect data values
• avoid exaggerated visuals
• don’t zoom in on axes to dramatize small differences
• use consistent axes
• compare data using the same scale and units across charts
• don't use dual axes or shifting baselines that can mislead viewers
• avoid manipulative scaling
• use zero-baseline on bar charts 
• use logarithmic scales sparingly
• design for usability
• intuitive interaction
• at-a-glance perception
• use contrast for clarity
• use familiar patterns
• use consistent formats the audience already knows
• design with the audience in mind
• analytical vs managerial perspectives (e.g. dashboards)
• use different level of data aggregations
•  in-depth data exploration 
• encourage scrutiny
• give users enough context to assess accuracy
• provide raw values or links to the source
• explain anomalies, outliers or notable trends
• via annotations
• group related items together
• helps identify and focus on patterns and other relationships
• diversify 
• don't use only one chart type
• pick the chart that reflects the best the data in the conrext considered
• show variance 
• absolute vs relative variance
• compare data series
• show contribution to variance
• use familiar encodings
• leverage (known) design patterns
• use intuitive navigation
• synchronize slicers
• use tooltips
• be concise
• use hover effects
• use information buttons
• enhances user interaction and understanding 
• by providing additional context, asking questions
• use the full available surface
• 1080x1920 works usually better 
• keep standards in mind 
• e.g. IBCS
• state the assumptions
• be explicit
• clearly state each assumption 
• instead of leaving it implied
• contextualize assumptions
• explain the assumption
• use evidence, standard practices, or constraints
• state scope and limitations
• mention what the assumption includes and excludes
• tie assumptions to goals & objectives
• helps to clarify what underlying beliefs are shaping the analysis
• helps identify whether the visualization achieves its intended purpose 
• show the data
• be honest (aka preserve integrity)
• avoid distortion, bias, or trickery
• support interpretation
• provide labels, axes, legends
• emphasize what's meaningful
• patterns, trends, outliers, correlations, local/global maxima/minima
• show what's important 
• e.g. facts, relationships, flow, similarities, differences, outliers, unknown
• prioritize and structure the content
• e.g. show first an overview, what's important
• make the invisible visible
• think about what we do not see
• know your (extended) users/audience
• who'll use the content, at what level, for that
• test for readability
• get (early) feedback
• have the content reviewed first
• via peer review, dry run presentation
• tell the story
• know the audience and its needs
• build momentum, expectation
• don't leave the audience to figure it out
• show the facts
• build a narrative
• show data that support it
• arrange the visuals in a logical sequence
• engage the reader
• ask questions that bridge the gaps
• e.g. in knowledge, in presentation's flow
• show the unexpected
• confirm logical deductions

Previous Post <<||>> Next Post

🧭Business Intelligence: A Software Engineer's Perspective (Part VII: Think for Yourself!)

Business Intelligence Series

After almost a quarter-century of professional experience the best advice I could give to younger professionals is to "gather information and think for themselves", and with this the reader can close the page and move forward! Anyway, everybody seems to be looking for sudden enlightenment with minimal effort, as if the effort has no meaning in the process!

In whatever endeavor you are caught, it makes sense to do upfront a bit of thinking for yourself - what's the task, or more general the problem, which are the main aspects and interpretations, which are the goals, respectively the objectives, how a solution might look like, respectively how can it be solved, how long it could take, etc. This exercise is important for familiarizing yourself with the problem and creating a skeleton on which you can build further. It can be just vague ideas or something more complex, though no matter the overall depth is important to do some thinking for yourself!

Then, you should do some research to identify how others approached and maybe solved the problem, what were the justifications, assumptions, heuristics, strategies, and other tools used in sense-making and problem solving. When doing research, one should not stop with the first answer and go with it. It makes sense to allocate a fair amount of time for information gathering, structuring the findings in a reusable way (e.g. tables, mind maps or other tools used for knowledge mapping), and looking at the problem from the multiple perspectives derived from them. It's important to gather several perspectives, otherwise the decisions have a high chance of being biased. Just because others preferred a certain approach, it doesn't mean one should follow it, at least not blindly!

The purpose of research is multifold. First, one should try not to reinvent the wheel. I know, it can be fun, and a lot can be learned in the process, though when time is an important commodity, it's important to be pragmatic! Secondly, new information can provide new perspectives - one can learn a lot from other people’s thinking. The pragmatism of problem solvers should be combined, when possible, with the idealism of theories. Thus, one can make connections between ideas that aren't connected at first sight.

Once a good share of facts was gathered, you can review the new information in respect to the previous ones and devise from there several approaches worthy of attack. Once the facts are reviewed, there are probably strong arguments made by others to follow one approach over the others. However, one can show that has reached a maturity when is able to evaluate the information and take a decision based on the respective information, even if the decision is not by far perfect.

One should try to develop a feeling for decision making, even if this seems to be more of a gut-feeling and stressful at times. When possible, one should attempt to collect and/or use data, though collecting data is often a luxury that tends to postpone the decision making, respectively be misused by people just to confirm their biases. Conversely, if there's any important benefit associated with it, one can collect data to validate in time one's decision, though that's a more of a scientist’s approach.

I know that's easier to go with the general opinion and do what others advise, especially when some ideas are popular and/or come from experts, though then would mean to also follow others' mistakes and biases. Occasionally, that can be acceptable, especially when the impact is neglectable, however each decision we are confronted with is an opportunity to learn something, to make a difference! 

Previous Post <<||>> Next Post

🧭Business Intelligence: A Software Engineer's Perspective (Part II: Major Knowledge Gaps)

Business Intelligence Series

Solving a problem requires a certain degree of knowledge in the areas affected by the problem, degree that varies exponentially with problem's complexity. This requirement applies to scientific fields with low allowance for errors, as well as to business scenarios where the allowance for errors is in theory more relaxed. Building a report or any other data artifact is closely connected with problem solving as the data artifacts are supposed to model the whole or parts of what is needed for solving the problem(s) in scope.

In general, creating data artifacts requires: (1) domain knowledge - knowledge of the concepts, processes, systems, data, data structures and data flows as available in the organization; (2) technical knowledge - knowledge about the tools, techniques, processes and methodologies used to produce the artifacts; (3) data literacy - critical thinking, the ability to understand and explore the implications of data, respectively communicating data in context; (4) activity management - managing the activities involved. 

At minimum, creating a report may require only narrower subsets from the areas mentioned above, depending on the complexity of the problem and the tasks involved. Ideally, a single person should be knowledgeable enough to handle all this alone, though that's seldom the case. Commonly, two or more parties are involved, though let's consider the two-parties scenario: on one side is the customer who has (in theory) a deep understanding of the domain, respectively on the other side is the data professional who has (in theory) a deep understanding of the technical aspects. Ideally, both parties should be data literates and have some basic knowledge of the other party's domain. 

To attack a business problem that requires one or more data artifacts both parties need to have a common understanding of the problem to be solved, of the requirements, constraints, assumptions, expectations, risks, and other important aspects associated with it. It's critical for the data professional to acquire the domain knowledge required by the problem, otherwise the solution has high chances to deviate from the expectations. The general issue is that there are multiple interactions that are iterative. Firstly, the interactions for building the needed common ground. Secondly, the interaction between the problem and reality. Thirdly, the interaction between the problem and parties’ mental models und understanding about the problem. 

The outcome of these interactions is that the problem and its requirements go through several iterations in which knowledge from the previous iterations are incorporated successively. With each important piece of knowledge gained, it's important to revise and refine the question(s), respectively the problem. If in each iteration there are also programming and further technical activities involved, the effort and costs resulted in the process can explode, while the timeline expands accordingly. 

There are several heuristics that could be devised to address these challenges: (1) build all the required knowledge in one person, either on the business or the technical side; (2) make sure that the parties have the required knowledge for approaching the problems in scope; (3) make sure that the gaps between reality and parties' mental models is minimal; (4) make sure that the requirements are complete and understood before starting the development; (5) adhere to methodologies that accommodate the necessary iterations and endeavor's particularities; (6) make sure that there's a halt condition for regularly reviewing the progress, respectively halting the work; (7) build an organizational culture to support all this. 

The list is open, and the heuristics aren't exclusive, so in theory any combination of them can be considered. Ideally, an organization should reflect all these heuristics in one form or another. The higher the coverage, the more mature the organization is. The question is how organizations with a suboptimal setup can change the status quo?

Previous Post <<||>> Next Post

💼Project Management: Project Execution (Part III: Projects' Dynamics - An Introduction)

Despite the considerable collection of books on Project Management (PM) and related methodologies, and the fact that projects are inherent endeavors in professional as well personal life (setups that would give in theory people the environment and exposure to different project types), people’s understanding on what it takes to plan and execute a project seems to be narrow and questionable sometimes. Moreover, their understanding diverges considerably from common sense. It’s also true that knowledge and common sense are relative when considering any human endeavor in which there are multiple roads to the same destination, or when learning requires time, effort, skills, and implies certain prerequisites, however the lack of such knowledge can hurt when endeavor’s success is a must and a team effort. 

Even if the lack of understanding about PM can be considered as minor when compared with other challenges/problems faced by a project, when one’s running fast to finish a race, even a small pebble in one’s running shoes can hurt a lot, especially when one doesn’t have the luxury to stop and remove the stone, as it would make sense to do.

It resides in the human nature to resist change, to seek for information that only confirm own opinions, to follow the same approach in handling challenges, even if the attempts are far from optimal, even if people who walked the same path tell you that there’s a better way and even sketch the path and provide information about what it takes to reach there. As it seems, there’s the predisposition to learn on the hard way, if there’s significant learning involved at all. Unfortunately, such situations occur in projects and the solutions often overrun the boundaries of PM, where social and communication skills must be brought into play. 

On the other side, there’s still hope that change can be managed optimally once the facts are explained to a certain level that facilitates understanding. However, such an attempt can prove to be quite a challenge, given the various setups in which PM takes place. The intersection between technologies and organizational setups lead to complex scenarios which make such work more difficult, even if projects’ challenges are of organizational rather than technological nature. 

When the knowledge we have about the world doesn’t fit our expectation, a simple heuristic is to return to the basics. A solid edifice can be built only on a solid foundation and the best foundation in coping with reality is to establish common ground with other people. One can achieve this by identifying their suppositions and expectations, by closing the gap in perception and understanding, by establishing a basis for communication, in which feedback is a must if one wants to make significant progress.

Despite of being explorative and time-consuming, establishing common ground can be challenging when addressing to an imaginary audience, which is quite often the situation. The practice shows however that progress can be made by starting with a set of well-formulated definitions, simple models, principles, and heuristics that have the potential of helping in sense-making.

The goal is thus to identify first the definitions that reflect the basic concepts that need to be considered. Once the concepts defined, they can be related to each other with the help of a few models. Even if fictitious, as simplifications of the reality, the models should allow playing with the concepts, facilitating concepts’ understanding. Principles (set of rules for reasoning) can be used together with heuristics (rules of thumb methods or techniques) for explaining the ‘known’ and approaching the ‘unknown’. Even maybe not perfect, these tools can help building theories or explanatory constructs.

||>>Next Post

📦Data Migrations (DM): Conceptualization (Part III: Heuristics)

Probably one of the most difficult things to learn as a technical person is using the right technology for a given purpose, this mainly because one’s inclined using the tools one knows best. Moreover, technologies’ overlapping makes the task more and more challenging, the difference between competing technologies often residing in the details. Thus, identifying the gaps resumes in understanding the details of the problem(s) or need(s), respectively the advantages or disadvantages of a technology over the other. This is true especially about competing technologies, including the ones that replace other technologies.

There are simple heuristics, that can allow approaching such challenges. For example, heavy data processing belongs usually in databases, while import/export functionality belongs in an ETL tool.  Therefore, one can start looking at the problems from these two perspectives. Would the solution benefit from these two approaches or are there more appropriate technologies (e.g. data streaming, ELT, non-relational databases)? How much effort would involve building the solution? 

Commercial Off-The-Shelf (COTS) tools provided by third-party vendors usually offer specialized functionality in each area. Gartner and Forrester provide regular analyses of the main players in the important areas, analyses which can be used in theory as basis for further research. Even if COTS tend to be more expensive and can have some important functionality gaps, as long they are extensible, they can prove a good starting point for developing a solution. 

Sometimes it helps researching on the web what other people or organizations did, how they approached the same aspects, what technologies, techniques and best practices they used to overcome the challenges. One doesn’t need to reinvent the wheel even if it’s sometimes fun to do so. Moreover, a few hours of research can give one a basis of useful information and a better understanding over the work ahead.

On the other side sometimes it’s advisable to use the tools one knows best, however this can lead also to unusable and less performant solutions. For example, MS Excel and Access have been for years the tools of choice for building personal solutions that later grew into maintenance nightmares for the IT team. Ideally, they can still be used for data entry or data cleaning, though building solutions exclusively based on (one of) them can prove to be far than optimal. 

When one doesn’t know whether a technology or mix of technologies can be used to provide a solution, it’s recommended to start a proof-of-concept (PoC) that would allow addressing most important aspects of the needed solution. One can start small by focusing on the minimal functionality needed to check the main aspects and evolve the PoC during several iterations as needed.

For example, in the case of a Data Migration (DM) this would involve building the data extraction layer for an entity, implement several data transformations based on the defined mappings, consider building a few integrity rules for validation, respectively attempt importing the data into the target system. Once this accomplished, one can start increasing the volume of data to check how the solution behaves under stress. The volume of data can be increased incrementally or by considering all the data available. 

As soon the skeleton was built one can consider all the mappings, respectively add several entities to build the dependencies existing between them and other functionality. The prototype might not address all the requirements from the beginning, therefore consider the problems as they arise. For example, if the volume of data seems to cause problems then attempt splitting the data during processing in batches or considering specific optimization techniques like indexing or scaling techniques like increasing computing resources. 

Previous Post <<||>> Next Post

💼Project Management: Tools (Part I: The Choice of Tools in Project Management)

“Beware the man of one book” (in Latin, “homo unius libri”), a warning generally attributed to Thomas Aquinas and having a twofold meaning. In its original interpretation it was referring to the people mastering a single chosen discipline, however the meaning degenerated in expressing the limitations of people who master just one book, and thus having a limited toolset of perspectives, mental models or heuristics. This later meaning is better reflected in Abraham Maslow adage: “If the only tool you have is a hammer, you tend to see every problem as a nail”, as people tend to use the tools they are used to also in situations in which other tools are more appropriate.

It’s sometimes admirable people and even organizations’ stubbornness in using the same tools in totally different scenarios, expecting though the same results, as well in similar scenarios expecting different results. It’s true, Mathematics has proven that the same techniques can be used successfully in different areas, however a mathematician’s universe and models are idealistically fractionalized to a certain degree from reality, full of simplified patterns and never-ending approximations. In contrast, the universe of Software Development and Project Management has a texture of complex patterns with multiple levels of dependencies and constraints, constraints highly sensitive to the initial conditions.

Project Management has managed to successfully derive tools like methodologies, processes, procedures, best practices and guidelines to address the realities of projects, however their use in praxis seems to be quite challenging. Probably, the challenge resides in stubbornness of not adapting the tools to the difficulties and tasks met. Even if the same phases and multiple similarities seems to exist, the process of building a house or other tangible artefact is quite different than the approaches used in development and implementation of software.

Software projects have high variability and are often explorative in nature. The end-product looks totally different than the initial scaffold. The technologies used come with opportunities and limitations that are difficult to predict in the planning phase. What on paper seems to work often doesn’t work in praxis as the devil lies typically in details. The challenges and limitations vary between industries, businesses and even projects within the same organization.

Even if for each project type there’s a methodology more suitable than another, in the end project particularities might pull the choice in one direction or another. Business Intelligence projects for example can benefit from agile approaches as they enable to better manage and deliver value by adapting the requirements to business needs as the project progresses. An agile approach works almost always better than a waterfall process. In contrast, ERP implementations seldom benefit from agile methodologies given the complexity of the project which makes from planning a real challenge, however this depends also on an organization’s dynamicity.

Especially when an organization has good experience with a methodology there’s the tendency to use the same methodology across all the projects run within the organization. This results in chopping down a project to fit an ideal form, which might be fine as long the particularities of each project are adequately addressed. Even if one methodology is not appropriate for a given scenario it doesn’t mean it can’t be used for it, however in the final equation enter also the cost, time, effort, and the quality of the end-results.

In general, one can cope with complexity by leveraging a broader set of mental models, heuristics and set of tools, and this can be done only though experimentation, through training and exposing employees to new types of experiences, through openness, through adapting the tools to the challenges ahead.

🔭Data Science: Trial and Error (Just the Quotes)

"One is almost tempted to assert that quite apart from its intellectual mission, theory is the most practical thing conceivable, the quintessence of practice as it were, since the precision of its conclusions cannot be reached by any routine of estimating or trial and error; although given the hidden ways of theory, this will hold only for those who walk them with complete confidence." (Ludwig E Boltzmann, "On the Significance of Theories", 1890)

"The discoveries in physical science, the triumphs in invention, attest the value of the process of trial and error. In large measure, these advances have been due to experimentation." (Louis Brandeis, "Judicial opinions", 1932)

"We know the laws of trial and error, of large numbers and probabilities. We know that these laws are part of the mathematical and mechanical fabric of the universe, and that they are also at play in biological processes. But, in the name of the experimental method and out of our poor knowledge, are we really entitled to claim that everything happens by chance, to the exclusion of all other possibilities?" (Albert Claude, [Nobel Prize Lecture], 1974)

"The natural as well as the social sciences always start from problems, from the fact that something inspires amazement in us, as the Greek philosophers used to say. To solve these problems, the sciences use fundamentally the same method that common sense employs, the method of trial and error. To be more precise, it is the method of trying out solutions to our problem and then discarding the false ones as erroneous. This method assumes that we work with a large number of experimental solutions. One solution after another is put to the test and eliminated." (Karl R Popper, "All Life is Problem Solving", 1999) 

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

"Technology is the result of antifragility, exploited by risk-takers in the form of tinkering and trial and error, with nerd-driven design confined to the backstage." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"We can simplify the relationships between fragility, errors, and antifragility as follows. When you are fragile, you depend on things following the exact planned course, with as little deviation as possible - for deviations are more harmful than helpful. This is why the fragile needs to be very predictive in its approach, and, conversely, predictive systems cause fragility. When you want deviations, and you don’t care about the possible dispersion of outcomes that the future can bring, since most will be helpful, you are antifragile. Further, the random element in trial and error is not quite random, if it is carried out rationally, using error as a source of information. If every trial provides you with information about what does not work, you start zooming in on a solution - so every attempt becomes more valuable, more like an expense than an error. And of course you make discoveries along the way." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"Another crowning achievement of deep learning is its extension to the domain of reinforcement learning. In the context of reinforcement learning, an autonomous agent must learn to perform a task by trial and error, without any guidance from the human operator." (Ian Goodfellow et al, "Deep Learning", 2015)

"A learning algorithm for a robot or a software agent to take actions in an environment so as to maximize the sum of rewards through trial and error." (Tomohiro Yamaguchi et al, "Analyzing the Goal-Finding Process of Human Learning With the Reflection Subtask", 2018)

"Reinforcement learning is also a subset of AI algorithms which creates independent, self-learning systems through trial and error. Any positive action is assigned a reward and any negative action would result in a punishment. Reinforcement learning can be used in training autonomous vehicles where the goal would be obtaining the maximum rewards." (Vijayaraghavan Varadharajan & Akanksha Rajendra Singh, "Building Intelligent Cities: Concepts, Principles, and Technologies", 2021)

"Methodologically, much of modern machine learning practice rests on a variant of trial and error, which we call the train-test paradigm. Practitioners repeatedly build models using any number of heuristics and test their performance to see what works. Anything goes as far as training is concerned, subject only to computational constraints, so long as the performance looks good in testing. Trial and error is sound so long as the testing protocol is robust enough to absorb the pressure placed on it." (Moritz Hardt & Benjamin Recht, "Patterns, Predictions, and Actions: Foundations of Machine Learning", 2022)

🔬Data Science: Heuristic (Definitions)

"Problem solving or analysis by experimental and especially trial-and-error methods." (Microsoft Corporation, "Microsoft SQL Server 7.0 Data Warehouse Training Kit", 2000)

"The mode of analysis in which the next step is determined by the results of the current step. Used for decision support processing." (Margaret Y Chu, "Blissful Data ", 2004)

"A type of analysis in which the next step is determined by the results of the current step of analysis." (Sharon Allen & Evan Terry, "Beginning Relational Data Modeling 2nd Ed.", 2005)

"The mode of analysis in which the next step is determined by the results of the current step of analysis. Used for decision-support processing." (William H Inmon, "Building the Data Warehouse", 2005)

"An algorithmic technique designed to solve a problem that ignores whether the solution can be proven to be correct." (Omar F El-Gayar et al, "Current Issues and Future Trends of Clinical Decision Support Systems", 2008)

"General advice that is usually efficient but sometimes cannot be used; also it is a validate function that adds a number to the state of the problem." (Attila Benko & Cecília S Lányi, "History of Artificial Intelligence", 2009) 

"These methods, found through discovery and observation, are known to produce incorrect or inexact results at times but likely to produce correct or sufficiently exact results when applied in commonly occurring conditions." (Vineet R Khare & Frank Z Wang, "Bio-Inspired Grid Resource Management", Handbook of Research on Grid Technologies and Utility Computing, 2009)

"Refers to a search and discovery approach, in which we proceed gradually, without trying to find out immediately whether the partial result, which is only adopted on a provisional basis, is true or false. This method is founded on a gradual approach to a given question, using provisional hypotheses and successive evaluations." (Humbert Lesca & Nicolas Lesca, "Weak Signals for Strategic Intelligence: Anticipation Tool for Managers", 2011)

"'Rules of thumb' and approximation methods for obtaining a goal, a high quality solution, or improved performance. It sacrifices completeness to increase efficiency, as some potential solutions would not be practicable or acceptable due to their 'rareness' or 'complexity'. This method may not always find the best solution, but it will find an acceptable solution within a reasonable timeframe for problems that will require almost infinite or longer than acceptable times to compute." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"An experience-based technique for solving problems that emphasizes personal knowledge and quick decision making." (Evan Stubbs, "Delivering Business Analytics: Practical Guidelines for Best Practice", 2013)

[heuristic process:] "An iterative process, where the next step of analysis depends on the results attained in the current level of analysis" (Daniel Linstedt & W H Inmon, "Data Architecture: A Primer for the Data Scientist", 2014)

"An algorithm that gives a good solution for a problem but that doesn’t guarantee to give you the best solution possible." (Rod Stephens, "Beginning Software Engineering", 2015)

"Rules of thumb derived by experience, intuition, and simple logic." (K  N Krishnaswamy et al, "Management Research Methodology: Integration of Principles, Methods and Techniques", 2016)

"Problem-solving technique that yields a sub-optimal solution judged to be sufficient." (Karl Beecher, "Computational Thinking - A beginner's guide to problem-solving and programming", 2017)

"An algorithm to solve a problem simply and quickly with an approximate solution, as compared to a complex algorithm that provides a precise solution, but may take a prohibitively long time." (O Sami Saydjari, "Engineering Trustworthy Systems: Get Cybersecurity Design Right the First Time", 2018)

🏗️Software Engineering: Heuristic (Just the Quotes)

"Heuristic reasoning is reasoning not regarded as final and strict but as provisional and plausible only, whose purpose is to discover the solution of the present problem. We are often obliged to use heuristic reasoning. We shall attain complete certainty when we shall have obtained the complete solution, but before obtaining certainty we must often be satisfied with a more or less plausible guess. We may need the provisional before we attain the final. We need heuristic reasoning when we construct a strict proof as we need scaffolding when we erect a building." (George Pólya, "How to Solve It", 1945)

"The aim of heuristics is to study the methods and rules of discovery and invention. [...] Heuristic, as an adjective, means 'serving to discover'." (George Pólya, "How to Solve It", 1945)

"An algorithm gives you the instructions directly. A heuristic tells you how to discover the instructions for yourself, or at least where to look for them." (Steve McConnell, "Code Complete", 1993)

"Heuristic (it is of Greek origin) means discovery. Heuristic methods are based on experience, rational ideas, and rules of thumb. Heuristics are based more on common sense than on mathematics. Heuristics are useful, for example, when the optimal solution needs an exhaustive search that is not realistic in terms of time. In principle, a heuristic does not guarantee the best solution, but a heuristic solution can provide a tremendous shortcut in cost and time." (Nikola K Kasabov, "Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"Heuristic methods may aim at local optimization rather than at global optimization, that is, the algorithm optimizes the solution stepwise, finding the best solution at each small step of the solution process and 'hoping' that the global solution, which comprises the local ones, would be satisfactory." (Nikola K Kasabov, "Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"Models of bounded rationality describe how a judgement or decision is reached (that is, the heuristic processes or proximal mechanisms) rather than merely the outcome of the decision, and they describe the class of environments in which these heuristics will succeed or fail." (Gerd Gigerenzer & Reinhard Selten [Eds., "Bounded Rationality: The Adaptive Toolbox", 2001)

"A heuristic is a rule applied to an existing solution represented in a perspective that generates a new (and hopefully better) solution or a new set of possible solutions." (Scott E Page, "The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools and Societies", 2008)

"There are two parts to learning craftsmanship: knowledge and work. You must gain the knowledge of principles, patterns, practices, and heuristics that a craftsman knows, and you must also grind that knowledge into your fingers, eyes, and gut by working hard and practicing." (Robert C Martin, "Clean Code: A Handbook of Agile Software Craftsmanship", 2008)

"A second class of metaphors - mathematical algorithms, heuristics, and models - brings us closer to the world of computer science programs, simulations, and approximations of the brain and its cognitive processes." (Diego Rasskin-Gutman, "Chess Metaphors: Artificial Intelligence and the Human Mind", 2009)

"[...] heuristics are simple, efficient rules - either hard-wired in our brains or learned - that kick in especially when we're facing problems with incomplete information." (David DiSalvo, "What Makes Your Brain Happy and Why You Should Do the Opposite", 2011)

"This is the essence of intuitive heuristics: when faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution." (Daniel Kahneman, "Thinking, Fast and Slow", 2011)

"Heuristics are simplified rules of thumb that make things simple and easy to implement. But their main advantage is that the user knows that they are not perfect, just expedient, and is therefore less fooled by their powers. They become dangerous when we forget that." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"A good heuristic decision is made by 1) knowing what to look for, 2) knowing when enough information is enough (the 'threshold of decision' ), and 3) knowing what decision to make." (Patrick Van Horne, "Left of Bang", 2014)

"Heuristic decision making is fast and frugal and is often based on the evaluation of one or two salient bits of information." (Amitav Chakravarti, "Why People (Don’t) Buy: The Go and Stop Signals", 2015)

"A heuristic is a strategy we derive from previous experience with a similar problem." (Darius Foroux, "Think Straight", 2017)

More quotes on "Heuristic" at the-web-of-knowledge.blogspot.com.