🧭💹Business Intelligence: Perspectives (Part XVIII: There’s More to Noise)

Business Intelligence Series

Visualizations should be built with an audience's characteristics in mind! Upon case, it might be sufficient to show only values or labels of importance (minima, maxima, inflexion points, exceptions, trends), while other times it might be needed to show all or most of the values to provide an accurate extended perspective. It even might be useful to allow users switching between the different perspectives to reduce the clutter when navigating the data or look at the patterns revealed by the clutter. 

In data-based storytelling are typically shown the points, labels and further elements that support the story, the aspects the readers should focus on, though this approach limits the navigability and users’ overall experience. The audience should be able to compare magnitudes and make inferences based on what is shown, and the accurate decoding shouldn’t be taken as given, especially when the audience can associate different meanings to what’s available and what’s missing. 

In decision-making, selecting only some well-chosen values or perspectives to show might increase the chances for a decision to be made, though is this equitable? Cherry-picking may be justified by the purpose, though is in general not a recommended practice! What is not shown can be as important as what is shown, and people should be aware of the implications!

One person’s noise can be another person’s signal. Patterns in the noise can provide more insight compared with the trends revealed in the "unnoisy" data shown! Probably such scenarios are rare, though it’s worth investigating what hides behind the noise. The choice of scale, the use of special types of visualizations or the building of models can reveal more. If it’s not possible to identify automatically such scenarios using the standard software, the users should have the possibility of changing the scale and perspective as seems fit. 

Identifying patterns in what seems random can prove to be a challenge no matter the context and the experience in the field. Occasionally, one might need to go beyond the general methods available and statistical packages can help when used intelligently. However, a presenter’s challenge is to find a plausible narrative around the findings and communicate it further adequately. Additional capabilities must be available to confirm the hypotheses framed and other aspects related to this approach.

It's ideal to build data models and a set of visualizations around them. Most probable some noise may be removed in the process, while other noise will be further investigated. However, this should be done through adjustable visual filters because what is removed can be important as well. Rare events do occur, probably more often than we are aware and they may remain hidden until we find the right perspective that takes them into consideration. 

Probably, some of the noise can be explained by special events that don’t need to be that rare. The challenge is to identify those parameters, associations, models and perspectives that reveal such insights. One’s gut feeling and experience can help in this direction, though novel scenarios can surprise us as well.

Not in every set of data one can find patterns, respectively a story trying to come out. Whether we can identify something worth revealing depends also on the data available at our disposal, respectively on whether the chosen data allow identifying significant patterns. Occasionally, the focus might be too narrow, too wide or too shallow. It’s important to look behind the obvious, to look at data from different perspectives, even if the data seems dull. It’s ideal to have the tools and knowledge needed to explore such cases and here the exposure to other real-life similar scenarios is probably critical!

Previous Post <<||>>  Next Post

⛁DBMS: Event Streaming Databases (More of a Kafka’s Story)

Event streaming architectures are architectures in which data are generated by different sources, and then processed, stored, analyzed, and acted upon in real-time by the different applications tapped into the data streams. An event streaming database is then a database that assures that its data are continuously up-to-date, providing specific functionality like management of connectors, materialized views and running queries on data-in-motion (rather than on static data). 

Reading about this type of technologies one can easily start fantasizing about the Web as a database in which intelligent agents can process streams of data in real-time, in which knowledge is derived and propagated over the networks in an infinitely and ever-growing flow in which the limits are hardly perceptible, in which the agents act as a mind disconnected in the end from the human intent. One is stroke by the fusing elements of realism and the fantastic aspects, more like in a Kafka’s story in which the metamorphosis of the technologies and social aspects can easily lead to absurd implications.

The link to Kafka was somehow suggested by Apache Kafka, an open-source distributed event streaming platform, which seems to lead the trends within this new-developing market. Kafka provides database functionality and guarantees the ACID (atomicity, concurrency, isolation, durability) properties of transactions while tapping into data streams. 

Data streaming is an appealing concept though it has some important challenges like data overload or over-flooding, the complexity derived from building specific (business) and integrity rules for processing the data, respectively for keeping data consistency and truth within the ever-growing and ever-changing flows. 

Data overload or over-flooding occurs when applications are not able to keep the pace with the volume of data or events fired with each change. Imagine the raindrops falling on a wide surface in which each millimeter or micrometer has its own rules for handling the raindrops and this at large scale. The raindrops must infiltrate into the surface to be processed and find their way to the beneath water flows, aggregating up to streams that could nurture seas or even oceans. Same metaphor can be applied to the data events, in which the data pervade applications accumulating in streams processed by engines to derive value. However heavy rains can easily lead to floods in which water aggregates at the surface. 

Business applications rely on predefined paths in which the flow of data is tidily linked to specific actions found themselves in processual sequences that reflect the material or cash flow. Any variation in the data flow from expectations will lead to inefficiencies and ultimately to chaos. Some benefit might be derived from data integrations between the business applications, however applications must be designed for this purpose and handle extreme behaviors like over-flooding. 

Data streams are maybe ideal for social media networks in which one broadcasts data through the networks and any consumer that can tap to the network can further use the respective data. We can see however the problems of nowadays social media – data, better said information, flow through the networks being changed as fit for purposes that can easily diverge from the initial intent. Moreover, information gets denatured, misused, overused to the degree that it overloads the networks, being more and more difficult to distinguish between reliable and non-reliable information. If common sense helps in the process of handling such information, not the same can be said about machines or applications. 

It will be challenging to deal with the vastness, vagueness, uncertainty, inconsistency, and deceit of the networks of the future, however data streaming more likely will have a future as long it can address such issues by design. 

🔭Data Science: Rare Events (Just the Quotes)

"We must rather seek for a cause, for every event whether probable or improbable must have some cause." (Polybius, "The Histories", cca. 100 BC)

"There is nothing in the nature of a miracle that should render it incredible: its credibility depends upon the nature of the evidence by which it is supported. An event of extreme probability will not necessarily command our belief unless upon a sufficiency of proof; and so an event which we may regard as highly improbable may command our belief if it is sustained by sufficient evidence. So that the credibility or incredibility of an event does not rest upon the nature of the event itself, but depends upon the nature and sufficiency of the proof which sustains it." (Charles Babbage, "Passages from the Life of a Philosopher", 1864)

"Events with a sufficiently small probability never occur, or at least we must act, in all circumstances, as if they were impossible." (Émile Borel, "Probabilities and Life", 1962)

"Most accidents in well-designed systems involve two or more events of low probability occurring in the worst possible combination." (Robert E Machol, "Principles of Operations Research", 1975)

"[…] all human beings - professional mathematicians included - are easily muddled when it comes to estimating the probabilities of rare events. Even figuring out the right question to ask can be confusing." (Steven Strogatz, "Sync: The Emerging Science of Spontaneous Order", 2003)

"Bell curves don't differ that much in their bells. They differ in their tails. The tails describe how frequently rare events occur. They describe whether rare events really are so rare. This leads to the saying that the devil is in the tails." (Bart Kosko, "Noise", 2006)

"A Black Swan is a highly improbable event with three principal characteristics: It is unpredictable; it carries a massive impact; and, after the fact, we concoct an explanation that makes it appear less random, and more predictable, than it was. […] The Black Swan idea is based on the structure of randomness in empirical reality. [...] the Black Swan is what we leave out of simplification." (Nassim N Taleb, “The Black Swan”, 2007)

"A forecaster should almost never ignore data, especially when she is studying rare events […]. Ignoring data is often a tip-off that the forecaster is overconfident, or is overfitting her model - that she is interested in showing off rather than trying to be accurate."  (Nate Silver, "The Signal and the Noise: Why So Many Predictions Fail-but Some Don't", 2012)

"[…] according to the bell-shaped curve the likelihood of a very-large-deviation event (a major outlier) located in the striped region appears to be very unlikely, essentially zero. The same event, though, is several thousand times more likely if it comes from a set of events obeying a fat-tailed distribution instead of the bell-shaped one." (John L Casti, "X-Events: The Collapse of Everything", 2012)

"[…] both rarity and impact have to go into any meaningful characterization of how black any particular [black] swan happens to be." (John L Casti, "X-Events: The Collapse of Everything", 2012)

"Black Swans (capitalized) are large-scale unpredictable and irregular events of massive consequence - unpredicted by a certain observer, and such un - predictor is generally called the 'turkey' when he is both surprised and harmed by these events. [...] Black Swans hijack our brains, making us feel we 'sort of' or 'almost' predicted them, because they are retrospectively explainable. We don’t realize the role of these Swans in life because of this illusion of predictability. […] An annoying aspect of the Black Swan problem - in fact the central, and largely missed, point - is that the odds of rare events are simply not computable." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"Behavioral finance so far makes conclusions from statics not dynamics, hence misses the picture. It applies trade-offs out of context and develops the consensus that people irrationally overestimate tail risk (hence need to be 'nudged' into taking more of these exposures). But the catastrophic event is an absorbing barrier. No risky exposure can be analyzed in isolation: risks accumulate. If we ride a motorcycle, smoke, fly our own propeller plane, and join the mafia, these risks add up to a near-certain premature death. Tail risks are not a renewable resource." (Nassim N Taleb, "Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applications" 2nd Ed., 2022)

"But note that any heavy tailed process, even a power law, can be described in sample (that is finite number of observations necessarily discretized) by a simple Gaussian process with changing variance, a regime switching process, or a combination of Gaussian plus a series of variable jumps (though not one where jumps are of equal size […])." (Nassim N Taleb, "Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applications" 2nd Ed., 2022)

"[…] it is not merely that events in the tails of the distributions matter, happen, play a large role, etc. The point is that these events play the major role and their probabilities are not (easily) computable, not reliable for any effective use. The implication is that Black Swans do not necessarily come from fat tails; the problem can result from an incomplete assessment of tail events." (Nassim N Taleb, "Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applications" 2nd Ed., 2022)

"Once we know something is fat-tailed, we can use heuristics to see how an exposure there reacts to random events: how much is a given unit harmed by them. It is vastly more effective to focus on being insulated from the harm of random events than try to figure them out in the required details (as we saw the inferential errors under thick tails are huge). So it is more solid, much wiser, more ethical, and more effective to focus on detection heuristics and policies rather than fabricate statistical properties." (Nassim N Taleb, "Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applications" 2nd Ed., 2022)

🔭Data Science: Events (Just the Quotes)

"[…] chance, that is, an infinite number of events, with respect to which our ignorance will not permit us to perceive their causes, and the chain that connects them together. Now, this chance has a greater share in our education than is imagined. It is this that places certain objects before us and, in consequence of this, occasions more happy ideas, and sometimes leads us to the greatest discoveries […]" (Claude A Helvetius, "On Mind", 1751)

"But ignorance of the different causes involved in the production of events, as well as their complexity, taken together with the imperfection of analysis, prevents our reaching the same certainty about the vast majority of phenomena. Thus there are things that are uncertain for us, things more or less probable, and we seek to compensate for the impossibility of knowing them by determining their different degrees of likelihood. So it was that we owe to the weakness of the human mind one of the most delicate and ingenious of mathematical theories, the science of chance or probability." (Pierre-Simon Laplace, "Recherches, 1º, sur l'Intégration des Équations Différentielles aux Différences Finies, et sur leur Usage dans la Théorie des Hasards", 1773)

"[…] determine the probability of a future or unknown event not on the basis of the number of possible combinations resulting in this event or in its complementary event, but only on the basis of the knowledge of order of familiar previous events of this kind" (Marquis de Condorcet, "Essai sur l'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix", 1785)

"Probability has reference partly to our ignorance, partly to our knowledge [..] The theory of chance consists in reducing all the events of the same kind to a certain number of cases equally possible, that is to say, to such as we may be equally undecided about in regard to their existence, and in determining the number of cases favorable to the event whose probability is sought. The ratio of this number to that of all cases possible is the measure of this probability, which is thus simply a fraction whose number is the number of favorable cases and whose denominator is the number of all cases possible." (Pierre-Simon Laplace, "Philosophical Essay on Probabilities", 1814)

"Things of all kinds are subject to a universal law which may be called the law of large numbers. It consists in the fact that, if one observes very considerable numbers of events of the same nature, dependent on constant causes and causes which vary irregularly, sometimes in one direction, sometimes in the other, it is to say without their variation being progressive in any definite direction, one shall find, between these numbers, relations which are almost constant." (Siméon-Denis Poisson, "Poisson’s Law of Large Numbers", 1837)

"Some of the common ways of producing a false statistical argument are to quote figures without their context, omitting the cautions as to their incompleteness, or to apply them to a group of phenomena quite different to that to which they in reality relate; to take these estimates referring to only part of a group as complete; to enumerate the events favorable to an argument, omitting the other side; and to argue hastily from effect to cause, this last error being the one most often fathered on to statistics. For all these elementary mistakes in logic, statistics is held responsible." (Sir Arthur L Bowley, "Elements of Statistics", 1901)

"The theory of chance consists in reducing all the events of the same kind to a certain number of cases equally possible, that is to say, to such as we may be equally undecided about in regard to their existence, and in determining the number of cases favorable to the event whose probability is sought." (Pierre-Simon de Laplace, "Philosophical Essay on Probabilities", 1902)

"Every theory of the course of events in nature is necessarily based on some process of simplification and is to some extent, therefore, a fairy tale." (Sir Napier Shaw, "Manual of Meteorology", 1932)

"The most important application of the theory of probability is to what we may call 'chance-like' or 'random' events, or occurrences. These seem to be characterized by a peculiar kind of incalculability which makes one disposed to believe - after many unsuccessful attempts - that all known rational methods of prediction must fail in their case. We have, as it were, the feeling that not a scientist but only a prophet could predict them. And yet, it is just this incalculability that makes us conclude that the calculus of probability can be applied to these events." (Karl R Popper, "The Logic of Scientific Discovery", 1934)

"Multiple equilibria are not necessarily useless, but from the standpoint of any exact science the existence of a uniquely determined equilibrium is, of course, of the utmost importance, even if proof has to be purchased at the price of very restrictive assumptions; without any possibility of proving the existence of (a) uniquely determined equilibrium - or at all events, of a small number of possible equilibria - at however high a level of abstraction, a field of phenomena is really a chaos that is not under analytical control." (Joseph A Schumpeter, "History of Economic Analysis", 1954)

"In fact, it is empirically ascertainable that every event is actually produced by a number of factors, or is at least accompanied by numerous other events that are somehow connected with it, so that the singling out involved in the picture of the causal chain is an extreme abstraction. Just as ideal objects cannot be isolated from their proper context, material existents exhibit multiple interconnections; therefore the universe is not a heap of things but a system of interacting systems." (Mario Bunge, "Causality: The place of the casual principles in modern science", 1959)

"Certain properties are necessary or sufficient conditions for other properties, and the network of causal relations thus established will make the occurrence of one property at least tend, subject to the presence of other properties, to promote or inhibit the occurrence of another. Arguments from models involve those analogies which can be used to predict the occurrence of certain properties or events, and hence the relevant relations are causal, at least in the sense of implying a tendency to co-occur." (Mary B Hesse," Models and Analogies in Science", 1963)

"In complex systems cause and effect are often not closely related in either time or space. The structure of a complex system is not a simple feedback loop where one system state dominates the behavior. The complex system has a multiplicity of interacting feedback loops. Its internal rates of flow are controlled by nonlinear relationships. The complex system is of high order, meaning that there are many system states (or levels). It usually contains positive-feedback loops describing growth processes as well as negative, goal-seeking loops. In the complex system the cause of a difficulty may lie far back in time from the symptoms, or in a completely different and remote part of the system. In fact, causes are usually found, not in prior events, but in the structure and policies of the system." (Jay Wright Forrester, "Urban dynamics", 1969)

"There are different levels of organization in the occurrence of events. You cannot explain the events of one level in terms of the events of another. For example, you cannot explain life in terms of mechanical concepts, nor society in terms of individual psychology. Analysis can only take you down the scale of organization. It cannot reveal the workings of things on a higher level. To some extent the holistic philosophers are right." (Anatol Rapoport, "General Systems" Vol. 14, 1969)

"[I]n probability theory we are faced with situations in which our intuition or some physical experiments we have carried out suggest certain results. Intuition and experience lead us to an assignment of probabilities to events. As far as the mathematics is concerned, any assignment of probabilities will do, subject to the rules of mathematical consistency." (Robert Ash, "Basic probability theory", 1970)

"Perhaps randomness is not merely an adequate description for complex causes that we cannot specify. Perhaps the world really works this way, and many events are uncaused in any conventional sense of the word." (Stephen Jay Gould,"Hen's Teeth and Horse's Toes", 1983)

"If you perceive the world as some place where things happen at random - random events over which you have sometimes very little control, sometimes fairly good control, but still random events - well, one has to be able to have some idea of how these things behave. […] People who are not used to statistics tend to see things in data - there are random fluctuations which can sometimes delude them - so you have to understand what can happen randomly and try to control whatever can be controlled. You have to expect that you are not going to get a clean-cut answer. So how do you interpret what you get? You do it by statistics." (Lucien LeCam, [interview] 1988)

"According to the narrower definition of randomness, a random sequence of events is one in which anything that can ever happen can happen next. Usually it is also understood that the probability that a given event will happen next is the same as the probability that a like event will happen at any later time. [...] According to the broader definition of randomness, a random sequence is simply one in which any one of several things can happen next, even though not necessarily anything that can ever happen can happen next." (Edward N Lorenz, "The Essence of Chaos", 1993)

"So we pour in data from the past to fuel the decision-making mechanisms created by our models, be they linear or nonlinear. But therein lies the logician's trap: past data from real life constitute a sequence of events rather than a set of independent observations, which is what the laws of probability demand.[...] It is in those outliers and imperfections that the wildness lurks." (Peter L Bernstein, "Against the Gods: The Remarkable Story of Risk", 1996)

"Events may appear to us to be random, but this could be attributed to human ignorance about the details of the processes involved." (Brain S Everitt, "Chance Rules", 1999)

"The subject of probability begins by assuming that some mechanism of uncertainty is at work giving rise to what is called randomness, but it is not necessary to distinguish between chance that occurs because of some hidden order that may exist and chance that is the result of blind lawlessness. This mechanism, figuratively speaking, churns out a succession of events, each individually unpredictable, or it conspires to produce an unforeseeable outcome each time a large ensemble of possibilities is sampled."  (Edward Beltrami, "What is Random?: Chaos and Order in Mathematics and Life", 1999)

"Entropy [...] is the amount of disorder or randomness present in any system. All non-living systems tend toward disorder; left alone they will eventually lose all motion and degenerate into an inert mass. When this permanent stage is reached and no events occur, maximum entropy is attained. A living system can, for a finite time, avert this unalterable process by importing energy from its environment. It is then said to create negentropy, something which is characteristic of all kinds of life." (Lars Skyttner, "General Systems Theory: Ideas and Applications", 2001)

"One can be highly functionally numerate without being a mathematician or a quantitative analyst. It is not the mathematical manipulation of numbers (or symbols representing numbers) that is central to the notion of numeracy. Rather, it is the ability to draw correct meaning from a logical argument couched in numbers. When such a logical argument relates to events in our uncertain real world, the element of uncertainty makes it, in fact, a statistical argument." (Eric R Sowey, "The Getting of Wisdom: Educating Statisticians to Enhance Their Clients' Numeracy", The American Statistician 57(2), 2003)

"Randomness is a difficult notion for people to accept. When events come in clusters and streaks, people look for explanations and patterns. They refuse to believe that such patterns - which frequently occur in random data - could equally well be derived from tossing a coin. So it is in the stock market as well." (Didier Sornette, "Why Stock Markets Crash: Critical events in complex financial systems", 2003)

"The basic concept of complexity theory is that systems show patterns of organization without organizer (autonomous or self-organization). Simple local interactions of many mutually interacting parts can lead to emergence of complex global structures. […] Complexity originates from the tendency of large dynamical systems to organize themselves into a critical state, with avalanches or 'punctuations' of all sizes. In the critical state, events which would otherwise be uncoupled became correlated." (Jochen Fromm, "The Emergence of Complexity", 2004)

"[myth:] Counting can be done without error. Usually, the counted number is an integer and therefore without (rounding) error. However, the best estimate of a scientifically relevant value obtained by counting will always have an error. These errors can be very small in cases of consecutive counting, in particular of regular events, e.g., when measuring frequencies." (Manfred Drosg, "Dealing with Uncertainties: A Guide to Error Analysis", 2007)

"[...] in probability theory we are faced with situations in which our intuition or some physical experiments we have carried out suggest certain results. Intuition and experience lead us to an assignment of probabilities to events. As far as the mathematics is concerned, any assignment of probabilities will do, subject to the rules of mathematical consistency." (Robert Ash, "Basic Probability Theory", 2008)

"Regression toward the mean. That is, in any series of random events an extraordinary event is most likely to be followed, due purely to chance, by a more ordinary one." (Leonard Mlodinow, "The Drunkard’s Walk: How Randomness Rules Our Lives", 2008)

"In the network society, the space of flows dissolves time by disordering the sequence of events and making them simultaneous in the communication networks, thus installing society in structural ephemerality: being cancels becoming." (Manuel Castells, "Communication Power", 2009)

"Without precise predictability, control is impotent and almost meaningless. In other words, the lesser the predictability, the harder the entity or system is to control, and vice versa. If our universe actually operated on linear causality, with no surprises, uncertainty, or abrupt changes, all future events would be absolutely predictable in a sort of waveless orderliness." (Lawrence K Samuels, "Defense of Chaos: The Chaology of Politics, Economics and Human Action", 2013)

"The problem of complexity is at the heart of mankind’s inability to predict future events with any accuracy. Complexity science has demonstrated that the more factors found within a complex system, the more chances of unpredictable behavior. And without predictability, any meaningful control is nearly impossible. Obviously, this means that you cannot control what you cannot predict. The ability ever to predict long-term events is a pipedream. Mankind has little to do with changing climate; complexity does." (Lawrence K Samuels, "The Real Science Behind Changing Climate", 2014)

More quotes on "Events" at the-web-of-knowledge.blogspot.com. 

♜Strategic Management: Resilience (Definitions)

"The ability to recover from challenges or to overcome obstacles. In a social-ecological context this refers to the innovation capacity of the organization to successfully address societal and environmental challenges." (Rick Edgeman & Jacob Eskildsen, "Social-Ecological Innovation", 2014)

"The quality of being able to absorb systemic 'shocks' without being destroyed even if recovery produces an altered state to that of the status quo ante." (Philip Cooke, "Regional Innovation Systems in Centralised States: Challenges, Chances, and Crossovers", 2015)

"The ability of an organization to quickly adapt to disruptions while maintaining continuous business operations and safeguarding people, assets, and overall brand equity. Business resilience goes a step beyond disaster recovery, by offering post-disaster strategies to avoid costly downtime, shore up vulnerabilities, and maintain business operations in the face of additional, unexpected breaches." (William Stallings, "Effective Cybersecurity: A Guide to Using Best Practices and Standards", 2018)

"A capability to anticipate, prepare for, respond to, and recover from significant multi-hazard threats with minimum damage to social well-being, the economy, and the environment." (Carolyn N Stevenson, "Addressing the Sustainable Development Goals Through Environmental Education", 2019)

"The ability of a project to readily resume from unexpected events, threats or actions." (Phil Crosby, "Shaping Mega-Science Projects and Practical Steps for Success", 2019)

"The ability of an infrastructure to resist, respond and overcome adverse events" (Konstantinos Apostolou et al, "Business Continuity of Critical Infrastructures for Safety and Security Incidents", 2020)

"The capacity to respond to, adapt and learn from stressors and changing conditions." (Naomi Borg & Nader Naderpajouh, "Strategies for Business Sustainability in a Collaborative Economy", 2020)

"The word resilience refers to the ability to overcome critical moments and adapt after experiencing some unusual and unexpected situation. It also indicates return to normal." (José G Vargas-Hernández, "Urban Socio-Ecosystems Green Resilience", 2021)

"Operational resilience is a set of techniques that allow people, processes and informational systems to adapt to changing patterns. It is the ability to alter operations in the face of changing business conditions. Operationally resilient enterprises have the organizational competencies to ramp up or slow down operations in a way that provides a competitive edge and enables quick and local process modification." (Gartner)

[Operational resilience:] "The ability of an organization to absorb the impact of any unexpected event without failing to deliver on its brand promise." (Forrester)

[Business resilience:] "The ability to thrive in the face of unpredictable events and circumstances without deteriorating customer experience or sacrificing the long-term viability of the company." (Forrester)

♜Strategic Management: Contingency Plan (Definitions)

"An identification of alternative strategies to be used to ensure project success if specified risk events occur." (Timothy J  Kloppenborg et al, "Project Leadership", 2003)

[contingency planning:] "A management process that analyses disaster risks and establishes arrangements in advance to enable timely, effective and appropriate responses." (ISDR, 2009)

"Specific planning designed to create a quick response after the occurrence of a risk event." (Annetta Cortez & Bob Yehling, "The Complete Idiot's Guide® To Risk Management", 2010)

"A plan that identifies alternative approaches to be used if the corresponding risk events occur." (Bonnie Biafore, "Successful Project Management: Applying Best Practices and Real-World Techniques with Microsoft® Project", 2011)

"A plan developed to mitigate the outcome of a risk, once the risk has materialised." (Mike Clayton, "Brilliant Project Leader", 2012)

"Mitigation plan alternative course(s) of action devised to cope with project risks." (Chartered Institute of Building, "Code of Practice for Project Management for Construction and Development" 5th Ed., 2014)

"A plan that allows an organization to respond appropriately to a specific type of unplanned event."(Rebecca Hamilton & Diane Brown, "Disaster Management and Continuity Planning in Libraries: Changes since the Year 2000", 2016)

"A plan for continued operation and execution of the most essential functions of a mission in the event of a disruptive failure, such as a natural disaster or a major cyberattack." (O Sami Saydjari, "Engineering Trustworthy Systems: Get Cybersecurity Design Right the First Time", 2018)

"A plan put in place before any potential emergencies, with the mission of dealing with possible future emergencies. It pertains to training personnel, performing backups, preparing critical facilities, and recovering from an emergency or disaster so that business operations can continue." (Shon Harris & Fernando Maymi, "CISSP All-in-One Exam Guide" 8th Ed., 2018)

[contingency planning:] "Management policies and procedures designed to maintain or restore business operations, including computer operations, possibly at an alternate location, in the event of emergencies, system failures, or disasters." (William Stallings, "Effective Cybersecurity: A Guide to Using Best Practices and Standards", 2018)

"A plan that is maintained for disaster response, backup operations, and post-disaster recovery to ensure the availability of critical resources and to facilitate the continuity of operations in an emergency situation." (NIST SP 800-57 Part 1)

"Management policy and procedures used to guide an enterprise response to a perceived loss of mission capability. The Contingency Plan is the first plan used by the enterprise risk managers to determine what happened, why, and what to do. It may point to the continuity of operations plan (COOP) or disaster recovery plan (DRP) for major disruptions." (CNSSI 4009-2015)

🕸Systems Engineering: Systems Thinking (Definitions)

"Systems thinking is a discipline for seeing the 'structures' that underlie complex situations, and for discerning high from low leverage change. That is, by seeing wholes we learn how to foster health. To do so, systems thinking offers a language that begins by restructuring how we think." (Peter Senge, "The Fifth Discipline", 1990)

"Systems thinking is a framework for seeing interrelationships rather than things, for seeing patterns rather than static snapshots. It is a set of general principles spanning fields as diverse as physical and social sciences, engineering and management." (Peter Senge, "The Fifth Discipline", 1990)

"A school of thought that focuses on recognizing the interconnections between the parts of a system and synthesizing them into a unified view of the whole." (Virginia Anderson & Lauren Johnson, "Systems Thinking Basics: From Concepts to Casual Loops", 1997)

"Systems thinking means the ability to see the synergy of the whole rather than just the separate elements of a system and to learn to reinforce or change whole system patterns." (Richard L Daft, "The Leadership Experience", 2002)

"A concept for describing a way of helping people view systems from a wide perspective, seeing overall structures, patterns and cycles in subsystems, rather than seeing only specific events in the main system." (Thomas Hansson, "Communication and Relation Building in Social Systems", 2008)

"Systems thinking is a mental discipline and framework for seeing patterns and interrelationships." (Richard L Daft, "The Leadership Experience", 2008) 

"A manner of thinking that takes into account how the things being studied relate and connect to each other. A key idea embedded in systems theory is that it can assist us in understanding of phenomena and that its holistic emphasis will promote orderly thinking. It is an apt approach to use when thinking about complex issues and interactions." (Deborah W Proctor, "Accessibility of Technology in Higher Education", 2009)

"An approach to analysis, based on the insight that components of a system or (sub)systems may act differently when isolated from the interacting environment and hence the basic concept for studying systems in a holistic way as a supplement to traditional reductionistic techniques." (Herwig Ostermann et al, "Benchmarking Human Resource Information Systems", 2009)

"Critical to this definition is the term ‘interaction’, in that systems thinking is a form of analysis that goes beyond specific causes and effects to the discernment of hidden patterns of behaviors and underlying systemic interrelationships." (Gerald Goodman & Anne Selcer, "Systems Thinking as the Model for Educating Future Healthcare Managers in Information Technology", 2009)

"Is thinking holistically and conscientiously about the world by focusing on the interaction of the parts and their influence within and over the system." (Kambiz E Maani, "Systems Thinking and the Internet from Independence to Interdependence", 2009)

"A holistic concept of tackling problems and events by taking into account the larger scope in the complete environment." (Nashon J Adero et al, "Flow-Based Structural Modelling and Dynamic Simulation of Lake Water Levels", 2011)

"An approach that emphasizes the interconnected nature of the different components that make up a system. Thus, to understand a problem with performance in an organization, you must analyze the whole organizational system not just the component (process, unit or individual) that on the surface seems to be the root of the problem." (Ian Douglas, "Organizational Needs Analysis and Knowledge Management", 2011)

"An approach to understanding the interconnectedness of components when grouped together in order to solve a problem and how the grouped components behave under different stimuli." (Kyle G. Gipson & Robert J Prins, "Materials and Mechanics: A Multidisciplinary Course Incorporating Experiential, Project/Problem-Based, and Work-Integrated Learning Approaches for Undergraduates", 2015)

"In a system dynamics context, a way of thinking based on system dynamics. It is also used to mean system dynamics analyses without quantitative definitions. It focuses on feedback loop structure in order to forecast the direction of performance and find pertinent elements for controlling systems. This is also called qualitative system dynamics." (Yutaka Takahashi, "System Dynamics", 2015)

"Systems thinking is a discipline or process that considers how individual elements interact with one another as part of a whole entity. As an approach to solving problems, systems thinking uses relationships among individual elements and the dynamics of these relationships to explain the behavior of systems such as an ecosystem, social system, or organization." (Karen L Higgins, "Economic Growth and Sustainability: Systems Thinking for a Complex World", 2015)

"The process and understanding of how items influence one another within a whole." (Reginald Wilson, "Outage Analysis and Maintenance Strategies in Hydroelectric Production", 2015)

"A perspective and approach to problem-solving that emphasizes understanding the world in terms of dynamic systems, the interrelationships among elements of systems, and how systems influence each other." (Elisabeth R Gee Kelly M Tran, "Video Game Making and Modding", 2016)

"A relevant scientific instrumentarium, based on principles of General Systems Theory, which uses the systems ideas in order to research and solve complex strategic problems/problem situations." (Dejana Zlatanović et al, "Higher Education Institutions as Viable Systems: A Cybernetic Framework for Innovativeness", 2020)

"The process of understanding how things influence one another. It refers rather to seeing overall structures, patterns and cycles in systems, and the connections between them, than specific events in the system." (The KPI Institute)

🚧Project Management: Project Planning (Just the Quotes)

"Planning starts usually with something like a general idea. For one reason or another it seems desirable to reach a certain objective, and how to reach it is frequently not too clear. The first step then is to examine the idea carefully in the light of the means available. Frequently more fact-finding about the situation is required. If this first period of planning is successful, two items emerge: namely, an 'over-all plan' of how to reach the objective and secondly, a decision in regard to the first step of action. Usually this planning has also somewhat modified the original idea. The next period is devoted to executing the first step of the original plan." (Kurt Lewin, "Action research and minority problems", 1946)

"Every company has beloved projects on which if prices had held up, if the contractors had finished on time (or finished at all), if the plans hadn't been altered, if the thing had actually worked, the planned return would have been earned. But since some or all of these calamities [things that don't go as expected] usually happen, any manager who neglects to allow for them is not planning - merely thinking wishfully. Desire for the project has, as usual, overtaken desire for profit." (Ernest Dale, "Planning and developing the company organization structure", 1952)

"Project management is the process by which it is assured that the objective is achieved and resources are not wasted. Planning is one of the two parts of project management. Control is the other. [...] Each project must first be planned in detail. Control is involved with comparing actual progress with the plan and taking corrective action when the two do not correspond. Without the plan, true control is not possible; the need for corrective action, its nature, extent, and urgency cannot he accurately determined." (Robert D Carlsen & James A Lewis, "The Systems Analysis Workbook: A complete guide to project implementation and control", 1973)

"Since software construction is inherently a systems effort - an exercise in complex interrelationships - communication effort is great, and it quickly dominates the decrease in individual task time brought about by partitioning [increasing the workers]. Adding more people then lengthens, not shortens, the schedule." (Frederick Brook, "The Mythical Man-Month", 1975)

"Because one has to be an optimist to begin an ambitious project, it is not surprising that underestimation of completion time is the norm." (Fernando J Corbató, "On Building Systems That Will Fail", 1991)

"If we decide to plan not to lose, we take a defensive posture in which we expend huge amounts of effort trying to prevent and track errors. This will lead us to a very heavyweight planning process in which we try to plan everything up front in a much detail as possible. Such a process will have many review steps, sign-offs, authorizations, and phase gates. Such a planning process is highly adept at making sure that blame can be assigned when something fails; but takes no direct steps towards making sure that the right system is delivered at a reasonable cost." (Kent Beck & Martin Fowler, "Planning Extreme Programming", 2000)

"One of the purposes of planning is we always want to work on the most valuable thing possible at any given time. We can’t pick features at random and expect them to be most valuable. We have to begin development by taking a quick look at everything that might be valuable, putting all our cards on the table. At the beginning of each iteration the business (remember the balance of power) will pick the most valuable features for the next iteration." (Kent Beck & Martin Fowler, "Planning Extreme Programming", 2000)

"Planning is not about predicting the future. When you make a plan for developing a piece of software, development is not going to go like that. Not ever. Your customers wouldn’t even be happy if it did, because by the time software gets there, the customers don’t want what was planned, they want something different." (Kent Beck & Martin Fowler, "Planning Extreme Programming", 2000)

"Projects sometimes fail long before they deliver anything. At some point they may be determined to be too expensive to continue. Or perhaps they took too long to develop and the business need evaporated. Or perhaps the requirements change so often that the developers can never finish one thing without having to stop and start all over on something new. Certainly these are planning failures." (Kent Beck & Martin Fowler, "Planning Extreme Programming", 2000)

"There are two ways to approach prevention of these planning failures. We can plan not to lose, or we can plan to win. The two are not identical. Planning not to lose is defensive; while planning to win is aggressive. [...] the problem that planning is supposed to solve is simply, to build the right system at the right cost. If we take a defensive posture by planning not to lose, we will be able to hold people accountable for any failures; but at an enormous cost. If we take an aggressive posture and plan to win, we will be unafraid to make errors, and will continuously correct them to meet our goals." (Kent Beck & Martin Fowler, "Planning Extreme Programming", 2000)

"We plan because: We need to ensure that we are always working on the most important thing we need to do. We need to coordinate with other people. When unexpected events occur we need to understand the consequences for the first two." (Kent Beck & Martin Fowler, "Planning Extreme Programming", 2000)

"When we plan to win we take direct steps to ensure that we are building the right system at the best possible cost. Every action we take goes towards that end. Instead of trying to plan everything up front, we plan just the next few steps; and then allow customer feedback to correct our trajectory. In this way, we get off the mark quickly, and then continuously correct our direction. Errors are unimportant because they will be corrected quickly." (Kent Beck & Martin Fowler, "Planning Extreme Programming", 2000)

"If you have no plan, you cannot have control, by definition, because it is your plan that tells where you are supposed to be in the first place. Further, if you don’t know where you are, you can’t have control. This comes from your information system. Most organizations have difficulties with both of these." (James P Lewis, "Project Planning, Scheduling, and Control" 3rd Ed., 2001)

"No project can succeed when the team members have no commitment to the plan, so the first rule of project planning is that the people who must do the work should help plan that part of the project. You will not only gain their commitment to the plan, but also most likely cover all of the important issues that you may individually have forgotten."(James P Lewis, "Project Planning, Scheduling, and Control" 3rd Ed., 2001)

"The big fallacy in our assumptions is that the world will stand still while we execute our project plan." (James P Lewis, "Project Planning, Scheduling, and Control" 3rd Ed., 2001)

"Project planning is the key to effective project management. Detailed and accurate planning of a project produces the managerial information that is the basis of project justification (costs, benefits, strategic impact, etc.) and the defining of the business drivers (scope, objectives) that form the context for the technical solution. In addition, project planning also produces the project schedules and resource allocations that are the framework for the other project management processes: tracking, reporting, and review." (Rob Thomsett, "Radical Project Management", 2002)

"If you've been in the software business for any time at all, you know that there are certain common problems that plague one project after another. Missed schedules and creeping requirements are not things that just happen to you once and then go away, never to appear again. Rather, they are part of the territory. We all know that. What's odd is that we don't plan our projects as if we knew it. Instead, we plan as if our past problems are locked in the past and will never rear their ugly heads again. Of course, you know that isn't a reasonable expectation." (Tom DeMarco & Timothy Lister, "Waltzing with Bears: Managing Risk on Software Projects", 2003)

"The pathology of setting a deadline to the earliest articulable date essentially guarantees that the schedule will be missed." (Tom DeMarco & Timothy Lister, "Waltzing with Bears: Managing Risk on Software Projects", 2003)

"Ending up somewhere entirely different from where you expected to go is the norm in this world. Software projects are prime illustrations of the law of unintended consequences, and their innovations and breakthroughs are more often side effects than planned outcomes." (Scott Rosenberg, "Dreaming in Code", 2007)

"[…] in software development, as in all things, plans get dodgier the farther into the future one looks. Any developer who has been around the block will admit that the cavalcade of methodologies over three decades of software history has left the field richer and given programmers useful new tools and ways of thinking about their work. But finding a developer or team that actually subscribes to a particular methodology isn’t easy." (Scott Rosenberg, "Dreaming in Code", 2007)

"The picture of digital progress that so many ardent boosters paint ignores the painful record of actual programmers’ epic struggles to bend brittle code into functional shape. That record is of one disaster after another, marking the field’s historical time line like craters. Anyone contemplating the start of a big software development project today has to contend with this unfathomably discouraging burden of experience. It mocks any newcomer with ambitious plans, as if to say, What makes you think you’re any different?" (Scott Rosenberg, "Dreaming in Code", 2007)

"Users may be annoyed by bugs, and software developers may be disappointed by their inability to perfect their work, and managers may be frustrated by the unreliability of their plans. But in the end, none of that matters as much as the simple fact that software does not work the way we think, and until it does, it is not worth trying to perfect." (Scott Rosenberg, "Dreaming in Code", 2007)

"And even if we make good plans based on the best information available at the time and people do exactly what we plan, the effects of our actions may not be the ones we wanted because the environment is nonlinear and hence is fundamentally unpredictable. As time passes the situation will change, chance events will occur, other agents such as customers or competitors will take actions of their own, and we will find that what we do is only one factor among several which create a new situation." (Stephen Bungay, "The Art of Action: How Leaders Close the Gaps between Plans, Actions, and Results", 2010)

"A project plan is a prediction. It predicts that a team of N people will complete X amount of work by Y date." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"Development is a design process. Design processes are generally evaluated by the value they deliver rather than a conformance to plan. Therefore, it makes sense to move away from plan-driven projects and toward value-driven projects. […] The realization that the source code is part of the design, not the product, fundamentally rewires our understanding of software." (Sriram Narayan, "Agile IT Organization Design: For Digital Transformation and Continuous Delivery", 2015)

"The planning fallacy is the systematic tendency for project plans and budgets to undershoot. […] The reasons for the planning fallacy are partly psychological, partly cultural, and partly to do with our limited ability to think probabilistically." (Paul Gibbons, "The Science of Successful Organizational Change",  2015)

"An effort estimate is not complete without including its assumptions. Estimate assumptions include any and all underlying factors the estimate relies upon. Assumptions are especially important in more rigid estimation environments, but they are a good practice even where expectations are more flexible. Explicitly listing all assumptions helps to remove ambiguity and avoid misunderstandings during project delivery." (Morgan Evans, "Engineering Manager's Handbook", 2023)

"Plans allow us to think through objectives beforehand in the hope of being prepared for delivery. Plans are useful when they preempt conflict, direct efforts in harmony, and align expectations. Plans are not useful when they waste valuable build time or provide a false sense of security, for example, by missing unknown unknowns." (Morgan Evans, "Engineering Manager's Handbook", 2023)

🛢DBMS: Trace (Definitions)

"The process of recording the sequence in which the statements in a program are executed and, optionally, the values of the program variables used in the statements." (Sybase, "Glossary", 2005)

"The SQL Profiler method for recording server events." (Thomas Moore, "EXAM CRAM™ 2: Designing and Implementing Databases with SQL Server 2000 Enterprise Edition", 2005)

"This is a record of data that has been captured about events in Profiler." (Joseph L Jorden & Dandy Weyn, "MCTS Microsoft SQL Server 2005: Implementation and Maintenance Study Guide - Exam 70-431", 2006)

"A collection of events and related performance data returned by SQL Server’s database engine." (Victor Isakov et al, "MCITP Administrator: Microsoft SQL Server 2005 Optimization and Maintenance (70-444) Study Guide", 2007)

"A trace is a collection of events and data. SQL Profiler is used to collect and monitor events. Creating a trace is sometimes referred to as capturing events." (Darril Gibson, "MCITP SQL Server 2005 Database Developer All-in-One Exam Guide", 2008)

"A collection of events and data returned by the Database Engine." (Microsoft, "SQL Server 2012 Glossary", 2012)

"A record of the processing of a computer program or transaction. The information collected from a trace can be used to assess problems and performance." (IBM, "Informix Servers 12.1", 2014)

"In DB2 replication, a facility that is used to collect monitoring, auditing, and performance data for the Capture program, the Q Capture program, the Apply program, the Q Apply program, or the Replication Alert Monitor." (Sybase, "Open Server Server-Library/C Reference Manual", 2019)

🌁Software Engineering: Event-Driven Architecture (Definition)

"A software architecture pattern promoting the production, detection, consumption of, and reaction to events. Some consider EDA to be an extension of or complement to SOA; others consider EDA to be part of the SOA approach (a special message exchange pattern where the service provider sends a message to multiple consumers)." (Nicolai M Josuttis, "SOA in Practice", 2007)

"An architectural style in which one or more components in a software system are event-driven and minimally coupled. 'Minimally coupled' means that the only relationship between the event producer and the event consumer is a one-way, “fire and forget” notification. The producer does not get a response associated with the notification back from the consumer, and a notification does not prescribe the action the consumer will perform. Something is event-driven without being EDA if it is not minimally coupled." (W Roy Schulte & K Chandy, "Event Processing: Designing IT Systems for Agile Companies", 2009)

"A software architecture pattern promoting the production, detection, consumption of, and reaction to events. " (David Lyle & John G Schmidt, "Lean Integration: An Integration Factory Approach to Business Agility", 2010)

"A design where various parts of the system respond to events as they occur." (Rod Stephens, "Beginning Software Engineering", 2015)