🤖〽️Prompt Engineering: Copilot Unabridged (Part 174: How We Can Make AI Aware of Its Knowledge Gaps)

Prompt Engineering Series

 

Prompt: "write a post of 600 words on how we can make AI aware of its knowledge gaps"

Introduction

One of the most persistent challenges in artificial intelligence is that modern systems do not know what they do not know. They generate answers with confidence, even when those answers are incomplete, speculative, or simply wrong. This inability to recognize knowledge gaps is not a minor flaw - it is a structural limitation rooted in how AI systems are built. Yet if we want AI to become a more reliable partner in reasoning, decision‑making, and discovery, we must find ways to help it identify, signal, and manage its own uncertainty. Making AI aware of its knowledge gaps is not about giving it human‑like introspection; it is about engineering mechanisms that approximate epistemic awareness.

The first step is explicit uncertainty modeling. Current AI systems generate text based on probability distributions, but they do not expose those probabilities in a meaningful way. They treat every answer as equally deliverable, regardless of how confident the underlying model actually is. By contrast, a system designed to surface its uncertainty - through calibrated confidence scores, probability ranges, or structured 'uncertainty tokens' - would be able to distinguish between strong knowledge and weak inference. This does not give the AI self‑awareness, but it gives users a window into the model’s internal landscape. When an AI can say, 'I am 40% confident in this answer', it becomes far easier to judge when to trust it and when to verify.

A second approach involves retrieval‑anchored reasoning. One of the reasons AI hallucinates is that it relies solely on internal patterns rather than external verification. Retrieval‑augmented generation (RAG) changes this dynamic by forcing the model to ground its answers in real documents, databases, or authoritative sources. When the system cannot retrieve relevant information, it can explicitly acknowledge the gap: 'I could not find supporting evidence for this claim'. This creates a form of externally enforced epistemic humility. The model becomes less of a storyteller and more of an evidence‑seeking agent.

Another promising direction is meta‑cognitive scaffolding - structures that help the AI evaluate its own reasoning steps. Chain‑of‑thought prompting, self‑critique loops, and multi‑agent debate frameworks allow the system to inspect its own output before presenting it. These mechanisms do not give the AI genuine introspection, but they simulate a process of internal review. When one reasoning path contradicts another, the system can flag the inconsistency as a potential knowledge gap. This mirrors how humans detect uncertainty: not through perfect self‑knowledge, but through the friction between competing interpretations.

We can also make AI more aware of its limits by training it on examples of uncertainty. Most datasets reward confident answers, not cautious ones. If we expose models to high‑quality examples of expert uncertainty - scientific papers that acknowledge limitations, medical notes that describe diagnostic ambiguity, or philosophical texts that explore unknowns - the AI learns that 'I don’t know' is a legitimate and often necessary response. This shifts the model’s behavior from overconfident prediction to calibrated judgment.

Equally important is user‑driven feedback. When humans correct AI errors, challenge assumptions, or request justification, they teach the system where its blind spots lie. Reinforcement learning from human feedback (RLHF) can incorporate these corrections into the model’s behavior, gradually shaping it into a system that recognizes patterns of uncertainty. Over time, the AI becomes better at identifying situations where it historically struggled - complex reasoning tasks, ambiguous queries, or domains with sparse training data.

Finally, making AI aware of its knowledge gaps requires rethinking the interface between humans and machines. If AI systems are designed to present answers as definitive, users will treat them as such. But if the interface encourages exploration - offering alternative interpretations, highlighting missing information, or prompting users to refine their questions - then uncertainty becomes part of the interaction rather than a hidden flaw.

The path to AI that understands its own limits is not about giving machines consciousness. It is about engineering systems that can recognize ambiguity, signal uncertainty, and collaborate with humans in the shared pursuit of knowledge.

Disclaimer: The whole text was generated by Copilot (under Windows 11) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🤖〽️Prompt Engineering: Copilot Unabridged (Part 173: How the Gap Between AI’s Answers and Real Knowledge Can Be Bridged)

Prompt Engineering Series

 

Prompt: "write a post of 600 words on how the gap between AI's answers and real knowledge can be bridged"

Introduction

The gap between AI’s answers and real knowledge is one of the defining challenges of our technological era. AI systems can generate fluent, persuasive responses, yet they lack the grounding, understanding, and epistemic commitment that characterize human knowledge. This gap is not merely a technical limitation; it reflects a deeper mismatch between statistical prediction and meaningful comprehension. Bridging it requires more than incremental improvements. It demands a rethinking of how AI systems are built, how humans interact with them, and how knowledge itself is represented in computational systems.

The first step toward narrowing the gap is grounding AI in the real world. Current models operate almost entirely in the domain of text, learning patterns from language without direct access to physical experience. This creates a form of 'disembodied intelligence' that can describe reality but cannot verify it. Integrating AI with sensory data - vision, sound, spatial awareness, and even embodied robotics - can provide the grounding that language alone cannot. When an AI system can connect words to objects, events, and interactions, its answers become anchored in something more than statistical likelihood. Grounding does not give AI human understanding, but it moves the system closer to a world-model rather than a word-model.

A second pathway involves explicit reasoning mechanisms. Today’s AI excels at pattern completion but struggles with logic, causality, and multi-step inference. Hybrid architectures that combine neural networks with symbolic reasoning, constraint solvers, or causal models can help bridge this divide. These systems allow AI to not only generate answers but also justify them, trace their logic, and detect contradictions. When an AI can explain why it reached a conclusion, the gap between output and understanding begins to narrow. Reasoning does not guarantee correctness, but it introduces structure, consistency, and transparency - qualities essential to real knowledge.

Another crucial element is epistemic humility. Humans know when they do not know; AI does not. One of the most dangerous aspects of current systems is their tendency to produce confident answers even when they are improvising. Bridging the gap requires AI to model uncertainty explicitly. Techniques such as probabilistic calibration, confidence scoring, and retrieval‑based fallback mechanisms can help systems signal when they are unsure. An AI that can say 'I don’t know' or 'I need more information' behaves more like a knowledgeable agent and less like a fluent guesser. Humility is not a weakness; it is a form of intellectual honesty.

Equally important is human‑AI collaboration. The gap between AI’s answers and real knowledge shrinks when humans remain in the loop - not as passive consumers of AI output but as active partners. When experts guide, correct, and contextualize AI responses, the system becomes part of a larger cognitive ecosystem. Tools that allow users to inspect sources, challenge assumptions, and refine prompts transform AI from an oracle into a collaborator. Knowledge emerges not from the model alone but from the interaction between human judgment and machine synthesis.

Finally, bridging the gap requires rethinking how AI is trained. Models trained on undifferentiated internet text inherit biases, errors, and superficial patterns. Curated datasets, domain‑specific corpora, and reinforcement learning from expert feedback can push AI toward deeper, more reliable forms of knowledge. The goal is not to eliminate uncertainty but to align AI’s learning process with the structures of real expertise.

The gap between AI’s answers and real knowledge is significant, but it is not insurmountable. By grounding AI in the world, enhancing its reasoning, cultivating uncertainty awareness, fostering human collaboration, and improving training methods, we can move toward systems that do more than imitate understanding. We can build systems that support, extend, and enrich human knowledge rather than merely simulating it.

Disclaimer: The whole text was generated by Copilot (under Windows 11) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🧭Business Intelligence: Perspectives (Part 31: More on Data Visualization)

Business Intelligence Series

There are many reasons why the data visualizations available in the different mediums can be considerate as having poor quality and unfortunately there is often more than one issue that can be corroborated with this - the complexity of the data or of the models behind them, the lack of identifying the right data, respectively aspects that should be visualized, poor data visualization software or the lack of skills to use its capabilities, improper choice of visual displays, misleading choice of scales, axes and other elements, the lack of clear outlines for telling a story respectively of pushing a story too far, not adapting visualizations to changing requirements or different perspectives, to name just the most important causes.

The complexity of the data increases with the dimensions associated typically with what we call currently big data - velocity, volume, value, variety, veracity, variability and whatever V might be in scope. If it's relatively easy to work with a small dataset, understanding its shapes and challenges, our understanding power decreases with the Vs added into the picture. Of course, we can always treat the data alike, though the broader the timeframe, the higher the chances are for the data to have important changing characteristics that can impact the outcomes. It can be simple definition changes or more importantly, the model itself. Data, processes and perspectives change fluidly with the many requirements, and quite often the further implications for reporting, visualizations and other aspects are not considered.

Quite often there's a gap between what one wants to achieve with a data visualization and the data or knowledge available. It might be a matter of missing values or whole attributes that would help to delimit clearly the different perspectives or of modelling adequately the processes behind. It can be the intrinsic data quality issues that can be challenging to correct after the fact. It can also be our understanding about the processes themselves as reflected in the data, or more important, on what's missing to provide better perspectives. Therefore, many are forced to work with what they have or what they know.

Many of the data visualizations inadvertently reflect their creators' understanding about the data, procedures, processes, and any other aspects related to them. Unfortunately, also business users or other participants have only limited views and thus their knowledge must be elicited accordingly. Even then, it might be pieces of data that are not reflected in any knowledge available.

If one tortures enough data, one or more stories worthy of telling can probably be identified. However, much of the data is dull to the degree that some creators feel forced to add elements. Earlier, one could have blamed the software for it, though modern software provides nice graphics and plenty of features that can help graphics creators in the process. Even data with high quality can reveal some challenges difficult to overcome. One needs to compromise and there can be compromises in many places to the degree that one can but wonder whether the end result still reflects reality. Unfortunately, it's difficult to evaluate the impact of such gaps, however progress can be made occasionally by continuously evaluating the gaps and finding the appropriate methods to address them.

Not all stories must have complex visualizations in which multiple variables are used to provide the many perspectives. Some simple visualizations can be enough for establishing common ground on which something more complex (or simple) can be built upon. Data visualization is a continuous process of exploration, extrapolation, evaluation, testing assumptions and ideas, where one's experience can be a useful mediator between the various forces. 

Previous Post <<||>> Next Post

🧮ERP: Implementations (Part XVI: It’s All About Politics)

ERP Implementations Series

An ERP implementation takes place within a political context and politics can make or break implementations. Politics occurs whenever individuals or organization groups interact to make decisions that affect parts or the whole organization. Besides decision-making there are further components that revolve around the various types of resources allocation and management, resulting in power dynamics that shape and pull organizations in politically charged directions.

Given the deep implications of ERP systems, probably in no other type of projects the political aspects are that visible and stringent to all employees to the degree that they pull decisions in one direction independently of the actual requirements. It may seem incredible, though there are cases in which ERP systems were selected just because the organization’s CEO played golf with the vendor’s CEO. In the end, the gaps between systems should be minimal nowadays, at least in theory, isn’t it?

Of course, just because one meets certain strange behaviors, it doesn’t mean that this is common practice! There are higher chances of selecting an inadequate system just because the sales representative did a good job and convinced the audience that the system can do anything they want. It probably does if coins are used for each missing feature, and in the long term it can be a lot of coins. Conversely, even if a system satisfies nowadays’ requirements, it doesn’t mean it will continue to do the same with future requirements. Only the future can tell whether the choice of a system over the others was a good one.

The bigger the gaps between the various interests, the more difficult it becomes to pull the project in the right direction. Probably the best way to demonstrate why one system is better than another is by bringing facts and focusing on the main requirements of the organization. This supposes the existence of an explicit list of requirements with a high-level description of how they can be addressed by the future system. This might not be enough, though it’s a good start, a good basis for discussion, for making people aware of the implications. However, doing this exercise for 2-3 or more systems is not cost effective, as such analysis can become time-consuming and expensive.

One way to address political resistance is by discussing openly with the stakeholders and addressing their concerns, arguing why the system is a good choice, what can be done to address the gaps, and so on. It will not always be enough, though it’s important to establish common ground for further discussions. Further on, it’s important to keep the same openness and disposition for communication given that the further the project progresses, the higher the likelihood of other concerns to appear. It’s a never-ending story if there are gaps between needs and what the system provides.

It's important to establish clear and honest communication with the stakeholders, informing them proactively about the challenges faced, independently in which area they are faced. Conversely, too much communication can be disruptive and can create other challenges. One way to cope with this is by identifying the communication needs of each stakeholder and trying to identify what’s the volume of information, respectively the communication needs of each of them. That’s project management 1:1.

The Project Manager and his team should ideally anticipate and address the potential conflicts timely, before they propagate and reach a broader audience. It’s questionable how much can be achieved proactively, especially when the project keeps everybody busy. The tendency is to answer politics with politics, though brainstorming sessions, open communication and a few other approaches can reach deeper where politics can’t.

Previous Post <<||>> Next Post

🧭Business Intelligence: Perspectives (Part 29: Navigating into the Unknown)

Business Intelligence Series

One of the important challenges in Business Intelligence and the other related knowledge domains is that people try to oversell ideas, overstretching, shifting, mixing and bending the definition of concepts and their use to suit the sales pitch or other related purposes. Even if there are several methodologies built around data that attempt to provide a solid foundation on which organizations can build upon, terms like actionable, value, insight, quality or importance continue to be a matter of perception, interpretation, and quite often be misused. 

It's often challenging to define precisely such businesses concepts especially there are degrees of fuzziness that may apply to the different contexts that are associated with them. What makes a piece of signal, data, information or knowledge valuable, respectively actionable? What is the value, respectively values we associate with a piece or aggregation of information, insight or degree of quality? When do values, changes, variations and other aspects become important, respectively can be ignored? How much can one generalize or particularize certain aspects? And, many more such questions can be added to this line of inquiry. 

Just because an important value changed, no matter in what direction, it might mean nothing as long as the value moves in certain ranges, respectively other direct or indirect conditions are met or not. Sometimes, there are simple rules and models that can be used to identify the various areas that should trigger different responses, respectively actions, though even small variations can increase the overall complexity multifold. There seems to be certain comfort in numbers, even if the same numbers can mean different things to different people, at different points in time, respectively contexts.

In the pursuit to bridge the multitude of gaps and challenges, organization attempt to arrive at common definitions and understanding in what concerns the business terms, goals, objectives, metrics, rules, procedures, processes and other points of focus associated with the respective terms. Unfortunately, many such foundations barely support the edifices built as long as there’s no common mental models established!

Even if the use of shared models is not new, few organizations try to make the knowledge associated with them explicit, respectively agree on and evolve a set of mental models that reflect how the business works, what is important, respectively can be ignored, which are the dependent and independent aspects, etc. This effort can prove to be a challenge for many organizations, especially when several leaps of faith must be made in the process.

Independently on whether organizations use shared mental models, some kind of common ground must be achieved. It starts with open dialog, identifying the gaps, respectively the minimum volume of knowledge required for making progress in the right direction(s). The broader the gaps and the misalignment, the more iterations are needed to make progress! And, of course, one must know which are the destinations, what paths to follow, what to ignore, etc. 

It's important how we look at the business, and people tend to use different filters (aka glasses or hats) for this purpose. Simple relationships between the various facts are ideal, though uncommon. There’s a chain of causality that may trigger a certain change, though more likely one deals with a networked structure of cause-effect relationships. The world is more complex than we (can} imagine. We try to focus on the aspects we are aware of, respectively consider as important. However, in a complex world also small variations in certain areas can shift the overall weight to aspects outside of our focus, influence or area of responsibility. Quite often, what we don’t know is more important than what we know!

Previous Post <<||>> Next Post

📦Data Migrations (DM): Conceptualization (Part VII: Data Import Layer)

Data Migrations Series

The data requirements for the Data Migration (DM) and Data Quality (DQ) are driven by the processes implemented in the target system(s). Therefore, a good knowledge of these requirements can decrease the effort needed for these two subprojects considerably. The needed knowledge basis starts with the entities and their attributes, the dependencies existing between them and the various rules that apply, and ends with the parametrization requirements, respectively the architecture(s) that can be used to import the data.

The DM process starts with defining the entities in scope and their attributes, respectively identifying the corresponding entities and attributes from the legacy systems. The attributes not having a correspondent in the legacy system need to be provided by the business and integrated in the DM logic. In addition, it’s needed to consider also the attributes needed by the business and not available in the target system, some of them more likely available in the legacy systems. For such attributes is needed either to misuse an attribute from the target or to extend the target system.

For each entity is created a data mapping that basically documents the data transformations needed for migrating the data. In the process is needed to consider also attributes’ data types, the (standard) formatting, their domain of definition, as well the various rules that apply. Their implementation belongs into the DM layer from which the data are exported in a standard format as needed by the target system.

Exporting the data from the DM layer directly into the target system’s tables has in theory the lowest overhead even if the rejected records are difficult to track, the rejections resulting only from records’ ‘validation against database’s schema. For this approach to work, one must have a good knowledge of the database schema and of the business rules implemented into the target system.

To solve the issue with errors’ logging, systems have a further layer on top of the database model, which also allow running data validation against target system’s business rules. Modern import frameworks allow loading the data via a set of standard files with a predefined structure. The data can be thus imported manually or via load jobs into the system a log with the issues being generated in the process. Some frameworks allow even the manual editing of failed records, respectively to import the data. Unfortunately, calling the layer from the DM layer is not possible from a database, though this would bring seldom a benefit. Some third-party tools attempt to improve the import functionality by calling the target system’s import layer.

The import files must be generated from the DM layer in the required structure with the appropriate formatting. The challenge however resides in identifying all the attributes that should make scope of the load. It’s an iterative process which sometimes is backed by try-and-error heuristics. Unless target system’s validation rules are known beforehand, the rules need to be discovered in this process, which can prove time-consuming. The discoveries need to be integrated also in the DM and from here results the big number of changes that need to be performed.

Given the dependencies existing between entities the files need to be generated and loaded in a predefined order. These dependencies are reflected also in the data processing and the validation rules considered in the DM layer.

A quality checkpoint can be implemented between the export from the DM layer and import to enforce the four-eyes principle. It’s normally the last opportunity for trapping the eventual issues. A further quality check is performed after import by validating on whether the data were imported as expected.

Previous Post <<||>> Next Post

🧮ERP: Implementations (Part V: It’s all about Partnership - An Introduction)

ERP Implementations Series

Unless the organization (customer) implementing an ERP system has a strong IT team and the knowledge required for the implementation is available already in-house, the resources need to be acquired from the market, and probably the right thing to do is to identify a certified implementer (partner) which can fill the knowledge and skillset gaps, respectively which can help splitting the risks associated with such an implementation.

In theory, the customer provides knowledge about its processes, while the partner comes with expertise about the system to be implemented and further technologies, industry best practices, project methodologies, etc. Further on, the mix is leveraged to harness the knowledge and reach project’s objectives. 

In praxis however finding an implementer which can act as partner might be more challenging than expected. This because the implementer needs to understand customer’s business and where it’s heading, bridge the gap between functional requirements and system’s functionality, advise on areas of improvement, prepare the customer for the project and lead the customer through the changes, respectively establish a basis for the future. Some of the implications are seldom made explicit even if they are implied by what is needed by the project. 

Technology is seldom the issue in an ERP implementation, the challenges residing in handing the change and the logistics required. There are so many aspects to be considered and handled, and this can be challenging for any implementer no matter how long has been on the market or how experienced the resources are. Somebody needs to lead the change and the customer seldom has the knowledge to handle the change. In some cases, the implementer must make the customer aware of the implications, while in others needs to take the initiative and lead the change, though the customer needs to play along, which can be challenging also. 

Many aspects need to be handled at management level from a strategical point of view on customer’s side. It starts with assuring that the most important aspects of the business where considered, that the goals and objectives are clear, that the proper environment is created, and ends with the timely decision-making, with assuring that the resources are available when needed, that the needed organization structures and roles are in place, that the required knowledge is available before, during and after implementation, that the potential brought by the ERP system is harnessed for the years to come. 

A partnership allows in theory splitting the implementation risks as ERP implementations have a high rate of failure. Quite often the outcomes of such projects don’t meet the expectations, the systems being in extremis unusable or a bottleneck for the organization. Ideally one should work with the partner(s) and attempt solving the issues, split eventually the incurred cost overruns, find a middle way. Most of the times it’s recommended to find a solution together rather than coming to a litigation. 

Given the complex dependencies existing between the various parts of the project, the causes that lead to poor implementations are difficult to prove, as there are almost always grey areas. Moreover, the litigations can require a considerable time and resources to settle. These can be just extreme situations, and as long one has a good partner, there’s no need to think that far. On the other side, even if undesirable, one must be prepared also for such outcomes, even if the countermeasures may involve an additional effort. Therefore, one must address such issues in contracts by establishing the areas of accountability/responsibilities for each party, document adequately the requirements and further (important) communication, make sure that the deliverables have the expected quality, etc.

Previous Post <<||>> Next Post

♜Strategic Management: Gap Analysis (Definitions)

"In the managerial planning process, this is the analysis taken following an exercise to determine what improvements in the process are required." (Robert McCrie, "Security Operations Management 2nd Ed.", 2006)

"An assessment of a system in comparison with another system or a set of requirements, listing those items that are not common between them." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"A technique to evaluate the current portfolio mix of components and determine changes needed so components may be added, changed, or terminated to rebalance the portfolio." (Project Management Institute, "The Standard for Portfolio Management" 3rd Ed., 2012)

"Describes the difference between current results and consequences and desired results and consequences." (Joan C Dessinger, "Fundamentals of Performance Improvement 3rd Ed", 2012)

"A formal analysis of the differences between what the policy or regulation requires and what’s actually being done in the organization. Used to generate a list of action items required to become compliant with the policy or regulation." (Mark Rhodes-Ousley, "Information Security: The Complete Reference" 2nd Ed., 2013)

"A comparison between the actual outcome and the desired outcome." (Weiss, "Auditing IT Infrastructures for Compliance" 2nd Ed., 2015)