VANKPA
The useful takeaway
Improve knowledge ownership and retrieval before expecting a chatbot to produce trustworthy answers.
A knowledge assistant can make approved information easier to find and use. It can also amplify confusion if the underlying documents are outdated, contradictory, or available to the wrong people. The first project question should be what information the assistant may use and what a trustworthy answer must contain.
Choose a specific audience and task. Helping employees find a current operating procedure is different from answering customer questions or searching confidential project records. Each scenario needs its own permissions, source rules, evaluation cases, and response when the evidence is incomplete.
Understand retrieval
IBM describes retrieval augmented generation as connecting a generative model with external knowledge to provide relevant context. It can support source-linked answers and reduce some errors, but it does not make a model error-proof. IBM research The quality of retrieval and the interpretation of the retrieved material still require evaluation.
Our recommendation is to begin with a small, well-maintained collection. Assign an owner, establish which version is authoritative, and remove obsolete duplicates from the assistant's eligible sources. If the business cannot agree which policy is current, the interface should expose that uncertainty rather than conceal it behind a confident answer.
VanKpa knowledge-assistant pattern
Ground the answer
- Authorize
Resolve the user and allowed information sources.
- Retrieve
Find relevant, current passages within that boundary.
- Respond
Ground the draft in evidence and expose sources.
- Review
Handle uncertainty, corrections, and escalation.
Enforce access when information is retrieved
The assistant should only retrieve information the current user is authorized to access. Apply boundaries to the underlying records, including attachments and derived indexes. A search result, citation, or preview can disclose information even when the final answer avoids quoting it.
Test the same question under different user roles and accounts. Check what happens after access is revoked or a document is removed. Include exports, conversation history, and retained logs in the design. Do not assume that adding a login screen resolves the information-access problem.
Test realistic questions
Build a reviewed question set covering routine tasks, ambiguous wording, missing facts, contradictory sources, and requests outside the intended scope. Define what a good answer should include and when it should decline to infer an answer. Evaluate whether the cited source actually supports the statement, not merely whether a citation is present.
NIST's AI Risk Management Framework provides voluntary guidance for examining AI risks over the system lifecycle. NIST research Our practical application is to maintain an evaluation record that connects each important risk to a test, an owner, and an operating response. This is a working control, not a claim of certification.
Design a useful no-answer state
When the assistant lacks enough evidence, it should explain what is missing and point to the next appropriate action. That could mean refining the question, opening a source document, or asking a responsible person. The goal is to advance the task without inventing information.
After launch, use unresolved questions to improve the knowledge collection and the product. Measure verified answer usefulness, source coverage, correction effort, and escalation—not only conversation volume. An assistant that honestly identifies a knowledge gap can be more valuable than one that always returns an answer.
- Start with a bounded task and an approved source collection.
- Identify the owner and authoritative version of each source.
- Test retrieval and citations under real permission boundaries.
- Include missing and contradictory information in evaluations.
- Give unanswered questions an operational destination.
Is documentation still needed?
It should make documentation easier to use, not remove the need for it. People still need authoritative records, clear ownership, and a way to inspect the material behind important decisions. The assistant is an access layer within that system.
Evidence behind the guidance
Sources & context
Published research informs this article. VanKpa's frameworks and recommendations are practical applications; illustrative data is labeled where used.
- IBM — Retrieval augmented generation ↗Accessed September 11, 2026
RAG connects generation with external knowledge; it does not eliminate model errors.
- NIST — AI Risk Management Framework ↗Accessed September 11, 2026
Voluntary risk-management resource; not a certification or legal-compliance determination.
What could this change?
Bring the question, the current workflow, and the result you want to improve. We can help define a useful next step.
A worked scenario
Consider an organization testing an internal document assistant. The useful outcome is to retrieve approved evidence before drafting an answer. This is a planning example, not a reported client result. The team needs a decision that can be checked against real work, rather than a feature list that looks complete during a presentation. The starting question is whether the proposed approach changes that particular task in a way the people doing it can recognize.
In this situation, a citation existing but failing to support the answer is the failure to guard against. Ask the responsible person to demonstrate an ordinary case and one difficult case using current records or safe test data. Record what they expect to happen, what actually happens, and where they need another person to intervene. Those observations establish the scope for this example; they do not justify an assumed improvement percentage or a guaranteed business result.
Decision checkpoints
| Checkpoint | Practical action | Evidence to retain |
|---|---|---|
| Prepare | Organize sources with ownership and access boundaries. | The approved scope, relevant source records, and unresolved questions. |
| Verify | Test relevant and deliberately unanswerable questions. | The test case, expected result, observed result, and correction needed. |
| Operate | Show the source passage used for each material claim. | The responsible owner, completion record, and next review trigger. |
Use these checkpoints to retrieve approved evidence before drafting an answer; they are a sequence of decisions, not a promise of a particular schedule. A completed document or screen is not enough if the underlying action still fails. Keep unresolved items visible and describe which ones prevent progression. The evidence can be a small test record, an approved mapping, or a reviewed example. It should be understandable to someone who was not present when the work happened.
Measure the useful result
A useful check for this topic is supported sampled claims divided by sampled material claims. The numerator is supported sampled claims; the denominator is sampled material claims. Define the sampling window, exclusions, and source of each count before interpreting the result. If only selected examples can be reviewed, describe them as a sample. Do not present a small reviewed group as a complete picture of the business, and do not assign a target simply because a round number looks persuasive.
The measure helps reveal whether the team can retrieve approved evidence before drafting an answer, but it does not explain every cause of success or failure. Inspect the underlying cases alongside the summary. If the count changes after test relevant and deliberately unanswerable questions, check whether the operating result changed or the counting method changed. Retain enough context to explain the difference. When records are incomplete, state the limitation and use a direct task review instead of manufacturing a precise-looking estimate.