This is the pass I run to catch what a knowledge base gets wrong before a customer does. Run it on a set cadence, monthly is a sensible default, and treat the first pass as read-only: diagnose everything before you change a thing. That way you meet the stale corners on your own schedule instead of in front of someone who's paying you.
Which context failure are you seeing?
When the assistant gets something wrong because of what it read, it's almost always one of these four. Name the one you're seeing, because each has a different fix.
- Poisoning. A wrong fact really sits in the base, and the assistant repeats it, confidently. The fix is verification: cross-check the claims that matter against a live source before it commits, and keep a person on anything it isn't sure of.
- Bloat. Nothing is wrong; there's just too much, and the one fact that counts is buried under thousands that don't. Answers get vaguer and slower. The fix is to segment: split distinct, growing areas into their own stores, and keep only what earns its place.
- Confusion. The fact is missing or irrelevant, so instead of saying "I don't have that," the assistant invents something to cover the hole. This is the classic hallucination. The fix is to fill the real gap, or let it say "I don't know" instead.
- Clash. Two sources disagree. An old policy and its replacement both still live in the base, and the assistant can't tell which to trust. The fix is freshness: track which source is current and retire the one that isn't.
What to keep always vs. what to fetch on demand
Most bloat and clash start with one decision: what loads every single time versus what gets pulled in only when a question calls for it. Sort your own material into two piles.
Standing knowledge (always at hand):
- Who you are and what the business does.
- Your policies and how you want work done.
- Your goals for the quarter.
- The past decisions you keep referring back to.
Situational knowledge (fetched when needed):
- Yesterday's support ticket.
- One customer's record.
- A single meeting note from last week.
- The details of one order or one project.
Before any fact earns a permanent place, ask it one question: is this true all the time, or only right now? Standing knowledge stays. Situational knowledge gets pulled in the moment you need it and let go again.
The audit checklist (read-only)
Tick a box only if it's genuinely true today. Don't fix anything yet; just mark what fails, then read the failures before you touch a thing. The line to hunt for is the one that would hand someone a wrong answer today.
- [ ] Map and routing integrity. Does everything the top file points to still exist? A "no" means the assistant is being sent to files and folders that were moved or deleted, and it guesses to cover the gap.
- [ ] Index truth. Do your lists and indexes match what's actually on disk? A "no" means an index insists you hold one set of things while the real state holds another, and the assistant trusts the index.
- [ ] Source freshness. Is each source fresh, drifting, or frozen? A "no", a feed that quietly went stale, means today's questions get answered with last quarter's facts, stated with full confidence.
- [ ] Duplicates and clashes. Is the same fact sitting in two places? A "no" means that the moment one copy changes, the two disagree and the assistant can't tell which to follow.
- [ ] Context placement. Is any situational data sitting in the base permanently? A "no" means yesterday's one-off facts are adding volume, going stale, and waiting to contradict something newer.
Maintenance rhythm
An audit tells you what's already wrong. Three habits keep things from going wrong so often to begin with.
- Automate the updates. Some knowledge arrives on a rhythm: a weekly meeting recap, a recurring report. Paste it by hand and you'll forget after the one meeting that mattered. Set the recurring inputs to flow in on a schedule. Wire it up once, then let it run.
- Segment as it grows. When one area is distinct and keeps expanding, everything for a single client, say, split it into its own store rather than letting it swell the main pile. The payoff is a narrower search. Do it whenever a section outgrows the rest.
- Backtrack to the root cause. When the assistant gets something wrong, don't scold it. Have it retrace its own steps: where did it look, what did it search, why did it miss the thing sitting right there? Then fix the cause, not the symptom: correct the map, move the misfiled item. Do this every time it's wrong, before you move on.
A knowledge base is worth only what you can trust in it: a big one you can't rely on is worse than a small one you can, because it hands you wrong answers with the same confidence as the right ones. The discipline was never building more; it was keeping what you have true.