Enterprise Search and Website Answer Agents

When a Retrieval Project Is the Wrong Answer

Not every finding problem is a retrieval problem. If the answer people want lives in one system, is current, and is two or three clicks from where they start, the fix is information architecture: better titles, better navigation, and a synonym list in the search box you already own. Putting an index and an answer layer on top of a small, well organized body of content adds moving parts without changing what people experience.

The harder case to admit is that the answer does not exist anywhere. Retrieval returns what has been written down. It cannot produce a policy nobody wrote or a procedure that lives in one person's head, and no amount of engineering changes that. When we look at what people are failing to find, some share of it is undocumented rather than unfindable, and that share is a writing assignment for whoever owns the subject, not a project for us.

There is a cheap way to tell the difference before committing to anything. Take the questions people actually asked, in their own words, and mark each one with where its answer lives.

• Answer sits in one system, and the asker had access the whole time: this is a navigation and labeling problem.
• Answer requires reading two or three documents together, or the sources are spread across systems with different search boxes: this is what retrieval is for.
• Answer does not exist in any system: nothing to retrieve, and building an index will only tell you that more expensively.
• Answer exists in several conflicting versions: settle which one is authoritative first, because retrieval will happily return the wrong one.

What the Source Systems Have To Provide

Every connector needs three things from the system it reads: a way to enumerate what is there, a way to fetch each item as text, and a way to learn who is allowed to see it. The first two are usually straightforward. The third decides the shape of the whole project.

Permission-aware retrieval depends on the source answering the entitlement question per item, and on an identity system that can report a person's group membership at the moment they ask, not at the moment the content was crawled. Membership baked into the index at crawl time goes wrong the day somebody changes teams, and it goes wrong silently. Some sources cannot answer the entitlement question at all: a shared drive whose access has been inherited and hand edited for years, a mailbox archive with no per item access control, an old document store with permissions only at the container level. Where that is the case, such a source is either restricted to an audience already entitled to all of it, or left out. That call gets made per source, in writing, during scoping rather than during rollout.

Before a source is connected, these are the things that have to be settled about it:

• A named owner who can approve access and say what is authoritative when two documents in that source disagree.
• A read scoped service account, so the connector cannot write even by accident.
• Whether entitlements are readable per item or only per container.
• Whether the content carries a text layer at all, or is scanned images and screenshots that would need OCR before any of it is reachable.
• Whether the source is live content or an archive that should be excluded so superseded material stops competing with current material.

Judging Whether It Works

Fluent output is not evidence of anything. The only measure that matters is whether the cited source actually supports the sentence it is attached to, which means evaluation is a reading exercise, not a demo. Build a fixed set of real questions with their known correct sources, and score answers against that set. Rerun it whenever content, retrieval settings, or the model changes, because a change that improves one family of questions routinely degrades another, and that is invisible without a fixed set to compare against.

Separate the two failure modes early, since they look identical to a user and have nothing in common underneath. In the first, the right document was never retrieved: the fix lives in chunking, indexing, and how the question is expanded before search. In the second, the right document was retrieved and the answer misstates it: the fix lives in how much context is passed, how the answer is assembled, and how output is constrained. Treating both as one problem is the most common way effort gets spent without moving the result.

Grounding failures also come in a version that no retrieval tuning can fix: the source itself is wrong, contradicts another source, or was superseded and never withdrawn. The system will cite it faithfully. Which is why the audit trail is useful beyond compliance: a wrong cited answer points straight at a document somebody needs to correct.

Where These Projects Break

The failures are almost always content failures wearing a technical costume. Retrieval quality is bounded by the state of what it reads, so the work that determines the outcome tends to be unglamorous and belongs to the customer as much as to us.

Near duplicates are the worst of them. Several versions of the same policy sit in different systems, most of them superseded, and retrieval returns whichever one most resembles the question rather than whichever one is current. No ranking change fixes this. Someone has to mark the authoritative copy or remove the rest.

Vocabulary is the second. People search using project code names, team acronyms, and internal shorthand that appear nowhere in the documents themselves, so a semantically correct search returns nothing. This is solvable with a maintained mapping between how people speak and how documents are written, but it is a living list with an owner, not a one time configuration.

The last one is procedural. Unanswered-question output only has value if a specific person is expected to act on it. Assign that to a role during scoping, or it becomes a report that accumulates and is never opened, and the documentation gaps it identified stay exactly where they were.