- Why search can bring back incompatible instructions
- Consider an imaginary 30-day versus 90-day dispute
- What the retrieval system must know
- 1. The source’s status
- 2. Scope, effective dates and version lineage
- 3. Permission to see each source
- 4. The rule for resolving conflicts
- What a trustworthy answer sounds like
- How to test a policy-aware RAG system
- Sources and further reading
Ask an AI assistant about a company policy and it may find two documents giving two different answers. One says an approval lasts 30 days; another says 90. Both passages look relevant. Both may carry official-looking titles. The critical question is not which passage sounds more confident. It is which document has authority for this user, action and date.
Retrieval-augmented generation (RAG) can locate passages that match a question. It does not, by itself, make an assistant a policy administrator. Without trustworthy document metadata and explicit conflict rules, a fluent answer may conceal a genuine contradiction.
Why search can bring back incompatible instructions
Two policies may refer to different business units, countries, employee groups or time periods. A former policy may remain indexed after it was superseded. A draft may be more detailed than the approved version. And a training handbook can paraphrase a rule that is formally controlled elsewhere.
Search engines rank relevance. Microsoft’s Azure AI Search documentation explains how semantic reranking considers whether a result answers the query. That is useful for finding evidence, but a high relevance score is not proof of legal or organizational precedence.
Consider an imaginary 30-day versus 90-day dispute
A fictional company has an older security handbook requiring a 30-day review interval. A newly approved department-specific standard says 90 days for a narrow class of internal systems. The assistant retrieves both. Selecting the newer number is not automatically correct: perhaps the user’s system is outside that department, or the new standard applies only after a transition date.
Conversely, choosing the older handbook because it has more keyword matches can be wrong too. Document publication time, effective date and applicability are different fields. Even apparently simple instructions can require interpretation by the responsible policy owner.
What the retrieval system must know
1. The source’s status
Useful records identify a document as draft, approved, archived or superseded. Merely storing the filename “final-policy-v2.pdf” does not constitute reliable status metadata. Ideally, authority comes from a controlled document-management process, not from an LLM guessing from the text.
2. Scope, effective dates and version lineage
A single policy can differ by region, job role, system class or contractual terms. Retrieval filters can exclude irrelevant versions and scopes before generation. Microsoft’s freshness-aware retrieval guidance explicitly distinguishes a recency ranking boost from a hard filter. Newness can help rank a result but cannot replace a rule defining which results are eligible.
3. Permission to see each source
An assistant should not reveal a confidential policy merely because it ranks highly. Access control must apply at retrieval and answer construction, with relevant permissions maintained as sources change. A correct policy summary disclosed to the wrong user is still a failure.
4. The rule for resolving conflicts
A system might be configured to prefer a controlled master policy over a local guideline, or to defer to the most recent approved version within the same scope. Such rules require an accountable owner and explicit exceptions. Different organizations have different hierarchies. An assistant should not silently impose one.
What a trustworthy answer sounds like
Instead of a crisp but unsupported “The policy is 90 days,” the system might say: “I found two requirements. The approved department standard lists 90 days for certain internal systems; the older general handbook lists 30. I cannot verify which scope applies to your system. Here are both passages and the policy owner to consult.”
That response is less convenient but more useful than confident misinformation. It keeps source identity visible and separates evidence from interpretation. A well-designed assistant can also ask a narrow question about location, system class or effective date, then retrieve again with the correct scope filters.
How to test a policy-aware RAG system
- Feed it a superseded document and its replacement; check whether the old version remains eligible.
- Introduce two approved documents with different scopes; verify that the assistant asks for missing context.
- Check a document whose effective date is in the future, not merely whose upload date is recent.
- Remove a user’s access to one source and confirm that neither the citation nor the answer leaks it.
- Require citations to the exact passages actually used, with version and authority metadata.
For background, IL.DIGITAL has examined why deleted files can survive in search indexes and how AI assistants obtain access to connected files. Both problems can change which sources a policy assistant sees.
The lesson is not to trust the newest document or the top search result. A useful answer needs a source that is relevant, current, authorized and governing for the particular case. If that cannot be established, uncertainty belongs in the answer—not hidden behind it.







