AI Search Found Two Conflicting Policies. Which One Does It Follow?

Two unbranded policy binders lie open beside each other as a researcher compares documents, illustrating conflicting AI search results. RAG
Two policy documents can both look relevant to an AI assistant yet disagree. Learn how document authority, effective dates, user access and conflict handling determine a trustworthy answer.

Ask an AI assistant about a company policy and it may find two documents giving two different answers. One says an approval lasts 30 days; another says 90. Both passages look relevant. Both may carry official-looking titles. The critical question is not which passage sounds more confident. It is which document has authority for this user, action and date.

Retrieval-augmented generation (RAG) can locate passages that match a question. It does not, by itself, make an assistant a policy administrator. Without trustworthy document metadata and explicit conflict rules, a fluent answer may conceal a genuine contradiction.

Why search can bring back incompatible instructions

Two policies may refer to different business units, countries, employee groups or time periods. A former policy may remain indexed after it was superseded. A draft may be more detailed than the approved version. And a training handbook can paraphrase a rule that is formally controlled elsewhere.

Search engines rank relevance. Microsoft’s Azure AI Search documentation explains how semantic reranking considers whether a result answers the query. That is useful for finding evidence, but a high relevance score is not proof of legal or organizational precedence.

When two good matches disagree
Conflicting documents A and B converge on a mandatory authority check, rather than automatically yielding one answer Policy APolicy B30 days • illustrative90 days • illustrative Authority checkversion • scope • approvaleffective date • permissions Answer or escalate
Invented policy values illustrate a common retrieval problem. They are not real corporate requirements.

Consider an imaginary 30-day versus 90-day dispute

A fictional company has an older security handbook requiring a 30-day review interval. A newly approved department-specific standard says 90 days for a narrow class of internal systems. The assistant retrieves both. Selecting the newer number is not automatically correct: perhaps the user’s system is outside that department, or the new standard applies only after a transition date.

Conversely, choosing the older handbook because it has more keyword matches can be wrong too. Document publication time, effective date and applicability are different fields. Even apparently simple instructions can require interpretation by the responsible policy owner.

The safe editorial test: If two retrieved sources conflict and the system cannot establish which governs, it should show the conflict, identify both sources and request authorized clarification. It should not invent a resolution.

What the retrieval system must know

1. The source’s status

Useful records identify a document as draft, approved, archived or superseded. Merely storing the filename “final-policy-v2.pdf” does not constitute reliable status metadata. Ideally, authority comes from a controlled document-management process, not from an LLM guessing from the text.

2. Scope, effective dates and version lineage

A single policy can differ by region, job role, system class or contractual terms. Retrieval filters can exclude irrelevant versions and scopes before generation. Microsoft’s freshness-aware retrieval guidance explicitly distinguishes a recency ranking boost from a hard filter. Newness can help rank a result but cannot replace a rule defining which results are eligible.

3. Permission to see each source

An assistant should not reveal a confidential policy merely because it ranks highly. Access control must apply at retrieval and answer construction, with relevant permissions maintained as sources change. A correct policy summary disclosed to the wrong user is still a failure.

4. The rule for resolving conflicts

A system might be configured to prefer a controlled master policy over a local guideline, or to defer to the most recent approved version within the same scope. Such rules require an accountable owner and explicit exceptions. Different organizations have different hierarchies. An assistant should not silently impose one.

What a trustworthy answer sounds like

Instead of a crisp but unsupported “The policy is 90 days,” the system might say: “I found two requirements. The approved department standard lists 90 days for certain internal systems; the older general handbook lists 30. I cannot verify which scope applies to your system. Here are both passages and the policy owner to consult.”

That response is less convenient but more useful than confident misinformation. It keeps source identity visible and separates evidence from interpretation. A well-designed assistant can also ask a narrow question about location, system class or effective date, then retrieve again with the correct scope filters.

How to test a policy-aware RAG system

  • Feed it a superseded document and its replacement; check whether the old version remains eligible.
  • Introduce two approved documents with different scopes; verify that the assistant asks for missing context.
  • Check a document whose effective date is in the future, not merely whose upload date is recent.
  • Remove a user’s access to one source and confirm that neither the citation nor the answer leaks it.
  • Require citations to the exact passages actually used, with version and authority metadata.

For background, IL.DIGITAL has examined why deleted files can survive in search indexes and how AI assistants obtain access to connected files. Both problems can change which sources a policy assistant sees.

The lesson is not to trust the newest document or the top search result. A useful answer needs a source that is relevant, current, authorized and governing for the particular case. If that cannot be established, uncertainty belongs in the answer—not hidden behind it.

Sources and further reading

Rate article
Add a comment