- Seven days is still in the system somewhere
- The document changed. The index did not.
- The file has to travel through another system
- A correct citation can still point to stale evidence
- What the AI actually read
- New research reveals an even stranger failure
- One old answer, several possible causes
- The most revealing test happens before generation
- Reindexing fixes only the part it actually replaces
- A date is useful. A validity boundary is better.
- The corrected answer needs its own proof
- Sources
A company changes an important rule in an internal document. The corrected file says requests can take 14 days to process. Someone asks the company’s AI assistant about the rule, and it confidently answers seven days—the previous limit.
The document is correct. The answer is outdated. The assistant may even cite the correct document by name.
One explanation is that the AI is searching an older, indexed version of the file. In retrieval-augmented generation (RAG), saving a document and updating the information available to search are separate operations. The original file can change before the system has processed and replaced its searchable passages.
There are other possibilities too. Both versions might be searchable, but the old one ranks higher. Or the search system may have the current text while the application reuses an earlier conversation or cached answer.
To understand the error, the key question is what information actually reached the model when it answered.
Seven days is still in the system somewhere
Consider a hypothetical company policy called Request Handling Procedure. Its original version, V1, allowed seven days to handle a request.
An employee edits the file, changing that requirement to 14 days, and saves the replacement as V2. Anyone opening the authoritative document now sees the new rule.
But the AI assistant was built around a searchable knowledge base. Its application previously extracted the document’s text and stored smaller passages in a search index.
Those passages were created when V1 was still correct.
If the next ingestion run has not happened, has failed or has not recognized the modification, the search index may continue returning the old requirement. The model then answers using information that no longer matches the original file.
This differs from the problem of an AI assistant finding a deleted document. Here, the source still exists and its contents have changed. The failure concerns which version of that source is available at question time.
Version synchronization
The document changed. The index did not.
Two timelines describe different states of the same information. In this hypothetical example, the question arrives after the document has changed but before its search records have been refreshed.
Swipe horizontally to explore the full timeline.
The file has to travel through another system
Many RAG assistants do not read the original PDF, Word document or internal wiki page from scratch every time a question arrives. Instead, they prepare information for searching ahead of time.
A typical pipeline extracts readable text, divides it into passages and creates search records. For semantic retrieval, an embedding model converts passages into numerical vectors that help locate related information.
The vectors are not miniature copies of the document. Search records may also retain readable passages, document identifiers and metadata alongside those vectors.
When a question arrives, a retrieval system finds relevant records and supplies selected passages to the language model as context. The model then generates an answer using the information it received.
This is the basic mechanism described in our guide to building a private local RAG knowledge base. The important issue here is what happens after the source content changes.
What exactly is a document chunk?
A chunk is a portion of extracted document content stored for retrieval. A long manual might become dozens of chunks. If one paragraph changes, the ingestion system needs to ensure that the searchable content representing that paragraph is updated appropriately. Different systems handle chunk boundaries and replacement differently.
Microsoft’s Azure AI Search documentation explains that indexers can run on schedules or on demand. Incremental indexing depends on detecting changes in the underlying source. A regular run processes changes it identifies; a reset changes the tracking state and is followed by a run to perform the actual processing.
For supported Azure Storage sources, change detection can use modification timestamps. Other data sources require different mechanisms or configuration.
That is a concrete example of why saving a file and refreshing its searchable representation are distinct events. It does not mean all RAG products use Azure-style indexers, timestamps or the same synchronization schedule.
OpenAI’s Retrieval documentation, for example, describes vector-store files that undergo chunking and embedding. Some file-processing operations are asynchronous and expose status information that applications can check before relying on the newly indexed content.
The general engineering requirement is the same: the application needs a reliable way to turn a source change into the appropriate change in its search records.
Does changing the document require retraining the AI model?
Usually not in a RAG system. Updating the source and its searchable representation changes which external evidence can be passed to the model. Retraining or fine-tuning the model’s weights is a separate operation. A current search index can improve answers without changing the underlying language model.
A correct citation can still point to stale evidence
There is a subtle distinction between finding the right document and finding the current text within that document.
Suppose the assistant displays a citation labeled Request Handling Procedure. The title matches the file that was edited. The link may also lead to the current document when the reader opens it.
Neither fact proves which passage was used to generate the answer.
The retrieval result could have been created from V1 and still carry the same source title or stable document identifier. A document’s identity can remain unchanged even as its content evolves.
Now imagine that the application exposes a detailed retrieval trace, including source version metadata. The discrepancy becomes much easier to identify.
Evidence inspection
What the AI actually read
The document and the retrieved passage agree on the source identity, but disagree on the rule.
This distinction becomes especially important when the same document is updated repeatedly. Search records need to preserve the relationship between their content and the source version that produced them.
Without reliable version handling, a document title can look reassuring while the underlying evidence has already expired.
| Layer | Before edit | At question time | After verified refresh |
|---|---|---|---|
| Original source | V1 — 7 days | V2 — 14 days | V2 — 14 days |
| Searchable passage | V1 — 7 days | V1 — 7 days | V2 — 14 days |
| Evidence selected | Old passage | Old passage | Current passage, if retrieved |
| Generated answer | 7 days | 7 days | Should reflect 14 days; verify separately |
Notice that the final row does not promise a correct answer after reindexing. Even when the current passage becomes searchable, the retrieval system still has to select it, and the model still has to use it correctly.
New research reveals an even stranger failure
What if the model already has the correct answer, but outdated evidence makes it wrong?
A research preprint released on September 25, 2026, Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers, investigates precisely that problem.
Researchers Md Shamim Ahmed, Lukas Galke Poech and Richard Röttger assembled a benchmark of 317 verified changes to facts, guidance or rules across medicine, law, software and platform policy.
The researchers focused on cases where the same model answered correctly without retrieval, then examined what happened when it was supplied with genuine but outdated evidence.
In a controlled experiment with neutral document presentation, outdated evidence changed 30% of previously correct Llama answers and 37% of previously correct Qwen answers in the evaluated subsets. When models were explicitly instructed to follow the retrieved evidence, those rates rose to 66% and 75%, respectively.
Here is the important distinction: the research was not primarily a test of whether an ingestion pipeline failed to notice a document update. The experiment supplied outdated evidence to the model and studied how the model treated it.
That makes it a second failure mode layered on top of document synchronization. A stale index can put obsolete material in front of an AI assistant, and the model may then trust that material even when its answer without retrieval would have been correct.
The study also found that adding accurate information about when evidence stopped applying improved the ability of some larger models to distinguish historical from current answers. Dates alone were much less reliable.
This matters because the newest document is not automatically the most applicable one. A recently uploaded copy of an old policy is still an old policy. Reliable retrieval needs information about validity, not just upload time.
One old answer, several possible causes
A stale index is only one explanation for a persistent old answer. The actual failure depends on the system’s architecture and what its logs or retrieval tools reveal.
The following comparisons help separate failures that look almost identical from the user’s point of view.
Swipe horizontally to compare possible causes.
| Observed behavior | Possible layer | Evidence worth checking |
|---|---|---|
| The source is updated, but search still returns only V1. | Change detection or ingestion | Indexer runs, processing failures, source timestamps and indexed content. |
| Both V1 and V2 are searchable, but the old passage is selected. | Version management or ranking | Retrieved record IDs, version filters, relevance scores and superseded records. |
| Direct search returns V2, but the assistant repeats V1. | Application context or generation | The exact passages supplied to the model, prior conversation and instructions. |
| A fresh retrieval works, but repeated questions return an identical old response. | Possible response cache | Cache configuration, keys, invalidation behavior and request traces. |
| V2 is indexed, but the assistant uses a different old document. | Retrieval scope or conflicting sources | Top search results, source identifiers, effective dates and authority rules. |
None of these observations alone proves a particular cause. For example, an identical repeated answer is not sufficient evidence of caching; a deterministic model can also repeat itself.
They do, however, identify different places to investigate. They also explain why a full reindex or a larger language model is not necessarily the right first response.
The most revealing test happens before generation
If the assistant provides access to its retrieval results, compare those results directly with the current source.
Begin by confirming the authoritative document really contains the revised rule. Check the actual file content, not just its filename or modification timestamp.
Then run the relevant search query against the knowledge base. Inspect the passages and source metadata returned by the search system.
If the retrieved passage still says seven days, the problem is already visible before the model generates anything. Investigate the ingestion pipeline, indexed versions and retrieval filters.
If the retrieved passage says 14 days but the model answers seven, examine the final prompt context, earlier messages and any application logic that reuses previous results.
OpenAI’s File Search documentation gives one concrete example: search results are not included in a standard response by default, but applications using the Responses API can request them through include: ["file_search_call.results"].
That option is specific to the documented API. Other products expose retrieval evidence differently, and some user-facing assistants provide no detailed trace at all.
A fresh conversation can help distinguish persistent conversational context from retrieval problems, but it is not conclusive by itself. A newly opened chat can still search the same outdated index or use a shared application cache.
What does an embedding actually store?
An embedding is a numerical representation of text used to compare semantic similarity. It is not itself the source paragraph in readable form. A retrieval system often stores the passage, metadata or a reference alongside the vector. Updating text may require new embeddings, but the exact replacement mechanism depends on the index and application.
Reindexing fixes only the part it actually replaces
Suppose the developer confirms that an old indexed passage survived after the source changed.
The immediate solution is to bring the affected searchable records into agreement with the current source, using the supported update or replacement mechanism.
But reliable replacement is more subtle than uploading another file.
If every ingestion run creates new records without correctly identifying and retiring the previous versions, both versions can remain searchable. The assistant may then encounter two conflicting statements from a document with the same title.
Systems can manage this through stable source identifiers, change tracking, content hashes and cleanup rules. The LangChain RecordManager documentation describes an indexing abstraction that records document hashes, write times and source identifiers so that indexing runs can recognize existing records and help manage obsolete entries.
That is a design pattern rather than a universal feature of every vector database or RAG product.
Rebuilding an entire index can also be unnecessary or insufficient, depending on the failure. For example, Microsoft’s Azure AI Search documentation distinguishes resetting an indexer’s change-tracking state from actually running the indexer, and notes that reset-and-run operations do not automatically delete unrelated orphaned records.
The correct fix follows from evidence about where the mismatch occurs.
A date is useful. A validity boundary is better.
A document’s creation date, modification time and effective period answer different questions.
A file modified today could contain a rule that applied last year. An older file could describe a requirement that remains in force. And a document may contain several rules with different effective periods.
Therefore, simply sorting everything by the most recent modification timestamp is not a complete freshness strategy.
The September 2026 stale-document study provides supporting evidence. In its retrieval-side experiments, a recency-aware reranker reduced poisoning by 4.6 to 10.0 percentage points in the evaluated conditions when date metadata was accurate. Gains were smaller when dates were missing, and incorrect dates could favor outdated evidence.
For a production knowledge system, this points toward a more meaningful goal: identifying which version of a particular instruction applies to the user’s question.
That may involve storing revision IDs, effective dates or supersession relationships, then testing retrieval with questions whose answers have recently changed. These are architectural choices, not requirements that every document-aware assistant already meets.
The corrected answer needs its own proof
Return to the company policy. The authoritative document now says 14 days. The search index has been refreshed. The old passage is no longer selected for current-policy questions.
Those are encouraging signs, but the final step is to ask the assistant again and inspect the evidence it actually used.
When the assistant now answers 14 days and the retrieved passage belongs to the applicable revision, the update has been verified for that test case.
This is the difference between assuming an AI assistant knows about a document correction and establishing that the correction reached the answer.
The file’s latest version is only the beginning. A dependable document-aware assistant also needs a way to establish that the evidence it retrieves is the version that still applies.
Sources
- Ahmed, Poech and Röttger — Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers. Research preprint, September 25, 2026. Experimental results on outdated retrieval, temporal validity and evidence-following behavior.
- Microsoft Learn — Run or Reset Indexers, Skills, or Documents. Incremental indexing, change detection, synchronization and reset behavior.
- OpenAI API Documentation — Retrieval. Vector stores, file processing, chunking, embeddings and search behavior.
- OpenAI API Documentation — File Search. Retrieval integration and inspection of file search results.
- LangChain Reference — RecordManager. Document hashes, source identifiers, indexing records and management of outdated documents.







