📌 Highlights
The paper's central claim is that chain dense retrieval suffers from exposure bias on the query side: the model is trained on gold histories but deployed on histories containing its own retrieval mistakes.
The proposed on-policy state training method preserves the existing chain architecture and contrastive objective while refreshing half of the training states from the current retriever at every epoch.
The paper reports 49.3 C@5 on MuSiQue, 20.8 C@5 on HotpotQA-Long, and 27.7 C@5 on 2Wiki-Long in the five-seed comparison, exceeding the listed baselines on those columns.
The main caveat is evaluation scope: the long-chain benchmarks are constructed by the authors, and some graph-based baselines use reduced corpora.
- State mismatch becomes severe late in a chain: 70% of step-three and 89% of step-four states on four-hop MuSiQue contain a wrong passage.
- On-policy training improves recovery after one wrong passage from 25.0% to 35.8% and after two or more wrong passages from 14.5% to 29.2% on fixed states.
- With a fixed reader, the retrieved passages produce 44.1 EM and 53.7 F1 on MuSiQue.
🎯 Introduction
The task is multi-hop retrieval-augmented generation in which the evidence needed to answer a question is distributed across several passages. The paper frames chain embedding retrieval as a practical alternative to graph-based systems and language-model search agents because it uses a dense passage index and does not require a language-model call at retrieval time.
The prior limitation is a train-test state mismatch. Teacher-forced training exposes the query encoder to gold passage prefixes, whereas deployment feeds it the passages it actually retrieved. On four-hop MuSiQue, the paper reports wrong-passage histories in 19% of feasible step-two states, 70% of step-three states, and 89% of step-four states. The paper's objective is to determine whether changing only the training-state distribution improves retrieval and whether the gain specifically comes from learning to recover from the retriever's own mistakes.
The paper positions its contribution as a dense-supervision analogue of on-policy imitation learning: every visited state can be labeled from the gold evidence set, including states that contain non-gold passages, without estimating a sparse episode-level reward.
🔬 Methodology
A question q is paired with a corpus C and an ordered gold passage set G(q). The retriever serializes the question and previously retrieved passages into a state text, embeds that text with a query encoder, and compares it with document embeddings computed once by a fixed document encoder. Greedy retrieval appends the highest-scoring unseen passage at each hop.
The supervision rule is defined for any state whose gold evidence is incomplete. The target is the earliest missing gold passage, while other missing gold passages are removed from the denominator because retrieving them is also valid progress. This allows contaminated histories to receive the same per-step contrastive supervision as clean histories.
On-policy state training uses a mixture of teacher-forced states and states visited by the current encoder. At the start of each epoch, the current encoder rolls out on all training questions; visited states are deduplicated, sampled, and combined with an equal-sized teacher-forced set. The visited set is discarded and recollected every epoch, so the training distribution follows the evolving policy. The document index, negative-selection rule, and query-side LoRA adaptation remain fixed across the comparison.
equation 1
Meaning: Scores a candidate passage by the inner product between the state-conditioned query embedding and the fixed document embedding.
equation 2
Meaning: Defines the greedy chain policy: retrieve the highest-scoring passage not already in the history, then append it to the next state.
equation 3
Meaning: Selects the earliest gold passage still missing from the current state as the positive target.
equation 4
Meaning: Applies the contrastive training loss while excluding other missing gold passages from the negative denominator.
📊 Experiments
The experiments train on 1,800 MuSiQue questions, select checkpoints on 200 questions, and evaluate on the official 1,000-question development set with an 11,656-passage corpus. The transfer benchmarks are HotpotQA-Long, with 100 test questions over 554,812 passages, and 2Wiki-Long, with 300 test questions over 220,000 passages. The retriever uses four chain steps on MuSiQue and eight on the long-chain benchmarks, with no further training on the transfer datasets.
The evaluation reports recall and complete coverage at k = 5 and k = 10. Recall measures the fraction of gold passages retrieved, while complete coverage counts a question only when all gold passages appear in the top-k ranking. Baselines include BM25, frozen Dense retrieval with Qwen3-Embedding-0.6B, G-Retriever, HippoRAG 2, LinearRAG, MDR, Q-RAG, and Search-R1. Search-R1 uses a 7B language model at every step, whereas the proposed retriever uses a 0.6B encoder and nearest-neighbor retrieval.
In the main comparison, on-policy state training reaches 73.5 R@5 and 49.3 C@5 on MuSiQue, 49.6 R@5 and 20.8 C@5 on HotpotQA-Long, and 59.2 R@5 and 27.7 C@5 on 2Wiki-Long. On MuSiQue, the fixed-reader evaluation reaches 44.1 EM and 53.7 F1. The ablations show that removing visited states reduces MuSiQue C@5 from 50.5 to 44.6, while removing refresh reduces HotpotQA-Long C@10 from 28.0 to 22.7 and 2Wiki-Long C@10 from 34.7 to 28.3.
The controlled recovery tests support the proposed mechanism. On fixed states, clean-history next-step success is similar across encoders, but after one wrong passage it increases from 25.0% with teacher forcing to 35.8% with on-policy training, and after two or more wrong passages from 14.5% to 29.2%. Synthetic contamination closes about half of the gap, whereas refreshing states from the current encoder reaches 50.5 C@5, indicating that both exposure to contaminated histories and policy-following contribute.
tab:main
Caption: Retrieval results. Our retriever is trained on MuSiQue and evaluated on the long-chain benchmarks without further training; its results are the mean over five seeds, with the standard deviation in gray. Dense is the frozen Qwen3-Embedding-0.6B encoder with the question as the query; the structure-augmented baselines, Search-R1 and our retriever use the same encoder. Search-R1 calls a 7B language model at every step. The best result in each column is in bold and the second best is underlined. † HippoRAG 2 and G-Retriever on the long-chain benchmarks are run on reduced corpora, an easier setting than the full corpora used by the other rows.
| Method | MuSiQue R@5 | MuSiQue R@10 | MuSiQue C@5 | MuSiQue C@10 | HotpotQA-Long R@5 | HotpotQA-Long R@10 | HotpotQA-Long C@5 | HotpotQA-Long C@10 | 2Wiki-Long R@5 | 2Wiki-Long R@10 | 2Wiki-Long C@5 | 2Wiki-Long C@10 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BM25 | 43.4 | 50.8 | 9.8 | 15.5 | 23.3 | 25.5 | 0.0 | 0.0 | 27.9 | 31.5 | 0.0 | 0.0 |
| Dense | 55.1 | 62.4 | 18.9 | 27.1 | 29.0 | 31.9 | 0.3 | 1.0 | 31.3 | 34.5 | 0.0 | 2.0 |
| G-Retriever | 36.5 | 43.7 | 9.2 | 15.9 | 16.1† | 20.5† | 0.0† | 2.0† | 18.9† | 22.2† | 0.0† | 1.7† |
| HippoRAG 2 | 58.2 | 66.0 | 23.1 | 32.3 | 30.9† | 33.4† | 1.0† | 1.3† | 32.1† | 35.7† | 0.0† | 2.3† |
| LinearRAG | 43.9 | 55.8 | 17.7 | 28.7 | 19.0 | 22.1 | 0.7 | 0.7 | 23.2 | 29.3 | 0.3 | 1.0 |
| MDR | 58.3 | 66.0 | 31.9 | 38.3 | 51.0 | 56.2 | 11.0 | 14.7 | 46.9 | 51.5 | 10.0 | 14.3 |
| Q-RAG | 61.7 | 67.8 | 36.2 | 44.9 | 25.2 | 28.0 | 3.3 | 5.7 | 14.7 | 27.8 | 1.7 | 2.3 |
| Search-R1 | 66.7 | 74.7 | 36.5 | 49.5 | 43.6 | 58.5 | 3.3 | 25.7 | 50.3 | 68.1 | 6.0 | 37.0 |
| On-policy state training | 73.5 ± 0.4 | 80.5 ± 0.5 | 49.3 ± 0.4 | 59.8 ± 0.9 | 49.6 ± 1.3 | 56.6 ± 0.7 | 20.8 ± 2.1 | 29.6 ± 0.9 | 59.2 ± 2.9 | 66.1 ± 2.9 | 27.7 ± 4.5 | 32.8 ± 5.2 |
tab:qa
Caption: Question answering on MuSiQue with a fixed reader; trained retrievers use the shared seed.
| Passages from | EM | F1 |
|---|---|---|
| Dense | 25.9 | 36.3 |
| G-Retriever | 21.2 | 29.2 |
| HippoRAG 2 | 28.2 | 39.4 |
| LinearRAG | 22.3 | 30.5 |
| Q-RAG | 32.4 | 42.0 |
| Search-R1 | 35.5 | 44.8 |
| On-policy state training | 44.1 | 53.7 |
Table 3
Caption: Ablations; all rows are trained with the same seed.
| Method | MuSiQue R@5 | MuSiQue C@5 | HotpotQA-Long R@10 | HotpotQA-Long C@10 | 2Wiki-Long R@10 | 2Wiki-Long C@10 |
|---|---|---|---|---|---|---|
| On-policy state training | 74.2 | 50.5 | 56.4 | 28.0 | 67.0 | 34.7 |
| w/o refresh | 71.8 | 48.4 | 52.6 | 22.7 | 64.6 | 28.3 |
| w/o on-policy states | 69.9 | 44.6 | 51.5 | 22.3 | 61.3 | 24.3 |
| w/o chain | 54.6 | 18.9 | 29.8 | 1.0 | 31.4 | 1.7 |
align 8
Meaning: Defines recall at k and complete coverage at k; complete coverage is stricter because every gold passage must appear in the top-k ranking.
fig:overview
Caption: Overview. (a) Teacher forcing trains on gold histories, while at test time the retriever's own mistakes stay in the query and its states drift. (b) On-policy state training rolls in the current encoder, labels every visited state with the earliest missing gold passage, and trains on a mix of gold and visited states, refreshed every epoch.
Why It Matters: This figure gives the central implementation picture: the method changes the training-state distribution while retaining the chain encoder, target rule, and contrastive-learning structure.
fig:bytype
Caption: Retrieval by question type, with the number of questions under each column. Colors are scaled within each column, and the best result of each column is framed. Our retriever is the mean over the five seeds of Table 1; † reduced corpora.
Why It Matters: The breakdown shows where complete-coverage gains persist or saturate across hop counts and question types rather than hiding all behavior behind aggregate scores.
fig:diagnosis
Caption: (a) Share of the teacher-forced retriever's states whose history contains a wrong passage, at each step of four-hop MuSiQue questions (gold passage still missing). (b) Rate at which each encoder retrieves a missing gold passage next on the same states, by the number of wrong passages in the history. All encoders are trained with the same seed.
Why It Matters: The diagnostic directly connects error accumulation with recovery: contaminated states become common at later steps, and on-policy training improves next-step success most strongly after mistakes.
fig:traj
Caption: Query trajectories of the two retrievers on six MuSiQue questions, each on its own chain. Gold passages sit on the top row in hop order and the non-gold passages retrieved by either chain on the bottom row. The query vector at step t is drawn at the mean of these positions weighted by the softmax of its cosine to each passage at the training temperature τ = 0.05, so it sits next to the passages it points at; filled nodes are steps that retrieve a gold passage.
Why It Matters: The qualitative trajectories illustrate the proposed mechanism: teacher-forced queries can remain attracted to non-gold passages after an error, while the on-policy retriever can move back toward the gold chain.
🔮 Conclusion
The paper concludes that chain embedding retrievers are limited by the gap between clean training histories and self-generated test histories. On-policy state training addresses this gap without changing the architecture, loss, document index, or requiring a language model at retrieval time.
The strongest evidence is mechanistic rather than purely aggregate: gains concentrate on contaminated states, refreshing visited states matters more as chains become longer, and deleting wrong passages from teacher-forced states exposes a substantial portion of the failure caused by query contamination.
For builders, the practical takeaway is to treat the distribution of retrieval histories as a first-class training choice alongside targets and negatives, especially when downstream success requires complete evidence rather than high average passage recall.
🛠️ Future Research Improvements
A natural next step is to evaluate on additional real-world multi-hop corpora with naturally occurring retrieval trajectories, rather than relying mainly on constructed long-chain variants. This would test whether the measured state mismatch persists under different corpus, question, and distractor distributions.
Future work should examine alternative dynamic-oracle policies for questions whose evidence order is ambiguous, as well as settings where the gold evidence set is incomplete or noisy. The current rule selects one earliest missing gold passage and masks other missing gold passages, which is robust to some ordering ambiguity but still depends on reliable gold evidence.
The paper also motivates larger studies of joint state and negative-distribution adaptation. Its shared-negative experiment shows that most gains come from histories, while a smaller part comes from the state-dependent negative pools; separating these effects across encoders and corpora remains an open engineering question.
🏭 Potential Industry Use Scenarios
For web search assistants and enterprise search, the method can serve as a lightweight multi-hop retrieval stage that gathers evidence across several passages before a reader or language model generates an answer. Its dense-index design avoids a language-model call at every retrieval step.
For retrieval-augmented generation pipelines, on-policy states can be added during retriever fine-tuning to improve resilience when an early retrieval introduces an irrelevant passage that remains in later query representations.
For agentic research or browsing systems, the same principle can be used when a retriever or tool policy repeatedly conditions on its previous outputs. The paper's results support dense per-step supervision as a complement to sparse outcome feedback, but deployment quality would still need validation under live, changing corpora and imperfect evidence labels.
💬 Critical Analysis
The experimental design is strong in isolating the proposed mechanism: the method keeps the encoder family, document index, negative rule, target construction, and training-state budget controlled while changing the visited-state distribution. Fixed-state probes further show that the benefit is not simply a general retrieval improvement; it is concentrated after the model has already made mistakes.
The main comparison should nevertheless be interpreted carefully. Search-R1 uses a 7B language model at every step, Q-RAG uses a different encoder and value-based reinforcement learning, and some graph baselines receive reduced long-chain corpora. These differences make the results useful as system-level comparisons but less clean as claims of universal superiority.
The complete-coverage metric is appropriate for multi-hop evidence gathering, but it is unforgiving on long chains: a single failed hop can make an entire question fail. The reported four-hop MuSiQue results and the small 2Wiki-Long four- and five-hop groups show that the method improves recovery without eliminating the compounding-error problem.
Overall, the paper provides a convincing diagnosis and a practical intervention for chain retrieval. The remaining question for production use is whether the same state-following advantage survives noisy or partially observed gold evidence, corpus updates, different document encoders, and naturally distributed query trajectories.