📌 Highlights
CacheGraft converts a larger model's offline understanding of a retrieved document into a receiver-space KV cache that a smaller model can use at query time. The central architectural claim is supported by broad improvements across five QA benchmarks, but the evidence is a proof of concept rather than a general deployment recipe.
- Uses Qwen3-4B-Base as the sharer and Qwen3-0.6B as the receiver.
- Performs cache fusion once per document and stores the result for later retrieval.
- Improves receiver-only answer quality in 29 of 30 reported comparisons, with 17 statistically significant improvements.
- Benefits are weaker when the sharer has little quality headroom, especially for retrieved QASPER.
- Storage and query-time verbosity are the main practical costs.
🎯 Introduction
Prior KV-cache translation work focused on latent communication between heterogeneous language-model agents. CacheGraft extends that idea to RAG: a larger sharer can enrich a smaller receiver's representation of each corpus passage during indexing, even though the larger model is unavailable during serving.
The paper positions this approach between cache-based RAG, which precomputes a single model's document cache, and textual document enrichment, which rewrites or augments documents before retrieval. Its paper-level objective is to test whether an off-the-shelf C2C fuser trained on general chat distillation can transfer useful document signal to knowledge-intensive QA without retrieval-specific training.
🔬 Methodology
The system represents a RAG prompt as x = [p,d,q], where p is a fixed preamble, d is the retrieved document, and q is the question. Offline, the sharer and receiver independently prefill [p,d]. A terminal layer-alignment strategy maps receiver layers to sharer layers, and a lightweight per-layer fuser combines their KV states. The resulting cache has the receiver's sequence length and representation space, so the sharer is not needed after indexing.
The fusion operation is defined by Equation 1. Because causal attention prevents document positions from depending on later question tokens, the fused cache for [p,d] is identical to the corresponding prefix cache obtained from [p,d,q]. This makes document-only preprocessing possible. Online, dense top-1 retrieval selects a document using bge-small, the corresponding cache is loaded by content hash, and the receiver generates from the question using Equation 2.
The pipeline figure makes the offline/online boundary explicit: only the fused KV cache crosses that boundary, while the sharer and fuser remain offline.
equation 1
Meaning: For each receiver layer n, the fuser combines the receiver cache with the sharer cache from the mapped sharer layer G(n), producing a fused cache in the receiver's KV space.
equation 2
Meaning: During online generation, the receiver predicts the next token using the fused document cache concatenated with its own cache for the generated prefix.
📊 Experiments
The evaluation uses SQuAD v1.1, AdversarialQA, CovidQA, QASPER, and TriviaQA. The authors retain one question per gold passage, evaluate both dense top-1 retrieval and oracle retrieval, and run standard greedy decoding on one NVIDIA A100-40GB. The receiver is Qwen3-0.6B, the sharer is Qwen3-4B-Base, and the released C2C projector was never trained on retrieval data.
Answer quality is measured with two locally run open-weight LLM judges, phi-4 and Gemma-2-9B, plus contains, which tests whether the generated answer includes the dataset gold answer. Higher values indicate better answer quality or answer inclusion. Statistical significance uses an exact two-sided paired McNemar test, with * marking p < 0.05. The authors avoid Exact Match and F1 because fused-cache generations are often more verbose.
Table 1 shows the main pattern: CacheGraft improves over receiver-only on nearly every metric, with gains up to +7.9 under phi-4 for retrieved SQuAD and +8.7 under contains for oracle AdversarialQA. QASPER retrieved is the main exception: CacheGraft improves phi-4 by +0.4 and contains by +1.1 but decreases Gemma by -2.0. Table 3 supports the interpretation that gains track the sharer's headroom over the receiver, while Table 2 demonstrates both successful steering toward relevant spans and incorrect span selection.
The cost experiment uses batch size 1, warm execution, greedy decoding, 24 maximum new tokens, and 100 SQuAD questions. CacheGraft takes 574 ms and 1.27 GB, compared with 334 ms and 1.22 GB for receiver-only text RAG and 652 ms and 7.62 GB for sharer text RAG. Thus it approaches receiver memory usage and avoids the 4B model online, but it does not minimize latency.
Table 1
Caption: Table 1: Performance of CacheGraft in comparison to using only the 0.6B receiver model and only the 4B sharer model across two LLM judges and the contains metric, which measures if the correct answer is included in the output text. CacheGraft improvements over the receiver-only baseline are marked in bold. Δ reports CacheGraft minus Receiver-Only for each metric.
| Dataset | Setting | n | Top-1 Recall | phi-4 Receiver-Only | phi-4 CacheGraft | phi-4 Sharer-Only | phi-4 Δ | Gemma Receiver-Only | Gemma CacheGraft | Gemma Sharer-Only | Gemma Δ | contains Receiver-Only | contains CacheGraft | contains Sharer-Only | contains Δ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AdversarialQA | retrieved | 416 | 59.4 | 30.8 | 36.8 | 41.8 | +6.0* | 34.9 | 38.9 | 48.3 | +4.1 | 24.5 | 30.3 | 33.7 | +5.8* |
| AdversarialQA | oracle | — | — | 42.1 | 49.3 | 59.1 | +7.2* | 44.2 | 51.4 | 63.9 | +7.2* | 33.9 | 42.5 | 48.3 | +8.7* |
| SQuAD | retrieved | 800 | 68.1 | 51.8 | 59.6 | 66.4 | +7.9* | 59.2 | 62.5 | 70.8 | +3.2* | 48.6 | 55.5 | 61.1 | +6.9* |
| SQuAD | oracle | — | — | 73.0 | 79.4 | 86.5 | +6.4* | 78.6 | 81.8 | 89.4 | +3.1 | 69.9 | 76.9 | 81.2 | +7.0* |
| CovidQA | retrieved | 800 | 53.2 | 33.1 | 36.9 | 41.9 | +3.8* | 50.8 | 53.8 | 59.8 | +3.0 | 21.0 | 22.6 | 24.4 | +1.6 |
| CovidQA | oracle | — | — | 53.7 | 59.1 | 61.1 | +5.4* | 71.7 | 76.9 | 80.0 | +5.2* | 32.6 | 35.4 | 37.2 | +2.8 |
| TriviaQA | retrieved | 800 | 54.3 | 38.1 | 41.6 | 57.8 | +3.5* | 49.2 | 52.4 | 74.1 | +3.1* | 45.9 | 48.9 | 65.5 | +3.0* |
| TriviaQA | oracle | — | — | 50.8 | 54.0 | 70.1 | +3.2 | 62.9 | 63.4 | 85.0 | +0.5 | 61.0 | 62.8 | 81.5 | +1.8 |
| QASPER | retrieved | 583 | 36.2 | 15.3 | 15.7 | 17.2 | +0.4 | 35.9 | 33.9 | 35.5 | -2.0 | 17.7 | 18.8 | 18.2 | +1.1 |
| QASPER | oracle | — | — | 34.7 | 37.5 | 37.9 | +2.9 | 57.4 | 60.8 | 59.2 | +3.4 | 32.7 | 39.0 | 34.8 | +6.3* |
Why It Matters: The table provides the main benchmark comparison and shows that gains are strongest when the larger sharer has more headroom, while QASPER retrieved results are mixed.
Table 2
Caption: Table 2: Example outputs of CacheGraft and the receiver-only Text-RAG baseline. The last two rows are failure cases: in each, CacheGraft answers with a different span of the same passage.
| Question | Gold | text-RAG | CacheGraft |
|---|---|---|---|
| Over how many species of trees can be found in the Saxon Garden? | 100 | 15.5 × | 100 ✓ |
| Where was the Continental Edison Company located? | France | New York City. × | The Continental Edison Company was located in France. ✓ |
| Who did Rollo sign the treaty of Saint-Clair-sur-Epte with? | King Charles III | Rollo. × | Rollo signed the treaty of Saint-Clair-sur-Epte with King Charles III of West Francia. ✓ |
| Where were missing patents later found? | patent archives | The missing patents were later found in patent archives. ✓ | The missing patents were found in the United States, Britain, and Canada. × |
| Whose writings were widely quoted by the Third Reich? | Luther | Luther's writings were widely quoted by the Third Reich. ✓ | The writings of Heinrich Himmler were widely quoted by the Third Reich. × |
Why It Matters: The qualitative examples show both the intended behavior—steering the receiver toward the relevant passage span—and a failure mode in which latent enrichment selects an incorrect span from the same document.
Table 3
Caption: Table 3: Relationship between CacheGraft’s gains and the opportunity for improvement, all computed in the oracle-retrieval setting. Headroom is Sharer-Only minus Receiver-Only under contains. Answer-in-doc is the sharer’s contains improvement when given the gold passage versus answering closed-book (no passage). Gain is CacheGraft’s improvement over Receiver-Only under the phi-4 judge.
| Dataset | Headroom | Answer-in-doc | Gain |
|---|---|---|---|
| AdversarialQA | +14.4 | +34.6 | +7.2 |
| SQuAD | +11.3 | +69.0 | +6.4 |
| CovidQA | +4.4 | +34.1 | +5.4 |
| TriviaQA | +20.5 | +40.7 | +3.2 |
| QASPER | +2.0 | +30.2 | +2.9 |
Why It Matters: This analysis links CacheGraft's benefit to two conditions: the sharer must outperform the receiver, and the document must provide factual signal that the sharer can exploit.
fig:pipeline
Caption: The sharer and fuser run only offline, once per corpus passage, with no knowledge of any query. The fused KV cache is the only artifact that crosses the offline/online boundary: at query time the receiver loads it and decodes alone, reading the passage $[p\,d]$ in latent space and the question $[q]$ as text.
Why It Matters: This figure captures the central systems split: expensive cross-model fusion occurs during indexing, while the compact receiver handles online question answering alone.
🔮 Conclusion
The paper's main result is that offline KV-cache fusion can make a 0.6B receiver more effective at knowledge-intensive RAG without running the 4B sharer during online inference. The evidence is strongest on SQuAD, AdversarialQA, CovidQA, and TriviaQA, while QASPER demonstrates that the method is not uniformly robust across datasets or evaluators.
The practical tradeoff is favorable when online memory is more constrained than offline storage and indexing compute: CacheGraft uses 1.27 GB in the reported serving test versus 7.62 GB for the sharer, but costs 574 ms versus 334 ms for receiver-only text RAG. The authors therefore present the system as an initial demonstration rather than a finished production architecture.
🛠️ Future Research Improvements
The clearest next step is retrieval-specific fuser training. The current projector was trained on general chat distillation, so a fuser optimized for document grounding, answer extraction, and concise generation could address the observed verbosity and improve transfer. The paper also raises the question of how large the sharer-receiver model gap can become before fusion fails.
The system should be extended beyond single-document top-1 retrieval. Top-k cache composition, position handling, cache compression, and interactions with textual document enrichment are all unresolved. These improvements should be evaluated with separate measurements for answer quality, storage, indexing cost, latency, and memory.
- Train fusers with retrieval-specific objectives and datasets.
- Test larger or more diverse sharer-receiver architecture gaps.
- Support top-k retrieval and multi-document cache composition.
- Combine latent enrichment with textual document enrichment.
- Apply KV-cache compression to reduce corpus storage.
🏭 Potential Industry Use Scenarios
A plausible deployment scenario is a knowledge assistant with a large offline indexing budget but strict online GPU limits. A larger model could preprocess enterprise documents into receiver-compatible caches, while a small model serves employee questions with lower memory requirements than running the larger model for every request.
Other possible uses include support or documentation assistants where documents change less frequently than queries, allowing cache generation to be amortized. The evidence does not yet establish production reliability: the reported system is limited to one retrieved document, carries substantial storage overhead, and can select the wrong span even when the correct passage is available.
- Enterprise knowledge assistants with recurring queries over relatively stable corpora.
- Customer-support systems where offline indexing can use larger models but serving must remain compact.
- Domain-specific document stores where factual grounding is more important than minimum latency.
- Edge or cost-sensitive serving environments where online GPU memory is the primary constraint.
💬 Critical Analysis
The strongest aspect of the work is the clean systems insight: causal attention allows a document's fused cache to be computed without knowing the future question, turning cross-model KV translation into an indexing operation. The benchmark coverage, retrieved-versus-oracle comparison, two independent judges, contains metric, qualitative examples, and cost measurement provide a useful first evaluation.
The evidence is less decisive about generality. All experiments use one C2C projector, one model pair, one-document top-1 retrieval, one GPU configuration, and a small set of QA datasets. The largest gains occur where the sharer already has substantial headroom, so the method may be transferring the larger model's document-reading ability rather than providing a broadly model-independent enrichment mechanism.
The reported cost profile also complicates the efficiency claim. CacheGraft reduces GPU memory relative to the 4B sharer but increases latency and token generation relative to receiver-only text RAG, while corpus-scale KV storage may dominate operational cost. A convincing follow-up should report cache size and indexing throughput, test compression, evaluate top-k retrieval, and compare against strong textual enrichment and cache-only RAG baselines under matched quality and cost budgets.