RECAP: Context-Aware Prediction Recycling with Request-Local and Persistent Memories for Multimodal Speculative Decoding

📌 Highlights

RECAP recycles a vision-language model's own target predictions to propose draft tokens for speculative decoding, avoiding a separately trained drafter. The paper's central claim is that reuse needs to preserve both textual prefix context and the current image condition. It addresses that requirement with one-, two-, and three-token keys, two request-local contextual memories, and a persistent token-level memory. Entropy-confidence weighting combines available rows; best-first search and suffix retrieval produce a candidate tree that one packed target pass verifies under greedy decoding.

The main reported single-turn results are mean speedups of 1.66× on Qwen2.5-VL-7B-Instruct and 1.56× on LLaVA-1.6 Vicuna-7B, across MMT-Bench, SEED-Bench, ScienceQA, OCRBench, ChartQA, MathVista, and TextVQA. Token Recycling, the strongest training-free baseline by mean speedup, reports 1.51× and 1.37× respectively. The reported gains are generation-throughput comparisons in greedy batch-one evaluation; they do not establish downstream task accuracy or behavior under concurrent serving.

• Core mechanism: fuse predictions retrieved at multiple textual-context resolutions and memory lifetimes.
• Reported single-turn mean speedups: 1.66× for Qwen2.5-VL-7B-Instruct and 1.56× for LLaVA-1.6 Vicuna-7B.
• Main evidence covers two 7B VLMs, seven single-turn benchmarks, and the MMSpec evaluation.

🎯 Introduction

Vision-language models generate autoregressively, requiring a sequential target pass for each output token. Speculative decoding reduces these passes by proposing multiple tokens and verifying them together. Multimodal approaches often use a separately trained drafter, which adds training and serving cost for each target model. Prediction recycling offers a training-free alternative: tree verification evaluates distributions at candidate nodes, and those target predictions can later serve as proposals.

The paper's motivating limitation is that existing recycling can index predictions by the last token alone. The same ending token can follow different textual prefixes or occur with different images, so a cached distribution may be invalid for the current condition. In a counterfactual diagnostic on 700 Qwen2.5-VL images, replacing the image changes the root prediction for 34.9% of fully rejected drafts, compared with 12.4% of drafts accepting five or more tokens. Longer keys can reduce prefix aliasing but occur less often, creating a specificity-coverage trade-off.

RECAP's objective is to preserve useful coverage while conditioning reuse more carefully. It stores predictions under keys of one, two, and three tokens; the contextual maps reset per request, and the one-token map persists across requests. It combines available predictions instead of selecting a single context length, then constructs and verifies a tree while retaining the target's greedy commitment rule.

🔬 Methodology

At each evaluated position, the target model produces logits and RECAP retains a top-K candidate set with a normalized sparse distribution. It indexes that row by the final one, two, or three history tokens. Longer keys distinguish more prefixes, while shorter keys remain available when longer matches are absent. The two longer-context maps reset at each request, making the request's image and prompt implicit in their contents; the token-level map persists across requests. Prefill seeds memory, and verification refreshes all maps with predictions from evaluated tree nodes.

For the rows available at a history, RECAP assigns each a specificity prior and an entropy-confidence factor. Concentrated rows receive more confidence, and the normalized weights define a mixture; its top-K mass is retained and renormalized for proposal use. The paper interprets the mixture as a weighted KL center. The confidence measures distribution concentration under the source condition, not direct agreement with the current image, so it complements rather than proves visual alignment.

Best-first search repeatedly retrieves and fuses rows at each proposed path context, composing local transitions into a candidate tree until a node or depth budget is reached. A suffix automaton over the prompt and committed history can insert a contiguous observed continuation while sharing existing tree prefixes. The target verifies the packed tree in one pass with ancestral attention. RECAP commits the longest path that agrees with greedy target choices plus one target-selected token. Every evaluated node updates memory, including uncommitted branches; only committed tokens extend sequence history. This distinction lets later compatible contexts reuse target predictions without treating hypothetical branches as committed output.

📊 Experiments

Evaluation uses Qwen2.5-VL-7B-Instruct and LLaVA-1.6 Vicuna-7B on an NVIDIA A100-SXM4-80GB GPU. Decoding is greedy, batch size one, with at most 200 new tokens. The seven single-turn benchmarks are MMT-Bench, SEED-Bench, ScienceQA, OCRBench, ChartQA, MathVista, and TextVQA, with 1,000 inputs per benchmark. MMSpec contributes 600 inputs across six categories and includes 223 generation turns from multi-turn conversations, for 723 turns total. Persistent memory is carried across each benchmark's inputs after warmup and cleared at benchmark boundaries; contextual maps reset per request.

The paper reports mean speedups of 1.66× for Qwen and 1.56× for LLaVA across the seven single-turn benchmarks, compared with Token Recycling at 1.51× and 1.37×. RECAP's mean acceptance lengths are 3.02 and 2.92, versus 2.71 and 2.55 for Token Recycling. On MMSpec, overall speedups are 1.89× and 1.59×, against 1.56× and 1.43× for Token Recycling. Speedup is the mean of paired generation-throughput ratios against autoregressive decoding; higher is better. Acceptance length counts accepted draft tokens plus one target-selected token; higher is better. The reported trained-drafter comparison has a different autoregressive harness, so its speedups are not directly comparable.

Component ablations on a 120-input MMSpec subset with Qwen report +4.90% throughput for best-first versus fixed expansion, +3.03% for contextual backoff versus unigram memory, +2.80% for specificity priors versus uniform fusion, and +5.42% for persistent versus reset token memory (the persistence result averages five runs). Entropy confidence adds +0.73% on Qwen and +1.09% on LLaVA in four-dataset controls. Suffix insertion adds +0.61% in its separate four-dataset control. These results support several design choices, while the confidence and suffix gains are modest and controls use distinct subsets.

A replay over 23,811 states from 400 inputs reports the specificity-coverage trade-off: on OCRBench, one-token coverage is 54.9% and three-token coverage is 18.6%, while conditional Recall@8 rises from 63.6% to 90.6%. The evaluation also reports that matching-image rows outperform swapped-image rows in fused Recall@8 by 0.89–6.35 percentage points across datasets. These are retrieval and replay measures, distinct from online throughput.

fig:overview

Caption: The \ framework. (A) Three-, two-, and one-token keys retrieve Top-$K$ prediction rows from $ $, $ $, and $ $; only $ $ survives request boundaries. (B) Entropy-confidence weighting fuses the retrieved rows under specificity priors and normalizes the retained candidates. (C) Best-first tree search repeats retrieval and fusion under each node's path context. (D) Suffix retrieval inserts a contiguous continuation from the prompt and committed history. (E) One packed target pass commits the matching path plus one target-selected token. All evaluated nodes refresh prediction memory; only committed tokens extend sequence history. Tokens and scores are illustrative.

Why It Matters: This overview connects the memory scopes, fusion rule, tree construction, suffix reuse, and verification pass into one implementable decoding loop.

fig:granularity

Caption: The specificity--coverage trade-off and contextual backoff. (a) Query coverage and conditional Recall@$8$ across $23,811$ states from $400$ inputs. Each benchmark connects $ $ (circle), $ $ (square), and $ $ (diamond). Finer keys raise conditional recall while reducing coverage. (b) Paired online throughput gains of $ + $ backoff against $ $.

Why It Matters: The figure shows why finer contextual keys need shorter-key backoff: finer keys improve recall among matches but cover fewer states.

fig:intro

Caption: The role of context in prediction recycling. Top: a cached her$ $head transition fails when reused after a different prefix. Middle: shorter accepted paths accompany stronger root image sensitivity (left), and source preference varies with candidate visual alignment (right). Bottom: \ retrieves and fuses predictions at multiple contextual resolutions.

Why It Matters: The examples and diagnostics motivate contextual keys and show how the better prediction source can vary with visual alignment.

🔮 Conclusion

The paper concludes that jointly controlling key specificity and memory lifetime makes recycled target predictions more useful for multimodal speculative decoding. RECAP combines request-local contextual rows with persistent token-level rows, fuses them with entropy-confidence weighting, and verifies proposed trees with the target model. It reports mean speedups of 1.66× and 1.56× across seven single-turn benchmarks on Qwen2.5-VL-7B-Instruct and LLaVA-1.6 Vicuna-7B, and 1.89× and 1.59× overall on MMSpec.

The practical result is a training-free proposal mechanism that avoids an auxiliary drafter while retaining target-controlled greedy commitment. The evidence supports the approach in the tested image-based, batch-one setting; it does not establish the same gains for larger models, concurrent serving, video, or application-level latency and quality.

🛠️ Future Research Improvements

The paper explicitly identifies larger models, concurrent serving, and video as untested operating regimes. Evaluations in these settings would determine whether request-local memory remains practical when many streams compete for resources, and whether the same lexical keys work for longer or temporally structured visual inputs. The paper also notes that priors, confidence weighting, and tree settings may shift latency-memory trade-offs across hardware and request lengths; systematic sweeps could characterize those trade-offs.

Further work should distinguish proposal efficiency from task outcomes. The reported experiments evaluate throughput, acceptance, and retrieval recall, but do not establish downstream task accuracy or application-level latency distributions. A deployment-oriented study could measure quality equivalence, tail latency, memory growth, and isolation behavior under realistic request mixes. Since entropy confidence reflects concentration rather than direct visual alignment, a future mechanism could test whether visual-condition signals improve source selection, while retaining the paper's target verification guarantee. These are suggested research directions based on stated gaps and method properties, not claims made by the experiments.

• Evaluate larger VLMs, concurrent request streams, and video inputs.
• Measure quality and application-level latency distributions alongside throughput.
• Sweep key priors, confidence rules, and tree budgets across hardware and request lengths.

🏭 Potential Industry Use Scenarios

A plausible use case is reducing generation time for image-question answering services that already serve a compatible VLM and can preserve a small prediction cache. RECAP proposes from target predictions rather than requiring a separate trained draft model, which may simplify model-specific deployment and avoid extra drafter weights. Its demonstrated task mix includes general visual question answering, OCR and text-centric inputs, charts, and reasoning benchmarks, making those workload types reasonable targets for a controlled pilot.

The main-body results do not establish production throughput under concurrent traffic, tail latency, or performance on video and larger models. A team considering deployment should first reproduce the reported greedy, batch-one setup, then test its actual hardware, request lengths, concurrency, and memory limits. Since contextual maps are request-local while token-level memory persists across a stream, deployment owners should define cache lifetime and user or service isolation explicitly. Treat RECAP as a decoding optimization whose output remains subject to the target model's existing quality and safety evaluation; the paper does not report application-level outcomes.

💬 Critical Analysis

The paper's strongest design argument is that key length and memory lifetime address different parts of multimodal conditioning. Contextual keys distinguish textual prefixes, and request-local scope binds stored predictions to the current image implicitly. Persistent unigram memory retains coverage where contextual rows are missing. The ablations report positive throughput effects for backoff, specificity priors, persistence, and best-first expansion; replay results also show higher conditional recall for finer keys at lower coverage. Together, these measurements support the specificity-coverage motivation and the use of backoff.

Several results deserve qualification. Entropy-confidence weighting adds only +0.73% and +1.09% in its stated four-dataset controls, and suffix insertion adds +0.61% in a separate control, so their gains appear smaller than persistence and best-first expansion in the reported settings. Ablations use different subsets and protocols, which limits direct comparison among every row. The trained-drafter table also uses faster autoregressive baselines and explicitly warns that those speedups are not directly comparable.

Reproducibility depends on implementation details such as memory lifecycle, tree budget, and request stream. The headline results are measured at batch size one on two 7B image-based VLMs, not under concurrent serving. Throughput and acceptance length do not measure task accuracy or production tail latency. Finally, a concentrated cached row is not necessarily visually aligned with the current image; the request-local scope and target verification address parts of that risk, but the evidence does not establish behavior for video or broader serving conditions. Those boundaries should guide any engineering claim beyond the paper's tested setup.