TraceSpec: Understanding and Improving Speculative Decoding for LLM Agents

📌 Highlights

TraceSpec treats speculative decoding for agents as a trajectory-conditioned problem: the relevant unit is an assistant decision embedded in a changing history, not merely an entire task or context-length bucket.

The strongest evidence is the controlled replay and inference evaluation. TraceSpec improves replay throughput over DSpark by 12.5% to 22.6% across target sizes and temperatures, and its Qwen3-4B throughput reaches 53.44 and 49.37 tokens/s on TraceSpec-Traj at temperatures 0 and 1, compared with 44.25 and 40.28 for DSpark.

The central analytical finding is a composition effect: aggregate online acceptance can rise with context length while matched late browsing actions show lower acceptance at longer contexts.

The main caveat is scope. The experiments cover two agent workloads, two temperatures, three Qwen3 target sizes, and a DSpark-based draft adaptation, so transfer to other serving stacks remains to be tested.

10,000 training states and 2,500 test states in TraceSpec-Traj.
Dynamic execution plus identical-prefix replay.
Full-turn supervision with state-conditioned allocation.
68 of 72 phase-level accepted-length comparisons favor TraceSpec.
Adaptive allocation adds 3.3% to 10.0% over fixed full-turn supervision.

🎯 Introduction

Interactive agents repeatedly call a language model while histories accumulate reasoning, actions, tool results, and feedback. Web research agents and software engineering agents therefore expose many distinct prediction states within one task. Speculative decoding can reduce verification rounds, but aggregate acceptance compresses these states and cannot determine whether a trend is caused by context length, workload composition, execution phase, or task progress.

TraceSpec addresses this measurement problem by retaining intermediate decision histories and evaluating them under aligned conditions. The paper's objective is both diagnostic and operational: characterize where target-draft agreement varies, then use those observations to select and weight assistant turns under a fixed supervision budget.

🔬 Methodology

TraceSpec-Traj is constructed from DeepResearch-traj, Nebius SWE-agent trajectories, and R2E-Gym development and testing-agent collections. It contains 10,000 training states and 2,500 test states, equally divided between web research and software engineering, with disjoint source tasks across splits. Each state records actual tokenized prefix length, workload, one of four phases—planning, action, reflection, or answer—preceding tool interactions, and trajectory position.

The evaluation has two complementary modes. Dynamic execution follows generated actions and tool feedback on BrowseComp-Plus and SWE-bench Verified. Fixed-prefix replay feeds every drafter the same saved histories from the TraceSpec-Traj test set. This separates behavior induced by the workload trajectory from agreement measured at identical decision boundaries. Episode latency is decomposed as

Training expands target-generated trajectories into assistant-turn segments. The target is frozen, while the DSpark draft is adapted using the original distillation objective. For a sampled block, token overlap is defined by equation 2, prefix-aware segment difficulty by equation 3, and state-conditioned smoothing by equation 4. Equations 5 and 6 convert difficulty into bounded adaptive shares, while equation 7 corrects sampling probabilities for unequal effective event counts. The resulting masks consume a fixed number of prediction events rather than a fixed number of segment visits.

equation 2

Formula:

\begin{aligned} a_{b,k}&=\sum_v\min\{p_{b,k}(v),q_{b,k}(v)\}\\ &=1-{\operatorname{TV}}(p_{b,k},q_{b,k}). \end{aligned}

Meaning: Measures target-draft distribution overlap at each prediction position.

equation 3

Formula:

D_s=1-\frac{\displaystyle\sum_{b\in\mathcal A_s} \sum_{k=1}^{m_b}\prod_{j=1}^{k}a_{b,j}} {\displaystyle\sum_{b\in\mathcal A_s}m_b}.

Meaning: Scores segment difficulty using consecutive prefix agreement.

equation 4

Formula:

\widehat D_s(t)= \frac{n_s(t)\overline D_s(t)+\kappa\overline D_{g(s)}(t)} {n_s(t)+\kappa}.

Meaning: Combines segment evidence with a state-group prior.

equation 5

Formula:

r_s(t)={\operatorname{clip}}\!\left( \frac{\widehat D_s(t)+\epsilon} {\sum_j b_j\widehat D_j(t)+\epsilon}, r_{\min},r_{\max}\right),

Meaning: Bounds relative difficulty used for adaptive weighting.

📊 Experiments

Dynamic evaluation uses 100 tasks from BrowseComp-Plus and 100 from SWE-bench Verified. Replay evaluates the 2,500 TraceSpec-Traj test states. The comparison includes DFlash, EAGLE-3, DSpark, LongSpark, DLite, and TraceSpec with Qwen3-4B, Qwen3-8B, and Qwen3-14B targets at temperatures 0 and 1, producing 108 configurations. DFlash, DSpark, and TraceSpec use seven-token draft blocks; dynamic episodes allow 50 action steps and 32,768 context tokens, while replay allows 36,864 context tokens and both modes allow up to 4,096 new tokens per call.

The reported metrics are accepted length, verification yield, and generation throughput. Accepted length counts emitted accepted tokens per verification round, verification yield divides accepted tokens by proposed draft tokens plus verification rounds, and throughput divides output tokens by model-call time including prefill. The paper reports pointwise 95% bootstrap intervals for baseline acceptance. TraceSpec leads 68 of 72 phase-level accepted-length comparisons, including all 24 replay comparisons and 44 of 48 dynamic comparisons. On TraceSpec-Traj, its gains over DSpark range from 12.1% to 25.3% across phases.

The trajectory analysis explains why context-only conclusions can be misleading. For Qwen3-8B at temperature 0, DSpark's online browsing acceptance rises from 3.633 below 2k input tokens to 5.479 at 16k to 32k, while browsing throughput falls from 82.73 to 22.54 tokens/s. In matched late browsing actions, DSpark acceptance instead decreases from 6.194 to 5.322 tokens per round between 8k to 16k and 16k to 32k contexts, with a 95% confidence interval of [-1.205, -0.473].

The training ablations support the proposed mechanism. Full-turn supervision improves accepted length by 3.9% to 10.2% over final-turn training, and adaptive allocation adds 3.3% to 10.0% over fixed full-turn supervision. On SWE-bench Verified at temperature 1, adaptive allocation raises accepted length from 4.417 to 4.859. On Qwen3-4B TraceSpec-Traj replay, TraceSpec improves DSpark throughput from 44.25 to 53.44 tokens/s at temperature 0 and from 40.28 to 49.37 at temperature 1.

tab:latency

Caption: Mean episode latency (s), $T=0$. BC: BrowseComp-Plus; SWE: SWE-bench Verified. Gain is the latency reduction of DSpark over DFlash.

Content:

Workload	Target	DFlash	EAGLE-3	DSpark	Gain (%)
BC	4B	89.17	87.16	82.86	7.1
BC	8B	73.18	77.27	68.14	6.9
BC	14B	147.08	167.77	143.15	2.7
SWE	4B	63.26	63.15	59.96	5.2
SWE	8B	80.12	84.13	75.90	5.3
SWE	14B	95.60	107.77	91.90	3.9

Why It Matters: Episode latency includes costs beyond verification, including prefill and tools.

tab:late

Caption: Late-action acceptance: Qwen3-8B, $T=0$, step $\ge20$. Tasks are paired across 8k to 16k and 16k to 32k contexts; $\Delta L=L_{16:32}-L_{8:16}$.

Content:

Workload	Drafter	$L_{8:16}$	$L_{16:32}$	95% CI of $\Delta L$
BC	DFlash	5.661	4.326	[-1.650, -0.941]
BC	EAGLE-3	6.007	4.897	[-1.489, -0.612]
BC	DSpark	6.194	5.322	[-1.205, -0.473]
SWE	DFlash	5.333	4.762	[-0.852, -0.257]
SWE	EAGLE-3	6.149	6.304	[-0.347, 0.716]
SWE	DSpark	6.057	6.106	[-0.238, 0.379]

Why It Matters: Paired late-action evidence resolves the difference between aggregate context trends and within-state behavior.

tab:replay

Caption: Replay acceptance on ($L$). All methods use identical saved prefixes. Column groups separate temperatures; rows and highlighting follow Table~tab:online.

Content:

Target	Phase	T=0 EAGLE-3	T=0 DFlash	T=0 LongSpark	T=0 DLite	T=0 DSpark	T=0 TraceSpec	T=1 EAGLE-3	T=1 DFlash	T=1 LongSpark	T=1 DLite	T=1 DSpark	T=1 TraceSpec
Qwen3-4B	Plan	3.899	3.585	3.611	2.618	4.161	4.856	2.972	2.756	2.863	2.470	3.639	4.218
Qwen3-4B	Act	3.704	3.419	2.788	2.549	3.953	4.909	3.316	2.892	2.824	2.471	3.789	4.695
Qwen3-4B	Reflect	3.777	3.439	3.252	2.651	4.092	4.859	3.303	2.810	3.004	2.499	3.752	4.669
Qwen3-4B	Answer	3.089	2.803	3.013	2.578	3.294	4.126	2.788	2.507	2.881	2.488	3.248	3.993
Qwen3-8B	Plan	3.156	3.345	3.041	2.610	3.808	4.397	2.755	2.925	2.912	2.553	3.620	4.060
Qwen3-8B	Act	3.873	3.999	3.203	2.563	4.436	5.141	3.155	3.129	2.769	2.422	3.956	4.630
Qwen3-8B	Reflect	3.504	3.578	3.221	2.751	4.095	4.662	2.889	2.899	2.915	2.617	3.602	4.153
Qwen3-8B	Answer	2.764	2.963	2.968	2.611	3.425	4.081	2.528	2.615	2.878	2.592	3.319	4.086
Qwen3-14B	Plan	3.160	3.517	3.050	2.548	3.933	4.584	2.793	3.080	2.901	2.485	3.654	4.273
Qwen3-14B	Act	3.382	3.754	3.027	2.539	4.149	5.053	2.914	3.385	2.845	2.418	3.936	4.728
Qwen3-14B	Reflect	3.354	3.681	3.093	2.624	4.128	4.737	2.815	3.243	2.873	2.492	3.789	4.247
Qwen3-14B	Answer	2.921	3.313	3.088	2.586	3.714	4.282	2.600	3.028	2.941	2.537	3.536	4.149

Why It Matters: Identical-prefix replay provides the cleanest comparison of phase-specific draft agreement.

equation 8

Formula:

L=\frac{\sum_i A_i}{\sum_i V_i},\quad R=\frac{\sum_i A_i}{\sum_i(P_i+V_i)},\quad G=\frac{\sum_i N_i}{\sum_i t_i}.

Meaning: Defines accepted length, verification yield, and generation throughput used in the experiments.

fig:framework

Caption: TraceSpec overview. (a) Trajectory-state construction, coverage, and source-task splits. (b) Dynamic execution and fixed-prefix replay with trajectory diagnosis. (c) Full-turn supervision with state-conditioned allocation and event-normalized sampling under a fixed prediction budget.

Why It Matters: This figure connects the paper's data representation, evaluation protocols, and adaptive training loop in one view.

fig:acceptanceoverview

Caption: Baseline acceptance across workloads. Columns show dynamic execution on BrowseComp-Plus and SWE-bench Verified, followed by fixed-prefix replay on . Groups denote Qwen3-4B, 8B, and 14B; rows denote $T=0$ and $T=1$. Error bars show pointwise 95\% bootstrap intervals.

Why It Matters: It compares accepted tokens per verification round across workloads, target sizes, temperatures, and draft methods.

fig:diagnosis

Caption: Online baseline acceptance and throughput by context length. Qwen3-8B at $T=0$: upper panels show accepted tokens per round; lower panels show output tokens/s, including prefill.

Why It Matters: It shows why acceptance and serving throughput must be analyzed separately as context length increases.

fig:phases

Caption: Baseline acceptance by domain and phase. Qwen3-8B results on at $T=0$ and $T=1$, measured in accepted tokens per round. All panels share a common color scale.

Why It Matters: It exposes phase-specific drafting differences that aggregate workload metrics can conceal.

🔮 Conclusion

TraceSpec connects trajectory-resolved evaluation with adaptive draft supervision. Its central empirical result is that drafting difficulty varies by workload, phase, and task progress, and that aggregate context trends can reverse under matched late-action comparisons.

The controlled results support full-turn supervision, state-conditioned acceptance-gap estimation, prefix-aware scoring, and event-normalized allocation as complementary design choices. TraceSpec improves replay generation throughput over DSpark by 12.5% to 22.6%, while the latency table shows why these gains should be interpreted separately from complete interactive episode time.

🛠️ Future Research Improvements

The most direct extension is broader transfer testing. The current evidence covers BrowseComp-Plus, SWE-bench Verified, TraceSpec-Traj, Qwen3-4B/8B/14B targets, and temperatures 0 and 1. New experiments should test additional agent workloads, tool interfaces, target families, draft architectures, sampling policies, and longer histories.

A second direction is tighter systems evaluation. Since episode latency includes prefill, draft verification, tool execution, and other costs, future work should co-optimize the allocation policy with batching, KV-cache management, scheduling, and tool latency rather than optimizing accepted length or model-call throughput alone.

The allocation signal could also be evaluated for stability and responsiveness: how quickly state-group estimates adapt, how clipping and shrinkage affect coverage, and whether the same difficult states remain difficult after the draft improves.

🏭 Potential Industry Use Scenarios

TraceSpec's replay methodology could support regression testing for production agents by storing representative decision histories and comparing draft versions on identical prefixes. This would make phase-specific degradation visible even when aggregate task throughput appears stable.

The adaptive supervision approach is potentially useful for organizations serving research, coding, customer-support, or workflow agents whose calls differ substantially in length and role. A deployment team could prioritize draft training for states with persistent target-draft disagreement while maintaining coverage of workloads, context ranges, tool states, and agent phases.

The evidence is strongest for inference-efficiency optimization, not task-quality improvement. Any production adoption should therefore measure task success, tool-call correctness, end-to-end latency, cost, and failure behavior alongside speculative-decoding metrics.

💬 Critical Analysis

The paper's strongest design choice is the separation between dynamic execution and fixed-prefix replay. Dynamic runs preserve realistic workload composition and tool feedback, while replay controls the history so comparisons are attributable to the drafter and training method. The paired late-action analysis is especially valuable because it demonstrates that a rising aggregate curve need not imply better agreement within a fixed decision type.

The evidence also supports the proposed training mechanism rather than only reporting a final system comparison. Removing intermediate-turn supervision, adaptive allocation, state-conditioned estimation, or prefix-aware scoring lowers accepted length across the reported Qwen3-4B settings. The fixed-event-budget comparison makes these ablations more interpretable than a comparison that changes both data volume and training exposure.

The main open question is external validity. The paper's conclusions are grounded in two workloads, a balanced two-domain corpus, a DSpark-based draft, and a bounded set of temperatures and context limits. Moreover, TraceSpec's higher generation throughput does not guarantee lower episode latency when tool execution and history processing dominate. Reproduction should therefore preserve the paper's distinctions among accepted length, verification yield, throughput, and full-loop latency.

Strength: aligned replay histories make draft comparisons more causally interpretable.
Strength: ablations isolate coverage, allocation, state conditioning, and sequential scoring.
Caveat: workload composition can change aggregate trends.
Caveat: end-to-end latency depends on costs outside draft verification.
Open question: transfer beyond the evaluated targets, workloads, and draft architecture.

Original Abstract

LLM agents execute tasks through repeated model calls over evolving histories of reasoning, actions, and tool feedback. Aggregate speculative-decoding metrics merge these distinct decisions, providing limited guidance on where draft supervision is needed. We present TraceSpec, a framework connecting trajectory evaluation with adaptive draft training. It combines dynamic execution with fixed-prefix replay and introduces TraceSpec-Traj, a corpus of 10,000 training states and 2,500 test states spanning browsing and software engineering. Analysis of five speculative baselines reveals that aggregate acceptance can rise with context length while declining within matched late browsing actions. Workload and phase further distinguish these trends, motivating supervision guided by trajectory conditions. TraceSpec trains on intermediate assistant turns and allocates a fixed prediction budget using state-conditioned estimates of prefix-aware acceptance gaps. Across three target sizes and two temperatures, it improves replay throughput by 12.5% to 22.6% over its DSpark base. Qwen3-4B ablations show that adaptive allocation increases accepted length by 3.3% to 10.0% over fixed full-turn supervision, with additional comparisons substantiating the roles of state conditioning and consecutive-token agreement. The resulting evaluation resources, trajectory findings, and training strategy connect the analysis of agent execution to practical draft improvement.