Each section below summarizes one paper's reported method, experiments, and limits from its full review. The studies use different tasks and metrics; their results are not directly comparable.
arXiv:2611.08376
Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Muir is a linear-attention language-model architecture that reallocates gate capacity according to measured state-update sensitivity and uses matched-filter query readout to compensate for unequal state-row energies. It compresses decay, erase, write, output, and query-readout gates into one asymmetric low-rank projection, reinvests saved parameters into a wider feed-forward network, and introduces a boundary-pivoted chunkwise kernel for numerically stable large-chunk training.
How it works
The base recurrence decays the prior state row-wise, erases the current key prediction, and writes a gated value. Muir adds a fifth gate γ_t that scales the normalized query only at readout. This keeps the key unit-normalized for stable erasure while allowing the model to attenuate or amplify state rows according to token context and expected retrieval horizon. The paper's sensitivity analysis shows that erase-gate changes are weighted by k_t,j² and the existing row energy, while write-gate changes act directly on v_t.
What the paper evaluates
The evaluation covers held-out FineWeb-Edu validation NLL, WikiText and LAMBADA perplexity, zero-shot commonsense benchmarks including PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OpenBookQA, and SIQA, plus 5-shot WinoGrande and TriviaQA. The main table spans 166M, 406M, 834M, and 1.34B parameters after 100B training tokens. Retrieval evaluations include synthetic single-needle, 5-hop variable-binding, multi-needle, and multi-value tasks; RULER tasks; six real-world document-extraction tasks under the Based metric; S-NIAH-1 passkey retrieval; and long-context tests up to 10M tokens.
Limit to keep in view
The main scaling study reaches 1.34B parameters, so the behavior of the rank allocation, readout gate, and FFN reallocation at substantially larger language-model scales remains unverified.
arXiv:2611.06425
D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding
D3D is a standalone diffusion drafter for asynchronous speculative decoding. It predicts candidate token blocks from the token prefix alone, using blockwise denoising combined with auxiliary autoregressive supervision on responses generated by the target model. Because drafting does not depend on fresh target hidden states, D3D can prepare successor blocks while the target verifies the current block. The authors report average decoding speedups of 5.00× with Qwen3-8B and 5.71× with Qwen3-32B across seven benchmarks. Progressive pruning from 36 to 31 drafter layers retains over 99% of mean acceptance length across three benchmarks.
How it works
D3D starts from pretrained autoregressive weights. For each prompt, the target model generates a response, and the drafter is trained on that response. The method samples anchor positions, places each visible anchor token at the start of a block, masks the following positions, and predicts the next response tokens in parallel. Multiple blocks are packed into a sequence: the prompt and response prefix share causal computation, while each block sees the relevant prefix and has bidirectional attention within itself.
What the paper evaluates
The authors evaluate Qwen3-8B and Qwen3-32B targets on GSM8K, MATH-500, and AIME25 for mathematical reasoning; HumanEval, MBPP, and LiveCodeBench-v6 for code generation; and MT-Bench for dialogue. They report decoding speedup relative to autoregressive decoding and mean output tokens per verification round, τ, including the target-generated token. The main D3D configuration uses block size 16, coverage C = 12, and fan-out F = 2. Evaluation uses the Transformers backend with greedy decoding in non-thinking mode. Asynchronous D3D, SSD, and PARD-2 use two H800 GPUs; synchronous DFlash, DSpark, and EAGLE-3 use one.
Limit to keep in view
The authors report that D3D, SSD, and PARD-2 use separate GPUs for target and drafter, and do not compare throughput per GPU with alternatives such as tensor-parallel target inference.
arXiv:2611.06372
Give Up to Get More: Priors, Not Policies, Govern Test-Time Compute Allocation
This paper studies how to divide a fixed budget of repeated LLM attempts across many inputs. It shows that giving harder inputs more attempts is only appropriate when the model already solves most inputs at the planned budget. For hard math, code, and formal-proof workloads, the optimal static allocation is inverted-U shaped: hopeless inputs receive little or no additional compute, while moderately solvable inputs receive the most. With a verifier, round-robin is difficult to beat without an informative prior from related-model runs on the same inputs. A related-model history prior raised solved share from 14% to 27% on the hardest reported workload, while cheap text and proxy-model priors usually failed.
How it works
The static problem assumes N inputs, pass rate p_i for input i, independent repeated attempts, and a total budget S=uN. If input i receives k_i attempts, its probability of being solved is 1-(1-p_i)^{k_i}. The paper maximizes expected solved-input coverage subject to the budget and a minimum of one attempt per input. The incremental value of the (k+1)-th attempt is Δ(p,k)=p(1-p)^k. Because this value is concave in the number of attempts for a fixed input, a greedy marginal-value allocation is optimal under the stated assumptions.
What the paper evaluates
The experiments cover MATH, GSM8K, CodeContests, MiniF2F formal proofs, PRM800K MATH, MATH500, GPQA, MMLU-Pro, and freshly generated hard MATH-500 samples from the Qwen2.5 family. The static evaluation uses public stored samples, with each problem's pool pass rate defining its coverage curve. Every allocation spends uN attempts and gives every input at least one attempt; uniform allocation is the baseline, while the oracle uses true pass rates and greedy marginal gains. The compared static rules include IAS calibration, k(1-p), another difficulty-proportional rule, and 21:30:45 pass-rate tiers. Table 1 shows the core hardness effect.
Limit to keep in view
Most evidence comes from offline replay of stored samples with independent attempts and binary, verifiable success; correlated retries, interactive self-correction, and open-ended generation are not tested.
arXiv:2610.42112
Subquadratic Attention with a Hierarchical Tree-Structured Indexer
Zyphra Hierarchical Attention (ZHA) replaces full query-to-KV-cache scoring with a recursively compressed tree indexer. Learned reference queries summarize contiguous key chunks, and each query traverses the tree using adaptive top-p routing before standard attention over selected leaves. On Qwen3.6-35B-A3B, ZHA retained near-dense performance across reasoning, agentic coding, and long-context retrieval while attending to 1%-6% of the KV cache in a deep configuration; the paper's roofline model predicts subquadratic scaling and large modeled advantages at billion-token contexts, although those ultra-long-context results are not empirically validated.
How it works
Tree construction is incremental during decode. With chunk size C, a first-level summary is added every C decode steps, a second-level summary every C^2 steps, and so on. The paper's roofline model estimates amortized construction overhead at around 1% or less. Each reference query produces a compressed key and bias that approximate the chunk's log unnormalized attention mass; summaries are recursively merged at higher levels. Routing starts at the top of the tree.
What the paper evaluates
The main evaluation converts Qwen3.6-35B-A3B using deep C4 L6 and wide C256 L1 trees, each with R=16 reference queries and 4B-token KL warmup. Routing thresholds p are evaluated at 0.8, 0.9, and 0.95. The suite covers AIME 2026, MMLU-Pro, GPQA Diamond, LongBench v2, RULER, and agentic coding evaluations including SWE-bench Verified; the paper also converts Nemotron-3.5-Lightning-30B-A3B to test a hybrid-NoPE architecture. ZHA stays close to dense attention on AIME 2026, MMLU-Pro, and GPQA Diamond, and both geometries are reported to remain competitive on agentic coding and long-context retrieval.
Limit to keep in view
The predicted latency and speedups at contexts approaching 1 billion tokens are model extrapolations; the paper explicitly states that ZHA's reasoning and retrieval quality at those lengths has not been empirically validated.
Read the full reviews
- Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout · arXiv:2611.08376
- D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding · arXiv:2611.06425
- Give Up to Get More: Priors, Not Policies, Govern Test-Time Compute Allocation · arXiv:2611.06372
- Subquadratic Attention with a Hierarchical Tree-Structured Indexer · arXiv:2610.42112
Source note: findings and limitations above come from the linked full paper reviews. Follow each original arXiv paper for the authors' complete methods and results.