Each section below summarizes one paper's reported method, experiments, and limits from its full review. The studies use different tasks and metrics; their results are not directly comparable.
arXiv:2611.06395
GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR
GIST derives a two-level data-selection strategy for offline reinforcement learning with verifiable rewards (RLVR) from the mismatch between a policy and the Gibbs-optimal target of KL-regularized RLVR. It selects verified correct responses from on-policy rollouts and incorrect responses from lower-temperature, near-mode rollouts, then masks tokens according to the initial policy’s probabilities: low-probability tokens in correct responses and high-probability tokens in incorrect ones. Across Qwen3-0.6B, Qwen3.5-2B, and Gemma-4-E2B-it on GSM8K, MATH-500, and MBPP-full, GIST improves the average benchmark accuracy for both BOLT and SFT. In a shared-pool comparison on Qwen3-0.6B/GSM8K, it leads seven selector baselines while supervising fewer tokens.
How it works
For a prompt, the paper defines RLVR as maximizing expected verifier reward while regularizing divergence from a reference policy. The resulting Gibbs-optimal policy reweights reference responses by exponentiated reward. The objective gap is a reverse KL to this target, while offline fitting can be expressed as a weighted likelihood corresponding to forward-KL target fitting. The authors therefore analyze a bidirectional KL mismatch and a curvature score that tracks where probability differences contribute most.
What the paper evaluates
The evaluation uses GSM8K and MATH-500 for mathematical reasoning and MBPP-full for code generation. The paper evaluates Qwen3-0.6B, Qwen3.5-2B, and Gemma-4-E2B-it. For math it reports Strict accuracy, which requires the prescribed answer format, and Flexible accuracy, which also credits correctly extracted answers; for code it reports pass@1. The two offline objectives are SFT and BOLT, with BOLT using \(\beta=0.1\). Trained methods share 32 candidates per prompt, with 16 samples per temperature, and use validation-based checkpoint selection. The paper defines the reported average as the mean of GSM8K Flexible, MATH-500 Flexible, and MBPP-full pass@1.
Limit to keep in view
The response quotas and eligibility rule are practical design choices, not values fixed by the theoretical analysis; excluding prompts without enough correct and incorrect responses may omit difficult cases.
arXiv:2611.05869
On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces
This paper develops convergence theory for constrained TD with Regularized Corrections and eligibility traces, including state-dependent and history-dependent trace parameters under non-i.i.d. off-policy data. It also introduces TDRC(λ)-Huber, which clips the correction term to reduce sensitivity to large Bellman errors while retaining the same target fixed point as TDRC(λ). The analysis uses coupled mean ordinary differential equations and stochastic-approximation weak convergence. For the Huber method, an integral small-gain condition gives uniqueness and global asymptotic stability; computable sufficient conditions show how sufficiently strong regularization can ensure convergence when the mean TD matrix is positive stable.
How it works
The paper considers policy evaluation with target-policy transition matrix P, behavior-policy transition matrix Pᵒ, state-dependent discounting, importance-sampling ratios, and a feature matrix Φ of full column rank. Eligibility traces obey e_t = λ_tγ_tρ_{t−1}e_{t−1}+ϕ(S_t). In the state-dependent case, λ_t=λ(S_t); in the history-dependent case, a finite memory state evolves as y_t=f(y_{t−1},S_t) and λ_t=λ(y_t,e_{t−1}). The resulting generalized Bellman operator has the fixed point v_π=T^(λ)v_π.
What the paper evaluates
The main body reports a numerical illustration, not a suite of empirical learning experiments. The central setup is a four-state off-policy prediction MDP: the target policy moves clockwise, while the behavior policy chooses clockwise and counterclockwise actions with equal probability. The behavior chain has a uniform stationary distribution; the target policy's transition-level importance ratios are 2 for clockwise transitions and 0 for counterclockwise transitions. The paper specifies γ=0.7, a two-dimensional feature matrix Φ, and a reward vector r_π.
Limit to keep in view
The convergence results concern constrained algorithms with diminishing stepsizes and a sufficiently large projection radius; the radius bounds may be conservative.
arXiv:2611.07375
When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO
This paper diagnoses constraint drift in Direct Preference Optimization when pairwise preferences conflict with pointwise criteria such as correctness, safety, or response length. Its analysis separates conflict pairs, which increase constraint cost, from matching pairs, which reduce it, and derives a pre-alignment diagnostic, rho*_eq, from dataset statistics. Conflict-aware DPO (CoDPO) uses that diagnostic to down-weight conflict pairs and applies an adaptive margin shift when residual drift appears during training.
How it works
Let the policy start at the reference policy, and let the implicit DPO reward margin be the difference between policy-to-reference log-ratios for the preferred and rejected outputs. The independent scorer is oriented so larger values are better, while the cost c is nonnegative and decreases with scorer quality. For each pair, Delta s = s(x, y_w) - s(x, y_l) and Delta c = c(x, y_w) - c(x, y_l). Conflict pairs have Delta s < 0 and Delta c > 0; matching pairs have Delta s >= 0.
What the paper evaluates
The evaluation spans six domains with different pairwise and pointwise signals. HelpSteer2 combines helpfulness ratings with response length; GSM8K combines explanation detail with answer correctness; ARC combines reward-model preference with answer correctness; Math-Calc is constructed from GSM8K with calculation density as the preference signal; TL;DR combines human preference with summary length; and PKU-SafeRLHF combines helpfulness with safety score. The first five are treated as conflict-dominated domains, while PKU-SafeRLHF is the self-correcting control. The primary full comparison uses HelpSteer2, GSM8K, ARC, and Math-Calc.
Limit to keep in view
The diagnostic relies on first-order approximations and an assumption that the geometry of matching and conflict updates is comparable; the paper reports a gap of 0.05 to 0.30 between rho*_eq and observed zero crossings.
arXiv:2611.06476
The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention
This empirical study tracks how length-normalized Direct Preference Optimization (DPO) with an SFT retention term changes structured-task capability as its preference margin grows. Across eight seeds of sparse LoRA fine-tuning on Qwen3.5-4B, the effect is task-selective: extraction remains equivalent to its baseline within a preregistered five-point margin, JSON tool calling declines slightly but does not establish equivalence, and router accuracy falls substantially. Router damage includes a transient format-break spike around margin 3. The study finds that almost all margin growth comes from suppressing rejected completions, while chosen reward stays approximately flat.
How it works
The study fine-tunes Qwen3.5-4B with LoRA on the eight full-attention layers, using rank 8 and 180 fixed optimizer steps across eight seeds. Length normalization averages token log probabilities over completion tokens. For each prompt and chosen/rejected pair, the implicit reward is policy log probability minus the SFT reference log probability; the preference margin scales their reward difference by β = 0.5. Training adds an assistant-only cross-entropy retention term on the chosen completion with weight λ = 0.5.
What the paper evaluates
The evaluation uses three 160-item structured tasks derived from SEC-filing queries: JSON tool calls with a function name and typed arguments; router classification into a fixed five-label set; and extraction of ticker, section, and filing date. The paper reports no verbatim overlap between evaluation prompts and training prompts, while noting domain overlap by design. Extraction shares ticker-by-section provenance with preference source filings for at least 81 of 160 items where provenance could be recovered; JSON shares a ticker for at least 131 of 160 items. These details matter when interpreting generalization.
Limit to keep in view
The evidence covers one 4B model, sparse LoRA on 8 of 32 layers, one objective with an SFT retention weight of 0.5, three structured tasks, and synthetic, easily separable preferences.
Read the full reviews
- GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR · arXiv:2611.06395
- On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces · arXiv:2611.05869
- When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO · arXiv:2611.07375
- The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention · arXiv:2611.06476
Source note: findings and limitations above come from the linked full paper reviews. Follow each original arXiv paper for the authors' complete methods and results.