GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR

📌 Highlights

GIST asks which parts of a fixed offline RLVR rollout pool deserve supervision. Its answer comes from analyzing the mismatch between the current policy and the Gibbs-optimal target of KL-regularized RLVR. The resulting selection rule treats verified outcomes asymmetrically: it draws correct responses from on-policy sampling and incorrect responses from sampling closer to the model’s mode, then selects tokens according to their probabilities under the frozen initial policy.

The authors report results for Qwen3-0.6B, Qwen3.5-2B, and Gemma-4-E2B-it on GSM8K, MATH-500, and MBPP-full, using both BOLT and SFT. GIST raises the reported average accuracy in every model/objective setting. In the controlled Qwen3-0.6B/GSM8K selector comparison, it leads seven baselines on Strict and Flexible accuracy; the teaser reports 66.6% fewer supervised tokens than training on all candidates and 31.5% fewer than random selection. These findings are evidence for the evaluated configurations, not a general guarantee that selective supervision will improve every offline RLVR task.

• Correct responses: sample on-policy; supervise low-probability tokens.
• Incorrect responses: sample near the policy mode; supervise high-probability tokens.
• Reported evidence covers three models, three benchmarks, and two offline objectives.
• The selection guarantee concerns mismatch at a fixed policy, not downstream training outcomes.

🎯 Introduction

Offline RLVR trains a language model on a fixed pool of responses whose outcomes can be checked automatically. Common approaches supervise every token in retained responses, although examples and token positions differ in how much they reveal about the target behavior. The paper frames the practical question as two linked choices: which responses should be kept, and which tokens in those responses should receive loss.

Prior approaches address portions of this design space, including rejection sampling of verified correct responses, reward-weighted objectives such as BOLT, and rollout or token subsampling based on other signals. The paper argues that these choices are not independent: whether a token is informative depends on both the verifier outcome and how the response was sampled. A low-probability token in a high-temperature sample may be noise, while one in a verified correct response may indicate a useful step the policy underweights; errors from low-temperature sampling can expose behavior near the model’s mode.

The paper’s objective is to derive response- and token-level selection principles from the mismatch between the policy and the Gibbs-optimal policy of KL-regularized RLVR, then approximate those principles using observable data-construction signals. It proposes GIST and evaluates it with BOLT and SFT across three models and three benchmarks. The main body reports that GIST improves the average accuracy in all model/objective settings and improves nearly all individual benchmark results.

🔬 Methodology

For a prompt, the paper defines RLVR as maximizing expected verifier reward while regularizing divergence from a reference policy. The resulting Gibbs-optimal policy reweights reference responses by exponentiated reward. The objective gap is a reverse KL to this target, while offline fitting can be expressed as a weighted likelihood corresponding to forward-KL target fitting. The authors therefore analyze a bidirectional KL mismatch and a curvature score that tracks where probability differences contribute most. At the reference policy, the analysis separates the two outcome classes: correct responses are underrepresented by the target and concentrate forward mismatch at low-probability tokens; incorrect responses are overrepresented by the policy and concentrate reverse mismatch on commonly generated responses and high-probability tokens.

GIST approximates these quantities without estimating the target probabilities or soft values directly. For each prompt, the initial policy generates 16 responses at temperature 1.0 and 16 at 0.3, without top-p or top-k truncation. A verifier labels each response. The prompt is retained only if it has at least four correct responses in the higher-temperature pool and four incorrect responses in the lower-temperature pool. GIST samples four of each class, uniformly without replacement, treating duplicates as distinct instances. It then scores token probabilities with the frozen initial policy and masks positions based on each response’s mean token probability, excluding EOS. The correct-response mask retains low-probability tokens, and the incorrect-response mask retains high-probability tokens. Masks are fixed during training; excluded tokens remain available as context but receive no loss.

The selected responses can be used with different offline objectives. In the reported experiments, BOLT weights responses using rewards and the full candidate pool, with weights computed before selection and not renormalized afterward. SFT assigns unit weight to each retained response. Both use weighted mean negative log-likelihood over masked positions. The paper notes that quotas are pragmatic choices, do not follow from the analysis, and exclude prompts that lack enough examples from both classes.

📊 Experiments

The evaluation uses GSM8K and MATH-500 for mathematical reasoning and MBPP-full for code generation. The paper evaluates Qwen3-0.6B, Qwen3.5-2B, and Gemma-4-E2B-it. For math it reports Strict accuracy, which requires the prescribed answer format, and Flexible accuracy, which also credits correctly extracted answers; for code it reports pass@1. The two offline objectives are SFT and BOLT, with BOLT using \(\beta=0.1\). Trained methods share 32 candidates per prompt, with 16 samples per temperature, and use validation-based checkpoint selection. The paper defines the reported average as the mean of GSM8K Flexible, MATH-500 Flexible, and MBPP-full pass@1.

Across the three benchmarks, adding GIST improves average accuracy for all three models under both objectives. The authors report average gains of 1.26–2.55 points over BOLT and 1.11–4.41 points over SFT. BOLT + GIST has the highest average for each model. On GSM8K, reported gains over BOLT reach 2.50 points Strict and 3.26 Flexible; gains on MATH-500 are smaller, 0.40–1.80 points. On MBPP-full, GIST gains 4.80 points for Qwen3-0.6B and 0.60 for Gemma-4-E2B-it, but trails BOLT by 0.60 for Qwen3.5-2B. The reported comparison with SFT shows gains for every model, benchmark, and metric; the paper notes that the large Gemma MATH-500 Strict improvement is mostly associated with answer-format compliance.

The controlled selector comparison holds the candidate pool and BOLT objective fixed on Qwen3-0.6B/GSM8K, using 1,593 eligible prompts. GIST scores 45.79 Strict and 49.73 Flexible, the highest values among Random, PODS, RFT-style, RSI-S, ETS, ProFit, and HES. Against all-candidate training on the same prompts, it improves Strict and Flexible by 1.97 and 3.03 points; against Random it improves by 2.65 and 1.13 points while supervising 31.5% fewer tokens. The teaser reports 66.6% fewer supervised tokens than training on all candidates. In ablations, removing pairing, masking, or Boltzmann weighting lowers Strict/Flexible results by 2.20/0.98, 2.04/2.12, and 1.97/1.59 points respectively. The authors also report that GIST retains 12,744 of 50,976 candidates and reaches its selected checkpoint in 2,394 optimizer steps versus 4,779 for all-candidate training; masked tokens still participate in forward and backward passes.

fig:teaser

Caption: Test Strict accuracy versus supervised tokens on Qwen3-0.6B/GSM8K. All methods use BOLT on the same candidate pool. GIST reaches the highest accuracy while supervising 66.6\% fewer tokens than training on all candidates and 31.5\% fewer than random selection.

Why It Matters: This controlled plot presents the paper’s central accuracy–token-budget comparison on a shared candidate pool.

fig:method

Caption: Overview of GIST (schematic). (a)~Rollouts on-policy ($ _h=1$) and near the mode ($ _l$). (b)~Correct responses are taken from $ _h$ and incorrect ones from $ _l$. (c)~The frozen initial policy scores tokens to mask both classes. (d)~Selected responses and masks are passed to an offline training objective.

Why It Matters: The schematic summarizes the rollout pairing, verifier-based response selection, token scoring, and downstream offline training.

🔮 Conclusion

The paper’s main conclusion is that offline RLVR data selection can be guided by where the policy differs from its Gibbs-optimal target. GIST operationalizes this with response selection conditioned on outcome and sampling temperature, followed by token masking based on initial-policy probabilities. Across the reported models and benchmarks, GIST raises average accuracy with both SFT and BOLT, and the shared-pool experiment favors GIST over seven selectors while using fewer supervised tokens than all-candidate training.

The evidence supports the method in the evaluated fixed-pool settings. The authors suggest that the analysis may apply to other offline objectives and to settings where data are re-selected as the policy changes, but those extensions are not demonstrated in the main-body experiments.

🛠️ Future Research Improvements

A direct next step suggested by the paper is to study data re-selection as the policy changes. GIST evaluates mismatch at the policy used to select the data, and the selection is performed once before training; refreshing candidates or masks could change both the mismatch being targeted and the cost of selection. The paper’s fixed-policy guarantee does not establish how such an iterative procedure would affect learning, so that outcome requires experiments.

The analysis leaves the class balance to a practical choice, and the implementation uses equal quotas of correct and incorrect responses. Future work could test how the quota and eligibility rule affect coverage of prompts, especially since prompts lacking enough examples from both classes are excluded. Additional evaluations could also examine whether the observed behavior transfers to more model families, task types, and offline objectives. These are validation directions motivated by the evidence boundary: the reported findings cover three models, three benchmarks, and BOLT and SFT, while the broader applicability is proposed rather than measured.

• Evaluate iterative candidate and mask refresh during training, tracking its added selection cost and effects on learning.
• Measure sensitivity to the equal response quotas and prompt eligibility criterion.
• Test transfer to other offline objectives, model families, and verifiable task domains before claiming broad applicability.

🏭 Potential Industry Use Scenarios

A plausible use is offline post-training for systems with automatically checkable outputs, such as mathematical reasoning or code generation, where a verifier can distinguish successful from unsuccessful responses. A team that already collects multiple rollouts per prompt could use GIST’s outcome and temperature pairing to build a smaller supervised subset, then apply an existing offline objective such as SFT or a reward-weighted likelihood objective. The shared-pool GSM8K comparison provides evidence for this kind of workflow in one evaluated setting, including fewer supervised tokens than all-candidate training.

Operational adoption would still require local validation. The evidence does not establish improvements for production traffic, tasks without reliable verifiers, changing user distributions, or latency-sensitive online systems. Engineers would need to account for verifier quality, the requirement that a prompt supply enough responses of both outcomes, and the initial-policy pass needed for token scores. The measured reduction in supervised tokens also does not imply an equal reduction in compute, since masked positions still enter the forward and backward passes. This makes GIST a candidate for controlled offline pipeline trials, with task-specific quality and compute accounting.

• Consider a pilot where responses have reliable binary verification and rollouts are already available.
• Track eligibility coverage, verifier errors, supervised-token counts, optimizer steps, and end-task metrics separately.
• Validate actual training compute because masked tokens remain in model passes.

💬 Critical Analysis

The paper offers a coherent link between theoretical mismatch analysis and a concrete data-construction procedure. A strength is that it treats response selection and token masking as coupled decisions and tests both through ablations. Its controlled selector comparison shares the candidate pool and objective, making the comparison more informative than a comparison where each method sees different data. The wider evaluation also checks two offline objectives and three model configurations, and the average metric improves in every reported model/objective setting.

The evidence has important limits. Most selector-level comparisons are on Qwen3-0.6B/GSM8K, while the wider benchmark table evaluates GIST against the unselected SFT and BOLT setups rather than all selector baselines. The response quotas and prompt eligibility rule are not derived by the theory, and eligibility excludes prompts without enough responses from both outcome classes. The theoretical guarantee concerns mismatch captured at the selection policy, not post-training performance; the empirical experiments are therefore essential and should not be treated as a direct consequence of the guarantee.

There are also implementation tradeoffs. GIST reduces the number of retained responses and supervised tokens in its controlled comparison, but masked positions still participate in forward and backward passes, and token scoring requires an additional initial-policy forward pass. The paper reports fewer optimizer steps to its selected checkpoint in the all-candidate comparison, while noting that the selected checkpoint occurs after more epochs; readers should distinguish these measurements from wall-clock or total compute claims. Finally, selection and masks remain fixed during training. The proposed extension to data re-selection as the policy changes is a reasonable research direction, but its value and cost remain unmeasured in the supplied main-body evidence.