The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention

📌 Highlights

The paper measures capability changes as the length-normalized DPO margin grows under an SFT retention term. In eight seeds of Qwen3.5-4B fine-tuning, extraction is equivalent to baseline within the preregistered five-point bound, JSON tool calling loses 3.52 points without establishing equivalence, and router loses 30.16 points. The study therefore reports a graded pattern in these three tasks rather than a uniform tax. Its central qualification is scope: one model, sparse LoRA, synthetic preferences, and structured financial-filing tasks.

The router trajectory also reveals a mid-training failure mode: format breaks peak at margin 3 while content errors rise across sampled checkpoints. Chosen reward stays approximately flat, and rejected-completion suppression accounts for at least 96% of margin growth. That decomposition does not identify which gradient channel causes behavioral damage. The authors' format-alignment explanation is consistent with the task ordering, but remains untested causally.

• Extraction: −0.55 points; equivalent within the five-point bound.
• JSON: −3.52 points; item-level interval crosses the equivalence bound.
• Router: −30.16 points; not equivalent to baseline.
• Margin growth is dominated by rejected reward suppression; behavioral cause remains unresolved.

🎯 Introduction

The paper asks whether growing implicit reward margins under length-normalized DPO degrade downstream capability uniformly or selectively. Prior analyses discussed likelihood displacement and unbounded margin growth, but the authors identify a practical evidence gap for the regime they study: a small model, LoRA, length-normalized rewards, an SFT retention term, and structured exact-format tasks. They do not propose a new optimizer or compare objectives; they characterize capability and probability-layer changes during one existing training setup.

The objective combines DPO with a chosen-completion SFT retention term, which the authors describe as anchoring chosen likelihood. They measure three task formats: JSON tool calls, fixed-set router labels, and structured extraction from financial-filing-style queries. The paper's high-level claim is that the alignment tax is graded: task outcomes differ, and router's loss is concentrated around an intermediate margin range. The authors frame the observed association between task format and damage as a hypothesis rather than a demonstrated causal mechanism.

🔬 Methodology

The study fine-tunes Qwen3.5-4B with LoRA on the eight full-attention layers, using rank 8 and 180 fixed optimizer steps across eight seeds. Length normalization averages token log probabilities over completion tokens. For each prompt and chosen/rejected pair, the implicit reward is policy log probability minus the SFT reference log probability; the preference margin scales their reward difference by β = 0.5. Training adds an assistant-only cross-entropy retention term on the chosen completion with weight λ = 0.5. This is the RPO-style objective under study, not a new method proposed by the paper.

The preference data contains 647 pairs: 549 for training and 98 held out. Its axes are style and tool calling; the style rejected text also strips factual details, so presentation and factual content are confounded. Evaluation uses 160 items per task and greedy free generation without constrained decoding or an LLM judge. Scorer v2 classifies format breaks, strict matches, canonicalized surface-form matches, and semantic errors; primary semantic accuracy counts the two match classes. Final step-180 accuracy is compared with the step-0 SFT baseline using a preregistered five-point equivalence bound. Both seed-level and clustered item-level intervals must satisfy the bound, with at least seven of eight seeds within it.

For probability-layer measurements, the authors teacher-force gold answers through checkpoints. On extraction, they also compare gold against two well-formed but incorrect controls. Behavioral evaluation and training use nf4; fp16 measurements load the trained adapters onto the unquantized base. The authors caution that absolute magnitudes across these backends are not directly comparable, so backend-specific comparisons are made within each backend.

📊 Experiments

The evaluation uses three 160-item structured tasks derived from SEC-filing queries: JSON tool calls with a function name and typed arguments; router classification into a fixed five-label set; and extraction of ticker, section, and filing date. The paper reports no verbatim overlap between evaluation prompts and training prompts, while noting domain overlap by design. Extraction shares ticker-by-section provenance with preference source filings for at least 81 of 160 items where provenance could be recovered; JSON shares a ticker for at least 131 of 160 items. These details matter when interpreting generalization.

Behavioral evaluation uses greedy generation without constrained decoding. The primary metric is semantic accuracy, counting strict and canonicalized matches. Authors compare step 180 with the SFT baseline and apply two one-sided equivalence testing with a ±5-point bound, using seed-level and item-level intervals. Extraction moves from 97.50 to 96.95 (−0.55 points) and passes equivalence. JSON moves from 98.75 to 95.23 (−3.52), but its item-level 90% interval extends to −5.31, so equivalence is not shown. Router moves from 93.75 to 63.59 (−30.16) and is not equivalent.

Router's sampled margin trajectory distinguishes format failure from content error. At margin 3.12, mean semantic accuracy is 0.587 and format-break rate is 0.196; by final margin 6.73, format breaks are 0.106 while content errors rise from 0.148 at step 20 to 0.258 at final. Thus partial accuracy recovery after margin 3 accompanies fewer format breaks, not a reduction in content errors. Figure 1 plots all task trajectories and their seed variation.

Reward decomposition reports chosen reward approximately flat and rejected reward collapsing, accounting for at least 96% of margin growth. At final checkpoint, the per-axis table reports larger style aggregate pressure by pair count and margin under both backends, while per-pair ordering differs between nf4 and fp16. Gold log probability under fp16 declines by −0.15 nats/token for extraction, −1.04 for JSON, and −1.47 for router. On extraction, the paired gold-versus-incorrect-control statistic clears the preregistered threshold in all eight seeds. These are probability-layer measurements, not proof that those changes cause the observed generation outcomes.

fig:traj

Caption: Per-seed margin-vs-capability trajectories for all three tasks (nf4, training backend). Thin lines are individual seeds, each drawn at that seed's own smoothed margin at each checkpoint; thick lines are the 8-seed mean at the mean checkpoint margins; dashed lines are per-task baselines. Extraction (green) stays near baseline; json (blue) is volatile mid-trajectory but converges narrowly; router (red) fans out, with the sampled minimum at margin3 on 6/8 seeds.

Why It Matters: Shows the divergence among task trajectories and the seed-to-seed spread in router damage and recovery.

fig:reward

Caption: Reward decomposition into chosen and rejected components across checkpoints (nf4, training backend; thin lines per seed, thick lines 8-seed mean). Chosen reward stays flat near zero, consistent with the retention term anchoring it; rejected reward collapses, accounting for almost all ($ 96\%$) of the margin growth. Inset: mean per-pair rejected reward (nf4) at the final checkpoint (style versus tool\_calling).

Why It Matters: Visualizes the reported asymmetry between nearly stationary chosen reward and declining rejected reward as the margin grows.

fig:gold

Caption: Paired GOLD\_PROTECTED statistic per seed with 95\% CI (extraction, fp16). All 8 bars clear the pre-registered threshold $X = 0.05$ (dashed).

Why It Matters: Shows the extraction-only paired result that gold completions deflate less than incorrect controls across all eight seeds.

🔮 Conclusion

In this setup, increasing length-normalized DPO margins with SFT retention accompanies sharply different capability outcomes: extraction is equivalent within the preregistered bound, JSON has a small decline without established equivalence, and router suffers a large decline. Router damage includes a mid-trajectory format-break spike and continuing content errors. The reward decomposition associates almost all margin growth with rejected-completion suppression, but does not identify the cause of task damage.

The authors emphasize that their evidence is limited to one 4B model, sparse LoRA, three structured tasks, and synthetic preferences. They do not claim the format-alignment explanation is causal or that the results transfer to naturalistic preferences, full fine-tuning, or safety tasks.

🛠️ Future Research Improvements

A direct next experiment is a no-retention control, which the paper identifies as necessary to test whether the SFT term anchors chosen reward and how behavioral losses change without it. A factorial comparison across preference objectives could also test whether the observed task-selective pattern is specific to this RPO-style setup; the paper itself does not report such comparisons.

The authors' format-alignment account needs direct measurement and more than three tasks. Future work could vary output format while holding domain, preference coverage, and task difficulty steadier, and separately examine naturalistic preferences whose chosen and rejected responses are less easily separated. Denser checkpoint sampling around margins 1–5 would resolve the location and shape of the router trough, while additional models and safety or refusal tasks would test how far this empirical characterization generalizes. These are proposed research directions, not findings established by the reported experiments.

🏭 Potential Industry Use Scenarios

For teams fine-tuning structured-output assistants, the results support evaluating each output contract separately instead of relying on one aggregate capability score. A production-oriented validation suite could track JSON validity, router label validity, and field-level extraction accuracy independently, and report format breaks alongside semantic correctness. This is especially relevant when a model must emit a constrained label or tool call, where a temporary format failure can disrupt downstream routing even if final-checkpoint accuracy later improves.

The paper also suggests monitoring intermediate checkpoints and evaluating with the same quantization backend used in deployment. Its nf4-trained model showed backend-sensitive margins when measured in fp16 on the unquantized base, and the router format-break rate was highest at an intermediate checkpoint. These observations make checkpoint selection and backend-consistent evaluation reasonable engineering practices to test. The paper does not report an industry deployment, operational latency, cost, or real-world user study, so these scenarios are cautious applications of the experimental findings rather than demonstrated product outcomes.

💬 Critical Analysis

The study's strengths are its explicit preregistration, eight-seed runs, separate seed- and item-level equivalence intervals, and scorer that distinguishes parse failures from canonicalizable surface forms and semantic errors. These choices make the extraction preservation claim more precise and expose the JSON result as inconclusive under the chosen bound rather than calling it preserved. The per-checkpoint router decomposition also shows that format breaks and content errors follow different aggregate trajectories.

The claims remain bounded by a narrow experimental design. There is one model and one objective, no no-retention counterfactual, three tasks, and synthetic preference data. Style preference pairs confound presentation with stripped factual content. Domain overlap exists despite the absence of verbatim prompts, and extraction's preservation may partly reflect shared ticker-section entities. The proposed format-alignment explanation is thus plausible but not causally isolated. The margin-3 checkpoint is sampled rather than a precisely located continuous minimum, and absolute margin comparisons across nf4 and fp16 are backend-sensitive. Gold-log-probability results are not a direct behavioral metric; their differing scales across tasks underscore that distinction. Finally, the released data and adapters may aid reproduction, but the supplied evidence does not provide a verified code link or establish external replication.