When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO

📌 Highlights

The paper identifies a concrete failure mode in heterogeneous preference supervision: a pairwise-preferred response can be worse on a pointwise criterion, and standard DPO then pushes probability toward a response with higher constraint cost. The proposed explanation is not simply that conflicts are common. Instead, drift depends on the balance between conflict-side pressure and matching-side repair. This distinction leads to rho*_eq, a pre-alignment diagnostic based on the conflict rate and average cost gaps.

CoDPO uses the diagnostic in two stages. Before training, it down-weights conflict pairs by severity while preserving matching pairs. During training, it shifts the DPO margin according to each pair's cost difference and increases the shift if the estimated constraint risk rises. The main evidence is consistent across the paper's six-domain diagnostic study: drift increases with conflict exposure on five conflict-dominated domains, while PKU-SafeRLHF behaves as the predicted self-correcting control.

The strongest practical result is the four-domain comparison on HelpSteer2, GSM8K, ARC, and Math-Calc. CoDPO reports lower drift than vanilla DPO while increasing P(chosen), and it retains more preference learning than methods that aggressively enforce or nearly eliminate constraint changes. The principal caveat is that the diagnostic is a first-order approximation whose zero-crossing prediction is close but not exact.

• Conflict and matching examples exert opposing effects on expected constraint cost.
• rho*_eq decides whether intervention is needed before alignment begins.
• CoDPO retains conflict data instead of deleting every conflicting pair.
• Adaptive margin shifting addresses residual drift after coarse calibration.
• The reported gains are strongest as a drift-preference trade-off, not as absolute cost minimization.

🎯 Introduction

Preference optimization methods learn from pairwise comparisons in which y_w is preferred to y_l, but practical alignment systems frequently impose additional pointwise requirements. A response may be preferred because it is more detailed or helpful while simultaneously being longer, incorrect, or less safe under an independent scorer. The paper calls this heterogeneous supervision. In the studied data, such conflicts are substantial: the preferred response is the longer one in 57% of HelpSteer2 pairs, and it is incorrect in 71% of GSM8K pairs and 58% of ARC pairs.

Standard DPO applies the same preference objective to matching and conflict pairs. The authors argue that this creates constraint drift: conflict pairs tend to raise expected constraint cost, whereas matching pairs tend to repair it. Existing constrained DPO, Lagrangian, multi-objective, and filtering approaches provide ways to trade preference performance against a pointwise cost, but their intervention strength is generally chosen by the practitioner or adjusted from an aggregate target. The paper's objective is earlier and more specific: determine from the dataset whether intervention is required, estimate how strong coarse correction should be, and then correct residual drift during optimization.

The paper therefore frames the problem as a pressure-repair balance. This framing matters because conflict frequency by itself is insufficient. Two datasets with similar conflict rates can have different cost-gap magnitudes and consequently different drift behavior. The proposed diagnostic and CoDPO method are designed to expose and control that balance before and during training.

🔬 Methodology

Let the policy start at the reference policy, and let the implicit DPO reward margin be the difference between policy-to-reference log-ratios for the preferred and rejected outputs. The independent scorer is oriented so larger values are better, while the cost c is nonnegative and decreases with scorer quality. For each pair, Delta s = s(x, y_w) - s(x, y_l) and Delta c = c(x, y_w) - c(x, y_l). Conflict pairs have Delta s < 0 and Delta c > 0; matching pairs have Delta s >= 0. With conflict rate p_c, the DPO gradient is partitioned into matching and conflict contributions.

The first-order analysis gives a lower bound on the expected change in constraint risk. The terms mu_0 and kappa_0 aggregate matching-side repair and conflict-side pressure, respectively. When p_c kappa_0 exceeds (1-p_c) mu_0 and the step is sufficiently small, the bound becomes positive, identifying a drift regime. At initialization the DPO learning signal is shared across pairs, and an assumption about comparable per-unit geometry allows the model-dependent terms to cancel approximately. The resulting rho*_eq compares average matching cost reduction with average conflict cost increase. A value below 1 calls for intervention; a value at least 1 leaves plain DPO active.

The coarse control uses the asymmetric weighting family w_(tau,p)(Delta s). Matching pairs retain weight 1, while conflict weights decrease exponentially with severity. For a fixed p, CoDPO finds tau* so the average conflict weight equals the clipped diagnostic. This is a data-level calibration solved by bisection and does not require gradient computation. The paper uses p = 8 in its experiments.

The fine control addresses the fact that the balance changes as the policy moves away from the reference. CoDPO shifts the DPO margin by lambda Delta c: costly preferred outputs receive a larger margin and are pushed less, while beneficial preferred outputs receive a lower margin and are pushed more. The shift can be implemented by tilting the reference policy against high-cost outputs, because the pairwise normalizer cancels. CoDPO estimates constraint risk from normalized policy probabilities over the two candidate responses. It initializes lambda at alpha lambda_tilt, checks the estimate every K steps, raises lambda when risk increases, and caps it at lambda_tilt. The final objective combines the fixed pair weights with the adaptive margin shift.

The resulting algorithm has the same per-optimization-step structure as vanilla DPO because weights, cost gaps, and lambda enter as scalar terms. The additional periodic risk evaluation uses fixed training pairs and no generation. This design makes the proposal suitable as a data-and-loss plug-in for DPO-style trainers, IPO, SimPO, and KTO, subject to the availability of a pointwise scorer or cost.

📊 Experiments

The evaluation spans six domains with different pairwise and pointwise signals. HelpSteer2 combines helpfulness ratings with response length; GSM8K combines explanation detail with answer correctness; ARC combines reward-model preference with answer correctness; Math-Calc is constructed from GSM8K with calculation density as the preference signal; TL;DR combines human preference with summary length; and PKU-SafeRLHF combines helpfulness with safety score. The first five are treated as conflict-dominated domains, while PKU-SafeRLHF is the self-correcting control. The primary full comparison uses HelpSteer2, GSM8K, ARC, and Math-Calc.

The default setup uses Qwen2.5-7B-Instruct with LoRA and DPO beta = 0.1; CoDPO uses p = 8, K = 500, and alpha = 0.1. The paper also tests DeepSeek-R1-Distill-Llama-8B, Mistral-7B-Instruct-v0.3, and Qwen3-8B, plus IPO, SimPO, and KTO as alternative preference losses. Evaluation is performed on the two candidate outputs in each preference pair. Expected constraint cost is computed from policy likelihoods over the candidates; drift is expected cost relative to the reference model; preference learning is P(chosen); and generation quality is assessed with position-debiased JudgeLM-13B pairwise comparisons.

The drift sweep varies average conflict weight rho over 0, 0.3, 0.5, 0.7, and 1.0. In all five conflict-dominated domains, drift rises monotonically with rho, is negative at rho = 0, and is positive at rho = 1. HelpSteer2 moves from -46 drift at rho = 0 to +20 at rho = 1 while P(chosen) rises from .489 to .566. GSM8K moves from -.138 to +.059 while P(chosen) rises from .358 to .453. ARC moves from -.116 to +.019 while P(chosen) rises from .473 to .532. Math-Calc moves from -.136 to +.026 while P(chosen) rises from .516 to .652. TL;DR moves from -5.1 to +1.0 while P(chosen) rises from .479 to .567. PKU-SafeRLHF remains near zero drift across the sweep, consistent with rho*_eq = 1.0.

In the four-domain baseline comparison, CoDPO reports +18.7 drift and .640 P(ch) on HelpSteer2, +.082 and .551 on GSM8K, +.046 and .622 on ARC, and +.066 and .757 on Math-Calc. Vanilla DPO reports larger drift in every domain: +59, +.129, +.055, and +.071 respectively, with lower P(ch) values of .566, .453, .532, and .652. SafeDPO and PD-DPO reduce cost below the reference but generally retain less preference learning; C2-DPO stays near the initial point; MODPO and Soups expose a tunable frontier. The authors report that among settings with drift comparable to or below CoDPO, CoDPO has the highest P(ch) in all four domains.

The component ablation supports both parts of the design. Coarse control reduces drift in every tested domain, model, and alternative loss; adding the margin shift reduces drift further in all four primary domains, with the largest extra reduction on GSM8K and the smallest on HelpSteer2. On HelpSteer2, the judge records 75.5% ties between CoDPO and vanilla DPO, with CoDPO wins at 14.1% and opposite wins at 10.5%. The p sweep reports drift reduction increasing from 22% to 75% while P(chosen) stays essentially constant at .640-.644. These results support the mechanism, although the strongest ablation claims are concentrated on the reported training-pair evaluation.

fig:teaser

Caption: Figure 1: Overview of CoDPO. (a) Under DPO, conflict pairs push the constraint cost up and matching pairs pull it down; rho*_eq, computed before training, compares the two. If the push dominates (rho*_eq < 1), CoDPO intervenes; otherwise the data are self-correcting and plain DPO is used. (b) Conflict pairs are down-weighted by severity, with tau* set from rho*_eq. (c) During training, the DPO margin is shifted, more strongly when the constraint-risk estimate rises. Curves in (b) and (c) are schematic.

Why It Matters: The figure summarizes the full control loop: diagnose pressure versus repair before training, calibrate conflict weights, then respond to residual drift during optimization.

🔮 Conclusion

The paper presents constraint drift as a balance between conflict-side pressure and matching-side repair in DPO. From this balance it derives rho*_eq, a pre-alignment diagnostic that correctly selects the intervention regime across the evaluated domains, including the self-correcting PKU-SafeRLHF control. CoDPO then combines diagnostic-calibrated conflict weighting with an adaptive cost-dependent margin shift.

The reported experiments show a favorable drift-preference trade-off: CoDPO reduces drift relative to vanilla DPO while retaining or improving P(chosen) across the four primary comparison domains. The method also transfers as a coarse data-level plug-in to other models and preference losses. Its practical promise is strongest for builders who need to preserve useful preference signals while constraining a measurable pointwise cost.

🛠️ Future Research Improvements

The paper explicitly identifies dynamic tracking of the pressure-repair balance as a natural next step. The current diagnostic is computed at initialization, while the reported HelpSteer2 analysis shows that the model-dependent balance moves from rho*(0) = 0.710 to 0.603 as training changes the learning signal. A future method could recompute a diagnostic during training or directly estimate the evolving conflict and matching contributions rather than relying on a fixed initialization approximation.

A second direction is to evaluate preference pairs generated during training. The current calibration and risk estimate use the fixed responses already present in the dataset, which makes the procedure computationally inexpensive but does not test whether the same balance remains valid as prompts, candidates, or annotator distributions change. Online pair generation would also require deciding how often to refresh cost statistics and how to prevent noisy scorer changes from causing unstable lambda updates.

Further work should examine held-out prompts and freely generated outputs, broader scorer families, and more systematic choices of p, K, and alpha. The existing p ablation on HelpSteer2 shows that larger p reduced drift while leaving P(chosen) nearly constant, but the experiments use p = 8 without domain-specific tuning. Establishing whether that behavior generalizes is important before treating the weighting shape as a default.

🏭 Potential Industry Use Scenarios

CoDPO is most directly applicable to production alignment pipelines where preference labels coexist with a measurable operational cost. A conversational assistant could use it when helpfulness preferences conflict with response length or correctness; an enterprise reasoning system could apply it when detailed explanations compete with answer correctness; and a summarization service could use it when human preference conflicts with summary length. In each case, the pointwise scorer supplies the cost gap used for diagnosis and control.

Safety-sensitive deployment is another plausible scenario, although the paper's PKU-SafeRLHF result is primarily a self-correcting control rather than evidence that CoDPO solves every safety problem. A team could first calculate rho*_eq to determine whether intervention is necessary, then retain useful preference examples while attenuating those that prefer a higher-cost response. The fixed-pair risk estimate can run inside an existing training loop without generation, which may reduce operational overhead compared with controls that require repeated sampling.

For engineering teams, the most attractive deployment pattern is an offline gate before expensive alignment runs: measure conflict rate and cost gaps, inspect whether the dataset is predicted to self-correct, and calibrate weights only when needed. Production adoption should still include held-out generation tests, scorer validation, and monitoring for cases where the pointwise cost is an imperfect proxy for the business or safety objective.

💬 Critical Analysis

The paper's strongest contribution is diagnostic clarity. Rather than treating conflict rate as a sufficient statistic, it separates how often conflicts occur from how strongly they push the pointwise cost. That yields an interpretable quantity that can be computed before training and tested against an observed drift curve. The experiments support the proposed mechanism: drift increases monotonically with conflict weight in the five conflict-dominated domains, the intervention regime matches the diagnostic in all six domains, and the self-correcting control remains nearly stable.

The method also has a coherent engineering decomposition. Coarse weighting preserves matching-pair repair while reducing conflict pressure, and the adaptive margin shift responds to drift that the initialization approximation cannot anticipate. The ablation shows that both components matter, with the combined method reducing drift beyond coarse control in each primary domain. The baseline table further supports the paper's intended objective: methods that enforce very low or negative drift can sacrifice preference learning, whereas CoDPO reports a more favorable compromise.

The main concern is the distance between the theoretical condition and the implemented diagnostic. The proof uses small-update behavior and regularity assumptions, while rho*_eq additionally assumes comparable geometry between matching and conflict examples. The reported zero-crossing gaps of 0.05 to 0.30 are consistent with a useful approximation, but they also show that the diagnostic is not an exact threshold. The method therefore should be treated as a calibration rule whose output requires empirical monitoring, not as a formal guarantee for arbitrary training regimes.

The evaluation leaves several questions for builders. Expected cost and P(chosen) are measured from the two candidate responses in each preference pair, so they do not fully characterize free-generation behavior, held-out generalization, or cases where the scorer is misaligned with the deployment objective. The p ablation is informative but conducted on one domain, and the choice of K = 500 and alpha = 0.1 may interact with dataset size and optimization schedule. Finally, the paper tests alternative models and losses but does not establish whether the same diagnostic remains reliable when preference data are generated online. These gaps do not invalidate the reported results, but they define the validation work needed before relying on CoDPO in a high-stakes production alignment pipeline.