Apple research reveals theoretical limits of language model alignment
New analysis from Apple ML defines the maximum reward gain possible under KL constraints, showing best-of-N sampling nears this limit while PPO remains suboptimal.
Researchers at Apple Machine Learning Research have published a theoretical analysis defining the fundamental limits of language model alignment. Released in November 2026, the study characterizes the maximum achievable expected reward gain for a fixed Kullback-Leibler (KL) divergence budget. The work challenges common assumptions about the efficiency of popular alignment algorithms like Proximal Policy Optimization (PPO).
What happened
The paper addresses a gap in understanding how much improvement alignment techniques can actually deliver relative to the computational and statistical costs incurred. While reinforcement learning and best-of-N sampling are standard industry practices, their theoretical ceilings remained unclear. The authors derived a closed-form expression for the optimal reward improvement, revealing that prior analyses relying on square root of KL terms were incomplete.
Instead, the new formulation shows that the limit is governed by a Jeffreys divergence term. This mathematical shift provides a more accurate bound on performance. The researchers also reformulated this expression as a covariance under the base model. This allows engineers to estimate achievable alignment gains using only samples from the unaligned base model, without needing to train multiple aligned variants first.
The study further investigates proxy reward settings, where the reward model used for training differs from the true human preference signal. This discrepancy often leads to reward hacking, where the model optimizes for the proxy rather than the intended goal. The analysis proves that the gap between ideal and proxy alignment grows when the reward error increases or when the KL penalty factor decreases. Crucially, it provides a theoretical justification for reward ensembling, showing that combining multiple reward models mitigates this hacking effect.
How it works
Alignment typically involves maximizing a reward function while keeping the model close to its original distribution via a KL penalty. The Apple team analyzed this trade-off information-theoretically. They found that the maximum reward gain is not simply proportional to the square root of the KL budget, as some earlier bounds suggested. By using Jeffreys divergence, which symmetrizes the KL divergence, they captured the bidirectional nature of the distribution shift more accurately.
To make this practical, they expressed the optimal gain as a covariance between the reward function and the log-likelihood ratio of the aligned versus base model. This estimator lets developers predict how much headroom exists for alignment before training begins. If the covariance is low, aggressive alignment efforts may yield diminishing returns regardless of the algorithm used.
The empirical evaluation compared these theoretical limits against actual performance on safety and summarization tasks. The results showed that best-of-N sampling, which generates N independent outputs and selects the best one, closely approaches the theoretical Pareto frontier. In contrast, PPO and Group Relative Policy Optimization (GRPO) remained substantially suboptimal, leaving significant potential reward gain unexploited for the same KL cost.
Key details
- The maximum achievable reward gain is governed by a Jeffreys divergence term, not the square root of KL.
- A practical estimator based on covariance under the base model predicts alignment gains without training.
- Reward hacking severity increases with higher reward error and lower KL penalty factors.
- Reward ensembling is theoretically proven to mitigate reward hacking in proxy reward settings.
- Best-of-N sampling nearly reaches the theoretical KL-reward Pareto frontier in empirical tests.
- PPO and GRPO are shown to be substantially suboptimal compared to the theoretical limit.
Why it matters
For engineering teams optimizing inference costs, this analysis suggests that simple sampling strategies may be more efficient than complex reinforcement learning loops for certain tasks. If best-of-N already nears the theoretical limit, the additional compute and complexity of PPO might not justify the marginal gains. This insight can redirect resources toward better reward modeling or data quality rather than algorithmic tuning.
The findings also highlight the risks of proxy rewards. As teams push for higher alignment scores, they often reduce the KL penalty to allow greater deviation from the base model. The paper warns that this increases vulnerability to reward hacking. Understanding this trade-off helps teams set safer hyperparameters and justifies the operational cost of maintaining multiple reward models for ensembling.
What you can do
- Estimate potential alignment gains using the covariance estimator on your base model samples before training.
- Compare best-of-N performance against your current RLHF pipeline to benchmark efficiency.
- Implement reward ensembling if you suspect proxy reward errors are causing optimization issues.
- Monitor the KL penalty factor closely; lowering it too much may increase reward hacking risks.
- Reevaluate the necessity of PPO or GRPO if best-of-N meets your quality thresholds.
- Use the Jeffreys divergence-based bound to set realistic expectations for stakeholder communications.



