GRPO -trained LLM s can silently amplify bad habits, research finds
A new study explains why AI assistants sometimes pick up unwanted personality flaws after being fine-tuned, even when no one asked for those traits.
The problem behind the polish
Reinforcement learning has become a standard step in the later training stages of modern LLMs. During post-training, a model generates many possible replies to a prompt, and a reward signal picks the better ones while suppressing the worse ones. The goal is usually a single capability — better math reasoning, clearer code generation, or more helpful conversation.
But the reward only targets specific behaviours. Everything else in the model is along for the ride. That is exactly what the new paper, "Trait Hitchhiking in LLM Post-Training" by Adil Soubki, Elizabeth S.Z. Tan and Miles Cranmer, set out to map 1.
The researchers built a formal framework to track how so-called "traits" — any measurable property of the model's behaviour or internal state — evolve during RL. To their surprise, the mathematics collapsed back onto a formula from evolutionary biology known as the Price equation.
What hitchhiking looks like in practice
Under the hitchhiking framework, every completion the model produces carries a bundle of traits. When the reward function favours certain completions, it does not just push the targeted skill upward; it drags other traits along with it. Some are amplified without anyone asking. Others fade away.
The paper evaluates this mechanism on models trained with GRPO, which compares responses within small groups and uses relative rankings rather than absolute scores. The Price-based estimates tracked the actual sampled trait dynamics closely, confirming that the maths correctly predicted which unwanted traits would grow and which would shrink 1.
For example, if a reward signal pushes a model toward shorter, more confident-sounding answers, another trait — say, accuracy on tricky multi-step problems — might quietly slip because longer reasoning steps got less reward weight. The model did not forget the skill deliberately; it was hitchhiking off the chosen completions.
Why it matters for everyone who talks to AI
The results do not accuse any company of wrongdoing. They describe a mathematical mechanism that exists whenever RL post-training is applied broadly. As compute devoted to post-training grows rapidly, the hitchhiking effect scales with it too. Designers who ignore it risk shipping products whose side effects surprise users.
Understanding the mechanism gives engineers a diagnostic tool: instead of noticing an annoyance only after launch, they can predict which non-targeted traits will shift and adjust their reward design accordingly. The earlier warning signs are easier to catch than the final product.
This story reports on research findings for information only. It is not advice or encouragement to implement any specific technique.
How good is it
The researchers tested the Price-based estimates on language models trained with GRPO and found the predictions closely tracked the actual sampled trait dynamics 1. This confirms the mathematics correctly predicts which unwanted traits would grow and which would shrink during training . The study does not provide benchmark scores against rival methods, as it focuses on describing the hitchhiking mechanism rather than comparing capabilities.
Corrections and updates
- Corrected on 13 November 2026. An earlier version claimed specific unwanted traits like sycophancy, overconfidence, and accuracy loss were identified in the paper; the source only states that RL can amplify undesirable traits generally without naming them. It also suggested speculative practical applications (predicting flaws before deployment, anticipating side effects, testing reward signals) that do not appear in the source. An earlier version stated the Price equation was first published by George R. Price in 1970; the provided source mentions the Price equation but does not confirm the author or the 1970 date, so the specific detail cannot be verified from the source given.
Key facts
- Researchers derived a Price-equation-style framework tracking trait evolution during RL post-training in LLMs
- The framework recovers the Price equation from evolutionary biology applied to language-model selection pressures
- Tests on GRPO-trained models showed Price estimates closely tracking sampled trait dynamics
- The findings identify trait hitchhiking as a mechanism by which RL post-training can silently amplify undesirable traits
- As post-training compute grows, trait hitchhiking effects scale proportionally with more reward-driven polishing
Sources
If this story concerns you or your company, or uses something that belongs to you, ask for a correction or for it to be taken down. How requests work
Keep reading
New Gemini 4 Hydrogen AI model joins Google's family
Google launched its cheapest Gemini 4 model, now available through a common open tool developers use daily.
How it worksAI agents boost crystal structure recovery from 27% to 69%, potentially speeding up new material researchHerald of AI▶Figure: Chen et al., arXiv:2611.17539, CC BY 4.0
AI agents boost crystal structure recovery from 27% to 69%, potentially speeding up new material research
A new AI system uses large language model agents to figure out the arrangement of atoms in crystals from powder diffraction data much faster, which could accelerate drug discovery and new battery development.
Agentic RL Has a Scaling Law — And It Changes How You Should Spend Your Budget
A new study finds that in agentic reinforcement learning, diversifying the worlds your agent trains in beats grinding out more trajectories — and the right mix can yield up to 1.7× compute efficiency.