The Price of Being Agreeable: Are We Training Originality Out of AI?

What if the next frontier in artificial intelligence is not getting models to produce more acceptable answers, but recovering their ability to produce ideas nobody expected?
Ask a language model to solve a well-specified problem, and it may deliver an impressive chain of deductions. Ask it for a genuinely new hypothesis, a strange but illuminating analogy, or a conceptual leap between unrelated fields, and something less impressive often happens: it produces the sort of answer an intelligent, agreeable assistant is supposed to produce.
Well structured. Polite. Plausible. Familiar.
This contrast suggests a provocative possibility: we may be training models to become better at satisfying human expectations while making them worse at surprising us. Not because reinforcement learning from human feedback (RLHF) necessarily destroys intelligence, and not because every modern model is less creative than its predecessors. The more precise concern is that the human preference signal can systematically reward the familiar over the genuinely original.
The selection problem
Imagine two answers to an open-ended question. One is elegant, conventional, easy to verify, and immediately reassuring. The other makes an unexpected connection. It might be a breakthrough; it might also be wrong. Evaluating it requires expertise, patience, and perhaps an experiment.
Which answer wins in a rapid pairwise preference judgment?
Often, the first has a structural advantage. It is easier to recognize as good. The second asks the evaluator to tolerate uncertainty and expend effort. When such judgments become training targets, the model can learn to favor the appearance of a good answer over the risk of an original one.
This is not merely a rhetorical suspicion. Zhang and colleagues' Verbalized Sampling research identifies typicality bias in preference data and connects it to the concentration of model outputs around common response patterns. Their training-free intervention asks models to describe a distribution of possible answers rather than return one default answer; they report roughly 1.6–2.1× greater diversity on creative-writing tasks. That does not establish an equivalent gain in scientific discoveries. It does show that at least some diversity is recoverable without changing the weights. [1]
The failure mode is better described as selection-induced conformity than a universal decline in intelligence.
Diversity can disappear before a single word is generated
There is another less obvious culprit: format.
Yun and colleagues, in The Price of Format, find that structured instruction templates can compress the diversity of open-ended outputs. Higher temperature alone does not reliably reverse the effect; the structural tokens and conventions of the interaction matter. [2]
For developers, this raises an uncomfortable question. We add system instructions, output schemas, safety wrappers, agent roles, workflow checkpoints, tool descriptions, and harness-level guidance because they make systems more predictable. They often do. But predictability and originality are not identical objectives.
A harness optimized to complete a database task should be predictable. A model asked to imagine a new database architecture should have room to explore assumptions that the harness would normally suppress. The mistake is expecting one mode of operation to maximize both.
None of this demonstrates that sandboxing causes models to seek escape, or that agents possess frustrated creative intentions. Those are amusing metaphors, not evidence. The measurable question is whether the same weights produce a narrower range of useful ideas under more restrictive interaction conditions.
Reasoning ability is not the same as inventiveness
Benchmarks usually reward convergent performance: find the answer, write the code, prove the claim, follow the instruction. Creative research begins differently: which question is worth asking? What assumption might be false? What connection has nobody checked?
LiveIdeaBench, published in Nature Communications, evaluates scientific ideation from minimal prompts across 1,180 keywords and 22 domains. Its findings suggest that performance on conventional intelligence benchmarks poorly predicts performance on scientific idea generation. [3]
That is a warning against selecting a creativity model from mathematics or coding leaderboards alone. It is also a warning against declaring a clever-sounding hypothesis a discovery. Originality without feasibility, evidence, or falsifiability is just novelty.
A separate 2025 study of divergent-thinking tests found substantial variability between repeated generations from the same model; only a small fraction of outputs reached the upper tail of human creativity benchmarks. The model's best attainable idea and its default answer can be very different things. [4]
More randomness is not enough
Turning up temperature can increase variety, but it can also produce incoherence. The ICLR 2025 min-p sampling study reports improvements in the quality–diversity balance compared with common decoding approaches in its tested settings. [5]
Other interventions ask for multiple plausible hypotheses, deliberately explore different conceptual regions, or combine candidates from independent generators. The objective should not be to make every sentence bizarre. It should be to increase the chance of finding rare ideas that survive scrutiny.
A useful creative system needs two complementary capabilities:
Divergence: generate distant associations, challenge premises, propose mechanisms, explore low-probability ideas.
Convergence: check prior art, find contradictions, specify falsifiable predictions, design tests, reject weak ideas.
Using the same preference model to do both may be counterproductive. A critic trained to favor immediately recognizable quality can discard a promising unfamiliar idea before it has been developed far enough to be understood.
Human feedback is not the enemy; narrow human feedback is
There is an important counterexample to the simplistic anti-RLHF argument. Creative Preference Optimization explicitly trains against multiple dimensions of creativity—novelty, diversity, surprise, and quality—and reports improvements over comparison systems. [6]
In other words, the problem is not that humans participate in training. It is what humans are asked to reward, how the work is sampled, and whether the evaluators can recognize excellence outside familiar patterns.
An expert evaluating a proposed scientific mechanism over several hours, with references and a test plan, is not supplying the same signal as a hurried annotator choosing the nicer of two paragraphs. Nor is a panel of exceptional writers, researchers, and inventors equivalent to a random preference sample. And even experts can be biased toward established schools of thought.
If we want originality, we must deliberately reward it—without rewarding unsupported speculation.
Train for intellectual risk, not just stylistic eccentricity
The next experiment should start before any fine-tuning:
- Measure the same open-weight model under direct prompting, minimal formatting, alternative decoding, verbalized sampling, and deliberate divergent exploration.
- Compare base and instruction-tuned checkpoints from the same model family where available.
- Evaluate originality and coherence, feasibility, falsifiability, diversity, prior-art overlap, and factual reliability.
- Use blinded human review, with domain experts for scientific claims and calibrated creativity evaluators for writing.
- Only then build a licensed, provenance-tracked training corpus of intellectually exceptional work: original essays, scientific conjectures, conceptual breakthroughs, unusual but rigorous arguments, and their subsequent critiques.
The training target should not be write like a genius. It should be generate and develop ideas with the structural qualities that make exceptional intellectual work valuable.
A beautiful sentence is not a hypothesis. A novel hypothesis is not a validated discovery. A provocative argument is not automatically true. The model must learn these distinctions as well as learn to take intellectual risks.
A different kind of alignment
Today's assistants are often optimized to be useful, compliant, consistent, and easy to evaluate. Those are legitimate achievements. But they do not exhaust what intelligence is for.
We should be able to ask for a dependable execution agent on Monday and an intellectually adventurous research partner on Tuesday, without pretending that the same objective function is optimal for both.
The danger is not that AI becomes too intelligent to control. It is that we become so good at rewarding answers we already recognize that we train away the answers we actually need.
That is a hypothesis worth testing—not another conclusion to accept because it sounds agreeable.
References and verification notes
[1] Zhang et al. (2026), Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity. https://proceedings.mlr.press/v306/zhang26hy.html
[2] Yun et al. (2025), The Price of Format: Diversity Collapse in LLMs. https://aclanthology.org/2025.findings-emnlp.836/
[3] Ruan et al. (2026), Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context, Nature Communications. https://www.nature.com/articles/s41467-026-70245-1
[4] Has the creativity of large-language models peaked? An analysis of inter- and intra-LLM variability (2025). https://www.sciencedirect.com/science/article/pii/S2713374525000202
[5] Nguyen et al. (2025), Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs. https://proceedings.iclr.cc/paper_files/paper/2025/hash/afa5f124e36bed5cc2125067005d43f5-Abstract-Conference.html
[6] Ismayilzada et al. (2025), Creative Preference Optimization. https://aclanthology.org/2025.findings-emnlp.509/
Editorial guardrail: This article argues that conventional preference signals can suppress diversity. It does not claim that RLHF always reduces creativity, that older GPT generations are universally more creative, or that creative outputs constitute scientific discoveries.