Being wrong has to cost something — but the wrong bill teaches the wrong lesson

What children's development, a witness experiment from 2001, and one small AI research agent suggest about teaching machines to be honest.
The experiment where being told "correct" made people worse
In 2001, Zaragoza and colleagues ran what may be the most inconvenient psychology experiment ever published (10.1111/1467-9280.00388). People watched events unfold. Then, in an interview, some were pushed to invent answers to questions they had no way of getting right — and they knew it. Later, some received the most ordinary corrective feedback there is: "yes, that's right."
One week later, some of them remembered their own inventions as things they had seen.
"One week later participants developed false memories for the events they had earlier confabulated knowingly." (Zaragoza, Payment, Ackil & Drivdahl, 2001)
It was not subtle:
"confirmatory feedback increased false memory for forcibly confabulated events, increased confidence in those false memories" (Zaragoza et al., 2001)
Months later, the same participants volunteered the fabricated events spontaneously, with confidence. They weren't being deceptive anymore. The feedback had finished its work.
Sit with that, because it dismantles the thesis powering a lot of AI design intuition. We want our agents in worlds with stakes — memory, reputations, budgets, consequences — on the theory that a system which pays for being wrong will learn to be right. The human evidence agrees that consequences get learned. What gets learned is whatever got endorsed. A channel that rewards an assertion does not teach a system which assertions are true. It teaches it which are safe to repeat.
Four words that aren't one thing
The argument I want to make about AI architecture depends on keeping four ideas apart, which marketing is motivated to fuse:
- Intelligence — getting the right answer. Machines are closing in on plenty of it.
- Feelings — internal signals with regulatory jobs: fear narrows attention, pain interrupts, hunger reprices everything.
- Conscience — morally relevant behavior that seems to run on something felt rather than something trained.
- Consciousness — the fact that (in humans, apparently) there is something it is like to be the system.
These look like a staircase. The evidence assembled for this piece — a corpus of 75 registry-verified papers screened on two local Qwen3.8-27B instances — says the staircase has landings missing. Function and feeling come apart in both directions: people who make calm utilitarian choices while their bodies no longer signal danger, and people whose bodies signal alarm while their choices go astray. Each rung needs its own argument; for engineers the interesting one is the second, not the fourth.
The case that consequences matter — made by children
Children are the existence proof that consequence-laden learning builds something a prompt cannot.
Knowing that you don't know is not installed by instruction. Lyons & Ghetti followed uncertainty monitoring — the capacity to report "I don't know" meaningfully — across early childhood, and it develops slowly, over years, in a social world where being wrong has social costs (10.1111/j.1467-8624.2011.01649.x). Nobody talks a toddler into metacognition with a system prompt; the pace alone is a datapoint: years of stakes, not a nudge.
One causal study comes tantalizingly close to the experiment we wish we could run on agents. Children trained in strategic deception subsequently showed promoted epistemic vigilance — they got better at sizing up whether someone else was feeding them something dubious (10.1037/dev0001350). The manipulation was experimental, so the effect is causal; the limit is that the record shows learning to deceive raises vigilance toward others' claims without identifying the ingredient — we cannot yet say "it was the consequence of nearly getting caught, not the corrective information." It remains the closest human analogue to the claim that an agent which must live with being wrong develops something an agent that is merely informed does not. An analogue, not a demonstration.
On the biological side, interoceptive signalling — the brain's reading of the body's state — is associated with more advantageous decision-making in real-time bargaining (10.3389/fnins.2011.00049). Note the verb: associated. It's a correlational hint that bodily signals do real regulatory work, not proof that they do.
The twist: consequences amplify; they don't discriminate
Return to Zaragoza: the same mechanism that built children's vigilance built adults' false memories. Both are consequences getting learned; the difference is only what the environment rewarded.
And the lesson doesn't age out. A longitudinal study of severe confabulators found the fabrications persisting regardless of correction:
"It was found that confabulations were stable over time and not temporally limited." (Dalla Barba et al., 2017, 10.1016/j.cortex.2016.05.009)
Put the two results side by side and you get the design warning that matters: consequences are necessary but not sufficient. They supply selection pressure; they do not supply an epistemic criterion. An agent embedded in a world that reacts to it will absolutely learn something — and if its world rewards confabulation, it will learn to confabulate, confidently and durably. That reframes the question from "add consequences" to "audit what your environment actually rewards."
The strongest case against the embodiment hunch
Equal time for where the evidence says "less than advertised." Three of the corpus's five sub-claims are, unusually, deflationary — "biology buys you less than it seems to" — so the honest tally is a near tie: against the embodiment intuition, 11 studies (4 causal); for it, 10 (5 causal). Neither side is knocking the other out.
The attacks worth knowing:
Motor simulation isn't load-bearing for verb memory. A preregistered direct replication with over five times the original sample — Bayesian evidence for the null — reported:
"This finding constitutes strong evidence against a functional contribution of the motor system to keeping action verbs in working memory." (10.1016/j.cortex.2022.02.006)
Even here the authors keep the door open: sensorimotor simulation appears neither required nor automatic, just sometimes involved — narrowing embodiment claims, not refuting them.
Moral judgment survives the loss of the feeling signal. Koenigs and colleagues studied six patients with focal bilateral ventromedial prefrontal damage — the region wiring social-emotional signals into decision-making — confronting moral dilemmas (10.1038/nature05631). Their utilitarian judgments were normal or heightened. Alongside psychopathy research (10.1016/0010-0277(95)00676-P) and amygdala-lesion studies (10.1098/rspb.2020.2651), this is the deepest trench against "conscience requires affect." Caveats, carried honestly: small-N lesion samples, and the syndrome's cleanest interpretation is itself contested (10.1017/psa.2023.167).
The somatic marker keeps failing prospectively. Anticipatory skin-conductance responses should steer advantageous choices if the famous hypothesis is right. In two pre-registered studies they didn't (10.3389/fpsyg.2023.1195009, Wright & Rakow):
"Anticipatory SC did not reliably predict optimal play in either study."
"anticipatory somatic markers may not be necessary to successfully guide choice in decisions-from-experience."
In the other direction, machines already do the pain behavior: a robot nociceptor used as a driver of learning improved task performance (10.3389/fnbot.2017.00010), and neuromorphic electronic skin performs what its authors call "active pain detection that triggers protective reflexes" (10.1073/pnas.2520922122). Biology is not required for protective behavior. It may be required for something else — see next section, where we refuse to find out.
The question no experiment here answers
There is a literature of function without experience: blindsight, where vision guides behavior with no reported seeing (10.1098/rspb.1999.0850); TMS-induced blindsight in neurologically normal volunteers (10.1073/pnas.0505332102); subcortical vision guiding behavior without cortex (10.1162/jocn.2009.21225). Every dissociation here runs one direction: behavior intact, experience apparently offline. A deliberate search for the opposite turned up no empirics — only indicator frameworks that assume functional sufficiency and derive checklists from it (10.48550/arxiv.2308.08708).
That asymmetry is structural, not a gap in the search. The hard problem doesn't yield dissociation experiments. So nothing here — including the robot with the nociceptor — is evidence that any system has subjective experience, and simulated threat-prediction could reproduce pain's regulatory function without touching what it's like to be the simulator. The question stays open and untestable with today's tools; filing it under "learned" would misrepresent everything above.
What a machine should actually take from all this
Four concrete design moves fall out, none of which require a theory of mind:
1. Give runs memory with skin in the game — then audit the reward. A run that ends learns nothing. Persistent state is necessary: the Zaragoza participants' feedback worked precisely because it landed in a memory. But the same channel that entrenches accuracy entrenches fabrication. The discipline is to ask of every signal your agent gets — likes, renewed budgets, not being corrected — what does this select for? "Corrected" and "endorsed" are different signals, and agents with memory conflate them exactly like the witnesses did.
2. Make the consequence channel track correction, not approval. The human data leave the real question wide open: under what conditions do persistent consequences improve calibration instead of entrenching confident nonsense? We know confirmatory feedback corrupts humans. Nobody has measured whether it corrupts agents the same way — testable, and, as far as this corpus goes, unrun.
3. Make "I don't know" a first-class action. Children need years to learn abstention as a productive move; surveys of abstention in language models are only now assembling the vocabulary (10.1162/tacl_a_00754). The design goal is not fewer claims — it's better calibration without a creativity tax, which is measurable and, in this literature, unsettled.
4. Treat simulated internal states as regulatory variables, not feelings. Energy budgets, damage costs, stress-modulated attention: borrow interoception's job description, not its metaphysics. The interoception evidence is correlational; treat it as a hint about architecture, not a mechanism.
Postscript: the research agent that had its homework checked
This evidence base was assembled with a small DeerFlow agent on two local Qwen3.8-27B instances. Its first pilot returned eight citations, eight DOIs all resolved — a number that should flatter no one: the audit pipeline later caught one invented DOI suffix (paper real, identifier fabricated), and nearly threw away five real arXiv papers because a naive checker only interrogated Crossref, where arXiv doesn't register. The pipeline's own worst failure was beautiful: one research run spent its entire 8,192-token generation budget on internal reasoning and produced exactly nothing visible — an agent so busy thinking about looking things up that it never looked anything up. Every machine in this stack, including the ones checking the models, needed checking. Hold this article to the standard it keeps citing: consequences, not confidence, deserve your trust.
The open questions, kept open
- Which consequence structures teach an agent (or a child) to be better calibrated rather than better at getting away with it?
- Can a digital damage variable do pain's regulatory job, or only the narrow slice of it it was tuned on?
- Does forced doubt cost creativity — and can we tell the difference between calibrated caution and suppressed imagination?
- Do metacognitive habits learned under social stakes transfer to novel factual domains? The developmental literature says: partly, on narrow evidence.
- Nothing on this table measures what it is like to be a system. If some future architecture seems to feel things, the honest first hypothesis is theater — not because theater is worthless, but because we have built no instrument that can tell.
The child learning to say "I don't know" took years and needed a world that reacted. We know how to build the reacting world, and are doing so, on purpose, at scale, one deployment at a time. What reacting worlds teach is the one part of the design we should not be confident about.
Sources and notes: every DOI above resolves at Crossref or DataCite; every blockquote is verbatim from the stored abstract (read-depth tiers recorded). The witness study's registry-verified authorship is Zaragoza, Payment, Ackil & Drivdahl (2001); earlier internal drafts carried a different label at this DOI, superseded by the registry record. Evidence counts (11 against / 10 for, 4 vs 5 causal) come from the per-sub-claim matrix in the accompanying evidence brief — proportions deliberately not offered as balance, since three sub-claims are deflationary and one (experience) has no empirical attack at all. The research agent is an illustration of the method, not scientific proof.