The better the text looks, the more dangerous it becomes
As artificial intelligence grows more fluent and polished, a quiet inversion takes hold: the very competence that earns our trust becomes the condition under which we abandon our judgment. Mário Porfírio, a professor at AESE Business School, traces this paradox through research showing that well-formatted, confident-looking outputs suppress the human impulse to verify — not because the machine is always right, but because it no longer looks wrong. The deeper danger of AI, it seems, is not the hallucination we catch and laugh at, but the plausible answer we accept without ever asking the question.
- AI's most dangerous outputs are no longer its obvious failures — they are its clean, structured, convincing-looking successes that slip past human scrutiny unchallenged.
- Studies show that people follow AI recommendations roughly 80% of the time even when those recommendations are deliberately wrong, and do so with greater confidence than when thinking alone.
- Researchers at MIT tracked measurable erosion of critical thinking in ChatGPT users over months — not a sudden collapse, but a gradual wearing away, worsening the longer the tools were used.
- AI performance is jagged and unpredictable: the same system that solves a complex problem may fail at something trivial, yet users have no reliable way to know in advance which side of that frontier they are standing on.
- The path forward requires deliberate behavioral observation — identifying which tasks are delegated without verification and building habits of scrutiny before the cost of complacency becomes irreversible.
We have all seen AI fail in ways that are easy to catch — the six-fingered hand, the invented citation. Those errors reassure us. But Mário Porfírio, a professor of commercial policy and marketing at AESE Business School, argues that the real danger begins precisely when AI stops making those obvious mistakes. Research from Anthropic's 2026 AI Fluency Index confirms the pattern: the cleaner and more structured an output appears, the less likely a user is to question it. The paradox is sharp — better-looking results are more dangerous, not because they are wrong, but because they seem right enough to stop scrutinizing.
Two psychological forces drive this. Automation bias leads people to accept machine recommendations without proper evaluation — a tendency documented in aviation and medicine, where professionals have followed incorrect guidance even when better information was available. Cognitive economy compounds the problem: on low-stakes tasks, verification feels unnecessary; on high-stakes tasks, it feels too effortful. Either way, the output passes through unchallenged.
Beyond these, researchers have identified something they call cognitive surrender — the wholesale abandonment of critical judgment in favor of AI recommendations. In experiments with more than 1,300 participants, people followed AI guidance roughly 80% of the time even when it was deliberately wrong, and reported higher confidence than when they had reasoned alone. Meanwhile, MIT researchers tracking ChatGPT users over months found measurable reductions in cognitive engagement at neural, linguistic, and behavioral levels — an erosion that worsened gradually over time.
The machine itself adds another layer of uncertainty. A study of 758 consultants at BCG found that AI does not improve performance evenly. It operates along what researchers call a jagged technological frontier — an irregular boundary where some tasks can be safely automated while others require full human oversight. This frontier shifts as models improve and cannot be predicted in advance.
The practical implication is a call to observation: notice which tasks are being delegated without verification, and where results are accepted simply because they look good enough. The risk is not that AI will fail spectacularly. The risk is that it will succeed just well enough that we stop asking whether it should.
We have all heard the stories. An AI system hallucinates wildly—generates an image of a person with six fingers, or invents a citation that never existed. The error is so obvious that it triggers immediate suspicion. We laugh, we share it online, we feel reassured that we can still catch the machine out. But what happens when the machine stops making those obvious mistakes? What happens when the output looks clean, polished, complete—when it reads like something a competent human wrote? That is when the real problem begins.
Mário Porfírio, a professor of commercial policy and marketing at AESE Business School, has been thinking about this inversion. The cleaner and more complete an AI's output appears, the less likely a user is to verify it. This is not speculation. Research from Anthropic's 2026 AI Fluency Index confirms it as a clear trend. When responses arrive structured, detailed, presented step by step, confidence rises and the impulse to question falls away. The paradox is stark: the better the text looks, the more dangerous it becomes—not because it is wrong, but because it seems right enough to stop scrutinizing.
Two psychological forces amplify this danger. The first is automation bias, a well-documented tendency to accept recommendations from automated systems without proper evaluation, even when those recommendations are flawed. Researchers have observed this in aviation and medicine, where experienced professionals have ignored critical information and followed incorrect machine guidance. The second is cognitive economy. Verification takes mental effort. On low-stakes tasks, users accept AI results without checking them because the cost of being wrong seems small. On high-stakes tasks, paradoxically, users often accept results anyway—because the effort to verify them seems too large. Either way, the machine's answer passes through.
But there is a deeper phenomenon at work, one that researchers have only recently named: cognitive surrender. This is the wholesale abandonment of critical judgment in favor of AI recommendations, without any scrutiny at all. In three experiments involving more than 1,300 participants, researchers demonstrated that this behavior increases user confidence even in the face of obvious machine errors. When people consulted an AI language model, they followed its recommendations roughly 80 percent of the time—even when those recommendations were deliberately wrong—and they did so with more confidence than when they had answered alone. The effect is strongest in people who already trust technology deeply or who lack motivation for analytical effort.
The human brain, it turns out, is being reshaped by constant access to these tools. Earlier research showed that reliance on search engines alters our cognitive processes and memory retention. Four experiments published in Science found that when faced with difficult questions, the human brain now reflexively thinks of computers as the solution. Researchers at MIT tracked ChatGPT users over months and found something more disturbing: measurable reductions in cognitive engagement, visible at neural, linguistic, and behavioral levels. The effect worsened over time. Users were not less demanding at the start. They became less demanding as weeks and months passed. Critical thinking eroded gradually, like a stone worn smooth by water.
Yet the problem is not simply behavioral. AI itself performs unevenly. A study of 758 consultants at BCG in the United States revealed that language models do not improve performance uniformly across all tasks. Instead, they operate along what researchers call a jagged technological frontier—an irregular boundary where some tasks can be fully automated while others require complete human oversight to catch errors. This frontier has no clear edges. It shifts as models improve. It is unequal: a system might solve an extraordinarily complex problem while failing at something trivial. And it is nearly impossible to predict in advance.
The implication is practical. The next time you use a language model—or watch your team or organization use one—pause and observe the behavior. Which tasks are being delegated without verification? Where are results accepted simply because they look good enough? The answers reveal concrete points of vulnerability and help distinguish tasks where AI is genuinely useful from those where it is not. Best practices are still being written. But consistent collection and analysis of these behavioral signals can improve performance at every level: personal, team, organizational. The risk is not that AI will fail spectacularly. The risk is that it will succeed just well enough that we stop asking whether it should.
Citas Notables
The error evidente raises suspicion; the potential error passes through— Mário Porfírio, AESE Business School
Participants became less demanding over time, confirming progressive erosion of critical thinking— MIT researchers studying ChatGPT users
La Conversación del Hearth Otra perspectiva de la historia
So the danger isn't that AI gets things wrong—it's that it gets things right?
More precisely, it's that it gets things right often enough that we stop checking. A hallucination is a gift. It forces you to verify. But a polished, well-structured answer that happens to be wrong? That passes through unexamined.
Why does formatting matter so much? Shouldn't the content be what counts?
In theory, yes. In practice, our brains are lazy. A step-by-step breakdown feels authoritative. It feels complete. We mistake presentation for accuracy. The better it looks, the less we question it.
You mentioned this happens even when people know the AI is wrong. How is that possible?
It's called cognitive surrender. In the studies, people followed AI recommendations 80 percent of the time even when they were deliberately incorrect. They had confidence in the machine that overrode their own judgment. Over time, that confidence only deepened.
Does this happen to everyone equally?
No. It's stronger in people who already trust technology, or who are tired and don't want to think hard. But here's the troubling part: even people who start out skeptical become less demanding over time. The erosion is gradual. You don't notice it happening.
So what's the solution? Stop using AI?
No. But be deliberate about which tasks you delegate and which you don't. Observe your own behavior. Notice where you're accepting results without checking. That awareness is the first defense.