Why Can't LLMs Say 'I Don't Know'?

Opening

“The downside of using deep research is that we can’t control the credibility of the sources, and it could lead to heavy hallucination.”

My tech lead paused after saying this during our POC discussion. We were debating whether to integrate OpenAI’s new deep research models—o3-deep-research and o4-mini-deep-research—into our product, and the conversation had taken that familiar turn—the one where everyone nods knowingly about “hallucinations” without really understanding what’s happening under the hood.

After some back-and-forth, we agreed the solution was “setting expectations with users.” Classic product management: when you can’t solve the technical problem, manage the human problem.

But as I walked away from that meeting, I remembered reading something from Anthropic about hallucinations that challenged my assumptions. That evening, I fell into a rabbit hole of research papers and blog posts, chasing a simple question that had been nagging at me:

If LLMs work like humans, why can humans confidently say “I don’t know” but LLMs seem to struggle with it?

What I discovered was both more fascinating and more practical than I expected.

What Anthropic Actually Found

Here’s where things get interesting. I expected to find research about making LLMs smarter or more cautious. Instead, Anthropic’s 2025 research revealed something counter-intuitive: Claude’s default behavior is actually to decline answering questions.¹

Let me repeat that: the model’s natural inclination is to not answer.

According to their paper “Tracing the thoughts of a large language model,” Claude has internal neural circuits that act like a decision-making system. Think of it as an internal debate: “Do I know enough about this to respond, or should I stay quiet?”

The researchers found that Claude only answers questions when something actively inhibits this default reluctance. It’s like having a naturally shy person who only speaks up when they’re really confident about something.

So if that’s the case, why do we still get hallucinations?

The circuit misfires.

Through intervention experiments, Anthropic could artificially activate the “known answer” features in Claude’s neural network. When they did this, they could make the model consistently hallucinate that “Michael Batkin plays chess” (a fictional fact). Sometimes, this sort of “misfire” happens naturally, without any intervention—and that’s when we get hallucinations.

It’s not that the model is trying to be helpful by making things up. It’s that the internal system responsible for saying “I don’t know” occasionally gets it wrong and thinks it knows something it doesn’t.

This discovery flipped my understanding completely. The problem isn’t that LLMs are too eager to answer—it’s that their internal “should I answer this?” mechanism sometimes malfunctions.

The Fundamental Difference: Human vs. LLM “Hallucination”

But this raised an even bigger question for me: if LLMs have mechanisms similar to humans for deciding when to speak, why do humans seem so much better at saying “I don’t know”?

The answer lies in a fundamental misunderstanding of what’s actually happening when LLMs “hallucinate.”

It’s Not Really Hallucination

Cognitive scientists have been pushing back on the term “hallucination” for LLMs. In humans, hallucination involves perceiving things that aren’t there—seeing, hearing, or feeling something that doesn’t exist in reality. But LLMs don’t perceive anything. They don’t have sensory experiences.

A more accurate term might be “confabulation”—when someone fills in gaps in their memory with plausible but false information.² This happens to humans too, but the underlying mechanisms are completely different.

System 1 vs. System 2 Thinking

Remember Daniel Kahneman’s “Thinking, Fast and Slow”? He describes two modes of human cognition:

• System 1: Fast, automatic, intuitive responses
• System 2: Slow, deliberate, analytical thinking

When humans say “I don’t know,” we’re often engaging System 2—we pause, reflect on our knowledge, and make a conscious decision about our certainty level.

LLMs, on the other hand, are essentially stuck in System 1. They generate responses by predicting the most likely next word based on patterns in their training data. There’s no deliberate reflection, no stepping back to assess confidence levels.

But Wait—What About Reasoning Models?

Here’s where things get really interesting, and frankly, concerning. You’d think that reasoning models like OpenAI’s o1 and o3—the ones that can engage in “System 2” thinking—would be better at saying “I don’t know,” right?

They’re actually worse.

According to OpenAI’s own internal testing, their newer reasoning models hallucinate more frequently than their predecessors:³

• o1: 16% hallucination rate on PersonQA benchmark
• o3: 33% hallucination rate (roughly double)
• o4-mini: 48% hallucination rate

For comparison, the non-reasoning GPT-4o had a 30% hallucination rate, while GPT-4.5 achieved 19%.

This completely breaks the historical trend where each new model incrementally hallucinated less than the previous one. OpenAI admits they don’t really know why this is happening, writing in their technical report that “more research is needed” to understand why hallucinations are getting worse as they scale up reasoning models.

The working theory? The reinforcement learning used for reasoning models may amplify issues that are usually mitigated by standard training pipelines. Because these models generate more detailed responses, they make “more accurate claims as well as more inaccurate/hallucinated claims.”

It’s like giving someone more time to think makes them more confident in both their correct and incorrect answers.

The Training Data Problem

Here’s where it gets really interesting. LLMs struggle with “I don’t know” partly because of how they’re trained. Think about it: when you’re browsing the internet (where most training data comes from), you either find information about a topic or you don’t. People rarely write “I don’t know” about things they don’t know—they just don’t write about them at all.

So LLMs rarely see examples of confident uncertainty in their training data. They’re optimized to always generate something, because that’s what their training taught them to do.

The Business Pressure Factor

There’s also a more pragmatic reason: user expectations. Companies deploying AI want their models to be helpful. A chatbot that frequently says “I don’t know” might be seen as ineffective. Users often prefer a model that tries to help, even if it occasionally gets things wrong, over one that regularly admits ignorance.

As one researcher put it: “Users will say your model is lazy and doesn’t even try, whereas if you train it to be more likely to produce something that looks like a solution more often, people will think ‘wow, the AI solved this problem I couldn’t solve.’”⁴

The Metacognitive Gap (And Why Humans Aren’t Perfect Either)

But here’s the key difference: humans have metacognitive awareness. We can think about our own thinking. When you ask me about quantum physics, I can reflect on my knowledge and confidently say “I don’t know much about that.” I have a sense of the boundaries of my knowledge.

But let’s be honest—humans make plenty of mistakes too.

Research shows that human error rates are often surprisingly high. In medical diagnosis, primary care doctors misdiagnose about 5% of patients. In journalism, American newspapers had factual errors in 61% of stories. When humans claimed to be 100% certain in knowledge tasks, they were still wrong 20% of the time.⁵

By comparison, ChatGPT hallucinates about 27% of the time according to 2023 estimates—not dramatically different from human error rates in many domains.

The crucial difference isn’t accuracy—it’s uncertainty acknowledgment.

When Columbia Journalism Review tested ChatGPT on 200 quotes, it falsely attributed 76% of them but only indicated uncertainty in 7 out of 153 cases where it was wrong.⁶ Humans, despite our overconfidence bias, still retain the capacity to say “I don’t know” when we recognize we’re at the limits of our knowledge.

LLMs don’t have this self-awareness. They can’t step back and assess their own knowledge state. They’re pattern-matching machines that generate the most statistically likely next word based on their training, without any real understanding of what they do or don’t “know.”

The goal isn’t perfection—it’s building AI systems that can acknowledge their limitations as humans naturally do, even if imperfectly.

What Can We Muggles Do About It?

So here we are: reasoning models that hallucinate more than their predecessors, training data that rarely includes “I don’t know,” and business pressures that reward confident responses over cautious ones.

Sounds hopeless? It’s not. While we can’t fix the underlying architecture, we can dramatically improve our interactions with these systems through better prompting techniques. Think of it as context engineering for uncertainty management.

1. “According to…” Prompting

This is probably the easiest and most effective technique. Instead of asking “What causes climate change?” try “According to scientific consensus, what causes climate change?”

This guides the model to ground its response in specific sources rather than generating from general training patterns. It’s like giving the model a specific knowledge base to draw from rather than asking it to freestyle.

Example:

• ❌ “How do vaccines work?”
• ✅ “According to medical literature, how do vaccines work?”

2. Progressive Prompting

Start broad, then narrow down. This helps you identify when the model is moving from confident territory into speculation.⁷

Example:

• First: “What do you know about renewable energy policy?”
• Then: “What specific policies were implemented in Germany in 2023?”
• Finally: “What was the exact impact on household electricity costs?”

Watch for the point where responses become vague or overly general—that’s often where hallucination risk increases.

3. Role-Based Prompting

Ask the AI to respond as a specific expert who would naturally acknowledge limitations.

Example:

• ❌ “Explain this medical condition.”
• ✅ “As a medical researcher, explain this condition, including what aspects are well-understood and what remains uncertain.”

4. Set Output Constraints

Shorter, more focused answers reduce the chances of drift into hallucination territory.

Example:

• ❌ “Tell me everything about artificial intelligence.”
• ✅ “In 100 words, explain the main types of machine learning.”

5. The 3-6 Second Rule (Applied to AI)

Remember that 3-6 second pause I developed for human communication? It works for AI too. Before hitting send, pause and ask yourself:

• Am I providing enough context for this request?
• Could this question be interpreted in multiple ways?
• Am I asking for information that might not exist or be verifiable?

This connects directly to the context engineering insights from my previous post—better context leads to better responses, whether you’re “prompting” humans or machines.

Enterprise Implications: When Stakes Are Higher

While these prompting techniques help individual users, enterprises dealing with customer support, financial research, or technical documentation need more systematic approaches. This is where architectural solutions become crucial.

Retrieval-Augmented Generation (RAG): The Double-Edged Sword

I’ll admit, I’m usually skeptical when people immediately jump to RAG as the solution for everything. But for hallucination reduction, it’s actually one of the most effective approaches we have.

RAG works by giving the model access to specific, verified information sources before it generates a response. Instead of relying purely on training data, the model can “look up” relevant information and ground its responses in those sources.

The results are significant: A 2024 Stanford study found that combining RAG with other techniques led to a 96% reduction in hallucinations compared to baseline models. In healthcare, RAG-powered tools reduced diagnostic errors by 15%.⁸

But there’s a catch: “hallucination with citations.” Models can still confidently generate incorrect responses while citing real sources that don’t actually support their claims. It’s like having a research assistant who finds the right papers but misunderstands what they say.

Grounding and Attribution Technologies

This is where newer approaches like Contextual AI’s “groundedness” technology become interesting. Instead of just retrieving information, these systems are designed to “stick to the knowledge sources you give them.”⁹

Key features include:

• Inline attributions: Source citations integrated directly into responses
• Faithfulness prioritization: Prioritizing retrieved knowledge over training data
• Controllable groundedness: Settings to manage how strictly the model adheres to sources

Think of it as the difference between a research assistant who paraphrases everything in their own words versus one who quotes directly from sources and tells you exactly where each piece of information came from.

The Enterprise Trade-off

For enterprises, the calculation is different than for individual users. When a hallucination could mean misrepresenting company policy to customers or providing incorrect financial advice, the cost of verification becomes worth it.

But these solutions aren’t magic bullets. They require:

• High-quality data sources (garbage in, garbage out)
• Robust data governance (who maintains the knowledge base?)
• Clear attribution chains (can we trace claims back to sources?)
• Regular auditing (are our sources current and accurate?)

The enterprise lesson? Reducing hallucinations isn’t just about better prompting—it’s about building systems that can reliably distinguish between what they know and what they’re guessing.

The Bigger Picture

Understanding hallucination mechanisms doesn’t just make us better at using AI—it makes us better at recognizing our own cognitive limitations. When I catch myself confidently stating something I’m not sure about, I think of Claude’s misfiring circuits. When I see a colleague jumping to conclusions without enough context, I think about System 1 vs. System 2 thinking.

The goal isn’t to eliminate uncertainty—it’s to become more conscious about when we’re operating in uncertain territory, whether we’re human or machine.

So why can’t LLMs say “I don’t know”? Because they’re designed to always generate something, trained on data that rarely models uncertainty, and rewarded for helpfulness over accuracy. But more importantly, they lack the metacognitive awareness that allows humans to recognize the boundaries of their knowledge.

The solution isn’t waiting for perfect AI. It’s becoming better context engineers—for both humans and machines.

Back in that POC meeting, when my tech lead mentioned hallucination concerns, we were right to take them seriously. But now I understand that “setting expectations with users” isn’t just about disclaimers—it’s about building systems and processes that acknowledge uncertainty as a feature, not a bug.

Whether we’re implementing RAG systems, training users on better prompting techniques, or just having more honest conversations about AI limitations, the path forward requires the same conscious effort I committed to in my first blog post.

The conscious effort to think deliberately about where the boundaries of knowledge lie—and to say “I don’t know” when we’re crossing them.

References