Writing is Thinking
Human metacognition in the age of AI
这篇暂时只有英文版。
The Nature editorial that caught my eye
A recent Nature Reviews Bioengineering editorial titled “Writing is Thinking”1 made a compelling case for preserving human-written scientific papers in the age of large language models. The editorial argues that writing isn’t merely about reporting results. It’s a cognitive tool that helps us uncover new thoughts and ideas.
As the editorial puts it: “Writing compels us to think—not in the chaotic, non-linear way our minds typically wander, but in a structured, intentional manner.” This resonated deeply with me, especially as someone who started this blog as a way to maintain “conscious effort” in my thinking.
What struck me most was the editorial’s suggestion that if writing is thinking, then when we read AI-generated text, are we reading the “thoughts” of an LLM rather than the researchers behind the paper? This question sent me down a rabbit hole, exploring the fundamental differences between human writing and AI text generation.
Expanding beyond context windows
As I reflected on the editorial, I realized something important about why humans write: we’re fundamentally driven to expand beyond our context windows.
When I write, I’m not simply organizing what I already know. I’m actively seeking to transcend my limitations. I write to discover connections I hadn’t previously made, to extend my understanding beyond what’s immediately visible to me. The very struggle of articulating thoughts forces me to reach beyond my current knowledge boundaries.
This is perhaps our defining characteristic as humans. We climb mountains “because they’re there,” we explore space, we create art that challenges perception, and we develop theories about realms we can’t directly observe.
The physical act of writing, whether typing on a keyboard or scribbling in a notebook, becomes the vehicle through which we expand our context window. It’s a tool for transcending our cognitive limitations.
Optimizing within constraints
LLMs, in contrast, are designed fundamentally differently. They operate by optimizing within constraints rather than trying to transcend them.
In my previous post, “Why Can’t LLMs Say ‘I Don’t Know’?”, I explored how current AI models struggle with recognizing their knowledge boundaries. They lack the intrinsic drive to question or expand their boundaries. They simply make the best statistical predictions possible within the parameters they’ve been given.
2025-07-09Why Can’t LLMs Say ‘I Don’t Know’?The Nature editorial mentions this phenomenon as “hallucination”: LLMs generating incorrect information confidently. This happens precisely because they lack what I think of as a proper loss function for being wrong.2 They face no real consequences for errors beyond what’s encoded in their training.
This fundamental difference shapes everything about how LLMs generate text versus how humans write. LLMs don’t write to discover. They write to predict the most statistically likely next word based on their training data and immediate context.
The missing element: intuition and embodied knowledge
There’s always something subtly missing in AI-generated text that’s difficult to articulate. Despite technical proficiency, LLM writing often lacks what we might call soul or intuition.
Intuition, knowledge without conscious reasoning, emerges from our embodied experiences, unconscious pattern recognition, and evolutionary adaptations. What makes it so powerful is that it integrates several forms of knowledge at once:
- 01ExplicitWhat we can articulate
- 02TacitAbsorbed, but hard to explain
- 03EmbodiedStored in physical responses
- 04SocialHow people actually behave
LLMs fundamentally lack physical presence. They can only “understand” concepts like gravity or exhilaration through textual descriptions rather than lived sensory experience. They have no skin in the game: no physical consequences for being wrong that might drive caution or accuracy.
The social dimension of writing
I’ve often noticed the peculiar tension we feel when others read our writing: a simultaneous desire for our thoughts to be found and fear of judgment. This productive embarrassment creates a unique cognitive pressure that machines don’t experience.
The anticipation of human readers drives us to refine our ideas more carefully than we might when thinking privately. We question our assumptions, strengthen our arguments, and clarify our reasoning precisely because we know others will scrutinize them.
Current instruction-tuned models are designed to agree and be helpful rather than to push back. They prioritize agreement over intellectual integrity, which is a very different relationship from a good editor who challenges our ideas through productive disagreement.
Humans write to expand their context window. Machines write to optimize within it.
人写作，是为了把上下文窗口撑大；机器写作，是在窗口里求最优。
Context engineers of our own understanding
When I sit down to write about a complex topic like this one, I’m not just organizing my existing thoughts. I’m actively expanding my understanding, making new connections, and discovering insights I didn’t have when I started typing. The struggle is productive precisely because it pushes me beyond what I already know.
As we navigate this new era of AI-assisted writing, we must remain the context engineers of our own understanding: using AI as a tool while preserving the uniquely human cognitive benefits that come from the conscious effort of writing.
After all, if writing is thinking, I’d rather do my own thinking, thank you very much.