The Recombination Illusion

[I’m trying something new…if you want to listen to a 16-minute audio podcast discussion of this article from NotebookLM, click here.]

Hey there,

A colleague and I stumbled onto something interesting over coffee a few weeks ago. We were discussing AI capabilities, and he made an observation that felt immediately true: "AI is phenomenal at going from many to few—summarizing, synthesizing, simplifying. But it's terrible at going from few to many—extrapolation, ideation, creation. That's when you start seeing hallucinations."

I nodded along. It made intuitive sense. Compression good, expansion bad.

Then I walked away and later found a counterexample that seems to break the framework: AI is absolutely brilliant at brainstorming.

Ask Claude or ChatGPT to generate 50 product positioning ideas, or to ideate on solutions to a complex user problem, or to explore alternative strategic frameworks. It excels. That's expansion, not compression. Few inputs, many outputs. The framework was wrong.

Except... I don't think it was wrong. I think it was imprecise.

So, after sitting with this tension for the last few weeks, I realized that AI doesn't struggle with expansion because of direction (few-to-many vs. many-to-few). AI struggles with expansion because of its architecture.

AI is extraordinary at compression with constraint. It fundamentally cannot do expansion without constraint.

When AI "brainstorms," it's not actually generating new ideas in the way humans do. It's performing sophisticated recombination of patterns compressed during training. It takes concept A from domain X, concept B from domain Y, and remixes them within the constraint of your prompt. That feels generative—and it is valuable—but it's still fundamentally bounded by the training distribution.

True creative expansion—the kind humans do when we invent genuinely novel frameworks, make intuitive leaps across distant domains, or create paradigms that didn't exist before—requires operating beyond the training distribution. And that's precisely where AI hallucinates, because hallucination isn't a bug in the system. Hallucination is what happens when you ask a compression architecture to expand beyond its compressed knowledge.

Let me explain what I mean by compression architecture.

AI training is fundamentally about compression: learning to represent the maximum amount of information in the minimum parametric space. This creates extraordinary capabilities:

But compression is inherently lossy. And what gets lost matters enormously.

When AI compresses information during training, it loses the metadata about how certain that information is. Think about how a human expert operates: you know the difference between "I've seen this pattern 1,000 times and I'm certain" versus "I've seen something vaguely similar twice and I'm guessing." That's an epistemic boundary—the edge of what you reliably know.

AI's compressed representation doesn't preserve these boundaries. Both patterns might have similar statistical weights in the model. So when you ask it to extrapolate or expand, it confidently generates plausible-sounding content beyond its reliable knowledge because it has no internal mechanism for knowing where its knowledge ends and speculation begins.

This is why you get such confident hallucinations. The architecture doesn't have a sense of "I don't know this, so I should signal uncertainty." It just keeps compressing patterns and generating outputs that are statistically plausible based on what it learned.

(If you're realizing that understanding AI's architectural limitations is critical to deploying it strategically, you might be interested in my private cohort program where I work directly with product leadership teams on exactly these frameworks. Learn more here.)

Yes, modern AI models hallucinate less obviously than earlier versions. GPT-4 and Claude rarely fabricate obviously false citations or completely made-up facts the way GPT-3 did.

But hallucinations haven't disappeared—they've gotten more dangerous because they've gotten subtler. Modern models hallucinate through confident extrapolation beyond their knowledge, through plausible-but-ungrounded reasoning, through filling gaps with statistically likely but unverified content.

We've mistaken the reduction in obvious hallucinations for solving the expansion problem. We haven't. We've just pushed the hallucination boundary further out.

Here's an analogy: Imagine you've compressed the entire internet into a lossy JPEG. You can decompress it and get a pretty good approximation of what was there. But if you ask that JPEG to generate new internet content that wasn't in the original compression, it can only recombine the patterns it learned. It has no mechanism to verify whether those recombinations correspond to reality—it just makes them statistically plausible based on the compression algorithm.

That's what modern LLMs do. Better training data and techniques (RLHF, constitutional AI, etc.) expand what's in the "compressed JPEG," reducing hallucinations in more territory. But they don't eliminate the fundamental issue: when you ask AI to operate beyond its training distribution, it will confidently create plausible-but-ungrounded outputs because that's what compression architectures do when asked to expand.

You might be thinking: "What about code generation? AI writes entire applications from natural language descriptions—isn't that expansion?"

Not quite. Code generation is sophisticated recombination within an extremely constrained solution space. Programming languages have rigid syntax, established patterns are heavily represented in training data, and outputs are verifiable (code either runs or it doesn't). AI isn't inventing new programming paradigms or novel algorithmic approaches—it's recognizing that your natural language description maps to patterns it's compressed from millions of code examples, then recombining them into syntactically correct, functionally appropriate code.

"Vibe coding"—having AI write code based on casual descriptions—feels magical because it's AI playing to its compression strengths: pattern recognition and recombination within well-defined boundaries. The Walmart team that's systematically applying this approach to coding, testing, PRDs, and design (which I wrote about in "The Legacy Game Paradox") isn't asking AI to create. They're asking, "Where else in our workflow can we leverage AI's compression and recombination strengths?" That's strategic thinking about AI capabilities, not wishful thinking about AI creativity.

What AI doesn't do well in vibe coding? Invent genuinely novel algorithms for unsolved problems, create new architectural patterns, or make strategic decisions that require understanding unstated business constraints. The human creativity is still required for the "what to build and why"—AI just excels at the "how to implement it using established patterns."

Will future AI architectures solve this limitation?

Current approaches are all variations on compression:

Theoretical alternatives exist—neurosymbolic AI, world models with causal reasoning, meta-learning architectures—but they're speculative and mostly assume that expansion is fundamentally about logic rather than creative leaps. I'm skeptical.

The deeper issue is that true creative expansion might require something we don't yet understand about human cognition. Humans don't just recombine patterns. We make intuitive leaps that violate statistical expectations. We say "what if the opposite is true?" or "what if we ignored this entire framework?" We abandon paradigms, not just optimize within them.

If that kind of thinking emerges from embodied experience, emotional states, or cognitive mechanisms we haven't reverse-engineered, then AI as currently conceived—disembodied statistical pattern matching—might hit a fundamental ceiling.

My honest assessment: We'll get increasingly sophisticated recombination that will feel creative for most use cases. But genuinely novel paradigm creation? That might require an architectural breakthrough we haven't even conceived of yet.

The current trajectory is making compression models bigger, better, and more multimodal. That's not a path to expansion—it's a path to more sophisticated compression.

If AI is fundamentally limited to recombination within its training distribution, and humans are uniquely capable of creative expansion beyond existing paradigms, that should change how we think about AI deployment, collaboration models, and—here's where this gets uncomfortable—our own cognitive development.

Because here's a question I can't stop thinking about: If the compressive work we're delegating to AI is the same cognitive work that builds our capacity for expansive thinking, what are we letting atrophy?

That's what I'm exploring next week. The paradox is sharper than you might think.

Break a Pencil,
Michael
www.breakapencil.com

P.S. This is Part 1 of a three-part series on AI's compression architecture and what it means for human cognitive development in an AI-saturated world. Next week: why the mental work you're delegating might be exactly what makes you irreplaceable.

P.P.S. Ready to build systematic AI capabilities across your team? My next "Build an AI-Confident Product Team" cohort on Maven starts soon. Or if you want a private cohort tailored to your organization's specific challenges, let's talk about that here.

P.P.P.S. Know a product leader wrestling with AI strategy? Forward this. They'll thank you.