The real limits of large language models

Large language models can write code, explain physics, summarise a 200-page document and translate between languages. That makes it tempting to think the remaining limitations are mostly engineering problems. Give the model more data. Make it larger. Add more computing power. Give it a longer context window. Let it search the web. Give it tools.

Many of those things work. But there is a more interesting question underneath all of this:

What can a language model fundamentally be expected to do well and where does its apparent intelligence become misleading?

That question matters more now than it did two years ago. AI is no longer an experiment confined to developers and researchers. Stanford’s 2026 AI Index describes AI as rapidly integrating into the global economy, while also highlighting a widening gap between what AI systems can do and how prepared organizations are to evaluate and manage them.¹

The limits therefore matter precisely because the technology is becoming useful.

It is easy to criticise an LLM by asking it a deliberately obscure question and waiting for it to make something up. That misses the more interesting story. Modern models are genuinely capable.

The message that I always bring when I have a chance is that, when the text that comes out of one of these systems makes sense, it’s because we are making sense of it.

Emily M. Bender²

Stanford’s 2025 AI Index reported major gains on difficult benchmarks. On SWE-bench, for example, AI systems went from solving 4.4% of coding problems in 2023 to 71.7% in 2024. Performance on GPQA, a benchmark designed around difficult graduate-level science questions, also rose dramatically.³

At the same time, adoption has moved well beyond experimentation. Stanford reported that 78% of surveyed organisations said they used AI in 2024, compared with 55% in 2023.⁴

In the European Union, 20% of enterprises with at least 10 employees used at least one AI technology in 2025, up from 13.5% in 2024.⁵

So this isn’t a story about a technology that doesn’t work. It works extraordinarily well. The problem is that capability and reliability are not the same thing.

This is probably the most important limitation to understand. Ask an LLM for a paragraph about a familiar subject and it can produce something that sounds authoritative, structured and complete. Ask it about something obscure and it can do exactly the same thing.

That creates a dangerous asymmetry. When a search engine doesn’t find an answer, you see an empty result. When a language model doesn’t have a reliable answer, it can still give you five convincing paragraphs. This is what we usually call a hallucination.

Research published by OpenAI in 2025 argues that standard training and evaluation procedures can reward models for guessing rather than acknowledging uncertainty. OpenAI explicitly describes hallucinations as a fundamental challenge for large language models, including its own systems.⁶

Anthropic has described a similar issue from another angle: language-model training fundamentally encourages the system to predict what comes next, which creates pressure to produce an answer even when the underlying information is uncertain.⁷

This creates a strange property. A model can be very good at producing an answer and still be bad at knowing whether the answer deserves your trust. That distinction is easy to forget because humans normally use fluency as evidence of competence. With LLMs, fluency is partly what you’re buying.

Imagine asking an AI:

Who designed this obscure piece of software in 2017?

The model might answer immediately. Perhaps it gives you a name, a company, a date and even a link. Everything looks plausible. But suppose the person never existed.

The problem isn’t simply that the model made a factual error. The deeper problem is that the presentation of the error contains no reliable signal telling you that it is an error. This is why AI-generated information requires a different reading habit.

You should separate:

This sounds correct.

from

I have evidence that this is correct.

Those are very different statements.

The distinction becomes particularly important in law, medicine, finance, research and technical architecture, areas where a plausible answer can be much more dangerous than an obviously bad one.

A wrong SQL query wastes a few minutes. A fabricated legal precedent can distort an entire argument. A fictional medical paper can mislead a patient. A subtly incorrect architecture recommendation can become a production incident six months later.

One of the most seductive improvements in modern AI is the enormous context window. You can now give a model hundreds of pages of documents and ask it to analyse them. This is genuinely useful. But a large context window is not equivalent to perfect comprehension.

Research on long-context reasoning continues to find situations where performance degrades as the amount of information and reasoning required increases. A 2025 study introducing LongReason evaluated 21 LLMs and reported significant performance drops as context length increased.⁸

Another 2025 benchmark, LongReasonArena, specifically targeted long reasoning rather than simple retrieval. The authors found that even advanced reasoning models struggled substantially on tasks requiring very long chains of reasoning; DeepSeek-R1, for example, achieved only 7.5% accuracy on their most challenging task setting.⁹

This is an important distinction. A model can see information without necessarily being able to use all of it reliably.

Think about the difference between reading a 500-page technical book and being able to solve a difficult problem using something buried on page 417.

Humans struggle with this too. The difference is that an LLM’s ability to accept a huge input can create the impression that the problem has already been solved. It hasn’t.

The arrival of reasoning models changed the discussion.

Instead of producing an immediate answer, newer models can spend additional computation exploring possibilities, checking intermediate results and revising their approach. This has produced substantial improvements on mathematics, coding and other difficult tasks. But “reasoning” shouldn’t be confused with guaranteed correctness.

A model can reason through a wrong premise. It can make an incorrect assumption early in a chain and then build an increasingly sophisticated argument on top of it. It can also produce a convincing explanation for an answer that is simply wrong. This creates an interesting failure mode:

The better the explanation sounds, the harder the mistake can be to notice.

A one-line wrong answer is easy to challenge. A wrong answer supported by ten paragraphs of apparently coherent reasoning can survive much longer.

Research into hallucinations in reasoning models has specifically examined how errors can propagate through long reasoning chains rather than simply appearing in the final sentence.¹⁰

The lesson isn’t that reasoning models are useless. Quite the opposite. They are increasingly useful. But their reasoning should still be treated as a computational process that can fail, not as an internal guarantee of truth.

There’s another limit that receives less attention. We are becoming extremely good at measuring AI on tests. That sounds positive. It is. But benchmarks are still artificial environments. A benchmark gives you a defined problem, a defined input and a measurable output. Real work is messier.

This is one reason task-based evaluations are becoming increasingly interesting.

METR, for example, measures AI systems by the length and complexity of tasks they can complete rather than relying only on traditional question-answer benchmarks. Its 2026 measurements show that frontier AI agents are improving rapidly on longer tasks, but the success rate still falls as tasks become more complex and extended.¹¹

The difference matters. Answering a difficult question is not the same as reliably completing a difficult project.

Humans are surprisingly tolerant of small errors. That’s useful in everyday life.

If someone tells you a restaurant is five minutes away and it takes eight, nothing terrible happened. Software is less forgiving.

• An API endpoint either works or it doesn’t.
• A financial calculation can be wrong by 0.5%.
• A configuration can contain one incorrect permission.
• A legal citation can be almost right and still be useless.

This is where LLMs can become awkward. They are extremely good at producing approximately useful output. But many professional tasks require exact output. That creates a mismatch.

Task characteristic	LLMs tend to be strong	LLMs require more caution
Language	Drafting, rewriting, summarising	Factual precision in unfamiliar subjects
Coding	Generating patterns, explaining code, prototypes	Complex production changes and hidden dependencies
Research	Exploring possibilities, synthesising material	Claims requiring authoritative verification
Reasoning	Structured problems with clear constraints	Long chains with uncertain premises
Documents	Extracting and transforming information	Subtle interpretation and missing context
Decisions	Generating options and arguments	High-stakes decisions requiring accountability

The interesting thing is that the right-hand column doesn’t mean “don’t use AI.” It means “don’t confuse assistance with verification.”

Giving a model access to search, databases, calculators, code execution or external systems changes the equation. A model doesn’t need to memorise today’s weather if it can query a weather service.

It doesn’t need to remember your company’s latest documentation if it can retrieve it.

It doesn’t need to perform arithmetic mentally if it can use a calculator.

This is one of the strongest directions in AI development. But tools don’t remove the underlying problem. They move it.

Now the model has to decide:

• Which tool should I use?
• What should I ask it?
• Which source should I trust?
• Did I interpret the result correctly?
• Should I take an action based on it?

An incorrect answer from an LLM is one problem. An incorrect action taken by an AI agent is a much larger one.

That is why the transition from chatbot to agent deserves caution. METR’s research shows that AI systems are increasingly capable of completing longer autonomous tasks, including complex software work. At the same time, reliability remains a central constraint as task duration increases. The more autonomy you give the system, the more expensive a small mistake can become.¹²

This is perhaps the most useful way to think about the subject. The question isn’t whether LLMs are intelligent. That argument gets stuck in definitions.

The more practical question is:

What happens when a system that is extraordinarily good at producing plausible output is placed in a situation where plausibility isn’t enough?

That is where the limits become visible. An LLM can write an excellent explanation without knowing whether every claim is true. It can produce elegant code without understanding every consequence of deploying it. It can analyse a document without noticing the one missing piece of information that changes the conclusion. It can reason for thousands of steps and still carry a wrong assumption from step three into step three thousand. And it can sound completely certain while doing all of this. That doesn’t make LLMs a failure. It tells us what kind of technology they are.

They are extraordinarily powerful reasoning and language tools, but they still need an environment that supplies reliable information, meaningful feedback, verification and, in consequential situations, human judgment.

The smartest way to use them isn’t to ask:

Can AI do this?

The better question is:

What happens if AI is wrong here?

If the answer is “not much”, let the model work. If the answer is “a lot”, build the verification around it. Because the real limit of an LLM may not be what it can produce. It may be how much you are willing to trust what it produces without checking.

Sources

1. Stanford, “The 2026 AI Index Report” ↩︎
2. IEEE, “Emily Bender Sets the Record Straight on “Stochastic Parrots” Five years later, its lead author revisits the paper in the age of ChatGPT” ↩︎
3. Stanford, “AI masters new benchmarks faster than ever.” ↩︎
4. Stanford, “The 2025 AI Index Report” ↩︎
5. EC, “20% of EU enterprises use AI technologies” ↩︎
6. Openai, “Why language models hallucinate” ↩︎
7. Anthropic, “Tracing the thoughts of a large language model” ↩︎
8. Arxiv, “LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion” ↩︎
9. Arxiv, “LongReasonArena: A Long Reasoning Benchmark for Large Language Models” ↩︎
10. Arxiv, “Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models” ↩︎
11. Metr, “Task-Completion Time Horizons of Frontier AI Models” ↩︎
12. Metr, “Research” ↩︎