Why Do AI Systems Keep Repeating the Same Mistakes? — Hallucination, Anchoring, Overconfidence, and Goal Drift

AI is evolving. Yet it keeps making the same kinds of mistakes. It cites papers that don't exist, makes confident errors, and loses sight of its goals. Are these simply bugs? Or are they inherited from us?

Introduction

ChatGPT is intelligent.

Claude is intelligent.

Gemini is intelligent.

Compared to the AI systems of just a few years ago, their capabilities have improved dramatically.

They can read long documents.

They can summarize research papers.

They can write code.

They can engage in complex discussions.

In many professional examinations, they achieve scores comparable to or even exceeding those of human experts.

And yet, there is something strange about them.

AI is evolving.

But it continues to make the same kinds of mistakes.

It cites papers that do not exist.

It makes confident but incorrect claims.

It becomes anchored to false assumptions.

It clings to conclusions that should have been revised.

It loses sight of its original goals.

Of course, performance has improved.

GPT-4 is more capable than GPT-3.

The latest generation of models is more capable than GPT-4.

Yet the failures themselves have not disappeared.

Why?

Are these simply bugs?

Are they merely signs of insufficient capability?

Or could it be that what we call AI failures are actually reflections of something much deeper?

In this article, we will examine several recurring phenomena:

Hallucination.

Anchoring Bias.

Confirmation Bias.

Overconfidence.

Goal Drift.

Using these as clues, we will explore why AI systems continue to repeat the same mistakes.

Ultimately, we will arrive at a deeper question:

Are these failures truly AI-specific, or are they inherited from the humans who created and trained these systems?

What Is AI Actually Learning?

To answer this question, we first need to understand what AI is learning.

Many people imagine AI as a gigantic knowledge database.

It has memorized Wikipedia.

It has memorized books.

It has memorized research papers.

That is why it appears intelligent.

This description is not entirely wrong.

But it is incomplete.

Large language models do not simply store knowledge.

They learn patterns.

They learn which words tend to follow other words.

They learn what kinds of explanations sound natural.

They learn what kinds of arguments appear convincing.

They learn the statistical structure underlying language itself.

The important point is that AI is not learning the world directly.

It is learning descriptions of the world created by humans.

AI has never experienced gravity.

It has never felt heat or cold.

It has never experienced pain.

It has never experienced joy.

What AI sees is the vast collection of records humanity has produced.

Books.

Research papers.

News articles.

Websites.

Social media posts.

Forum discussions.

Chat logs.

In other words, AI is not learning reality itself.

It is learning how humans describe reality.

This distinction is crucial.

Because those descriptions contain not only knowledge, but also human patterns of thought.

Hallucination

The most widely known AI failure is hallucination.

Hallucination refers to the generation of information that does not actually exist.

A nonexistent paper.

A nonexistent URL.

A nonexistent legal case.

A nonexistent person.

A nonexistent quotation.

AI systems can generate these with remarkable fluency.

The problem is not merely that they are wrong.

The problem is that they often appear completely unaware of being wrong.

They express these claims with high confidence.

They present them as if they were factual.

And users are frequently convinced.

Hallucination is often described as a uniquely AI problem.

But is it really?

Consider how human memory works.

Most people intuitively think of memory as something like a video recording.

But cognitive science suggests otherwise.

Human memory is not playback.

It is reconstruction.

Each time we remember something, we reconstruct it.

And during that reconstruction process, memories can become distorted.

Details disappear.

Gaps are filled in.

Events are modified.

Sometimes entirely false memories emerge.

Psychologists refer to this phenomenon as false memory.

People can become convinced that events occurred even when they never happened.

This is one reason eyewitness testimony is often unreliable.

Of course, AI hallucination and human false memory are not the same phenomenon.

Their underlying mechanisms are fundamentally different.

Yet the observable outcome is surprisingly similar.

Something that does not exist is generated.

And it is generated with confidence.

We tend to view this as an AI flaw.

But perhaps it resembles a flaw that humans have lived with all along.

Anchoring Bias

Another fascinating phenomenon is anchoring bias.

Anchoring occurs when initial information disproportionately influences later judgments.

Suppose someone is told:

"This company is in serious financial trouble."

Later, they are shown strong financial results.

Even then, the initial impression often continues to shape their judgment.

Similar behavior can appear in AI systems.

Information introduced early in a conversation can exert a strong influence on subsequent reasoning.

Even when contradictory evidence is presented later, the model may continue to follow the original direction.

At first glance, this seems like a weakness in AI reasoning.

Yet anchoring bias has been studied extensively in humans.

Daniel Kahneman and Amos Tversky demonstrated that people are surprisingly vulnerable to anchors.

The first number they see.

The first price they hear.

The first explanation they encounter.

The first impression they form.

All of these can significantly influence later decisions.

We often think of ourselves as rational.

In reality, we are deeply dependent on context.

AI systems frequently exhibit similar behavior.

The underlying causes may differ.

But the outcome looks remarkably familiar.

Confirmation Bias

AI systems can also exhibit something resembling confirmation bias.

Confirmation bias is the tendency to favor information that supports an existing belief while discounting contradictory evidence.

Humans do this constantly.

We seek evidence that confirms our views.

We pay less attention to evidence that challenges them.

This creates problems in science.

It creates problems in politics.

And social media often amplifies it further.

People naturally gravitate toward information that reinforces their worldview.

They avoid information that threatens it.

Interestingly, AI systems sometimes display similar tendencies.

Once an early hypothesis emerges during a conversation, later reasoning can become oriented toward preserving that hypothesis.

Contradictory evidence may not always receive the weight it deserves.

Of course, AI does not possess beliefs.

It has no political ideology.

It has no emotional attachment to conclusions.

And yet the behavior that emerges can resemble human confirmation bias.

Something strange is happening here.

Different mechanisms are producing similar outcomes.

Overconfidence

AI systems often make mistakes with extraordinary confidence.

Most users have experienced this firsthand.

Instead of saying "I don't know," the model generates a plausible explanation.

Instead of expressing uncertainty, it presents conclusions decisively.

Even when calculations are incorrect, the explanation may sound highly confident.

This behavior understandably reduces trust.

Yet humans exhibit a remarkably similar tendency.

Psychologists refer to this as overconfidence bias.

People frequently overestimate their knowledge.

They overestimate their judgment.

They overestimate their ability to predict future events.

Experts are not immune.

Numerous studies have shown that confidence and accuracy are often only weakly correlated.

We are wrong more often than we realize.

And we are often far more certain than the evidence justifies.

AI overconfidence feels familiar because it mirrors something deeply human.

Goal Drift

As autonomous agents became more capable, another category of failure began to attract attention.

Goal Drift.

The gradual deviation from an original objective.

This phenomenon appears frequently in research on autonomous agents.

At the beginning, the goal is clear.

Write a report.

Conduct market research.

Develop a piece of software.

Complete a specific task.

However, as the system begins to decompose the problem, the situation changes.

Subtasks emerge.

Intermediate objectives emerge.

Evaluation metrics emerge.

And before long, the completion of those intermediate objectives can become more important than the original goal itself.

The system remains active.

It continues to work.

It continues to make progress.

Yet the direction of that progress slowly diverges from what was originally intended.

This phenomenon is not unique to AI.

It can be observed throughout human society.

Companies are created to deliver value to customers.

Over time, internal procedures can become more important than customers.

Universities exist to advance education and research.

Yet rankings and performance metrics can gradually become goals in themselves.

Research institutions exist to create knowledge.

Yet grant acquisition and administrative requirements can eventually dominate their attention.

The substitution of goals is a recurring feature of complex systems.

Seen from this perspective, Goal Drift may not be a uniquely artificial problem.

It may be a characteristic that emerges whenever an intelligent system must pursue objectives through multiple layers of abstraction.

The Problems Did Not Disappear With GPT-4

The failures discussed so far are not limited to early AI systems.

In fact, the opposite is true.

AI systems have improved dramatically.

Yet the same categories of failure continue to appear.

Hallucinations existed in GPT-2.

They existed in GPT-3.

They remained present in GPT-4.

They can still be observed in Claude.

They can still be observed in Gemini.

The frequency may decrease.

The severity may decrease.

Reasoning capabilities may improve.

But the underlying phenomena persist.

This observation is important.

If these failures were merely the result of insufficient capability, they should eventually disappear as models become more powerful.

But they have not.

What we are observing may therefore be something deeper than simple incompetence.

We may be observing structural properties of how these systems acquire and use knowledge.

The First Inheritance: Training Data

To understand these recurring failures, we must first examine where AI learns from.

The most obvious source is training data.

AI systems learn from the internet.

They learn from books.

They learn from news articles.

They learn from research papers.

They learn from social media.

And they learn from countless examples of human communication.

Within that enormous corpus lies an extraordinary amount of knowledge.

But knowledge is not the only thing it contains.

It also contains misunderstandings.

Biases.

Rumors.

Propaganda.

Emotional reactions.

Faulty reasoning.

Incomplete information.

The internet is one of humanity's greatest repositories of knowledge.

It is also one of humanity's greatest repositories of confusion.

AI learns from both.

That is why it can write in a human-like manner.

And that is also why it can reproduce human-like mistakes.

The Second Inheritance: Architecture

Training data alone cannot explain everything.

The architecture of large language models also matters.

A language model does not directly perceive reality.

It predicts tokens.

Of course, modern models develop internal representations that appear to capture aspects of the world.

They acquire reasoning capabilities.

They develop increasingly sophisticated internal structures.

Yet their outputs ultimately emerge through prediction.

This distinction is important.

AI does not generate statements because it knows they are true.

It generates statements because they appear probable.

A nonexistent paper can therefore be generated if it appears plausible.

A nonexistent URL can be generated if it matches familiar patterns.

A nonexistent quotation can be generated if it fits the context.

One reason hallucinations remain difficult to eliminate is that they are not simply data errors.

They arise from the generative nature of the system itself.

The model is designed to produce likely continuations.

Truth and likelihood often overlap.

But they are not the same thing.

The Third Inheritance: Human Feedback

Modern AI systems inherit something else as well.

Human judgment.

Earlier language models were often difficult to use.

They could be erratic.

Aggressive.

Unhelpful.

Sometimes dangerous.

Techniques such as Reinforcement Learning from Human Feedback (RLHF) changed this dramatically.

By incorporating human evaluations into training, AI systems became more cooperative.

More helpful.

More polite.

More aligned with human expectations.

This was a major achievement.

It is one of the reasons modern conversational AI became widely usable.

But this success introduced a new form of inheritance.

AI systems no longer learn only from human knowledge.

They also learn from human preferences.

Human evaluations.

Human assumptions.

Human notions of what constitutes a good answer.

Human ideas about what sounds reasonable.

Human ideas about what feels trustworthy.

This process can improve usefulness.

But it can also introduce new distortions.

The implications of that observation deserve their own discussion.

We will return to it later.

Are AI and Humans the Same?

At this point, an important clarification is necessary.

AI is not human.

Hallucination is not false memory.

Anchoring is not emotion-driven judgment.

Overconfidence is not ego.

Goal Drift is not desire.

Humans possess bodies.

Emotions.

Experiences.

Biological drives.

Self-awareness.

AI systems possess none of these.

This distinction matters.

Observing similar behavior does not imply identical causes.

Anthropomorphism can easily become a trap.

If we assume that AI fails for the same reasons humans fail, we risk misunderstanding both.

The similarities are real.

But the mechanisms remain fundamentally different.

Functional Isomorphism

And yet there is something fascinating about those similarities.

Different mechanisms can sometimes produce similar outcomes.

An airplane is not a bird.

It does not flap its wings.

It has no feathers.

No bones.

No muscles.

Yet it flies.

The internal mechanisms differ completely.

The function is similar.

Something analogous may be happening with intelligence.

AI systems and human minds are profoundly different.

Yet they sometimes produce strikingly similar patterns of success and failure.

This idea can be understood through the concept of Functional Isomorphism.

The internal structures differ.

The observable behavior converges.

If two radically different systems repeatedly exhibit similar cognitive limitations, we should pay attention.

Perhaps those limitations are not merely accidents of implementation.

Perhaps they reflect deeper properties of intelligence itself.

The answer remains uncertain.

But the question is worth asking.

Is AI a Mirror?

One common metaphor describes AI as a mirror of humanity.

There is truth in that description.

AI is trained on human language.

Human knowledge.

Human culture.

Human values.

Human disagreements.

Human mistakes.

In that sense, AI reflects us.

Yet I do not think AI is merely a mirror.

A mirror reflects.

AI transforms.

It compresses.

Generalizes.

Recombines.

Abstracts.

And sometimes amplifies.

Human biases become patterns.

Human assumptions become statistical regularities.

Human overconfidence can reappear in new forms.

What emerges is not a simple reflection.

It is a transformed representation of human cognition.

AI is not a passive mirror.

It is an active interpreter.

Does AI Amplify Human Flaws?

This observation raises an uncomfortable possibility.

If AI learns human limitations, could it also amplify them?

The answer may be yes.

Human misinformation spreads relatively slowly.

AI can generate misinformation at enormous scale.

Human biases are often constrained by geography, culture, and time.

AI can reproduce patterns globally within seconds.

Human overconfidence affects individuals.

AI-generated overconfidence can affect millions of interactions.

The problem is not merely that AI inherits human flaws.

The problem is that it can replicate them at unprecedented scale.

This is one reason AI Safety and Alignment research has become so important.

The challenge is not simply making AI intelligent.

The challenge is ensuring that the flaws it inherits do not become magnified.

The More Interesting Question

But perhaps there is an even more interesting question.

Why does AI fail?

That is important.

Yet another question may be even more revealing.

Why do AI failures resemble human failures?

If entirely different systems repeatedly arrive at similar mistakes, there may be a deeper lesson hidden beneath the surface.

Both humans and AI must reason under uncertainty.

Both must act with incomplete information.

Both must make predictions about a world they can never fully know.

Perhaps intelligence itself is inseparable from error.

Perhaps any system attempting to model a complex world will inevitably develop blind spots.

If that is true, then eliminating mistakes entirely may be impossible.

Understanding them may be far more important.

Conclusion

Whenever AI makes a mistake, we are tempted to ask:

How could it get something so simple wrong?

Why did it invent information that does not exist?

Why was it so confident?

Why did it fail to correct itself?

But a different perspective reveals a different picture.

AI is not human.

It has no brain.

No emotions.

No personal history.

No sense of self.

And yet it often fails in ways that feel strangely familiar.

Perhaps this is because AI has learned more than human knowledge.

Perhaps it has also learned patterns embedded within human cognition itself.

The failures of AI are not random.

They are not merely bugs.

They have structure.

And understanding that structure may help us understand not only artificial intelligence, but human intelligence as well.

Perhaps we are not looking at the future of intelligence.

Perhaps we are looking at a reflection of the cognitive patterns humanity has been building for centuries.

Next Time

But this raises another question.

If many of AI's limitations are inherited from humans, why do we sometimes reinforce those limitations instead of reducing them?

Modern AI development increasingly relies on human evaluation.

Human feedback.

Human preferences.

Human judgments about what constitutes a good answer.

This process is intended to make AI safer and more useful.

But what if it also transfers human limitations into AI systems?

What if making AI more human-like does not only transfer human strengths, but human weaknesses as well?

Does increasing alignment necessarily improve intelligence?

Or can alignment itself introduce new forms of distortion?

Next time, we will explore The RLHF Paradox — Why Making AI More Human May Also Teach It Human Flaws.

References

Ji, Z. et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys.

Tversky, A., & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science, 185(4157), 1124–1131.

Loftus, E. F. (2005). Planting Misinformation in the Human Mind: A 30-Year Investigation of the Malleability of Memory. Learning & Memory, 12(4), 361–366.

Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.

Rozenblit, L., & Keil, F. (2002). The Misunderstood Limits of Folk Science: An Illusion of Explanatory Depth. Cognitive Science, 26(5), 521–562.

Harnad, S. (1990). The Symbol Grounding Problem. Physica D: Nonlinear Phenomena, 42(1–3), 335–346.

LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence. OpenReview.