Can AI spot its own mistakes? Why detecting hallucinations is harder than you might think

We’ve all seen the impressive things AI chatbots and language models like ChatGPT or Gemini can do, from writing stories and summarising reports to answering complex questions in seconds. But these systems aren’t perfect. Sometimes they produce answers that sound convincing but are actually completely wrong. In AI research, these false or misleading statements are called hallucinations.

Naturally, people are working hard to figure out how to spot these mistakes automatically. After all, if we want to trust AI in areas like healthcare, education, or the law, we need to know when it’s getting things wrong. But is it actually possible for AI to detect its own hallucinations?

A new study from Yale University (link at the bottom) tackles this very question. And the answer isn’t as straightforward as you might expect.

Why this is a big deal

When an AI makes things up, say, citing a non-existent study or giving incorrect medical advice, it’s not always easy to catch. The language sounds smooth and believable. Automated tools to detect these hallucinations would be hugely helpful, saving time and improving safety.

But the Yale team, led by Amin Karbasi and his colleagues, wanted to know if this goal is even achievable in theory - before worrying about building the perfect tool.

A theoretical deep dive (without the maths)

To explore this, the researchers created a theoretical model based on language identification. You can think of it like a classification challenge where the AI has to decide if a statement is true or not, based on examples it has seen.

Here’s where it gets tricky. The researchers proved that if you only train your AI detector on correct examples (without showing it what mistakes look like), it is mathematically impossible to build a reliable detector. In other words, if your detector has only ever seen what “right” looks like, it can’t confidently recognise “wrong”.

This might sound surprising at first, but it’s a bit like teaching someone to spot fake paintings without ever showing them a forgery. They might be able to guess, but they’ll never be truly reliable.

The power of showing both right and wrong

The study then explored what happens if you train your detector with both correct and incorrect examples. For instance, giving it examples of true facts and clearly labelled hallucinations.

This changed everything. With access to both types of examples, the researchers found that it is theoretically possible to build a reliable hallucination detector.

This theory matches what many AI companies already try to do in practice. For example, they often use methods like Reinforcement Learning with Human Feedback (RLHF), where human reviewers check AI outputs and provide feedback on what’s good and what’s bad. While the Yale study does not specifically discuss RLHF, its findings support the idea that exposing AI systems to both correct and incorrect examples is essential for improving safety and reliability.

Why this matters in the real world

The key takeaway is that AI can’t learn to spot its own hallucinations just by looking at the "good stuff". It needs to be exposed to its own mistakes, with human experts labelling them, to get better.

This has big implications for how we develop AI. It means:

• Human oversight is essential. We can't rely on AI to police itself without human input.

• High-quality labelled data is crucial. The more clear examples of both correct and incorrect outputs we provide, the better AI can learn to avoid mistakes.

• Trust in AI takes work. While AI might feel magical, there’s no shortcut to building safe and reliable systems.

  
  

So next time you see an AI confidently answering a tricky question, remember: behind the scenes, it takes a lot of expert guidance and training – not just clever algorithms – to help it get things right. And as always, take a moment to double-check the facts, just in case it got something wrong.

Read the study: (Im)possibility of Automated Hallucination Detection in Large Language Models