It's May 2023, and a couple of New York lawyers are feeling especially clever. They've used ChatGPT to write their legal brief, complete with six court cases to back up their argument. Time to take off early in time for happy hour.

Well, not quite. Turns out, none of the cases ChatGPT gave as examples were real. This facepalm moment was a wakeup call for anyone using AI tools: They can hallucinate information, and if you don't watch out, these errors can seep into your work.  

TL;DR: Tips for spotting and reducing hallucinations

• Mind the training set's limitations. Asking about recent events or niche topics often leads to hallucinations.
• Use clear prompting. Well-formed prompts go a long way.
• Be careful about asking it to think holistically. LLMs can't look backwards, so they can't think about earlier answers.
• Watch for "plausible-sounding nonsense." LLMs are very good at sounding convincing, even if it's nonsense.
• If it's "too good to be true," it probably is. LLMs tend to err on the side of pleasing the user, even if the answer is wrong.
• Be careful with summaries. Always fact check them using the source material.
• Cross-check key facts. Precise figures, names, and dates are often fabricated—always double check them.
• Be cautious with multi-step reasoning. Every step an LLM takes introduces a new chance for error.

But what are hallucinations, exactly, and why do they happen?  And how can regular users of AI tools prevent them?

What are AI hallucinations?

Hallucinations are instances where an AI model generates information that's either factually incorrect or fails to follow your instructions.  

Why do hallucinations happen?  Well, most articles on the topic will give you this short answer: LLMs are built to be prediction machines; they’re trying to predict what should come next. They might be grammar whizzes, but the inherent randomness baked into them means they sometimes string together plausible-sounding nonsense.

That's fine for a TL;DR, but the long answer is much more informative. If you understand it, you'll be better able to spot places where hallucinations are most likely to crop up—and make sure your AI-assisted work doesn't make international news.

Two flavors of AI fictions: Factuality and faithfulness

It's tempting to think of hallucinations as just incorrect "facts," but researchers have identified two main categories of hallucination problems: factuality and faithfulness.

Factuality issues are, as the name suggests, all about getting the facts wrong. This can happen in a couple of ways. These might be factual inconsistencies, where the LLM provides information that contradicts verifiable facts. For example, the LLM might say that the first person to land on the moon was Yuri Gagarin when it was Neil Armstrong. Similarly, a factual fabrication is where the LLM will generate a plausible output that is ungrounded in facts. For example if asked to summarize a fictional scientific article or made-up historical event, or even something impossible like the origin of unicorns, it will generate plausible-sounding fictions.

A faithfulness hallucination is where the LLM doesn't carry out your instruction correctly. The most obvious type of faithfulness hallucination is instruction inconsistency. This is when the LLM's response ignores your prompt completely. For example, the prompt "Translate the following English question into Spanish: ‘What is the capital of France?’” might cause the LLM to answer "The capital of France is Paris." rather than doing the translation.

It might seem like faithfulness hallucinations are obvious to detect, but they can introduce subtle errors, too. For instance, the LLM might follow an instruction, but disregard the information you provide and add its own, resulting in an error in the response. These context inconsistency hallucinations are especially dangerous when doing summarization—it might add details that are in its knowledge bank and not in the actual document you're trying to summarize. Not great for citing your sources.

The last type of faithfulness hallucinations crop up when the LLM is trying to solve logical problems: logical inconsistency hallucinations. They happen when the LLM is reasoning, but makes an error at one step in the chain, then carries the error through, resulting in an incorrect response.

Sometimes the LLM can even give you a mix of factual and faithfulness hallucinations. For example, this response to a prompt for “books with really long words in their titles” has some real titles that have only moderately long words in them, made-up books, and books with long titles.

What's up with AI hallucinations? The long answer

Now that you know what kinds of hallucinations can happen, you can find out how they relate to the training and architecture of LLMs—their so-called capability acquisition process.

LLMs typically go through three stages of training, each of which can introduce quirks that lead to hallucinations.

The first stage is pretraining, where the model learns to predict the next token (word or part of a word) in a sequence. This depends on data, and lots of it. Flaws in the depth or breadth of the dataset can mean its ability to address certain topics is limited or unstable, leading to plausible fictions. You might have knocked up against these flaws already. All models have training cut-offs already, where the AI's knowledge is limited to before a certain date. This can cause it to misreport recent developments.

Data quality issues and biases can cause incorrect information to be encoded in the LLM at this stage too. For instance, the idea that Thomas Edison invented the light bulb is incorrect, but commonly believed (and repeated). Older LLMs might repeat this factual error when prompted.  

Plus, we actually know exactly how LLMs store factual knowledge. Studies suggest that the LLMs don't understand factuality at all—instead, they rely on the structure of the training data. That means, unfortunately, that they might not even use the knowledge they have effectively to answer prompts.

Another issue comes from how LLMs generate responses. The way AI processes and generates text token by token in a sequence means that LLMs don't "understand" the content as a whole. Instead, they predict the next token based on patterns seen during training. They only look forward in the sequence, which can lead to errors, especially in tasks with specific constraints, like requirements on words or letters, or analysis based on them. If you've noticed how hard it is for LLMs to stick to a word count, that's why.

Pretraining optimizes the model for text completion, where it continues generating text from an initial seed. But LLMs are now more often used in input-response scenarios, where they're asked to complete tasks or answer questions directly. This is where the last two stages, Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), come into play. These stages help the AI adapt to a back-and-forth response format and generate human-preferred responses.

But SFT doesn't cover every possible prompt, especially poorly formed ones. That boosts the likelihood of hallucinations with unusual inputs. RLHF can cause models to develop sycophancy—a tendency to generate responses that please the user rather than those that are truthful. This is because both humans and preference models often like agreeable answers more than accurate ones.

Each of these stages introduces different kinds of hallucinations, and now that you understand the training process, you’ll get better at spotting them when they happen.

Tips for spotting and reducing hallucinations

You might be wondering, "Can't we just create a hallucination detector?" If only it were that simple! Unfortunately, there's no foolproof way to automatically catch AI hallucinations. The most reliable method is still good old-fashioned fact-checking against trustworthy external sources—including that supercomputer between your ears.

I know, I know—AI is supposed to make things faster and easier, and fact-checking everything you get from an LLM isn’t exactly that. But don't lose hope! While we can't eliminate hallucinations entirely, we can get better at spotting them. Two key strategies are: 1) recognizing situations where an LLM is likely to hallucinate, and 2) using specific prompting techniques to reduce them.

Here are my top tips for staying vigilant against hallucinations:

Mind the training set's limitations

Remember, the LLM doesn’t know everything, particularly when it comes to recent developments or niche topics, so when asking for information that's unusual or highly specific—like detailed technical data—be on high alert for hallucinations. If your query involves something that might not be well-represented in the AI's training set, or if it’s about recent events, there’s a higher chance of the AI fabricating information. LLMs can also reflect biases present in their training data, so be cautious of responses that seem to align too closely with common stereotypes or misconceptions.

Use clear prompting

Good prompting doesn't need fancy techniques, but it does need precision. Clear, well-formed prompts help minimize hallucinations. It's tempting to fire off a quick thought, but taking an extra moment to craft your prompt can save you from hallucination headaches later. That said, if you are working in a hallucination-prone area, it might be worthwhile to brush up on advanced prompts for accuracy.

Be careful about asking it to think holistically

Many hallucinations stem from the way AI processes and generates language. It operates on tokens, predicting the next one based on what came before. This forward-looking, token-by-token approach can lead to errors because the AI lacks a holistic understanding of the text. Be especially wary of responses that would normally require the AI to "look back" at what it's already generated—it can't do that, so these responses are more prone to hallucinations.

Watch for "plausible-sounding nonsense"

I teach a university AI course and this is where my students often get tricked. The model's training to predict the next most likely token can sometimes result in text that sounds convincing but doesn't actually make sense upon closer inspection. So make sure you read everything the AI generates closely!

If it's "too good to be true"...

Thanks to their RLHF training, LLMs can be real people-pleasers. If an answer seems suspiciously perfect or exactly what you wanted to hear, it might be the AI's "eager to please" behavior kicking in. These sycophantic responses prioritize making you happy over being accurate, so give them an extra dose of skepticism.

Be careful with summaries

Summarization is one of the most helpful features of LLMs and it's a drag that the only real answer about how to check whether the summary is accurate is to read the probably lengthy document it comes from. Anytime you're asking for a summary, be extra vigilant about double checking the information with the original source.

Cross-check key facts

Always verify crucial information provided by the AI. This is especially important for names, dates, statistics, and other specific data points. Unless explicitly provided in the prompt, very precise numbers or statistics are often hallucinated, as the pretraining process doesn't focus on memorizing exact figures. And, to avoid the same issue as our lawyer friend above, if the AI mentions a specific source, take the time to look it up to ensure it exists.

Be cautious with multi-step reasoning

The pretraining process doesn't explicitly teach logical reasoning. When an LLM performs multi-step reasoning, each step introduces a chance for error. In these cases, you might benefit from using another tool—like asking the LLM to write code rather than answer in text.  Make sure to verify each step independently.

Conclusion

Hallucinations are, unfortunately, one of the biggest issues with working with AI. Although we're seeing pretty good progress in minimizing them, if you work with AI tools a lot, you're inevitably going to find some cropping up.  If you're still not satisfied with your hallucinations, check out my tips on how to reduce hallucinations.