Arbitrary-fact Hallucinations | Post by Rodrigo Cardenete

LLMs come with a well-known failure that hit the fan when you plug them into an AI agent: "arbitrary-fact hallucinations".

This happens when an LLM generates a false statement that is not grounded in any training data or documented facts.

Basically, it makes things up when it doesn’t know the answer.

The worst part is that they are not even lying: they are not aware of their own mistake (if we can call that awareness in any sense).

This is common when an LLM is asked about chess moves, legal precedents, or math computations (now most AIs have attached calculators, but the LLM itself still hallucinates with basic maths most of the time).

The origin of this problem goes back to how models are trained.

Overly simplified: during training, models get rewarded when they “guess” the next correct word from a large training dataset. After a large number of iterations, that learning gets baked into their "weights", the very parameters they will use to make decisions in the future.

But they are not rewarded for saying “I don’t know,” which is exactly what a wise person would do.

As you can imagine, this becomes a major problem the moment an agent starts making decisions in a business process. To mitigate that, there are only a few options for now:

✅ 1. Wait until new models improve by rewarding "honesty" or uncertainty during training.

✅ 2. Some people are incorporating "confidence scores": only decisions above certain confidence threshold are considered. But this is impractical in most real-world cases, especially with AI agents where latency and token efficiency matter.

✅ 3. Use evals in your AI workflows: one model checks the previous model’s output for accuracy, focusing on known hallucination patterns (that's what we do)

The paper “Why Language Models Hallucinate”, published by OpenAI last month (link in the comments), is a great read. It studies most types of hallucinations, including this peculiar side effect from how models are rewarded.

This helps us understand the famous quote: "Hallucinations aren’t a bug, they're a feature".