In February 2023, Air Canada's AI chatbot told a grieving passenger that the airline offered a "bereavement fare" discount that he could apply for retroactively after purchase. No such policy existed. The chatbot had hallucinated it. Air Canada was subsequently ordered by Canadian courts to honour the discount and pay damages—a landmark case that established that businesses are liable for the false outputs of their own AI systems.
This is the hallucination problem in its real-world form. An LLM is not a database lookup—it is a sophisticated pattern-matching probabilistic system that generates text it believes is plausible given its training, without any internal mechanism for verifying the factual accuracy of what it produces. The model does not "know" it is wrong. It has no flag for uncertainty unless specifically engineered to produce one.
Why LLMs Hallucinate: The Technical Root Cause
LLMs are trained to predict the most statistically probable next token (word fragment) given a preceding sequence. They are not trained to retrieve facts from a verified database. This means that when the model encounters a question whose answer falls outside its confident training data distribution—a very recent event, an obscure company-specific fact, a precise numerical calculation—the model does not stop and say "I don't know." It generates what sounds like the correct answer based on the linguistic patterns in its training data.
The result can be convincingly formatted falsehoods: citations to academic papers that do not exist, statistics that were never published by the organizations cited, legal clauses from laws that have never been enacted. The output reads professionally and confidently—which is precisely what makes it dangerous.
The Enterprise Risk Categories
Not all hallucinations are equally dangerous. For enterprise risk management, it helps to classify them by consequence severity:
- Low-Risk Hallucinations: Minor factual errors in non-consequential contexts (e.g., an AI copywriter getting a historical date slightly wrong in a blog post). These are annoying but not operationally harmful. Human review catches most of them.
- Medium-Risk Hallucinations: Confidently incorrect product information in customer-facing chatbots, wrong pricing in AI-generated quotes, or incorrect technical specifications in AI-assisted documentation. These can lead to customer dissatisfaction, refund requests, and brand damage.
- High-Risk Hallucinations: False legal advice from AI legal assistants, incorrect medical information from healthcare AI, fabricated financial data from AI financial analysts, or—as in the Air Canada case—invented policies from customer service AI. These hallucinations create significant legal liability and financial exposure.
Seven Mitigation Strategies for Production AI Systems
1. Retrieval-Augmented Generation (RAG) for Factual Queries
For any AI system that must answer factual questions about your business, products, or policies, implement RAG. Rather than letting the model generate answers from its training data (which may be outdated or incorrect), RAG first retrieves the relevant content from your verified knowledge base and then asks the model to synthesize an answer from that retrieved content. If the information is not in the knowledge base, the model can be instructed to say so rather than generating speculative content.
2. System Prompt Guardrails
The system prompt (the instructions given to the model before the user conversation begins) is your first line of defense. Well-designed system prompts instruct the model to acknowledge uncertainty, refuse to answer questions outside its defined scope, cite its sources, and never fabricate specific statistics or policy details without a direct reference it can provide.
3. Confidence Scoring and Abstention
Certain fine-tuning and prompting techniques can instruct a model to output a confidence score alongside its answers, or to "abstain" (refuse to answer) when its internal estimate of answer reliability falls below a threshold. This is particularly important in high-stakes domains like healthcare or legal advice.
4. Output Verification Pipelines
For high-risk AI outputs (financial figures, contractual terms, medical recommendations), implement automated post-processing that scans the model's output for claims that can be verified against authoritative internal data systems. Flag any output containing a specific numerical claim that cannot be traced to a verified source before it reaches the end user.
5. Human-in-the-Loop for High-Stakes Decisions
Not everything should be fully automated. For any AI output that informs a consequential business decision (a contract clause, a credit approval, a medical diagnosis suggestion), design a mandatory human review checkpoint into the workflow. The AI accelerates the human's work—it does not replace their judgment on consequential matters.
6. Continuous Red-Teaming
Regularly and systematically attempt to make your own AI system hallucinate. Hire or assign a team to act as adversarial users, probing the model with questions designed to elicit false confident responses. Every hallucination discovered in red-teaming is a hallucination discovered before a real customer does.
7. Model-Level: Prefer Smaller, Fine-Tuned Models Over Large General Models for Constrained Domains
Counterintuitively, a smaller model fine-tuned specifically on your domain's verified data often hallucmates less frequently than a giant general-purpose model for domain-specific questions, because the fine-tuned model's "knowledge" is scoped to what it has been explicitly taught. For narrow, constrained use cases, smaller and more specialized is often safer than larger and more general.
The Governance Imperative
The Air Canada ruling sets a clear precedent: organizations cannot disclaim responsibility for their AI systems' outputs by treating them as a "separate entity." Before deploying any customer-facing or decision-support AI, your organization must have a clear framework for: defining acceptable use cases, establishing output quality standards, creating audit trails of AI decisions, and defining remediation processes when the AI causes harm to a customer.
AdaptNXT builds enterprise AI systems with responsible AI governance baked in from the architecture stage—not retrofitted after a crisis. Talk to us about building AI systems your business can trust.
