Air Canada’s chatbot became famous for the wrong reason: it gave a traveler bad information about bereavement fares, and the airline was later held responsible for that answer. For anyone building customer service automation, the story is a giant flashing airport sign that says, “Please verify your facts before takeoff.”
The case is also a perfect way to explain why Retrieval-Augmented Generation, or RAG, is often better than a plain large language model when the job involves policies, prices, refunds, legal terms, medical details, financial rules, or any other fact that cannot afford to be “close enough.” A standard LLM can sound confident. A RAG system can be designed to look up the right source before speaking. That difference matters.
In simple terms, an LLM is great at language. RAG is better at grounded answers. When a customer asks, “Can I apply for a bereavement fare after travel?” the safest answer should come from the current airline policy, not from a model’s statistical memory, optimistic guesswork, or charming ability to write a sentence that sounds like it came from a very polite airport employee.
What Happened With Air Canada’s Chatbot?
The Air Canada chatbot case involved a passenger, Jake Moffatt, who used the airline’s website chatbot while arranging travel after a family death. The chatbot reportedly told him he could apply for a bereavement fare refund within a certain period after booking. Based on that guidance, he bought tickets and later submitted his refund request. Air Canada denied the request, pointing to its actual bereavement travel policy, which did not allow that kind of post-travel refund.
The important part is not just that the chatbot was wrong. Mistakes happen. The important part is that the company tried to distance itself from the chatbot’s response. The tribunal did not accept the idea that the chatbot was somehow separate from the company’s website. In plain English: if your business puts a chatbot in front of customers, the chatbot is not an intern hiding in the broom closet. It is part of your customer experience.
The ruling sent a clear message to companies using AI chatbots: automation does not erase accountability. If a chatbot gives incorrect information about a company policy, the customer may reasonably rely on that information. That creates legal, reputational, and operational risk. It also creates a technical question every business should ask before launching a chatbot: where does the answer come from?
Why A Plain LLM Can Be Risky For Factual Answers
Large language models are powerful because they learn patterns from massive amounts of text. They can summarize, rewrite, classify, translate, brainstorm, and explain complex topics in friendly language. That is impressive. But a plain LLM is not automatically a source-of-truth system.
An LLM does not “know” company policy in the same way a database knows a refund rule. It generates likely text based on patterns, instructions, and available context. If the model has not been given the current policy, it may produce an answer that sounds plausible but is inaccurate. This is commonly called hallucination, though in business settings “expensive nonsense with excellent grammar” is sometimes the more accurate phrase.
Policy changes make the problem worse
Company policies change. Airline fare rules change. Insurance coverage changes. Software pricing changes. Return windows change. Legal disclosures change. A general LLM trained at one point in time cannot reliably know what changed yesterday, last week, or five minutes ago. Even if the model once saw a similar policy, the current version may say something different.
Confidence is not accuracy
LLMs are especially tricky because they can be fluent even when wrong. A clumsy error is easy to spot. A polished error is dangerous. A chatbot that says, “Yes, you may apply after travel by completing the online form within 90 days,” sounds helpful, specific, and official. If that sentence conflicts with the real policy, the customer does not care that the model had great conversational flow. The customer cares that the answer cost money.
What Is RAG?
Retrieval-Augmented Generation is an AI design pattern that connects a language model to an external knowledge source. Instead of asking the model to answer from memory alone, the system first retrieves relevant documents, policies, database records, help center articles, product manuals, or other approved material. Then the model uses that retrieved context to generate an answer.
A basic RAG workflow looks like this:
- The user asks a question.
- The system searches trusted content for relevant passages.
- The best matches are added to the model’s prompt.
- The model answers using the retrieved material.
- The system can display citations, source titles, timestamps, or confidence signals.
That last part is crucial. For factual use cases, the answer should not be a mysterious sentence floating out of the AI fog. It should be traceable to a policy, page, contract clause, product document, or approved knowledge base entry.
Why RAG Is Better Than An LLM For Facts
RAG is not better than an LLM at everything. If you want a catchy slogan, a product description, or a friendly email draft, a strong LLM may be enough. But when the question is factual, current, regulated, or tied to money, RAG has major advantages.
1. RAG grounds answers in approved sources
A customer service chatbot should not invent policy. It should retrieve policy. With RAG, the organization can choose the trusted documents the chatbot is allowed to use: fare rules, refund terms, help center pages, legal notices, tariff documents, HR policies, warranty details, or internal support playbooks.
That means the answer is based on the company’s actual source material, not on whatever the model thinks a policy probably says. The difference is like asking a lawyer to check the contract versus asking a confident stranger at baggage claim.
2. RAG can use updated information
RAG systems can be connected to regularly updated knowledge bases. When the refund policy changes, the document can be updated and re-indexed. The chatbot can then retrieve the latest version. A plain LLM, by contrast, may not know about the update unless the new information is explicitly included in the conversation or the model is retrained or otherwise refreshed.
This makes RAG especially useful for businesses with fast-moving information: airlines, banks, healthcare organizations, SaaS companies, retailers, universities, government agencies, and subscription services.
3. RAG makes answers easier to audit
When something goes wrong, companies need to investigate. What did the bot say? Which document did it use? Was the document outdated? Did retrieval fail? Did the model ignore the retrieved text? Was the policy itself unclear?
A well-built RAG system can keep logs showing the user question, retrieved sources, generated answer, and source version. That audit trail is priceless. Without it, debugging an AI mistake can feel like interrogating a parrot with a law degree.
4. RAG helps reduce hallucinations
RAG does not eliminate hallucinations completely. No serious AI architect should promise magic. But grounding the model in retrieved evidence can reduce the chance that it fabricates details, fills gaps with guesses, or blends old and new information into one very convincing mess.
The key is to design the system so the model is instructed to answer only from retrieved sources. If the sources do not contain the answer, the bot should say so and escalate. “I don’t know” is not a failure when the alternative is “Here is a refund policy I invented while wearing a tiny digital pilot hat.”
5. RAG supports citations and transparency
For high-stakes facts, users should be able to see where an answer came from. A RAG chatbot can provide a link to the relevant policy page, show a quoted passage, or identify the document used. This gives customers more confidence and gives the company a cleaner record of what was communicated.
The Air Canada Lesson: The Source Of Truth Matters
The Air Canada chatbot problem was not merely a “chatbot problem.” It was a source-of-truth problem. The correct policy existed elsewhere on the website, but the chatbot gave a conflicting answer. From the customer’s perspective, both were Air Canada communications. Why should a traveler assume the static policy page was official but the chatbot was not?
This is exactly where RAG architecture helps. If the chatbot had been required to retrieve the current bereavement travel policy before answering, the system could have responded differently. For example:
“According to Air Canada’s bereavement travel policy, bereavement fares must be requested according to the policy requirements and may not be available as a refund after completed travel. Please review the bereavement travel page or contact support before booking.”
That answer is less flashy, but it is safer. It also gives the customer a clear next step. In customer service, boring and correct usually beats charming and wrong. Nobody wants a chatbot with jazz hands when money is on the line.
How A RAG Chatbot Should Be Built For Customer Support
Building a reliable RAG chatbot is not just a matter of plugging documents into a model and hoping the robot behaves. A good system needs careful design, testing, monitoring, and governance.
Start with authoritative content
The first step is deciding what counts as truth. For an airline, that might include fare rules, refund policies, passenger rights pages, tariff documents, support scripts, and official announcements. For a healthcare company, it may include reviewed medical content, insurance details, appointment policies, and safety disclaimers.
If the knowledge base is messy, outdated, or contradictory, the chatbot will inherit that mess. RAG is not a washing machine for bad content. It is more like a very fast librarian. If the library shelves are chaos, the librarian may still bring back the wrong book.
Use retrieval that understands meaning
Modern RAG systems often use vector search, keyword search, semantic ranking, or hybrid retrieval. Vector search helps find passages with similar meaning, even when the user does not use the exact wording. Keyword search helps with specific terms such as “bereavement fare,” “refund,” “tariff,” or “completed travel.” Hybrid retrieval often works well because it combines both approaches.
Control the model’s behavior
The model should be instructed to answer from the retrieved context, avoid unsupported claims, ask clarifying questions when needed, and escalate when the policy is unclear. For example, if a customer asks about a refund after travel and the retrieved policy says post-travel refunds are not allowed, the bot should not soften that answer into “You may be eligible.” Friendly wording is fine. False hope is not.
Add guardrails for high-risk topics
Some questions should trigger extra caution. Refunds, cancellations, legal rights, health advice, financial recommendations, and identity verification all deserve stricter rules. A RAG chatbot can be configured to provide general information, cite the relevant policy, and route the user to a human agent before a final decision is made.
Test with real customer questions
Do not test a chatbot only with neat questions written by the product team. Real users type messy things. They use slang. They misspell. They ask three questions at once. They say “my grandma passed and I already flew, can I still get the discount?” A good RAG evaluation set should include realistic, emotional, ambiguous, and edge-case questions.
Monitor, log, and improve
RAG systems need ongoing review. Companies should monitor unanswered questions, low-confidence retrievals, customer complaints, and policy conflicts. When the bot gives a poor answer, the fix might involve updating the source document, improving retrieval, changing the prompt, adding a rule, or creating a human escalation path.
RAG Is Not Magic, But It Is A Better Seatbelt
RAG is not perfect. Retrieval can fail. The wrong document can be selected. A policy can be poorly written. The model can still misread context. Source documents can contain contradictions. That is why serious RAG systems use layered defenses: better content management, retrieval evaluation, source ranking, answer validation, human review, and clear escalation.
Still, RAG is a better architecture for factual answers than relying on an LLM alone. It gives the model something concrete to use. It makes answers more current. It creates a path for citations. It gives companies a way to audit what happened. Most importantly, it aligns the chatbot with the organization’s actual policies.
Think of the LLM as the friendly speaker and RAG as the fact-checking desk behind it. The speaker may have charisma. The fact-checking desk keeps the company out of trouble.
Business Takeaways From The Air Canada Chatbot Case
The Air Canada chatbot story should be required reading for any company that wants to automate customer service. The lesson is not “never use AI.” That would be like banning airplanes because turbulence exists. The lesson is “use AI with the right architecture, controls, and accountability.”
Here are the biggest takeaways:
- Do not let a chatbot answer policy questions without grounding. Connect it to approved, current content.
- Do not assume disclaimers will save you. If the bot is on your website, customers may treat it as your voice.
- Do not make the user resolve contradictions. If the chatbot and policy page disagree, the company has a problem, not the customer.
- Do not treat AI as a separate actor. The business owns the customer experience.
- Do build audit trails. Save source versions, retrieved passages, and generated responses.
- Do escalate when stakes are high. A human handoff can be cheaper than a legal dispute.
Where RAG Fits In The Future Of AI Customer Service
The future of AI support will not be one giant model answering everything from memory. The better future is a network of specialized systems: LLMs for language, retrieval systems for facts, databases for account information, workflow tools for actions, and human agents for judgment.
For example, a travel chatbot might use RAG to answer questions about fare rules, a booking system API to check a passenger’s itinerary, a policy engine to determine eligibility, and a human agent for exceptions. The LLM ties the experience together in natural language, but it should not be the only brain in the room.
This division of labor is healthier for users and safer for companies. Customers get faster answers. Businesses get better consistency. Support teams spend less time correcting avoidable mistakes. Legal teams sleep slightly better, which is good, because legal teams are not known for being cuddly when chatbots improvise refund policies.
Experience-Based Insights: What This Looks Like In Real AI Support Projects
In practical customer support projects, the biggest surprise is usually not that AI can answer questions. Everyone expects that now. The surprise is how quickly small content problems become big chatbot problems. A company may have five versions of the same policy scattered across a help center, PDF, internal wiki, archived announcement, and support macro. A human agent knows which one is current because they have experience. A retrieval system needs that knowledge to be encoded clearly.
That is why the most successful RAG projects often begin with content cleanup. Teams remove outdated pages, label official documents, add effective dates, create ownership rules, and decide who approves changes. It is not glamorous work. No one throws a launch party for “we deleted 37 stale refund articles.” But this is where chatbot accuracy is won.
Another common experience is that users ask questions differently than companies write policies. A policy might say, “Requests for post-completion fare adjustment are ineligible.” A grieving traveler might ask, “Can I get the bereavement discount after I already flew?” Those are the same issue in human terms, but not always in search terms. Good RAG systems bridge that gap by combining semantic retrieval, keyword matching, synonyms, and carefully written answer rules.
Testing also reveals how important refusal is. Many organizations want their chatbot to answer everything because unanswered questions feel like failure. In fact, the safest enterprise chatbots know when to stop. A good bot says, “I could not find an approved policy that answers this,” or “This request needs an agent.” That may feel less magical, but it builds trust. The worst chatbot is not the one that says it does not know. The worst chatbot is the one that invents a confident answer and sends the customer on a three-month email hike.
Another lesson is that citations change user behavior. When a chatbot includes the source of its answer, customers are more likely to trust it, and support agents have an easier time reviewing it. Citations also help internal teams spot weak content. If the bot keeps citing an old page, someone can fix the index or update the source. If the bot cannot find a policy, the company may discover that the policy was never written clearly in the first place.
Finally, RAG works best when companies treat it as an operational system, not a novelty feature. It needs owners, quality metrics, feedback loops, escalation paths, and regular audits. The technology can be impressive, but governance is what keeps it useful. In the Air Canada situation, the core issue was not whether chatbots are cool. They are cool. The issue was whether the chatbot could be trusted with a factual, policy-based answer. For that kind of task, RAG is not just a technical upgrade. It is common sense with a search index.
Conclusion
Air Canada’s chatbot case is a powerful reminder that businesses cannot outsource accuracy to a friendly interface. A chatbot may look like software, but to customers it feels like the company speaking. When that voice gives the wrong answer, the consequences can include refunds, complaints, legal exposure, and public embarrassment.
RAG offers a smarter path. By retrieving current, approved information before generating an answer, a RAG chatbot can be more accurate, transparent, and auditable than a plain LLM working from memory alone. It is not perfect, and it still requires careful design, but it is much better suited for facts, policies, and customer decisions.
The short version for businesses is simple: let the LLM handle the conversation, but let RAG bring the receipts. Your customers deserve accurate answers. Your legal team deserves fewer surprises. And your chatbot deserves a source of truth before it starts handing out refund advice like candy at a boarding gate.
