Skip to content

Why does our company chatbot invent prices, stock and policies?

  • Home
  • Blog
  • Why does our company chatbot invent prices, stock and policies?
Why does our company chatbot invent prices, stock and policies?
Short answer

If your chatbot confidently gives wrong prices, refund policies or stock levels, the problem is usually not the model itself but the context it is given. Language models fill a gap in their knowledge base with the statistically most likely text; NIST calls this behaviour “confabulation”. The lasting fix is clean, current source data, a RAG pipeline with sensible chunking and hybrid search, mandatory citations, explicit permission to say “I don’t know”, and regular evaluation.

Symptoms: the bot is confident but wrong

  • It tells a customer “You have a 30-day no-questions-asked return right on this product.”, a sentence that appears nowhere in your published returns policy.
  • It quotes the old price for an updated product, or mixes two price lists and produces a third figure that exists nowhere.
  • It says “In stock, ships today.” although the bot is not connected to the stock system at all.
  • Asking the same question twice returns two contradictory answers.
  • When asked for a source, it cites a document name or page number that does not exist.
  • An employee can learn information from a document they are not authorised to see (a salary table, a quote margin) simply by asking the bot.

Root cause: why does the language model make things up?

NIST’s Generative Artificial Intelligence Profile (NIST AI 600-1), published in July 2024, defines “confabulation” as generative AI confidently presenting erroneous or false content, and states that it is a natural result of how these models are designed: they generate outputs that approximate the statistical distribution of their training data. In other words, the model does not answer “record not found” like a database; it builds a fluent, convincing sentence. Your company-specific prices, stock and policies are not in the model’s training data; if they are not in the context either, the gap is filled with a guess.

The established answer to this is RAG (Retrieval-Augmented Generation). The 2020 paper by Lewis et al. combined the “parametric” memory stored in the model’s weights with a “non-parametric” memory, a searchable document index; according to the paper, this approach generates more specific and factual text and makes it easier to trace which source an answer is based on. But RAG is only as good as the passages it retrieves. A poorly fed RAG system makes the model even more confident with the wrong context.

Why “just give it our PDFs” does not work

  • Bad chunking: tables are split row by row, the product name ends up in one chunk and the price in another. The model matches them by guessing.
  • Stale and conflicting documents: last year’s price list sits in the same index as the current one; whichever the search returns first shapes the answer.
  • Retrieval misses: the user says “refund”, the document says “right of withdrawal”. Microsoft’s Azure AI Search documentation lists the mismatch between user wording and document terminology as one of the core challenges of RAG.
  • No live data: constantly changing information such as stock and prices lives in the ERP, not in a static PDF. A PDF starts going stale the moment it is uploaded.
  • No access control: if every document goes into one index, the bot can read everything to everyone. The same Microsoft document stresses that users must only retrieve content they are authorised to see, which is achieved through document-level security trimming.
  • Missing instructions: if the system prompt does not say “answer only from the provided sources and say you don’t know if the answer is not there”, the model fills the gap with general knowledge.

Step-by-step fix: building RAG properly

  1. Build a question inventory: list the types of questions the bot must answer and define a single “source of truth” for each. For example, the ERP price table for prices, the published policy page for returns, and the live stock service for availability.
  2. Clean the source data: remove old versions from the index; add version, effective date and content owner to every document. If two documents conflict, the content owner must decide, not the bot.
  3. Fetch volatile data through tools: for prices and stock, define a read-only API call (tool or function calling) to the ERP instead of a document. The model then reads the figure from the system rather than generating it.
  4. Chunk according to the content structure: keep headings, sections and table rows together; add metadata such as source document name, section heading and date to every chunk.
  5. Use hybrid search and re-ranking: run keyword search and vector search together and re-rank the results semantically. Add a synonym map for terms such as refund, withdrawal and return, and set a minimum threshold for low-scoring results.
  6. Enforce access control at query time: attach permission metadata to every chunk and filter results by the identity of the user asking the question.
  7. Tighten the system prompt: Anthropic’s documentation states that explicitly allowing the model to say “I don’t know”, grounding answers in direct quotes and having it verify each claim with a source reduce hallucinations, but do not eliminate them entirely. An instruction like the one below is a good starting point.
  8. Create an evaluation set: prepare a set of real customer questions, with personal data removed and known correct answers. Include questions the bot should not answer. After every change, run the set and compare the rates of correct answers, correct sources and correct “I don’t know” responses.
  9. Add guardrails and human approval: route answers that create commitments, such as price quotes, refund approvals or contract interpretations, to a human. Add a simple output check that compares figures in the answer with the source.
  10. Keep logs: record the question, retrieved chunks, answer given and user feedback together. This lets you tell whether a wrong answer came from retrieval, the source or the instructions. Set the retention period in line with your data protection obligations.
You are the customer assistant for [Company].
- Answer only on the basis of the information inside <sources>.
- End every sentence with the ID of the source you used, as [source-id].
- Take price and stock information only from the check_stock_price tool; never estimate.
- If the sources do not contain the answer, say: "I do not have verified information on this, I am transferring you to a colleague."
- Do not approve refunds, grant discounts or interpret contracts; hand these requests to a human agent.

Instructions alone are not enough. If retrieval returns the wrong chunk, even the best prompt will give the wrong answer, complete with a citation. Measure search quality with the evaluation set first.

Lasting prevention

  • Content ownership: give every document group an owner; when a document is updated, the index should refresh incrementally and automatically.
  • Regression testing: rerun the evaluation set whenever the model version, system prompt or chunking settings change; if results drop, do not release.
  • Uncited answer report: report answers given without a citation and “I don’t know” answers weekly; together they are your list of missing content.
  • Regular human review: pick random samples of conversations and have an expert check them. As in the NIST AI RMF approach, treat risk management as a continuous process, not a one-off task.
  • Transparency for users: state clearly that the bot is an AI assistant, which topics it can help with and how to reach a human agent.

When should you call in an expert?

If the bot gives customers information on prices, stock or contracts, if your data is scattered across several systems such as the ERP and file servers, or if employees with different permission levels use the same assistant, this is a data and architecture project rather than a prompt fix. That is also the picture we usually see in the field at businesses in Denizli: the model is fine, the source data is scattered. As part of its AI-based business solutions, Workas supports companies with organising source data, ERP integration, permission-aware RAG deployments and building evaluation sets.

Frequently Asked Questions

Will a bigger or newer model fix hallucinations?

Not on its own. NIST states that confabulation is a natural result of how generative models are designed. If company-specific information is missing from the context, even the best model will guess; the real improvement comes from correct source data and good retrieval.

What is the difference between RAG and fine-tuning?

Fine-tuning changes the model’s style and behaviour; RAG adds current documents to the context for every question. For frequently changing prices, stock and policies, RAG and tool calling are a better fit, because you do not need to retrain the model when the data changes.

Won’t customer satisfaction drop if the bot says “I don’t know”?

Promising a wrong price or a non-existent refund right is far more costly. If you combine “I don’t know” with a hand-over to a human agent, the customer is not left stranded and you learn which content is missing.

Do mandatory citations stop fabrication completely?

No, but they make answers auditable. Anthropic also notes that these techniques reduce hallucinations without eliminating them; critical answers still need human approval and an output check.

Sources

  1. Lewis et al. (2020) – Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv)
  2. NIST AI 600-1 – AI RMF: Generative Artificial Intelligence Profile
  3. Microsoft Learn – RAG and Generative AI (Azure AI Search)
  4. Anthropic – Reduce hallucinations

Let's Get Started

Let's Work Together!

Let's plan your company's IT needs together — we'll quickly find the right solution for you.

x

Since 2002, Workas Software Technology has provided businesses in Denizli with ERP, cybersecurity, network, server and backup, custom software and AI solutions.

Address
Kınıklı, Hüseyin Yılmaz Cd. No:67,
B Blok Z06, 20160 Pamukkale/Denizli, Türkiye
Get in Touch
0549 669 99 06
0535 089 29 29
Working Hours
Mon - Sat: 8:00 - 18:00