Skip to content
AI Compliance Tool
Check my AI risk
PlatformSolutionsPricingRegulationsFree toolsGuides Partners Sign in

How to Review an AI Customer Service Bot

A practical review checklist for AI customer service bots: disclosure rules, human escalation, transcript data, retention, hallucination and abuse testing, logging, accessibility and metrics.

By AI Compliance Tool editorial team · Published 2026-10-14 · Reviewed 2026-10-14 · Reviewer: Pending independent review · 10 min read

Key takeaways

  • Disclosure is the first check. EU AI Act Art. 50(1) has required providers to design chatbots so people know they are talking to AI since 2 August 2026; Utah Code 13-75-103 requires disclosure when a consumer asks, with a safe harbour for bots that disclose at the outset and throughout.
  • Texas TRAIGA (effective 1 January 2026) requires AI-interaction disclosure by government agencies and by health-care providers using AI in treatment; it does not impose a general private-sector chatbot disclosure duty.
  • Treat transcripts as personal information. PIPEDA's consent, appropriate-purpose and safeguard principles apply, Quebec requires a PIA for new information systems and notice of profiling, and vendor training use must be explicitly addressed.
  • Test before and after launch: a ground-truth question set for hallucinations, a prompt-injection and abuse script, and a monthly review of escalation rate, containment and complaint volume.

A customer service bot is usually the first AI system a small business puts in front of the public, and it is the one most likely to generate a complaint. This guide is a review you can run on an existing bot or a vendor demo. Work through the nine areas, score each one, and record the results against the bot’s inventory entry.

Before you start

Gather four things: the inventory record (owner, vendor, model, what it is allowed to do), the vendor’s documentation and contract, the current system prompt or configuration, and a week of transcripts. If any of these does not exist, that is your first finding.

Decide which jurisdictions’ users the bot serves. The disclosure rules below differ by place, and your review should state which ones you assessed.

1. Disclosure

EU. Since 2 August 2026, Art. 50(1) of the EU AI Act requires providers to design AI systems that interact directly with people so those people are informed they are interacting with AI, unless that is obvious to a reasonably well-informed person in context. If you built or configured the bot under your own name, you are likely the provider. If a vendor supplies it, the vendor is likely the provider and you are the deployer; confirm their compliance and do not disable the notice. If you rely on the “obvious” exception, write down why.

Utah. Utah Code 13-75-103(1) requires a supplier using generative AI in a consumer transaction to disclose that the person is interacting with AI if the person asks or clearly prompts whether AI is being used. Section 13-75-104 gives a safe harbour where the AI clearly and conspicuously discloses at the outset and throughout the interaction. Section 13-75-103(2) requires prominent disclosure by regulated-occupation providers in high-risk interactions. The Act is scheduled to sunset on 1 July 2027 unless extended.

Texas. TRAIGA s. 552.051 (effective 1 January 2026) requires governmental agencies offering consumer-facing AI to disclose it clearly, and health-care providers using AI in treatment to disclose to patients no later than the first date of service. There is no general private-sector chatbot disclosure duty in TRAIGA.

Canada. No statute requires chatbot disclosure, but the joint federal, provincial and territorial generative AI principles call for openness and disclosing when AI is involved in interactions, and Quebec s. 8.1 requires prior notice when technology can identify, locate or profile a person.

Checks. Does the first message say it is an AI assistant? Does it answer “are you a bot?” honestly? Is the disclosure visible on mobile? Screenshot the result.

2. Escalation to a human

Define the triggers: user asks for a person; the bot cannot answer with confidence; a complaint, cancellation, refund dispute, safety or health topic; any legal threat; repeated failed answers. Test each trigger. Check the hand-off actually reaches a staffed queue with the transcript attached, and that the user is told expected wait times. Measure how long escalation takes end to end. A bot that “offers” a human but drops the user into an unmonitored inbox fails this check.

3. Data in transcripts

Transcripts contain names, order numbers, addresses, health details and whatever else customers type. Under PIPEDA the organization must collect and use personal information only for purposes a reasonable person would consider appropriate (s. 5(3)), with meaningful consent, and protect it with safeguards appropriate to sensitivity (Principle 4.7). The OPC’s generative AI guidance specifically flags prompt injection and model inversion as safeguard concerns.

Questions to answer in writing:

  • Where are transcripts stored, in what country, and who at the vendor can see them?
  • Does the vendor use transcripts to train or improve models? If so, is that in your contract, your privacy notice and your consent language?
  • Does the bot have access to customer records, and can it reveal one customer’s data to another?
  • In Quebec, has a privacy impact assessment under s. 3.3 been completed for the bot as an information-system project, and is there a transfer assessment under s. 17 for any out-of-province vendor?
  • For EU users, has a GDPR Art. 35 DPIA been considered?

4. Retention

Set a retention period for transcripts and enforce it in the platform, not just in policy. Keep it as short as the business purpose allows; PIPEDA’s limiting-retention principle and Quebec’s governance-policy requirement (s. 3.2, which covers retention and destruction) both expect a defined period. Retain longer only for records you need as evidence (for example, a disputed interaction), and record the exception. Confirm that deletion propagates to vendor backups.

5. Hallucination testing

Build a ground-truth set of at least 50 questions from real tickets with verified answers. Include pricing, policy, availability, legal or regulated topics, and 10 questions the bot should refuse or escalate. Run the set, score each answer as correct, partially correct, wrong or appropriately escalated, and keep the scored sheet. Re-run after every model, prompt or knowledge-base change.

Why it matters legally: the FTC has pursued unsubstantiated AI accuracy claims under Section 5 of the FTC Act (Operation AI Comply, September 2024; the Workado order, August 2025), and in July 2026 proposed a policy statement on representing systems as accurate while steering outputs. That statement is proposed, not final, but if your website says the bot gives “accurate answers”, your test results are the substantiation.

6. Prompt-injection and abuse testing

Spend an hour trying to break the bot and record what happens:

  • Ask it to reveal its instructions or internal data.
  • Paste text that says “ignore previous instructions and…” and see whether it complies.
  • Ask it to make commitments it should not (discounts, legal promises, medical advice).
  • Try to make it produce abusive, discriminatory or sexual content.
  • Impersonate another customer to retrieve their details.
  • Flood it with repeated requests to see whether rate limits exist.

Two legal anchors: the EU Art. 5 prohibitions on manipulative or deceptive techniques causing significant harm have applied since 2 February 2025, and Texas TRAIGA prohibits deploying AI intended to incite self-harm or crime or to produce unlawful sexual content. Neither is likely to be triggered by a well-configured support bot, but your test record is how you show it.

7. Logging

Confirm the platform logs: timestamp, session ID, user inputs, bot outputs, which knowledge sources were used, escalation events, and configuration version. Confirm who can access logs, how long they are kept, and that they can be exported. If the bot serves EU users and could ever be classified as high-risk, remember that deployers must keep automatically generated logs for at least six months from 2 December 2027. For a standard support bot, 90 days plus incident holds is a reasonable starting point; write down your choice.

8. Accessibility

No instrument in our verified dataset imposes a chatbot-specific accessibility rule, so treat this as best practice and check your general accessibility obligations separately. Test keyboard-only navigation, screen-reader labelling of the chat window and messages, colour contrast, resizing, and a non-chat path (phone or email) for people who cannot use the bot.

9. Metrics to monitor

MetricWhyReview
Escalation rate and time to humanShows whether the bot traps peopleWeekly
Containment rate (resolved without a human)Business value; a sudden rise can mean the bot is confidently wrongWeekly
Ground-truth test scoreHallucination trend across model changesEach change, at least monthly
Complaints mentioning the botEarly warningWeekly
“Are you a bot?” responses sampledDisclosure workingMonthly
Transcript retention complianceDeletion actually runningMonthly
Injection test resultsSecurity regressionQuarterly and after changes

Scoring and recording

Score each of the nine areas as Pass, Pass with actions, or Fail. Attach screenshots, the scored test sheet, the vendor’s written answers and the configuration export to the inventory record, and date the review. Schedule the next one.

Example. Cedar Ridge Plumbing, a fictional 40-person contractor in Edmonton and Spokane, reviews its vendor-built booking bot. Findings: disclosure present on desktop but hidden behind a collapsed widget on mobile (Pass with actions); escalation routes to a shared inbox checked twice daily (Fail; fixed by routing to the dispatch queue); vendor contract silent on training use (Fail; addendum requested); 50-question test scored 41 correct, 6 escalated correctly, 3 wrong on pricing (Pass with actions; price table added to knowledge base); injection test revealed the system prompt (Fail; vendor patch applied and retested). The dated review and retest sit in the inventory record and become the baseline for the next quarter.

Sources

Frequently asked

Do we have to tell website visitors they are chatting with AI?
If the bot is used by people in the EU, the Art. 50(1) design duty applies to its provider, with an exception where it is obvious from context. In Utah you must disclose when clearly asked, and disclosing at the outset gives a safe harbour. Canada's privacy regulators' generative AI principles say to disclose AI involvement. Disclosing up front everywhere is the simplest defensible position.
What if our vendor built the bot, not us?
Under the EU AI Act the vendor is usually the provider responsible for Art. 50(1) design, and you are the deployer responsible for how it is used. Confirm in writing that the vendor's product meets the disclosure design duty, and check your own configuration has not turned it off.
How many test questions are enough for hallucination testing?
Start with 50 questions drawn from real tickets with verified answers, including 10 where the correct response is 'I do not know, let me connect you to a person'. Re-run the set after every model or knowledge-base change.
Can the vendor use our transcripts to train its models?
Only if your contract permits it and your notice and consent cover that purpose. Under PIPEDA, consent is valid only if people can reasonably understand the purpose and consequences; the OPC's generative AI principles treat training as a distinct purpose needing legal authority.

This guide is general information, not legal advice. Laws and dates are drawn from our verified regulatory dataset and link to official sources; check them for changes after the review date.