This isn’t a developer’s guide. It’s a practical checklist for teams commissioning or launching a chatbot. We encourage our clients to test their chatbot properly throughout the development process.
The gap between “it works” and “it works well for our customers” is where chatbot projects either succeed or become expensive mistakes. We’ve tested that the chatbot can answer questions, but has it handled the actual messy, frustrated, confused questions your real customers, service users will ask at 10 PM when they need help?
This is where you, as the client, can step in. You don’t need technical knowledge to test your chatbot, you just need to know your service users. Chatbot testing can be ineffective when people are too polite or polished in testing. Real users are not.
Why Your Perspective Matters More Than You Think
You know your customers in ways our development team doesn’t. We build what you asked for based on specifications and requirements. But specifications rarely capture the full reality of how people communicate when they need help.
You’ve read the support emails. You’ve heard the phone calls. You know that customers don’t ask “What are your business hours?” in a neat, predictable way. They ask “Are you guys open right now?” or “Can I still place an order?” or just type “hours???” with three question marks because they’re in a hurry.
And it’s not just about getting the right answer. Research shows that chatbots can actively harm the experience when they feel creepy, intrusive, or rigid. Customers are quick to lose trust if a bot feels like it’s prying, ignoring context, or forcing them down a narrow script that doesn’t match what they’re asking. A chatbot that technically works but makes users uncomfortable or frustrated is doing more damage than one that simply says “I don’t know.”
How patient are your service users? Studies confirm customers are less satisfied the slower a chatbot responds but adding typing indicators negates the effect. The most successful chatbot launches happen when clients take an active role in testing, bringing their customer knowledge into the process. You don’t need technical expertise to do this effectively.
How to Test Without Technical Knowledge
Start with the questions you know customers will ask, but don’t ask them the “right” way. Type them the way a frustrated customer would at the end of a long day. Use abbreviations. Make typos. Skip punctuation. Ask two questions in one message. Ask the same question in a variety of ways.
Test the handoff to human support extensively. Customers will inevitably ask questions your chatbot can’t answer, and how that transition happens determines whether people feel helped or abandoned. Try to reach a human agent. See what happens when you ask for help in the middle of the chatbot providing an answer. If your chatbot hands off to a human check whether the human agent has context about what you were asking the chatbot, determine if that is a pain point for your customers.
What You Need to Test Your Chatbot
Start by gathering sources that reflect how your customers actually interact with your business. We’d highly encourage you to review the resources from your onboarding workshops, specifically your user personas, to continue testing chatbot. You can then review conversations and sort them using our health score.
Some of the documents that might hold the answers to your customers question could include:
- Call logs and emails: Look at the questions and complaints your team receives most often. These show the real language, frustrations, and phrasing customers use.
- FAQs and support documentation: These highlight predictable questions and give your chatbot a solid foundation for answers.
- Customer feedback surveys/ customer reviews: Insights from surveys can reveal patterns, common pain points, and areas where people feel confused or frustrated.
Once you have these, you can use them to simulate realistic conversations. Type questions as your customers would, add typos or abbreviations, anything you test will go towards improving it. How may misspellings of ‘chlamydia’ do you think we came across when building ‘Pat’ the sexual health chatbot – a lot!
The Scenarios Your Chatbot Must Pass
Create a testing checklist based on your actual customer scenarios, not theoretical ones. What happens when someone is angry and uses inappropriate language? What happens when they ask about a product or service you discontinued last year? What happens when they’re trying to do something your business doesn’t offer, like asking for a refund on a non-refundable item?
If you’re aware that high risk conversations will be had with your chatbot, questions asked of your AI assistant could alert you to users greatest in need. Even if you’re not such an organisation, conversations with your chatbot can alert you to problems with your website or your service. So, your chatbot needs a plan for scenarios like this.
Try to confuse the chatbot on purpose. Ask the same question three different ways in a row. Start asking about one topic, then completely switch to another mid-conversation. Type responses that don’t answer the chatbot’s questions. This isn’t about being difficult, it’s about simulating real customer behaviour. People get distracted. They change their minds. They don’t always follow the conversational flow your chatbot is guiding them on.
Testing Across Devices
Open your chatbot on your phone and test the entire experience with your thumbs. Many chatbots are designed and tested primarily on desktop computers, but more of your customers might interact with it on mobile devices. Buttons that are easy to click with a mouse become frustratingly small on a phone screen. Responses that look concise on a laptop can require endless scrolling on a mobile device.
Finally
Before you approve your chatbot for launch, ask yourself these critical questions based on your testing experience:
- Can my chatbot handle the ten most common customer questions without human intervention?
- Does the chatbot recognise when it’s out of its depth and connect people to human help smoothly?
- Will customers feel heard and helped, even when the chatbot can’t fully solve their problem?
If you answered no to any of these questions, you’ve identified exactly where more work is needed before launch. This is the value of client-led testing. You’re catching these issues now, before they come back to bite you later.
