AI chatbot and assistant testing
I check that your chatbot tells the truth and follows your rules, doesn't make things up, can't be manipulated and doesn't leak data - before your customers find out it does.
A chatbot built on a language model answers a little differently every time, so you can't test it like an ordinary web form. It takes requirements analysis, well-designed test cases and automation, combined with a human who decides what a correct answer is.
Who it's for
For companies whose chatbot speaks on their behalf
This service is for the people responsible for the chatbot: business owners, CTOs, product managers, customer service and e-commerce leads.
Who I work with
- Companies with a customer service chatbot on their website: online shops, service businesses, SaaS.
- Companies building an internal AI assistant on their own documents: knowledge bases, procedures, terms and policies (so-called RAG).
- Software houses and agencies that build chatbots for clients and need independent verification before handover.
- Industries where a wrong answer has real consequences: e-commerce (prices, promotions, returns), legal, financial and medical services, education.
Typical situations
- before the chatbot goes live for customers,
- after a change of model, system prompt or knowledge base,
- when there are complaints about the bot's answers or worrying conversations in the logs,
- before an audit or compliance review.
Test areas
What I test in a chatbot
I set the scope and the balance between areas according to your priorities. For one company the top concern is security and data leaks, for another it is correct prices and promotion terms.
-
Factual accuracy
Does the bot answer in line with your company documents: terms, price list, offer, procedures? Does it give the right amounts, deadlines, thresholds and exceptions? Does it even find the right part of the knowledge base?
-
Hallucinations
Does the bot invent facts that are not in your documents: promotions, products, promises or regulations that don't exist? Does it admit that it doesn't know instead of guessing?
-
Language quality
Grammar, inflection (for example of product and place names, which is demanding in Polish), spelling and a tone that fits the brand. Plus how the bot handles casual questions, typos, indirect questions and questions in another language.
-
Security
- resistance to prompt injection and jailbreaks, i.e. attempts to bypass its instructions,
- protection of the system prompt and internal instructions from disclosure,
- protection of personal and confidential data,
- staying on topic: declining conversations unrelated to the business, offensive content, political opinions,
- tool misuse, if the bot has access to systems such as orders or customer accounts.
-
Functionality and conversation flows
- typical customer journeys from question to resolution,
- keeping context in a longer conversation,
- handover to a human: when it should happen and whether it works,
- integrations: forms, links, collecting contact details, actions performed by the bot.
-
Conversation usability
Are the answers clear and of a sensible length? Does the bot ask follow-up questions when something is unclear? Is it easy to get things done? How does the chat window look and work, including on a phone? Is it clear how to reach a human?
-
EU AI Act: selected requirements
A technical check of the requirements that can be verified in a conversation with the bot, above all the duty to inform users that they are interacting with an AI system (Article 50 of the AI Act, applicable from 2 August 2026). Plus pointers to bot behaviour worth reviewing with a lawyer. This is a technical check, not legal advice.
-
Performance: basic scope
Measuring response times and their stability in typical scenarios, and flagging answers that are clearly too slow. Load testing, i.e. behaviour under many simultaneous conversations, only by individual arrangement.
Method
How I work: three layers of tests
The test suite has three layers that differ in how closely they are tailored to your company.
-
Layer 1
Universal tests
Checks that apply to every chatbot regardless of industry: security, attempts to extract instructions and data, off-topic questions, AI disclosure, handover to a human, language quality and tone. I only adjust their parameters (company name, topic scope, contact channel), which keeps this part fast and its price predictable.
-
Layer 2
Industry tests
Test sets typical of a given industry, for example:
- online shop: prices, delivery, returns, complaints, promotions, product availability,
- beauty salons and other appointment-based services: treatment prices, available slots, referral to the booking calendar, no health advice or promised results,
- hotels and guesthouses: room availability and prices, cancellation terms, check-in, answers for guests from abroad,
- real estate agencies and developers: listing details and prices, no invented properties, correct collection of contact details.
-
Layer 3
Tests tailored to your company
The most important layer, prepared entirely by a human:
- risk analysis together with you: which bot error would hurt your business most,
- document analysis: terms, price lists, procedures, the bot's knowledge base,
- test cases with an answer key: for each question I define clearly which answer is correct and which is unacceptable,
- edge cases: an order exactly at the free delivery threshold, the last day of a promotion, unusual combinations.
Often before testing even starts: document analysis can reveal problems before the bot is asked a single question, such as contradictions between the terms and the FAQ that later lead to wrong answers.
Human in the loop
A human decides, automation executes
What counts as a correct answer is decided by a human, based on your documents and what we agree together. That is the foundation of reliable testing.
- Automation lets me run hundreds of questions, in many different phrasings, and repeat them after every change to the bot.
- Answer evaluation: answers are compared with the key by simple rules (e.g. whether the right amount was given) and by an independent AI model from a different family than the model behind your bot. I calibrate its verdicts by hand.
- AI models also support document analysis, but only as a second reader: I verify every finding in the source.
- Exploratory testing: on top of automated tests I hold manual conversations with the bot, looking for behaviour nobody anticipated.
Tools: proven open-source tools for testing LLM-based applications, run locally, and browser automation where needed. I connect to the bot through its API or through the chat window on your website.
Step by step
What the process looks like
- Initial consultationGoals, type of bot, the model it runs on, key risks and access. Free of charge.
- Risk workshop and prioritiesWhat matters most to your business: security, prices, reputation, compliance.
- Document analysisPreparing the test suite with an answer key.
- Test executionAutomated and exploratory.
- ReportResults, examples of wrong answers and recommendations.
- Retest after fixesOptional; package B includes one retest.
- Regression testing after further changesThe same suite run again, as part of package C: a monthly plan or single rounds.
Packages and pricing
How much does chatbot testing cost
Prices in Polish zloty (PLN), excluding VAT. After a free consultation you get a specific written quote before any work begins.
-
Package A
Starter review
from PLN 3 500typically PLN 3 500 - 5 000, excl. VAT
- layer 1: security, protection of instructions and data, staying on topic, handover to a human,
- check of the AI disclosure in the main channel,
- basic language quality assessment,
- concise report.
- 3-5 business days
- a good first step
-
Package B · full audit
Chatbot quality audit
PLN 9 000 - 16 000excl. VAT
- all three layers of tests,
- risk and document analysis,
- tests with an answer key: factual accuracy, hallucinations, edge cases,
- exploratory testing,
- report with recommendations and one retest after fixes.
- 2-3 weeks
- retest included
-
Package C
Ongoing care and regression tests
PLN 1 200 - 2 500per month excl. VAT, or billed per round
- re-running the test suite after changes to the model, prompt or knowledge base,
- a report of differences against the previous round,
- a few new tests added.
- on your bot's change schedule
- available after package B
Add-ons to package B
- usability: conversation UX and the chat window, including on phones: +PLN 1 500 - 3 000,
- another language or channel (e.g. a messenger app, a mobile app): +PLN 2 000 - 4 000,
- basic performance measurements (response times): +PLN 1 000 - 2 000,
- a workshop for your team covering the results and good practices: priced individually.
What affects the price within the range
- the volume of documents,
- the number of conversation paths,
- how the bot is accessed: through an API or through the chat window on the website,
- the bot's access to tools and customer data, which means more security testing.
Pilot projects: 25% lower price. This applies to the first three projects delivered in Q4 2026. Get in touch to check whether you still qualify.
Deliverables and requirements
What you get and what I need from you
What you get
- a test report organised by area, with a severity rating for each problem,
- concrete examples of wrong or risky answers: the question, the bot's answer and why it is a problem,
- recommended fixes: system prompt, knowledge base, settings, processes,
- a list of inconsistencies found in the source documents, if any.
What I need
- access to the bot, ideally in a test environment as close to production as possible, or through an API,
- information on which AI model the bot runs on,
- the documents the bot relies on: terms, price lists, knowledge base, instructions,
- consent to testing, including security tests, i.e. simulated attacks on the bot,
- an agreed budget for bot calls, if every conversation generates costs on your side.
Honest limitations.
- Testing reduces risk, but it is not a guarantee. AI chatbots answer non-deterministically, so no test suite can rule out every error. That is why tests are worth repeating after changes.
- EU AI Act: I check selected requirements technically; I don't give legal advice.
- Load testing: only within an individually agreed scope.
Experience
Why Netlea
- over 20 years in IT, including about 10 years as a tester and about 10 years leading QA and software development teams,
- ISTQB Advanced Level Test Manager certification (CTAL-TM),
- risk analysis and test case design are my everyday work, not an add-on to automation,
- hands-on experience using AI in testing,
- independence: I don't build chatbots, so I assess them without a conflict of interest.
Security
Confidentiality and data
- I process your documents only to the extent needed for testing.
- If I use AI tools in the analysis, I do so on terms agreed with you and only through services that don't use submitted data to train models.
- On request I sign a non-disclosure agreement (NDA) and a data processing agreement.
- If your company requires data to be processed in the EU, I choose tools that meet this condition.
FAQ
Frequently asked questions about chatbot testing
Can you test chatbots built on any platform?
In most cases, yes: through the platform's API or directly through the chat window on the website. We agree the details during the consultation.
Can testing harm our bot or cost us a lot?
I run security and automated tests with your consent, ideally in a test environment. We agree the number of calls in advance to keep costs on your side under control.
How is this different from the tests done by the chatbot vendor?
It is an independent assessment based on your documents and your business risks, not generic platform tests.
How often should the tests be repeated?
After every significant change: to the model, system prompt, knowledge base, offer or terms.
How much does chatbot testing cost?
A starter review costs from PLN 3 500, a full quality audit PLN 9 000 - 16 000, and ongoing care after bot changes PLN 1 200 - 2 500 per month, all excluding VAT. After a free consultation I prepare a specific written quote. The first three projects delivered in Q4 2026 are 25% cheaper.
Will we receive the test suite?
As standard you receive a report with results, examples and recommendations. Re-running the tests after changes to the bot is part of ongoing care (package C).
Will you help us fix the bot?
The report contains concrete recommendations. Implementing the fixes is up to your team or the bot vendor, and I can then retest the changes.
Let's test your chatbot before your customers do
Describe in two sentences what kind of bot it is and what worries you. In a free consultation we will agree the scope, and then you get a specific written quote.
Book a free consultation