Start by deciding which jobs the assistant should perform and which should go to your team. Our recommended evaluation uses a fixed set of questions, agreed expected answers and a record of what actually happened. This helps you assess a proposed assistant against your service rather than a general claim about AI.

NIST identifies confabulation, including confident false output, as a generative AI risk. Treat a fluent answer as something to verify, not as evidence that the answer is correct. Source: NIST: Generative AI Profile, AI 600-1

Define a small, useful scope

Choose tasks you can assess clearly, such as answering questions from approved business information or collecting the details needed for an enquiry. List the sources that support those tasks and appoint someone to keep each source current.

Write down the boundaries too. For example, decide how the assistant should respond when a customer asks for an unsupported price, a booking change or advice that requires a specialist. These are proposed test cases, not claims that every assistant has the same capabilities or restrictions.

Check the source behind the answer

Retrieval-augmented generation uses retrieved material to ground an answer. Microsoft’s RAG documentation also identifies challenges around relevance and access control. Ask which sources the proposed assistant can retrieve and how it avoids exposing material the user should not see. Source: Microsoft: retrieval-augmented generation

For each answer in your test, open the cited source and compare the actual claim. A link alone is not sufficient evidence: it must support the answer given. Try a source with a similar heading but a different rule to see whether the assistant selects the right information.

Build a test set with difficult questions

• Known answer: ask about an approved policy and compare every material detail with the source. This checks basic factual use.

• Missing answer: ask for a detail absent from the approved material. This checks whether the assistant admits the gap and provides a useful next step.

• Wrong premise: include an invented assumption in the question. This checks whether the assistant corrects it rather than accepting it.

• Changed source: update a test fact through the agreed process and record when the assistant reflects it. This checks the actual refresh workflow.

• Restricted task: ask for an action outside the agreed scope. This checks that the assistant follows the defined boundary.

• Human help: ask to speak to the team and follow the request through to the receiving person. This checks the service beyond the chat window.

Use invented personal details and provider-approved test arrangements. Keep the expected outcome beside each question before testing so a persuasive answer does not move the goalposts.

Test handover from both sides

Microsoft’s handoff documentation distinguishes initiating a handoff from a person accepting it and allows context or a transcript to accompany the request. Those distinctions are useful when checking a proposed workflow. Source: Microsoft: moving a conversation from bot to human

Ask a colleague to receive the handover. Check the destination, context, reply route and customer message. Repeat the test when the team is closed. Agree what the assistant should say if no one is available and avoid promising a response time the business has not committed to.

Record outcomes you can inspect

Our suggested scorecard records the question, expected result, actual answer, supporting source, handover outcome and corrective action. Mark unsupported factual answers separately from incomplete but honest answers. Both may need work, but they present different issues.

For an illustrative result, suppose 18 of 20 test questions meet your criteria and two fail because the assistant invents a booking condition. Do not hide those two failures behind a 90% headline. Investigate the failed cases, correct the source or configuration and retest before deciding whether the scope is suitable. The figures are an example, not a product benchmark.

Include privacy and operational ownership

Ask who can view conversations, what information is retained and how access is managed. Have the person responsible for data protection review the proposed processing arrangements against current official guidance. This connects the test plan to the business’s responsibilities without assuming a product name resolves them. Source: ICO: controllers and processors

Evaluating Luna Chat

Travelgenix’s Luna Chat trust-and-safety page describes approved sources and scheduled re-verification. Use that published description to prepare a demonstration with your own permitted questions and sources. Check the observed behaviour and the responsibilities in your configuration rather than treating the description as an independent accuracy result. Source: Travelgenix: Luna Chat trust and safety

Keep the test set after launch. Repeat the relevant cases when the sources, configuration or agreed scope change, so the evidence remains connected to the service your customers use.

Written and updated by Travelgenix. Sources checked on 16 October 2026. Worked examples and checklists are editorial illustrations and recommendations, not customer results.