A useful chat test starts with a question whose answer you can verify. Use the published website as the source of truth, then check retrieval, the generated answer, and its links separately.
Prepare a small question set#
Choose three to five common tasks: what the organization offers, eligibility or service area, how to get started, an important limitation, and a follow-up. Write the expected facts and the page that supports each answer.
For example, ask “How do I request a quote?” only after confirming that a public page describes the actual process. Avoid using private account details or customer transcripts as live test inputs. Tests create ordinary stored conversations.
Run a meaningful live check#
- Open the published site's widget.
- Ask one specific question from the set.
- Read the answer for factual agreement, missing qualifications, and unsupported claims.
- Follow its references and confirm the intended canonical hostname and page.
- Send a follow-up that depends on the first answer.
- Record the result as a short finding, without retaining private session identifiers.
A greeting, a generic apology, or a nonempty response is not a successful answer. A relevant reference title also does not prove the response explained the task correctly.
Locate the weak layer#
If the source page is missing publicly, fix its publication first. If the content is present only after browser JavaScript runs, ensure the important answer exists in the generated public HTML.
If a relevant page is not retrieved, inspect deploy warnings, indexing, page scope, and whether the intended site is selected. The widget's retrieval request uses /api/messages/get_pages; its answer request uses /api/messages. Relevant retrieval can succeed while generation fails.
If retrieval is correct but the answer is incomplete, improve the page's wording and context. Put the direct answer near a clear heading, state exceptions, and link to the canonical procedure. Publish the reviewed correction and repeat the same question.
Handle stale or incorrect answers#
Check the public page in a fresh tab, the entry's draft versus published state, and the current site's deploy. Do not assume saving a draft updates retrieval. For changes to templates or indexing, use the appropriate CMS or deployment workflow.
Do not promise that updating your site immediately changes answers inside unrelated AI products. Site chat, outside model answers, crawler requests, and referrals are separate systems and evidence.
For a service error, keep the failing URL, time, response status, and request ID when available. Avoid attaching credentials or full private conversations. See troubleshooting and content quality.