An AI answer looked better after you edited a page. Was the question the same? Did the engine use a new surface? Was the earlier claim actually wrong, or simply relevant to a different plan? Without a stable test record, a before-and-after screenshot cannot answer those questions.
The aim of a repeatable audit is an inspectable comparison. Keep the parts you can control stable, record the parts you cannot, and explain the remaining uncertainty. Exact output repetition is not a prerequisite for useful measurement, but consistent comparison conditions are.
Define the decision before collecting answers
Choose a buyer problem, not a generic ambition to “increase visibility.” A pricing audit asks whether applicable amounts and conditions are stated correctly. A suitability audit asks whether capability claims match the intended use case. A discovery audit asks whether relevant options appear under specified questions. Do not combine those outcomes without explaining their different meanings.
NIST’s Generative AI profile is a voluntary risk-management companion covering the design, use, and evaluation lifecycle. It supports the need for context-specific evaluation; it does not certify this checklist or provide a universal commercial accuracy score. NIST: Generative AI risk profile (2024).
Record the full experimental conditions
Version the question list and approved references before the first run. For every capture, preserve engine and user-facing surface, date and time, location or locale settings, relevant personalization settings, the full answer, citations, and any failure. Record unknown settings as unknown; do not guess a model version from branding.
Keep API observations separate from consumer-interface observations unless there is evidence they represent the same behavior. A provider name alone is too coarse for comparing results. If terms or practical access prevent a repeat capture, document that missing observation rather than substituting a different surface silently.
| Field | Reason |
|---|---|
| Question-set version | Shows whether the benchmark changed |
| Engine and surface | Prevents comparing different experiences as identical |
| Capture timestamp and locale | Provides context for freshness and applicability |
| Reference version | Preserves the evidence used for judgment |
| Outcome and reason | Allows a second reviewer to inspect the decision |
| Missing/failed captures | Makes coverage limitations visible |
Separate correctness from citation quality
The ALCE research evaluated fluency, correctness, and citation quality as different dimensions. That distinction is a useful model for a business review: a correct statement and an adequate supporting citation are separate judgments. ALCE: evaluating generated answers and citations (2023).
Use a short rubric: supported, contradicted, insufficient evidence, or not comparable. Maintain a separate status for collection failures. Ask a second reviewer to inspect disputed claims. If the source itself is ambiguous, that is a source-quality finding rather than automatic proof of an engine error.
Keep the denominator visible
Suppose a fictional run plans 20 captures, completes 18, and finds 12 supported answers. “12 of 18 completed captures were supported, with 2 collection failures” is more informative than a standalone percentage. Those invented numbers demonstrate reporting only; they are not a Pevatron performance result.
Decide whether your unit is a claim, an answer, or a question-engine pair. One answer can contain several claims, so changing units changes the denominator. Show coverage alongside outcomes and avoid averaging incomparable measures into a health score whose method nobody can inspect.
Rerun with an explicit change log
Record the source edit and when it became publicly accessible. Rerun the frozen questions and compare each new capture with the saved baseline. Keep natural wording variants as a separate test set so that a successful exact prompt cannot hide nearby failures.
Your conclusion should state what changed, how many observations support it, and what else could explain the change. For 2027 operations, schedule reviews around meaningful product or pricing changes as well as a regular cadence. A versioned evidence trail is more useful than a promise of perfectly deterministic third-party answers.
See how a claim and its evidence appear in the illustrative Tailscale sample report. When you are ready to compare options, review Pevatron audit plans and pricing.
Questions answered
Does repeatable mean the engine must return identical wording?
No. It means the audit conditions and review method can be inspected and repeated. The actual answers may still vary.
Should failed captures disappear from an accuracy report?
No. Report them as coverage failures and disclose how completed captures enter any outcome calculation.
Sources and research notes
Primary sources checked on 14 November 2026. Our checklists and examples are recommendations, not measured customer results. Recheck changing provider policies before acting in 2027.