What is span-level provenance? A practical audit guide for 2027

Why a citation is only the start

The 2023 ALCE paper evaluated fluency, correctness, and citation quality separately. On its ELI5 task, even the best evaluated models lacked complete citation support 50% of the time. That historical benchmark is not an error-rate estimate for current commercial engines. ALCE: evaluating generated answers and citations (2023).

The practical distinction is useful: an answer may read well, state a correct fact, and still cite evidence that does not establish that fact. Review those dimensions separately. Otherwise, a professional-looking citation becomes a shortcut around verification.

Build a review record another person can inspect

W3C describes provenance through the entities, activities, and agents involved in producing something. Our suggested record applies that general idea to answer review; it is not a claim of implementing the W3C PROV specifications. W3C: provenance overview.

Preserve the full answer before extracting claims. A shortened excerpt can lose a qualification such as annual billing or selected countries. Record both the exact text and its position, and keep the original version so that future edits cannot silently change what was audited.

Compare the conditions before marking a mismatch

Consider a fictional answer saying a plan costs $20 per user each month. The official page might say $20 per user per month when billed annually, with a different monthly rate. The number alone matches; the omitted billing condition can still mislead a buyer. This is a hypothetical teaching example, not an observed customer incident.

Write the comparison as a complete proposition: product, plan, amount, currency, billing unit, billing period, region, eligibility, and effective date. If one side does not specify a condition, label the uncertainty. Do not force the record into true or false merely to produce a cleaner score.

Turn the record into an actionable handoff

A useful handoff identifies what your team controls. If the official page is ambiguous, assign a content change with acceptance criteria. If a third-party page is outdated, attach the correct source when requesting a correction. If the answer cites no usable evidence, preserve that gap rather than inventing a causal source.

After a change, rerun the original question and compare the new capture with the earlier record. A corrected answer in that rerun is evidence of an observed improvement for that run. It does not establish that your edit caused the change, or that every buyer now receives the corrected answer.

Start with one commercially important claim

Choose one price, eligibility, or capability claim that repeatedly comes up in sales or support. Ask a second reviewer to reproduce your judgment using only the saved record. If they need to ask which plan, which date, or which passage you meant, improve the record before scaling the audit.

Keep restricted customer information out of public examples. An excellent internal evidence record can contain material that should never become an indexed sample. Publish a redacted illustration only when its provenance and limitations remain understandable.

Questions answered