Field notes. John Arndt, Soxoa. Published .
An AI result arrives with a neat table, a confidence score, and the number you expected. The temptation is to approve it and move on. That becomes expensive when the table contains the right total for the wrong reasons.
In our construction takeoff review work, we had to evaluate quantities, specifications, missing scope, and conflicting plan evidence together. The useful method was a claim ledger tied to the original source. The same approach applies to invoice extraction, research summaries, and reports assembled from several documents.
The evaluate-ai-output skill packages that method for Claude. The historical review process supplies the foundation. The new Claude package remains an untested draft.
Start with the whole assignment
Before checking an answer, write down what the task required and which source revisions were supplied. If the assignment covers an entire plan set, an excellent count on one electrical sheet establishes only that count. It says little about plumbing omissions or a missing schedule.
Create a source inventory and record what was actually inspected. Keep unreadable pages and missing attachments visible. This prevents an incomplete review from turning into an accidental claim about complete accuracy.
Define the acceptance criteria before scoring. If the criteria were never written, propose them and identify them as a new definition. Preserve older results under the definition used at the time. Moving the finish line makes comparisons useless.
Check claims in both directions
The review needs two passes. First, work from the source into the result: did every required fact receive a disposition? Second, work from the result back into the source: does every material claim have support?
An illustrative invoice example makes the distinction clear. Suppose the source shows a subtotal of $800, tax of $64, and an $864 total. The extraction returns the correct total but classifies all $864 as taxable merchandise. A total-only check passes an answer that may be unsuitable for the intended accounting task.
A small claim ledger catches the problem:
- Invoice total
- Source evidence: Total field: $864 · Result: $864 · Disposition: Supported
- Merchandise subtotal
- Source evidence: Subtotal field: $800 · Result: $864 · Disposition: Contradicted
- Tax
- Source evidence: Tax field: $64 · Result: Missing · Disposition: Omitted
The example is illustrative. The important choice is to check each required fact and its meaning separately.
Preserve the difference between correction and normalization
Converting inches to feet can preserve an answer. Adding a specification the model omitted changes the answer. Keep the original result unchanged and store reviewer corrections separately.
The same discipline applies to counts. Ten entries can contain the same object ten times. Ten valid objects can be assigned to the wrong floor. A citation can name a real page while pointing to an unrelated region. Matching a summary number does not resolve those problems.
Source conflicts also deserve their own category. When a schedule and drawing disagree, the reviewer should record the disagreement and the decision needed. The model should not silently pick whichever source produces a cleaner table.
Ask for a bounded review
Try this with an authorized document and an existing result:
“Use evaluate-ai-output to review this extraction against the original document. Preserve the raw result. Build a claim ledger covering required fields, omissions, unsupported additions, and source conflicts. Explain the denominator for any accuracy metric. State which conclusions the available evidence cannot support.”
Claude needs access to the original material, the result, and the task requirements. A screenshot of a summary may support a limited review, but it cannot substitute for the underlying pages.
Human accountability remains essential where the output affects an estimate, financial record, or other consequential decision. The skill organizes review and exposes gaps. It does not provide professional certification or a universal accuracy guarantee.
Start with one completed task. Compare the review findings with your own source check, then use the disagreements to improve the next evaluation.
Get the skill
evaluate-ai-output is part of the free Soxoa Ops plugin for Claude. In Claude Code, add the marketplace with /plugin marketplace add johnarndt/soxoa-claude-plugins, then run /plugin install soxoa-ops@soxoa-plugins. You can read the skill on GitHub before installing it, or copy its folder into ~/.claude/skills/.