An evaluation answers a specific question: can the assisted workflow complete this task to the required standard with an acceptable amount of effort? It is not a general intelligence score. A meaningful comparison uses the same task boundaries and checks for both the manual and assisted approaches.
Construct a representative test set
Collect examples of normal work, difficult cases and missing or contradictory inputs. Keep a small set of cases separate from those used while refining the prompt. Reusing only familiar examples can make a process look better than it performs on new work.
Record the source, date and permission status of each case. Remove confidential or personal information unless its use is authorized. Preserve the expected answer or acceptance criteria independently of the generated output. Do not let the system define its own success after seeing the result.
Define the rubric
| Dimension | Concrete check |
|---|---|
| Correctness | Are material statements supported by the input? |
| Completeness | Are required facts or actions missing? |
| Instruction adherence | Does the result follow the requested scope and format? |
| Review effort | How long does verification and correction take? |
| Critical failures | Did it expose restricted data or invent a consequential fact? |
Separate critical failures from routine quality scores. An output with excellent style but a fabricated payment instruction should not receive a passing overall result because its other dimensions scored well.
Calculate useful measures
Suppose an illustrative set contains 40 drafts. Thirty are usable without correction, eight need correction and two cannot be used. The uncorrected acceptance rate is 30 ÷ 40 = 75%. The correction rate is 8 ÷ 40 = 20%, and the unusable rate is 5%. Those labels describe different outcomes; do not combine them into a vague accuracy claim.
Now add effort. If corrected drafts take 12 extra minutes each and unusable drafts require 25 minutes of manual replacement, additional effort is 8 × 12 + 2 × 25 = 146 minutes. The headline acceptance rate alone hides that work. Include preparation and review time for the accepted drafts as well.
Interpret limits honestly
A small selected test set cannot establish performance across an entire organization. Report the task type, sample selection, prompt version, settings, reviewer process and failure examples. Compare the median effort and range, and investigate outliers rather than discarding them merely because they look inconvenient.
If two reviewers disagree, clarify the rubric before treating the scores as precise. Repeat the evaluation after a material change to the tool, prompt, source data or workflow. Keep a change log so an improvement can be traced to what actually changed.