Measuring an AI pilot: hours, errors, rework and maintenance
By Larry Lo
Published
Three key takeaways
- Use comparable volume and difficulty for the baseline and pilot.
- Include review, rework and maintenance in the workload.
- Agree stopping conditions before the demo, not success criteria afterwards.
Define the unit and protect the measurement data
Choose a completed unit, such as one reviewed receipt or one approved training pack. Separate active labour from elapsed waiting time. Record sample size, complexity, missing-input rate and the definition of an error. Use anonymised samples and restricted, time-limited logs; measurement is not a reason to collect more personal data.
Illustrative formulas, not customer outcomes
These are calculation methods, not measured results: manual hours per period = volume × manual minutes per item ÷ 60. Pilot hours = volume × (handling + review + rework minutes per item) ÷ 60 + maintenance hours per period. Net hours difference = manual hours − pilot hours; a negative value means more work. Record one-time setup and training separately.
Error rate = inspected outputs with confirmed errors ÷ inspected outputs. Rework rate = completed items needing correction ÷ completed items. Mark a zero denominator as not applicable. Log serious events such as data exposure or incorrect payments separately so averages cannot hide them.
Agree expand, change and stop decisions
Before testing, the owner and reviewer should agree acceptable errors, workload limits, sampling and the review date. Permission leaks or unapproved external sending can trigger immediate suspension. Log versions, correction reasons, maintainer and time. If the sample is small or its difficulty changes, extend observation instead of forcing an ROI calculation.
Agilizing's training-material production changed from 10 days to 3 hours in the specific January–February 2026 case confirmed by the client owner. This is not a forecast for other workflows or an ROI percentage. The ZOmate and Flower Nice Day cases provide no performance figures.
Questions
- Can we just time the model response?
- No. Include preparation, review, correction and delivery. Distinguish elapsed waiting from active labour to tell faster turnaround from reduced human work.
Limitations
The formulas are illustrative, not a full financial model or return guarantee. Track API/AI usage, integrations, setup, licences and opportunity costs separately. Differences in difficulty or sampling can invalidate a comparison.
Sources & references
Related services
Related cases
Bring one anonymised workflow example to a 30-minute business diagnosis. Identify the problem, owner and next step before choosing software.
Book a 30-minute diagnosis