Human baseline

When a company hires an employee, it compares the candidate with the person doing the work today, without demanding perfection. With an agent, the company forgets this and starts demanding 100% from a process that has never come close to that.

Before automating anything, the team measured 80 letters written by volunteers: 25 minutes per letter, 3.2% with some factual error, 8% returned during review, and an average queue of 11 days in peak season. Those four numbers became the baseline.

When the agent reached a 1.9% factual error rate, the conversation stopped being "it makes mistakes" and became "it makes fewer mistakes than we do, and the queue fell from 11 days to 4 hours". The decision to raise the level of autonomy started being made with numbers on the table.

Demanding a level of perfection from the agent that was never demanded from the human. Without a baseline, the agent’s first mistake becomes an argument for canceling the project, while the manual process, which makes more mistakes, keeps running without anyone measuring it.

The symmetrical mistake costs the same: without a baseline, no one notices when the agent gets worse, because no one ever defined what "better" meant.

• Measure before building, using a real sample of 30 to 100 cases.
• Use exactly the same criteria the evals will use later; otherwise, the comparison will not hold.
• Record at least four things: error rate, time per case, cost per case, and rework rate.
• If the current process is not measurable, that is already the project’s first finding, and usually the most useful one. Record the finding and continue with the measurement.

Measure 30 to 100 cases from the current process: error rate, time, cost, and rework. A spreadsheet is enough.