Ninety-Five Percent Accurate Compared to What?

An accuracy number means little until you know what a lazy guess would score on the same transactions.

Watermelon AI · Field Notes ·

Every bookkeeping automation tool quotes an accuracy figure. Ninety-five percent of transactions categorized correctly. It sounds strong. Before you trust it, ask one question: what would a simple guess have scored?

Run the lazy version

Here is a lazy method anyone could build in an hour: code every transaction the same way that vendor was coded last time. No intelligence. No reading. Just repetition.

On a typical small business's books, that method often scores in the high eighties, because most vendors are coded the same way every month. Rent is rent. Software is software. The power company is utilities.

So a tool that scores ninety-five isn't ninety-five points better than nothing. It's a few points better than repeating last month. That may still be worth having. But the real test is how it handles the transactions the lazy method gets wrong: new vendors, vendors that split across accounts, and one-off purchases. That is where the work is, and where the score should be measured.

Set the lazy score as the floor. A result means something only to the extent it rises above that floor.

A check that never fails tells you nothing

The same idea applies to review steps. Picture a monthly checklist with every box ticked, every time, or a review that has never flagged a thing. It runs on schedule. It produces a tidy result. It looks rigorous.

But if nothing ever tests whether it can catch a problem, there is no way to know that it would. A review that has never been wrong has never been tested.

A second look needs fresh eyes

One more version of this. A system codes a transaction and also writes a note explaining why. Then a review step compares the code to the note and finds that they agree. That looks like a second review. It isn't. Code and note were produced together, in one step, so of course they match.

A real second look goes back to the source, the receipt, the bill or the bank line, without assuming the first answer was right.

What we measure

When we report how well a setup is working, the numbers come with their baseline: how a simple repeat-last-time rule does on the same month, and how the system does on the hard cases specifically. Reviews go back to source documents, not to the system's own explanation. And every check we run has been handed a known error at some point, to prove it catches one.

The takeaway: When a tool quotes an accuracy number, ask what "code it like last time" scores on the same transactions, and how the tool does on new vendors.

Want these checks running on your books?

We set this up on your books and your rules, and nothing posts without your approval.

Book a call

More field notes

All field notes →