Check what the number counts before you send it
On one software product, a customer asked a simple question: how many of their people had been active on the product in the last six months, and in the last three?
An AI wrote a read-only query to answer it. The query counted distinct people by type of activity, such as signing in and taking certain actions in the app, plus a total for “any activity at all”. It ran against the live database without errors.
The answer was a handful of people. Out of a much larger group, almost nobody had done anything in either window.
If that number had gone to the customer, it would have read as “your people have stopped using this.” It was not true. The query was correct, the data was real, and the conclusion would still have been wrong.
What the number was actually counting
The AI flagged the result before anyone sent it, and the reasons were specific.
First, the query measured the wrong kind of activity for this customer. Most of their people got value from the product without visiting it often. Their use showed up as records that stayed active over time, not as sign-ins or clicks. A count built on visits and in-app actions was always going to be close to zero for them, even when use was healthy.
Second, one type of record the query relied on had stopped being created some time earlier, after an unrelated change elsewhere in the product. That part of the count was zero because nothing was being recorded, not because nothing was happening. The AI had written this down as a caveat when it wrote the query, before the results came back: if this part of the number looks like a drop, do not send it without explaining why.
The fix was not to adjust the first number. It was to ask a different question. A second read-only query counted the people who had an active record in effect during each window, which is what use actually looked like for this customer. That number was far higher, and it was the one that went into the reply, with a plain explanation of what it counted.
Why this is easy to miss
A database query does what it says, not what you meant. “Active users” sounds like one thing, but it can be written many ways: signed in, clicked something, had a record created, had something in effect. Each one is a reasonable definition. They can give very different answers for the same customer.
In practice, an AI writing a query from the database structure alone tends to use the most visible signals, such as sign-ins and events. It usually cannot know, from the schema alone, how a particular customer uses the product, or which records stopped being written after an earlier change. That knowledge sits in your head, your notes, and your history, not in the column names.
There is also a quieter trap. A zero looks like a fact. But a zero can mean “nothing happened” or “nothing was recorded”, and the query on its own usually cannot tell you which.
The habit: three questions before a number leaves the building
Before you send a number to a customer, an investor, a board or anyone who will make a decision on it, ask your AI to answer three questions in writing, next to the number.
- What exactly does this count? One sentence, in plain English: “People who signed in at least once between these dates.” Not the name of the metric, but the actual rule. Then ask yourself whether that rule matches the question you were asked. If the customer asked about participation and the rule is about sign-ins, you have a mismatch before you have a result.
- Is the data behind it still being recorded? For each kind of record the number depends on, ask for the date of the most recent one. A source that went quiet months ago will show up immediately. A recent record only shows the source is not completely dormant; it does not prove every relevant activity is being recorded. Any zero or sudden drop gets this check before anything else.
- What would this number look like if things were going well? Say it out loud before you look, or better, write it down, the same way you would write down the expected count before a bulk change. If a healthy customer would still score near zero on this measure, the measure is wrong for this customer.
If the answers do not line up, ask for a second query that measures the thing the question was really about, and send that one with a sentence explaining what it counts.
It also helps to have a second pair of eyes on the number before it goes out. Our guide covers getting a fresh reviewer to check work the original author is too close to, and a report going to a customer is a good place to use one.
How to check you are doing it
Pick the last number you sent outside your team: a usage report, a growth figure, a customer summary. Ask your AI:
- Write one sentence describing exactly what this number counted.
- Give the date of the most recent record each part of it relied on.
- Tell me whether a customer who was doing well could still have scored low on it, and why.
If the first answer surprises you, or the second shows a source that stopped months ago, the number you sent may not have meant what you thought. It may still have been right. But you would want to know that before the next report, not after a customer has made a decision on it.
On the product in this story, the check cost a few minutes and one extra query. The first number would have told a customer something about their own people that was not true.