One bug report is a sample, not the whole problem
On one business software product, an administrator at a customer wrote in to say that items she had approved on her dashboard did not seem to be going through. She had approved a batch earlier in the month. As far as the system could tell, none of that earlier batch had ever arrived.
The cause of her problem was found quickly. Next to each item waiting for approval, the dashboard showed a checkbox and a “Review” link. The checkbox was not connected to anything, so ticking it saved nothing. The “Review” link opened the general list of waiting items rather than the item she had clicked on, so it was easy to come away believing the job was done. Approvals made from the full list page did work.
So the reply could have been one line: approve from the other page and you are all set. That workaround was sent, and it was accurate. If it had been the whole response, though, the product would have kept the same trap for the next person, and nobody would have known how many people had already stepped in it.
Why the first fix tends to be the smallest one
A bug report describes one person’s experience. When you hand that report to an AI and ask it to fix the problem, the report quietly becomes the definition of the problem. In practice, an AI tends to do what was asked, and often does it well: find the cause, change the part that broke, confirm the reported case now works.
That is not wrong, but it is often narrower than the real problem. In the story, the same dashboard screen was part of the product for every user, not just this one. Anyone who had used that checkbox would have had the same result, and they had little reason to write in, because from their side the approval looked finished. The one person who reported it was the visible part of something that could have been much larger.
The habit: three questions before any fix
Before you accept a fix for a reported bug, ask your AI to answer three questions in writing. Ask before the fix, so the answers shape the work.
- Who else has this? “Is this specific to this one account, or could anyone who did the same thing have hit it? How could you count them? Give me a read-only way to get the number, meaning a check that looks at the data and changes nothing.”
- What is stuck because of it? “If this has been happening for a while, which records are now in the wrong state, and what would it take to put them right?”
- What stops the next one? “Where else in the product could the same kind of mistake exist, and how would you find out quickly if this happened again?”
In the story, the first two questions produced read-only database queries to run across every customer, not just the one who wrote in: items still marked as waiting even though an approval had been recorded somewhere, items marked approved with no follow-up record behind them, and how many items each customer had waiting and for how long. Those numbers are a diagnostic, not a confirmed list of victims. A control that saved nothing also left no trace of who clicked it, so no query can say for certain who tried and failed. An item can be waiting because nobody has looked at it yet, or because someone ticked a box that did nothing. What the counts do give you is a starting list: which customers have unusually many or unusually old waiting items, and which records are in a state that should not exist. Those are the people worth asking, and the records worth cleaning up. When a bug leaves no trace like this, say so plainly rather than treating the count as the full answer.
The third question produced a ticket with six parts instead of a one-line fix. The changes written for it included these: the “Review” link opens the exact item that was clicked; approving shows a clear confirmation; the dead checkbox is removed instead of patched; the approval itself is all-or-nothing, so it cannot be left half-saved; the dashboard lists recently approved items, so a person can see that their approval landed; and each approval is recorded as an event in the product’s usage analytics, so if approvals from that screen ever drop to zero again, there is a number that shows it.
The same report also mentioned a smaller problem: one record had been saved with no currency set. Asking where else that kind of mistake could live turned up two more places in the code where the currency was fixed to a single default value instead of being read from the record.
A note on what to tell the customer
While the wider work was going on, the reply to the customer stayed narrow and true. It confirmed which of her approvals had been saved, said plainly that the earlier ones had not reached the system, explained which controls did not approve, gave the workaround, and did not promise a date for the fix. Code that has been written, or even merged, is not necessarily something the customer can use yet.
How to check you did it
You can check this habit with one number. Before the fix goes in, have the “who else” and “what is stuck” queries run and write down what they find. After the fix and any cleanup, run the same queries again. The number of records in a state that should not exist should be the one you expected, often zero. Keep in mind what the number measures: it counts records that show signs of the problem, which may be more or fewer than the people it actually affected. If you cannot get a number at all, the “who else” question is probably not answered yet. There is more on that habit in Write down the number before you run it.
Then do what the customer did, in the live product, and look for the confirmation yourself. A fix you have seen work for one user, plus a count of where else the problem has left marks, is a much stronger answer than a fix that only makes the original report go away.
If you use a separate reviewer on your AI’s work, as our guide describes, add the three questions to what it checks on any bug fix.