Write down the number before you run it
On one software product, a batch of older records needed to be moved into a newer format so people could manage them in the new version of the app. An AI wrote the conversion. The rule for it had been agreed in plain English beforehand: convert the records that were still active, and skip the ones that had been rejected or cancelled.
That rule sounds hard to get wrong. The code that carried it out looked reasonable too. It checked whether each old record was still current, and it checked whether the record had been rejected.
Before anything was written to the live database, the conversion was run as a dry run: a version of the change that only counts what it would do and changes nothing. The dry run said only about one in ten of the records would be converted. Most would be skipped as “rejected”.
That would have been hard to spot by reading the code alone. What gave it away was a number written down earlier. An export of the same old records had already shown how many should convert and how many should be skipped. The dry run did not match it, so the real run did not happen.
What was actually wrong
The old records had a status field that, according to the code’s own definitions, meant “not approved” when it held one particular value. Most of the old records held that value. So the conversion read them as rejected and skipped them.
The AI did not guess at a fix. It ran read-only queries, first to group the records by that field, then to look at what had actually happened on those accounts over time. The history was clear: those records had been approved and active the whole time. The field on these old records was a leftover default that had never been updated. It looked like a decision and was not one.
The fix was one line. The conversion stopped trusting that field and used only the record’s own approval status. The next dry run matched the expected numbers exactly, row for row. Only then did the real change run, and its check afterwards matched too. The saved copy of the conversion script was corrected to match what actually ran, so the next time a similar conversion is needed it will not repeat the mistake.
Why this is easy to miss
A data change can be wrong in a way that produces no error at all. Skipping a record is not a failure as far as the software is concerned. The run finishes, reports success, and tells you how many rows it touched. If you do not already know how many it should have touched, that report reads as good news.
In practice, the risky part of a bulk change is often not the code but a hidden assumption about what the old data means. A field name, a code comment, or a definition elsewhere in the app can all say one thing while long real-world use has made the data say another. An AI writing the change can only work from what it can see, and what it can see is usually the definition, not the history.
The habit: expected numbers first, then a dry run, then the change
For any change that moves, converts, updates or deletes many records at once, ask your AI to work in this order, and do not let it skip a step.
- Write down the expected numbers before writing the change. Get them from somewhere other than the change itself: an export, a report, a count you trust, a quick read-only query written separately. “How many records should this convert, how many should it skip, and why?” Make sure the count covers the same set of records the change will touch, and get an exact count rather than a rough guess where you can, so the comparison later means something. Put the answer in your notes or the task, with the date.
- Run a dry run that only counts. Ask for a version of the change that changes nothing and reports how many records fall into each group: converted, skipped, and the reason for each skip.
- Compare, and stop on any mismatch. If the dry run and the expected numbers disagree, the real change does not run. Not “probably fine”, not “close enough”. A mismatch means one of the two numbers is wrong, and you do not yet know which.
- Explain the gap with evidence, not a guess. Ask the AI to show you, with read-only queries, which group the difference is in and what the real records look like. A fix should come with a reason you can repeat in one sentence.
- Rerun the dry run after the fix. Only run the real change when the dry run matches. Then check the result afterwards against the same numbers.
- Save the fix where it lives. If the change came from a saved script, make sure the saved script matches what actually ran.
If you want a second opinion on the plan before anything runs, our guide covers getting a fresh reviewer to look at work the original author is too close to.
How to check you are doing it
Look at the last bulk data change on your project and ask three questions:
- Is there a written expected number, from a separate source, dated before the change ran?
- Is there a dry run result next to it?
- Is there an “after” check, and does it match?
If any of the three is missing, the change may well have been right, and there may be other ways to check it, such as comparing the resulting records against the rule one by one. But you would have to do that work after the fact, when it is harder and the change has already happened. The next bulk change is the easier place to start.
A good test of the habit: when the dry run disagrees with your number, does the work actually stop? On the product in this story it did, and the cost was a short delay and a few read-only queries. Without the number written down first, the run would most likely have finished cleanly, reported success, and left most users unable to find their records in the new version.