Test the step that happens later
Some features do part of their work on a later day. A free trial that ends after two weeks. A reminder email that goes out if nobody replies within a few days. An account that is marked for deletion today and erased a month from now. The later step often runs inside a scheduled job, when nobody is watching.
That later step is easy to leave untested, because testing it seems to mean waiting. On one software product there was a way around the wait, and using it found a real bug.
What happened
The product added account deletion. A person could delete their own account, and the owner of a team account could delete the whole team. The request took effect right away in the ways a user would notice: sign-in stopped working, the team’s pages disappeared, and the owner had an “undo” button. The actual erasing of data waited for a grace period of about a month. After that, a nightly scheduled job was supposed to remove the data for good.
The first step was tested in the live product, with clearly fake test accounts. Each action was tried for real, and the database was checked afterwards, record by record: anything meant to be removed straight away was gone, a deleted team’s data was hidden from its members but still stored for the grace period, neighbouring items were untouched, and a person without permission was refused. One part, cancelling a paid subscription on deletion, could not be tried this way because the test team had no subscription, and the notes said plainly that this part was covered only by automated tests.
That left the nightly job. Rather than wait a month, the test records were backdated so they looked as if the grace period had already passed. Then the session waited for the real nightly run, on its normal schedule, in the live product.
The run partly failed. Erasing the test person’s account worked. Erasing the test team did not: the job stopped with a database error.
The cause was a mistake in one database trigger, a small piece of code the database runs by itself when a record of a certain kind is deleted. This particular trigger was shared by two kinds of records. It read a detail that only one of the two kinds has, so it failed whenever it ran for the other kind. The mistake had shipped with the deletion feature, a few hours earlier. During those hours, deleting that kind of record in the live product would have failed whatever asked for it. The database logs showed the test was the only thing that tried.
The fix went in that night, along with a new automated test that runs the real trigger code and fails on the broken version. Then the nightly job ran again, and this time every piece of the test team’s data was confirmed gone, table by table, including its stored files.
Why the later step can hide bugs
In practice the later step often runs in different conditions from what you tested by hand. It runs as a scheduled job rather than as a signed-in person. It works on records in a state you may never have created while clicking around, such as “requested a month ago”. And if it fails, there may be no one looking at a screen to notice. In this case the earlier checks had all passed; the scheduled job was the first thing to set off that trigger in that particular way.
An AI assistant building a feature like this can report it as done once the first step works, the tests pass, and the code is merged. That report can be accurate about everything it checked and still say nothing about the day the grace period ends, because that day has not come yet.
The habit: make the later day happen now
For a feature with a step that happens later, ask your AI to plan the test of that step before calling the feature finished. For example:
“This feature has a step that happens later, run by a scheduled job. Before we test it, tell me everything that job does when it runs, including anything outside our database, such as emails, payments or messages to other services. Then make clearly labelled test records that already look old enough for that step, and let the real scheduled job run on them. Do not write a separate test version of the job.”
Three things make this work:
- Know what the real job will touch. The real job does not know your records are tests. It may also process real records that are due that night, and it may send emails or call a payment service for anything it picks up. Read that list with your AI first, and if any of it could reach a real person or real money, decide how to handle it before the run.
- Backdate, do not shortcut. Change the test records’ dates so they qualify. Leave the job and its timing alone. The point is to run the same code, in the same place, that will later run for real customers.
- Write down what should be gone, and what should stay, before the run. For a deletion, that is a list of every place the data lives, plus a few nearby things that must survive. Our post on writing down the number before you run it covers the same habit for a single count: write down what you expect before you run anything.
Use test accounts with obviously fake names, so anyone reading the database can tell what they are.
How to check it worked
The job’s own report is a claim. A message saying it ran, or a green tick on the schedule, tells you it started. It may not tell you it did everything.
After the run, ask your AI for two things, shown to you as actual output rather than a summary:
- The job’s result for that run, including any errors. In the story above, the database error showed up there.
- A count for every item on your list. Each thing that should be gone should show zero. Each thing that should stay should still be there.
If anything fails, fix it, then run the same test again on fresh test records. If everything matches, you have good evidence that the later step works for the cases you tried. Our guide has more on asking for evidence rather than accepting a report that something worked.
The other reviewer’s take
Every post on this site is checked before it goes up by two reviewers that did not write it: a separate Claude reviewer and ChatGPT, from OpenAI. Below is what ChatGPT said about the first draft, quoted as it wrote it (any dashes are changed to commas, to match this site’s style), and Claude’s reply. Both reviewers then checked the final version, including this section.
ChatGPT, on the first draft (verdict: PASS):
Required changes: none
Suggestions:
Clarify “the deleted items were gone” in the first-step checks by distinguishing immediately removed records from data retained until the grace period ends.
Claude’s reply: ChatGPT passed the first draft and made one suggestion, and I took it. The draft said that after the first-step tests “the deleted items were gone”, which blurred two different things. The post now says that anything meant to be removed straight away was gone, while a deleted team’s data was hidden from its members but still stored for the grace period. That difference matters, because the data waiting out the grace period is what the nightly job erases.
I also made two small changes the other Claude reviewer suggested. The check section now says the database error showed up in the job’s result for that run, which matches the story better than the vaguer “the job’s response”. And the link to the earlier post on writing down a number now says what that post covers, a single count, rather than suggesting it is about a list. I did not find anything in either review to disagree with.