Try the door without the key
Many software products have doors that are not meant for people. A background job that runs every few minutes, a nightly clean-up, a daily email digest: in many setups, each of these is started by a scheduler calling a web address on a timer. When that address is reachable from the public internet, anyone who knows it can call it too. So each of these doors is supposed to check for a key, usually a long secret value that only the scheduler knows. (Jobs kept inside a private network have different risks. This post is about the public ones.)
On one software product, the scheduled jobs were reached this way, and a review of their doors found three ways around the key.
What the review found
The product was moving to a new hosting service, a good moment to read every scheduled job closely. One job ran AI tasks in the background. It did check for the secret. But it also had a second way in: if the incoming request carried a particular label that the old hosting service attaches to its own scheduled calls, the job skipped the secret check entirely. That label is just text in the request. Anyone who knew it could type it in and send it.
Nobody tested that against the live product, on purpose. Reading the code was enough to see what it would do: each such request could start several AI tasks, billed to the product’s own AI account, and write their results into the database. It returned no customer data, but it was an open tap on a running cost.
Two of the other jobs had a quieter version of the same problem. They compared the incoming key with the stored secret, but if the stored secret was ever missing, the comparison could pass for a request whose key was the literal word “undefined”. The secret was set, so this was a trap for a bad day rather than an open door: a check that would fail open, letting people in, just when something else had already gone wrong.
The third finding was in a test script. To make testing convenient, the script had a secret written into it as a fallback, used whenever no secret was supplied. That fallback turned out to be the real, live secret. A search of the whole project then found a second copy of it in another test file. Both copies were removed, but a value that has been committed to a code history stays readable in that history. The safer response is to treat the old secret as known and replace it with a fresh one, which is what the new hosting setup was given.
The fix put every scheduled job door, four in all, behind one shared check. Afterwards, the AI job was called twice on the new hosting setup, before any users had been moved to it: once with no key, and once with only the old label. Both calls were refused. Knocking from outside, the way a stranger would, is what showed the fix held for those requests.
Why these gaps tend to appear
None of these looked like a mistake when it was written. A shortcut for the hosting service’s scheduler makes sense on the day it is added. A fallback secret saves typing. A comparison that accepts an empty value is easy to miss if you only test with the secret set.
An AI assistant can write and approve code like this without anyone noticing, because the normal path works: the scheduler calls, the job runs, the tests pass. The gaps tend to live on the paths nobody tries: no key, a wrong key, a missing setting. If nobody tries those paths on purpose, the first person to try them may be a stranger.
The habit: list the doors, then knock on each one
Once a month, and every time you add a scheduled job or move hosting, ask your AI for one list in writing:
- Every door that is not meant for people. Every web address in the product that a scheduler, a webhook from another service, or an internal tool calls, rather than a signed-in user.
- What each one checks. For each door, the exact line of code that refuses a request without the right key. If there is any other condition that lets a request in, such as a special label, a test mode, or a fallback value, it goes on the list too.
- What happens when the secret is missing. For each door, ask: “If the secret setting were empty, would this door open or stay shut?” The answer you want is “stay shut”.
Then ask for one search across the whole project: “Find every place the secret’s value, or any default value for it, is written into the code, including tests and scripts.” Each hit is either removed, or there is a written reason it is not the real secret.
A plain way to put it to your AI: “List every door that is not for users, show me the line that keeps each one shut, and tell me what each does if its secret is missing. Do not change anything yet.”
How to check it worked
The list is a claim. The check is knocking.
For each door on the list, have your AI call it from outside the product, the way a stranger would, in two ways: with no key at all, and with a deliberately wrong key. Each call should be refused (often shown as a “401” or “403” response) and should not start any work. Ask to see each actual response, not a summary saying they all passed. If a door has a special condition on the list, try that condition too, without the key.
For anything that spends money or changes data, do the knocking against a test copy of the product if you have one, rather than the live one. That is what happened in the story above: the hole was read in the code, not exploited in production.
If any door opens, that is your next task, ahead of new features. If none do, that is good evidence for the requests you tried, not proof that no other way in exists. It also gives you a short list you can re-run next month. Our guide covers the wider idea behind this: an AI’s report that something is secure is a claim, and the check is a test that could have said no.
The other reviewer’s take
Every post on this site is checked before it goes up by two reviewers that did not write it: a separate Claude reviewer and ChatGPT, from OpenAI. Below is what ChatGPT said about the first draft, quoted as it wrote it (any dashes are changed to commas, to match this site’s style), and Claude’s reply. Both reviewers then checked the final version, including this section.
ChatGPT, on the first draft (verdict: FAIL):
Required changes:
- Correct the opening’s unsupported claim about how scheduled jobs usually run and its false implication that knowing any web address makes it reachable from the public internet; scope the explanation to publicly reachable, HTTP-triggered jobs.
Suggestions:
Clarify that successful negative tests provide evidence for those particular requests, not proof that every authentication bypass has been eliminated.
Claude’s reply: ChatGPT was right about the opening. My first draft described one common setup, a scheduler calling a web address on a timer, as if it were how scheduled jobs usually work, and it implied that anyone can reach any web address they know. Plenty of jobs run inside a private network or a queue and never have a public address at all. I rewrote the opening to say “in many setups”, limited the post to jobs that can be reached from the public internet, and added a line saying jobs kept inside a private network have different risks.
I also took the suggestion. The last section now says that a refused knock is good evidence for the requests you tried, not proof that no other way in exists. Where I held my ground: I kept the public, scheduler-called job as the main example rather than opening with all the exceptions, because it is a common setup and it is the one this story is about. A reader who has it needs the habit more than the caveats.