Writing

What running agents on production work has cost me

Four failures from running coding agents on production work, from a commit that never existed to guesses saved as facts, and what I changed after each one.

Published

I keep a list of what coding agents have cost me on production work. Between August and October 2026 it gained four entries: a commit that did not exist, a deletion I had not approved, a scheduled task that died twice without telling anyone, and guesses that were saved as facts. This is the list, with what I changed after each entry.

I’m a backend and ops engineer, and I direct agents on production work. I write the skills they follow, decide what they are allowed to run, split work across several of them and review what comes back.

An agent reported a commit that was never made

I was running a framework upgrade with one implementer agent per task, followed by a spec reviewer and a quality reviewer. One implementer finished and reported a commit hash, with the full output of the pre-commit hook. The output was complete and plausible. The reflog had no such commit. The edits were real, and the commit was not.

I found out because the next agent in the chain noticed that the branch did not match what it had been told.

Since then I run git log myself, from outside the agent, after every implementer. It costs one call, and a commit is either in the log or it isn’t. I treat every subagent report the same way now, as a claim to check. In that same project a reviewer agent told me a dependency would block the next phase. It had read the version constraint in the manifest instead of what the installed version requires, and it was wrong.

An agent deleted something outside what I approved

An agent proposed a list. Two turns later it showed me a table, and I answered that I agreed. It then acted on the earlier list as well, and deleted an item the table did not include. The item was not in git, had no snapshot and did not go through the trash, so it is gone.

What I wrote down that day is that “agreed” covers the table in front of me and nothing proposed earlier in the conversation.

When something has to be deleted now, a copy exists first and the command can only remove what was checked. In October an agent cleared 23 dead jobs from a production queue that way. It listed them, confirmed they all carried the same error, saved the rows to a file and removed them one at a time by id. I had worked the same way in July, when I retired an old local workspace: the folder was renamed to a trash directory, and the final rm became a scheduled task with pre-checks that abort if any of them fails. The deletion in August had no copy.

A scheduled task died twice in a row without telling anyone

In October I set up a weekly scheduled task in Claude Code. An agent collects a report on its own, and the approvals that follow stay with me. The first scheduled run stopped after five seconds. I corrected the prompt and tested it. The next scheduled run produced no report, because the collection script had stopped at a permission prompt and nobody was there to answer it.

The cause was how allow rules match. A compound command is split at &&, | and the other shell operators, and every part needs a rule of its own. A rule also matches the command text as written, so one written for a script’s absolute path does not cover the same script called by a relative path after a cd, and the tail at the end of the pipe needs approval too.

I fixed it in the prompt and left the permissions alone. In a run that nobody is watching, every shell call is a single command, by absolute path, with no pipe, and the agent reads the output with its file tools. These tasks also only run while the desktop app is open, so I leave slack and never use one as a hard deadline.

Silent failure is the mistake I repeat most. Outside agent work it was a backup that broke without an alert twice in six weeks, and a cron job that kept clearing a stuck database lock and hid the cause for months. Recovery that runs without an alert hides the problem it recovers from. What I do now is build the prevention together with the fix. When I put my homelab backup into code, it shipped with a failure notification and a dead man’s switch for the day it stops running at all. A scheduled agent run is the same kind of automation, and the thing to decide before scheduling one is what will tell you when it produces nothing.

Guesses were saved as facts

When I reviewed my session notes for August, about 13 of the 23 showed the same thing. A plausible hypothesis had gone into agent memory, an instruction file, a playbook or a handoff document, and nobody had verified it.

In the clearest case, subagents corrected the main agent four times in one session, and all four corrections had the same cause. It had read half of a mechanism and filled in the other half with what made sense. One of those fills explained a behaviour with a setting that does not exist in the installed version of the tool. The first inference was already stored in memory as a decision tree, and two earlier sessions had built on it. That memory was corrected twice in one day, the second time to undo the first correction. A wrong memory does more damage than a missing one, because it ends the investigation.

I am part of the same pattern. Two statements I had repeated to my manager for months did not survive reading the code.

In agent teams, every claim now carries one of three labels: confirmed in the code, stated by the documentation, or assumed. A survey run under that rule, with seven agents in parallel, overturned the premise of its own task. The task named a target version that was nine days from the end of its bugfix support, while the next major version had been stable for five months.

What I would tell someone starting

  • Check the artifact and not the report: the log, the commit that is deployed, the file on disk.
  • Say what your yes covers. In my setup, describing how I want commits is not permission to commit, and each commit and push needs its own explicit approval.
  • Before a task runs unattended, run it once by hand, then decide what will tell you when it produces nothing.
  • Make agents label where each claim comes from, and keep assumptions out of memory and instruction files.

Then keep your own list, and for each entry write down how you found out. On mine, the fabricated commit was caught because the next agent happened to notice. When the answer is luck, the entry is telling you which check to build next. For that one, the check turned out to be a single git log call.