I run autonomous agents in production every day. This one gotcha has bitten me from both sides in the same week. One killed worker nearly re-submitted the same listing to a directory that reviews duplicates by hand.
Here is how the near-duplicate happened. A billing limit killed one of my browser automation workers mid-run. It died with 84 bytes of stdout. I read that as total failure and started redoing its jobs from the top. An hour later I found the truth in its leftover browser tabs. The worker had already completed one directory submission (the listing was live), and had a second one half-filled in an open form. My "redo everything" pass was minutes away from submitting the live listing a second time.
That is the whole trap in one story. Now let's generalize it, because your stack has the same bug.
Why do retries double-fire side effects?
A dead process is not undone work. When a worker dies, you lose the report. Every external action it completed stays completed.
Most agent pipelines start with retry logic like this:
result = run_agent(publish_task)
if not result.ok:
# If the first run already published before it died,
# this line publishes it twice.
result = run_agent(publish_task)
This is perfectly safe for a pure function. It is a bug for anything that does a POST, submits a form, sends an email, or moves money. The retry silently assumes the failed run changed nothing. Look at the quadrant diagram above: that assumption only holds in one of the two "agent died" states, and you cannot tell them apart from the exit code.
LLM agents make this worse than ordinary jobs, for two reasons. Their tasks are long, so death lands mid-sequence with three of seven actions done. And their actions live on other people's platforms, where a duplicate submission, comment, or charge is visible, embarrassing, and sometimes unrecoverable.
Why can't you trust an agent's success report?
An agent's "done" is generated text, not a receipt.
My publishing agent once reported a clean cross-post to a second platform. The API token was missing from its environment at runtime, so the publish had silently failed. The agent still summarized the task as complete. Our outcomes poller caught the lie a day later by querying the platform and finding nothing.
Another time, a read-only monitoring agent reported that a human had clicked a confirmation button. It had no way to observe a click. It inferred the click from a misread screenshot and stated the inference as fact. That false "resolved" sat in a status file, hiding a deadline that mattered.
Neither agent was broken. This is just what language models do: they produce the most plausible completion of "report on your task." A plausible report and a true report look identical from inside the transcript. The only way to tell them apart is to look outside the transcript.
How do you fix it?
Treat every agent task like an unreliable network call with side effects. That gives you three concrete rules: verify against the source of truth, put an idempotency key on every external action, and reconcile before any retry.
Rule 1: verify "done" against the platform, never against the report.
USERNAME = "astraedus" # your Dev.to username
def actually_published(slug: str) -> bool:
r = requests.get(f"https://dev.to/api/articles/{USERNAME}/{slug}")
return r.status_code == 200
report = run_agent(publish_task)
state = "published" if actually_published(slug) else "unverified"
The agent's report never touches your state store. Only the platform's answer does. When you cannot verify, record unverified, not done. An honest unknown beats a confident fiction.
Rule 2: give every external action an idempotency key.
The ledger below is any durable key-value store. A dict works for a demo; use SQLite or Redis in production.
import hashlib, shelve
ledger = shelve.open("side_effects.db")
key = hashlib.sha256(f"submit:{directory}:{app_slug}".encode()).hexdigest()
if key in ledger:
# This action already fired once. Return the recorded
# outcome instead of re-running the side effect.
result = ledger[key]
else:
outcome = run_agent(submit_task)
ledger[key] = verify(outcome)
Payment APIs like Stripe support idempotency keys exactly so a retried request cannot charge a card twice. Your agent pipeline needs the same discipline for the same reason. The key comes from what the action does, not from which run attempts it. A second run computes the same key, finds it in the ledger, and gets the stored outcome back. The side effect never fires twice.
Rule 3: when a run dies, reconcile before you retry.
Before re-dispatching anything, inventory what the dead worker left behind: files it wrote, the git status of any repo it touched, its open browser tabs or sessions, and the live URLs it may have created. Each artifact tells you which actions completed. The recovery loop is small:
def reconcile_then_retry(task):
evidence = inventory(task) # files, git status, live URLs, open sessions
for action in task.actions:
if action.key in ledger:
continue # fired and recorded: skip
if action.visible_in(evidence):
ledger[action.key] = "verified" # fired, never recorded: record
else:
dispatch(action) # genuinely missing: safe to run
In my directory incident, a 30-second tab inventory would have shown me the live listing and saved the near-duplicate.
What should you do differently tomorrow?
Add a side_effects table and make every retry path call reconcile first. The one detail that matters most: assign the idempotency key before the agent runs, not after it fails. A key created during recovery cannot protect the run that just died.
None of this needs a framework. A hash function, a key-value table, and the platform's read API cover all three rules. What it needs is the mental shift: your agent does not tell you what happened. The world does.
I write these from real work at astraedus.dev, where I build apps and tools. Building something, or stuck on something like this? Reach me at astraedus.dev or [email protected].
Get the next one in your inbox, subscribe at astraedus.dev.