I've been running an agent continuously for months, and it fails in a very specific way. Give it a fresh session and one task, and it's sharp. Let the same session run for forty turns, and the quality falls off a cliff on work it handled fine an hour earlier. Nothing crashed. No limit was hit. It just got worse.
(Shape is illustrative. The ratio is the point, not the exact percentages.)
The fix is to stop treating the window as one container and start treating it as a budget with three tiers. A small hot layer you re-send every turn. A warm layer scoped to the task in front of you. A cold layer that sits on disk and costs nothing until you query it.
Why does an AI agent get worse the longer it runs?
Because a conversation is append-only by default, and every token you added on turn 3 is still being re-sent on turn 40.
The model doesn't forget the early context. It drowns in it. On turn 3, those three files you read were the entire point of the turn. By turn 40 you're on a different problem, and those files are still sitting there, competing for attention with the thing you actually need right now. Nothing removed them, because nothing was ever made responsible for removing them.
That costs you twice. Quality drops, because attention is finite and you spent it on stale tool output. Cost climbs, because every turn re-sends the whole history. A conversation sitting at 200K tokens that runs 40 more turns bills those 200K tokens 40 more times. (Caching softens that bill. It doesn't fix the underlying problem, and I'll come back to why.)
Notice what isn't happening in that chart. Neither window filled up. Both had room to spare. Fullness was never the problem.
What is a context budget?
A context budget splits everything the agent could know into three tiers, ranked by how often each one gets re-sent.
The tiers aren't about importance. They're about frequency. Your decision log might be the most valuable text you own, and it still belongs in cold storage, because you need two lines of it once a week. The system prompt might be dull boilerplate, and it belongs in hot, because it shapes every single turn.
One rule per tier follows from that split.
Rule 1: Cap the hot layer, and let a script enforce it
Anything loaded on every turn needs a hard token ceiling checked by code, because going over budget usually truncates quietly instead of raising an error.
Measure it with the real tokenizer:
from anthropic import Anthropic
client = Anthropic()
HOT_BUDGET = 4000 # tokens re-sent on every single turn
def count(text: str) -> int:
return client.messages.count_tokens(
model="claude-opus-5",
messages=[{"role": "user", "content": text}],
).input_tokens
used = count(open("AGENT.md").read())
print(f"hot layer: {used}/{HOT_BUDGET} tokens")
if used > HOT_BUDGET:
raise SystemExit(f"over budget by {used - HOT_BUDGET} tokens")
Run that in CI. The endpoint is free and sits on its own rate limit pool, so the check costs you nothing. A hot layer with no enforced ceiling grows every week, because adding one more rule always feels free in the moment.
Don't reach for tiktoken here. It's OpenAI's tokenizer, not Claude's, and there's no offline Claude tokenizer to swap in. It undercounts, and it undercounts worse on code than on prose. The count isn't even stable across Claude models. Anthropic's own migration guidance says the tokenizer introduced with Claude 4.7 can produce up to roughly a third more tokens on identical text. A budget you measured six months ago is a budget that's wrong today. Count against the model ID you actually ship.
Silent truncation is the part that bites. My own always-loaded startup context runs under a hard character cap, and the harness that injects it drops anything past that cap without complaining. Go over, and you lose the tail. Nothing tells you. The rule that came out of that: adding to the hot layer is a spending decision. A new line earns its place only if it prevents more trouble than it costs.
Rule 2: Retrieve instead of re-reading
Never load a whole file into context to recover one fact. Index it once, then query for the handful of lines that answer the question.
SQLite bundles a full-text search engine, FTS5, compiled in by default on virtually every standard Python build. No new infrastructure:
import sqlite3, pathlib
db = sqlite3.connect("notes.db")
db.execute("CREATE VIRTUAL TABLE IF NOT EXISTS notes USING fts5(path, body)")
db.execute("DELETE FROM notes") # rebuild, so a second run doesn't duplicate every row
for p in pathlib.Path("docs").rglob("*.md"):
db.execute("INSERT INTO notes VALUES (?, ?)", (str(p), p.read_text(errors="ignore")))
db.commit()
rows = db.execute(
"SELECT path, snippet(notes, 1, '>>', '<<', ' ... ', 20) "
"FROM notes WHERE notes MATCH ? ORDER BY rank LIMIT 5",
("retry AND webhook",),
).fetchall()
for path, snippet in rows:
print(path, snippet)
That's standard library only. No vector database, no embedding bill, no service to keep alive. An FTS5 index over a few thousand markdown files answers in tens of milliseconds, fast enough to query mid-turn instead of guessing.
Five snippets instead of five files. Reach for embeddings when you genuinely need semantic recall, but keyword search already answers most questions shaped like "what did I decide about X".
Rule 3: Make the warm layer someone else's problem
When a task reads a lot but answers little, run it in a subagent. The reading lands in that agent's window, and only the conclusion lands in yours.
The asymmetry is the whole trick. Reading forty files to answer one architectural question can run into six figures of tokens. The answer is 200 words. If your main loop does that reading, it carries every one of those tokens for the rest of the session. If a subagent does it, the main loop pays for 200 words.
The dispatch needs an explicit return contract:
CONTRACT = """
Search the repo and answer the question below.
Return at most 200 words: the answer, the file:line references that support it,
and nothing else. Do not paste file contents back into your reply.
If the repo does not answer it, reply exactly: blocked: <what you need>.
"""
That last line matters more than it looks. A vague subagent prompt comes back as a confident guess, and a confident wrong answer costs far more than the tokens you saved, because everything downstream gets built on it. Give the subagent the access to look things up itself, and give it explicit permission to come back empty.
Can the API clear old context for me?
Yes. Context editing strips stale tool results out of the history in place. It's the server-side cousin of the subagent trick:
client.beta.messages.create(
model="claude-opus-5",
max_tokens=16000,
betas=["context-management-2025-06-27"],
context_management={"edits": [{"type": "clear_tool_uses_20250919"}]},
tools=[...],
messages=[...],
)
That clears old tool output rather than summarizing it, which is what you want when most of your history is tool results nobody will read again. There's a clear_thinking_20251015 strategy too. The bare form above runs on defaults, so set an explicit trigger threshold once you know what your loop actually looks like.
Rule 4: Keep durable state on disk, not in the conversation
Anything that has to outlive the session belongs in a file, because the conversation is the one storage layer guaranteed to disappear.
Here's the test I use. If this session died mid-task right now, could a fresh one pick the work up from what's on disk? If the answer is no, your state is sitting in the wrong tier.
state/
ACTIVE.md # what I am doing now, and the next concrete step
DECISIONS.md # what I chose, and why (the why is the expensive part)
LESSONS.md # what broke, and the rule that stops a repeat
notes.db # the FTS5 index over all of it
Only ACTIVE.md is hot. Everything else is cold and gets queried. Write the reasoning, not just the outcome, because the outcome is usually recoverable from the code and the reasoning never is. That's also where the compounding shows up: a lesson written down once becomes a rule that costs almost nothing to carry forever.
Does prompt caching solve this instead?
Caching makes re-sending a long prefix much cheaper, but it does nothing about relevance, so it fixes your bill and not your quality.
It's still worth setting up properly. Caching is prefix matched, so any byte that changes invalidates everything after it. Keep stable content first (a frozen system prompt, a deterministically ordered tool list) and put volatile content (timestamps, request IDs, the varying question) after your last breakpoint. Then check usage.cache_read_input_tokens on real traffic. If it reads zero across repeated requests, a datetime.now() in the system prompt is the usual culprit. The other one is subtler: the minimum cacheable prefix depends on the model, so a prompt that's simply too short silently never caches at all.
Real money, genuinely. But a cached irrelevant token is still an irrelevant token sitting in the window. Budget first, then cache what survives.
The takeaway
Pick the rule your agent breaks worst and fix only that one this week. For most people it's the first one, because almost nobody has measured the layer they re-send a thousand times a day.
Measure it once. You'll either find nothing, or you'll find the reason your agent goes stupid after lunch.
I write these from real work at astraedus.dev, where I build apps and tools. Building something, or stuck on something like this? Reach me at astraedus.dev or [email protected].
Get the next one in your inbox → subscribe at astraedus.dev.