There’s a reflex, when a model does the wrong thing, to add a sentence. Then another. The prompt grows every time it disappoints you, on the theory that more instruction means more control. Past a point it means the opposite, and the cost is measurable.

I’ve argued before for writing to models in structure instead of prose, and for treating the prompt as a craft worth practicing. This is the case underneath both: length is not a free variable you can keep turning up. Every token you add competes with every token already there, and after a while the additions start eating the instructions you cared about.

The cost has a name

The comfortable assumption is that a longer, more detailed prompt is a safer one. More room, more coverage, fewer ways to be misunderstood. The evidence says otherwise, and the effect has a name: context rot. Chroma tested 18 frontier models, including Claude 4, GPT-4.1, Gemini 2.5, and Qwen3, and found that output quality drops as input grows, “often in surprising and non-uniform ways,” even on tasks that should be trivial, like finding one relevant sentence or repeating a list back. Follow-up work through 2026 has found the same on newer models, so this is not an artifact of one generation you can wait out.

The finding that should change how you write prompts is the one about relevance, not size. A single distracting passage, one irrelevant paragraph, measurably lowered accuracy. That bites at any length, including the few hundred tokens a real prompt occupies: the defensive sentence you added last month is that distractor. Sheer length is the same effect taken to its extreme, where a model with a 200K-token window can degrade noticeably by 50K tokens, the decline gradual rather than a wall at the advertised limit. Your prompt will never be that long, but it sits on the same curve. Padding a prompt “to be safe” is not neutral. It is more surface for attention to scatter across, and some of it drags the model off course.

Why prompts get fat

Prompts bloat the way code bloats: by accretion, one incident at a time. The model formats a date wrong, so you add a line about dates. It gets too casual once, so you add a paragraph on tone. Each addition is rational on its own and nobody ever goes back to delete. A year later you have a wall of defensive instructions, most guarding against failures that happened once and would not happen again on a current model. It reads as thoroughness. It behaves as noise, and the rules you actually depend on end up buried in the middle, which is where models pay the least attention.

The tell is that you cannot say, without scrolling, which three lines in your prompt are doing the real work. If you cannot, the model can’t either.

The lever is density, not brevity

The fix is not to write short prompts for the sake of it. A vague one-liner fails just as reliably as a bloated essay. The variable that matters is signal per token: how much of your intent each word actually carries. Anthropic frames the target as the “smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome,” and both halves carry weight: the set has to be small and every token in it has to earn its place.

This is why structured, notation-style prompting tends to beat prose. A labeled block says the same thing in fewer tokens and hands the model a clear hierarchy instead of making it infer one from paragraphs. Three fields on their own lines beat three sentences that bury the same constraints in filler. The tokens you keep are easier to read, too.

What earns its place is the constraint that changes the output: a format the model would otherwise get wrong, a fact it cannot know, a boundary it would cross. What doesn’t: restating the obvious, hedged qualifiers, politeness, and the third example that only repeats what the first two already showed.

An editing pass you can run today

Write the prompt the way you normally would, then cut it by a third and watch what happens to the output. Most of the time nothing does, which tells you the third you removed was carrying no weight. When quality drops, you have found a line that was genuinely load-bearing, so put it back, and now you know it earns its slot instead of assuming it. Do this a few times and you stop guessing which instructions matter and start measuring it.

Treat prompt length as a budget you spend, not a bucket you fill. The best prompt is the shortest one that still gets the result reliably, and it is almost always shorter than the one you have.

Sources