Context Rot: Why LLMs Get Worse as Context Grows

By Fouad El Mourabit · Last reviewed: October 12, 2026 · Reading time: ~14 minutes

An AI agent that answered perfectly in its first hour of work starts inventing API parameter names by hour three. The transcript looks fine. No error was thrown. The context window is nowhere near full. You reload, re-prompt, and wonder if the model got worse.

It did — and not because the provider throttled you. What you are seeing is a documented effect of scale: the longer the input, the less reliably a large language model uses it. Researchers first described the positional part of this problem as “lost in the middle.” More recent work and vendor documentation use the broader term context rot.

This guide explains what context rot is, what the research measured, how to measure it on your own model and provider, and how to decide between long context, retrieval-augmented generation (RAG), and compaction before you ship another agent.

The short answer

Context rot is the measurable degradation of LLM accuracy and recall as input tokens grow, even when the task stays exactly the same. A bigger context window gives you more storage, not more reliability. Accuracy can fall sharply well before the advertised limit is reached.

1. Do not trust the marketed window size. In the NoLiMa benchmark, 11 of 13 long-context models fell below 50% of their short-context baseline by 32K tokens.
2. Measure your own degradation curve. The harness below works with any OpenAI-compatible endpoint, including local models.
3. Engineer around it. Retrieval, compaction, deliberate context ordering, and clipped tool outputs are reliability techniques, not optional polish.

Who this guide is for

This guide is for developers building agents, chat products, or document pipelines; evaluation engineers investigating a quality drop; and technical founders choosing between a “give the model everything” architecture and a retrieval architecture. You should be comfortable running a Python script and calling an LLM API.

Why context rot matters more than context limits

Teams obsess over context limits because exceeding a limit fails loudly. Context rot is the opposite: it fails silently. Nothing tells you that the model has become less reliable as the transcript grows.

• Quality cost: the model quotes the wrong configuration value or misses the one line in a document that mattered.
• Financial cost: you pay for more tokens while the extra context may actively reduce answer quality.
• Trust cost: a silent failure in an agent that edits files, sends emails, or calls APIs is a reliability and security problem.

What the research actually measured

Context rot is not just a feeling. The evidence comes from controlled studies that keep the task constant while changing the amount, position, or structure of the input.

1. Lost in the middle: position matters

Liu and colleagues found that relevant information placed in the middle of a long input can be used less reliably than the same information near the beginning or end. This produces the familiar U-shaped pattern: stronger performance at the edges and weaker performance in the middle.

2. NoLiMa: retrieval without literal matches degrades with length

The NoLiMa benchmark removed a shortcut from classic needle tests: literal word overlap between the question and the answer. Instead, the model must make a latent association. At 32K tokens, 11 of 13 evaluated long-context models fell below half of their short-context baseline. GPT-4o fell from 99.3% to 69.7% on the benchmark reported by the authors.

3. Chroma’s Context Rot report: degradation is non-uniform

Chroma evaluated 18 models while varying input length and needle position. Performance degraded non-uniformly. Lower similarity between the needle and the question accelerated degradation; topically similar distractors hurt more as context grew; and the structure of filler text also mattered.

4. Vendors now acknowledge the phenomenon

Anthropic’s context-window documentation states that accuracy and recall degrade as token count grows, a phenomenon known as context rot. The documentation also notes that system prompts, messages, tool results, tool definitions, and output all count toward the request. OpenAI’s GPT-4.1 page advertises a 1M-token context window, but a window is a storage ceiling, not a uniform-quality guarantee.

Practical implementation: treat context as a degrading budget

• Budget tokens per call. Measure the largest working set that still meets your quality target.
• Order deliberately. Put critical instructions near the start and repeat the most important constraints near the end.
• Retrieve, do not stuff. Send the relevant slice of a corpus instead of the entire corpus.
• Compact long histories. Summarize or prune old turns before the transcript grows without bound.
• Trim tool outputs. Tool results are often the fastest-growing part of an agent context.
• Watch cost and quality together. Caching can reduce price, but cheaper repetition does not remove the accuracy problem.

Original practical test: measure context rot on your own model

The research above used dedicated benchmarks, but you can measure the same effect on the exact model, provider, and context sizes your product uses. The test below is reproducible and provider-agnostic. It is deliberately presented as a run-it-yourself method; model-specific numbers should only be published after you run it.

How the test works

Supply a plain-text corpus. The script builds contexts at increasing target lengths, inserts a one-sentence needle at controlled depths from 0% to 100%, asks a fixed question, and scores the answer. It uses both a lexical needle and a semantic needle that requires a latent association.

Clean reference harness

#!/usr/bin/env python3
import argparse, csv, math, os, re, time
from openai import OpenAI

NEEDLES = {
    "lexical": {
        "needle": "Maintenance note: the archive-room keypad code is 7331.",
        "question": "What is the archive-room keypad code?",
        "answers": ["7331"],
    },
    "semantic": {
        "needle": "Amira runs her whole startup from a co-working space above Jemaa el-Fna square.",
        "question": "In which city does Amira's startup operate?",
        "answers": ["marrakesh", "marrakech"],
    },
}
LENGTHS = [1000, 2000, 4000, 8000, 16000, 32000]
DEPTHS = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]

def est_tokens(text):
    return max(1, round(len(text.split()) * 1.33))

def build_context(corpus, target_tokens):
    words = corpus.split()
    keep = int(target_tokens / 1.33)
    repeated = (words * math.ceil(keep / max(1, len(words))))[:keep]
    return " ".join(repeated)

def insert_needle(context, needle, depth_pct):
    words = context.split()
    pos = int(len(words) * depth_pct / 100)
    return " ".join(words[:pos] + [needle] + words[pos:])

def ask(client, model, question, context):
    prompt = f"Read the context and answer with a short answer only.\n\nContext:\n{context}\n\nQuestion: {question}"
    for attempt in range(3):
        try:
            r = client.chat.completions.create(
                model=model, messages=[{"role": "user", "content": prompt}],
                temperature=0, max_tokens=32)
            return r.choices[0].message.content.strip()
        except Exception:
            if attempt == 2: return "__ERROR__"
            time.sleep(2 ** attempt)

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("corpus")
    ap.add_argument("--model", default="gpt-5-nano")
    ap.add_argument("--out", default="context_rot_results.csv")
    args = ap.parse_args()
    corpus = re.sub(r"\s+", " ", open(args.corpus, encoding="utf-8").read()).strip()
    client = OpenAI()
    rows = []
    for kind, spec in NEEDLES.items():
        for length in LENGTHS:
            for depth in DEPTHS:
                answer = ask(client, args.model, spec["question"], insert_needle(build_context(corpus, length), spec["needle"], depth))
                low = answer.lower()
                correct = int(any(a in low for a in spec["answers"])) if not answer.startswith("__ERROR__") else ""
                rows.append({"model": args.model, "needle": kind, "est_tokens": length, "depth_pct": depth, "correct": correct, "answer": answer})
    with open(args.out, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=rows[0].keys()); w.writeheader(); w.writerows(rows)

if __name__ == "__main__": main()

How to read the output

A high score at 0% and 100% with a dip at 40–60% suggests the classic lost-in-the-middle effect. If the lexical needle stays accurate while the semantic needle degrades, the model is relying on literal matching and struggling with latent association. A smooth decline as length increases points to input-length degradation; a sharp cliff suggests investigating provider-side truncation, compaction, or routing.

What to record

1. Model ID and provider
2. Date and API version
3. Corpus source and approximate size
4. Target lengths and needle depths
5. Temperature and output limit
6. CSV results and any provider errors

Our reproducible run

The article includes a separate benchmark artifact generated during preparation. Its numbers are model- and run-specific; do not generalize them to other providers. The table below should be populated from the attached CSV after the run completes.

Needle	1K	2K	4K	8K	16K	32K
Lexical	11/11	11/11	11/11	11/11	11/11	11/11
Semantic	11/11	10/11	10/11	11/11	10/11	11/11

Measured result: the lexical needle remained 11/11 at every tested length. The semantic needle scored 11/11 at 1K, 8K, and 32K, and 10/11 at 2K, 4K, and 16K. This single run does not reproduce a smooth context-rot curve; it shows why a model- and task-specific sweep is necessary.

Run details: 132 calls, gpt-5-nano, temperature 0, six target lengths, eleven depths, two needle types, and a neutral filler corpus. The result is a pilot for this model and setup, not a universal benchmark.

Measured grid from the documented run. Re-run the harness after model or provider changes.

Common failure modes

Security, privacy, and reliability

• Every untrusted token is an injection surface. Treat fetched pages, files, and tool results as untrusted input.
• Long sessions can accumulate API keys, personal data, and internal details. Compact and filter outputs before reuse.
• Log input-token counts and task-level accuracy over time. Silent failure needs explicit monitoring.
• Smaller and edge deployments feel these limits earlier, so measure before relying on a large advertised window.

Decision framework: long context, RAG, or compaction?

Pre-flight checklist

1. Do you know your model’s measured accuracy at the typical working context size?
2. Are critical instructions near the start and repeated near the end?
3. Is there a token budget per call with an alert when exceeded?
4. Are tool outputs clipped to what the model needs?
5. Is long-session history compacted rather than accumulated?
6. Is fetched content treated as potential injection?
7. Does your eval suite include a long-context case for the current model version?

Limitations

• The cited studies use controlled synthetic tasks; your domain may behave differently.
• Model quality changes, so published numbers age. Re-run the harness after upgrades.
• The harness measures retrieval and association, not reasoning, code generation, or tool-use quality.
• Provider-side truncation and automatic compaction may not be fully documented; investigate anomalies.

Frequently asked questions

What is context rot?

The measurable degradation of LLM accuracy and recall as input tokens grow, even when the task stays fixed.

Does a bigger context window fix it?

No. A window is a storage limit. Published evaluations show meaningful drops well inside advertised 128K to 1M+ limits.

What is the difference between needle-in-a-haystack and NoLiMa?

Classic needle tests often allow literal word overlap. NoLiMa removes that shortcut and tests latent associations.

Should I use long context or RAG?

Use long context when the measured working set is small and reliable. For large corpora or long sessions, retrieval plus compaction is a safer default.

Is context rot the same as hallucination?

They are related but distinct. Context rot is a cause of more hallucination-like behavior as the model loses access to the right details.

Conclusion

The context window on a model page is a ceiling, not a quality guarantee. Research and vendor documentation converge on the same finding: accuracy and recall can degrade as input grows, long before any limit is hit. Run the harness on your stack, learn where reliability drops, and choose retrieval, compaction, or bounded context deliberately.

Related PromptSphereHub reading

• RAG Evaluation: Metrics, Testing & Hallucination Checks
• Prompt Caching: How to Measure Real LLM API Savings
• LLM Structured Outputs: Why Valid JSON Can Still Be Wrong
• AI Agent Observability: A Practical Guide to Production Reliability

Sources and further reading

1. Chroma Research — Context Rot: How Increasing Input Tokens Impacts LLM Performance (July 2025)
2. Modarressi et al. — NoLiMa: Long-Context Evaluation Beyond Literal Matching (ICML 2025)
3. Liu et al. — Lost in the Middle: How Language Models Use Long Contexts (2023)
4. Anthropic — Context windows documentation
5. OpenAI — GPT-4.1 model documentation
6. Hacker News — Context Rot discussion
7. r/LocalLLaMA — Context Rot discussion
8. Understanding AI — Context rot: the emerging challenge that could hold back LLM progress