How information is lost from the middle of the context and what to do about it in production?
You give the model a long document and it confidently answers questions about the beginning and end, but fails when the answer is buried in the middle. This is not a bug in the implementation, it's a fundamental limitation of self-attention that breaks production RAG systems, agents and pipelines for analyzing long documents. A common mistake is to assume that the model will evenly distribute attention, and to not consider positional decay when designing prompts and architecture.
➡️ Why This Happens?
Self-attention itself is invariant to position without positional encodings, it doesn't see distance. Absolute encodings (Sinusoidal, Learnable) quickly decay in practice, while relative encodings (RoPE, ALiBi) add bias: the further a token is from the current one, the less its contribution to the attention score. In deep layers, middle tokens receive fewer gradients: information from the center of the context is replaced by noise from edge tokens. In production, with lengths of 8k-16k tokens, this leads to a 20-40% drop in recall for facts located between 30% and 70% of the sequence.
Production case: losing a fact in the middle of the context
Example: you pass a prompt with 10k tokens of context containing 5 facts about a customer, and ask it to answer fact #3, which is hidden in the middle. I ran an A/B test on GPT-4 and Llama-3 70B with synthetic data: accuracy on middle queries was 62% compared to 94% on edge queries. In a RAG pipeline, this means that the retriever can find the block, but the model simply ignores it and you get an answer based on noise, not on data.
➡️ Practical techniques for production
1️⃣ Multi-turn summarization with chunking: you split the context into blocks of 2-4k tokens, summarize each block with a separate call to the model, and pass the compressed summary + the last chunk. Trade-off: latency increases by 2-4x, but we reduced the error rate by 35% in production.
2️⃣ Sparse attention with a sliding window: use architectures like Mistral, LongLoRA, or LongRoPE. Global tokens (the first 512) hold the beginning and end, while the local window (4096) holds the middle. If you're taking a model into production, look at YaRN or NTK-aware scaling, they redistribute RoPE frequencies for even coverage.
3️⃣ Context augmentation through re-ranking: in RAG, duplicate critical facts at the beginning and end of the prompt. Or add a weighted positional bias: inject a position_id modification into the embeddings. Warning: don't do this on the entire pipeline: it can break attention for short queries (test on real data).
4️⃣ Fine-tune with samples from the center: add examples to the training where the answer is located between 30% and 70% of the length. But there's a nuance: bias towards the center worsens recall at the edges; adjust the ratio to no more than 1:5 (center : edges) and validate on both ends.
What's the Conclusion?
Positional decay is not a bug, but a property of attention design, so in production, either compress the context through summarization, or structure it by duplicating key facts at the edges, or change the architecture to RWKV or Mamba, where positional encoding does not create this effect.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore