Context Windows Explained: What They Are and Why They Matter

What is a Context Window?

The context window is an LLM's working memory—the maximum amount of text it can "see" at once during a conversation.

[System Prompt] + [Previous Messages] + [Current Input] = Context
                     ↑
              Must fit in window

Key Points

• Measured in tokens (roughly 4 characters each)
• Includes both input AND output
• Everything outside the window is forgotten
• Larger windows need more VRAM

Context Sizes by Model (2026)

Why Context Windows Matter

1. Conversation Memory

Longer context = remember more of the conversation

2. Document Analysis

Larger documents need larger context to analyze in full

3. Code Understanding

Full codebase context helps AI understand project structure

4. RAG Quality

More retrieved chunks = better informed responses

Context and VRAM: The Trade-off

Memory Scaling

The attention mechanism scales quadratically with context length:

Practical impact: Doubling context roughly quadruples attention memory.

VRAM Calculator

Rough formula:
Attention VRAM ≈ (context_length² × layers × heads × 2) / 1e9 GB

Optimizing Context for Local Use

Set Context in Ollama

# Default context (usually 2048 or 4096)
ollama run llama3.1:70b

# Reduced context (saves VRAM)
ollama run llama3.1:70b --num-ctx 4096

# Increased context (needs more VRAM)
ollama run llama3.1:70b --num-ctx 16384

Context vs VRAM Trade-offs

Use RAG Instead of Large Context

Instead of stuffing everything in context:

# Bad: Huge context
prompt = entire_document + question  # May exceed window

# Good: RAG retrieval
relevant_chunks = vector_db.search(question, k=5)
prompt = relevant_chunks + question  # Fits in 4K context

The "Lost in the Middle" Problem

Research shows LLMs pay more attention to:

• The beginning of the context
• The end of the context

Information in the middle may be partially ignored.

Solutions

1. Put important info at start/end
2. Use shorter, focused contexts
3. Summarize middle sections
4. Use RAG for targeted retrieval

Context Window Techniques

Sliding Window Attention

Some models (Mistral) use sliding windows for efficiency—each token only attends to nearby tokens, not the full context.

Sparse Attention

Attend to a subset of tokens using patterns (local + global), reducing memory.

RoPE Scaling

Extend context beyond training length by interpolating positional embeddings.

Context Caching

Reuse computed attention for unchanged context portions (faster, same VRAM).

When Do You Need Large Context?

Need Large Context (32K+)

• Analyzing entire documents
• Understanding full codebases
• Long-form content creation
• Multi-turn research sessions

Don't Need Large Context (4K-8K)

• Quick Q&A
• Code completion
• Simple chat
• Most daily tasks

Key Takeaways

1. Context window = model's working memory
2. Larger context needs exponentially more VRAM
3. Most tasks work fine with 4K-8K context
4. RAG is often better than huge context
5. Place important info at start/end of prompts
6. Reduce context (--num-ctx) to save VRAM

Next Steps

1. Set up RAG as an alternative to huge context
2. Choose your GPU based on context needs
3. Understand VRAM requirements better
4. Run Llama 4 with its 10M context

---

Understanding context windows helps you optimize local AI performance. Often, smarter use of smaller context beats brute-forcing larger windows.