Context Engineering: Managing What the Model Sees

Last Audited: 2026-08-24

NUP AI-Native Verified

ISO/IEC 42001 Cl. 8.3NIST AI RMF Measure 2.7IEEE 7000-2021 Cl. 6

In Plain Language

When systems work in short demos but degrade as conversations lengthen, teams blame 'the model getting dumber'—the true culprit is context window pollution. Layer 2 is the craft of managing what the model sees: token budgeting, just-in-time retrieval, scheduled turn compaction, and working scratchpads.

Layer 2 · Context

The “Model Getting Dumber” Fallacy

Every engineering team building multi-turn LLM features encounters the same baffling bug: an assistant answers flawlessly on Turn 1 in a unit test, but by Turn 7 in a live user session, it begins hallucinating facts, repeating itself, or violating strict formatting instructions.

Teams almost universally misdiagnose this failure. They claim “the model got dumber as the chat went on” and attempt to fix it by writing longer system prompts or switching model providers. In reality, language models have no internal fatigue or stateful memory. The degradation is caused 100% by what information was or was not assembled in the context window at the exact moment of inference.

The Canonical Definition of Context Engineering:

Context Engineering is everything assembled around a single instruction before it is dispatched to the model: system directives, retrieved knowledge chunks, tool calling schemas, pruned conversation turns, and working memory scratchpads. It is an attention-management and curation problem, not a writing problem.

The Core Tension

Why Bigger Context Windows Don’t Remove the Need to Curate

With frontier models offering 128k to 2M token context windows, many teams assume context management is obsolete: “Why bother chunking or filtering when we can dump all 300 pages into the prompt?”

This assumption creates severe production failures. Dumping raw material into a context window is never free:

1. Attention Dilution (Lost in the Middle)

Transformer self-attention distributes finite attention weights across all tokens. Irrelevant noise actively obscures critical facts.

2. Latency Penalty (TTFT Inflation)

Time-to-first-token (TTFT) scales with prompt token length. Processing 100k unneeded tokens adds seconds of user-facing lag.

3. Quadratic Financial Waste

Passing 50,000 unneeded tokens across 10 turns bills 500,000 input tokens for a simple $0.001 question.

Figure 2.3 · Context Anatomy & Eviction Priority

The 5-Part Context Payload & Deterministic Eviction Hierarchy

Section 508 Accessible

Retrieval Architecture

The 6-Stage Dynamic RAG Pipeline

Delivering high-precision facts without cluttering the window requires a deterministic 6-stage assembly pipeline:

Figure 2.1 · Dynamic Context Assembly

The 6-Stage Dynamic RAG & Context Pipeline

Section 508 Accessible

Token Allocation

Context Window Budgeting & Attention Curves

To prevent context starvation, set an explicit token budget partitioned across four distinct reserves:

Figure 2.2 · Attention & Token Budget Allocation

Context Window Partitioning & The “Lost in the Middle” Curve

Section 508 Accessible

Deterministic Eviction

Interactive 5-Tier Eviction Priority Inspector

When total tokens exceed your budget ceiling, what gets cut first? Click each tier to inspect the deterministic eviction hierarchy and safety guardrails:

Tier 1: Raw API & Tool Responses

Action: EVICT_FIRST

Raw JSON database dumps, full HTML scraping outputs, and verbose HTTP payloads from previous turns.

TOKEN SAVINGS IMPACT

Reduces 50% – 70% of payload tokens immediately.

SAFETY GUARDRAIL

Retain only the extracted key-value fields needed for the active user answer before discarding raw payload.

Operational Patterns

The 4 Concrete Context Craft Moves

To build resilient multi-turn applications, master these four foundational context engineering moves:

Craft Move 01Lazy-Loading Information Exactly When the Active Turn Requires It

Just-in-Time Context Assembly

The Rule: Never front-load documents or customer history at session start. Query and assemble context dynamically on each conversational turn based on active user intent.

✓ Failure Prevented: Eliminates context window pollution, stale state injection, and unneeded token spend on turns that do not require document lookups.

Just-in-Time Context Assembler (TypeScript)

export async function assembleJustInTimeContext(session: ChatSession, currentMessage: string) {
  // 1. Classify intent to determine if RAG or DB query is required
  const intent = classifyTurnIntent(currentMessage);
  
  let dynamicContext = "";
  if (intent.requiresOrderLookup) {
    const orderData = await db.orders.findUnique({ where: { id: intent.orderId } });
    dynamicContext += formatOrderJsonContext(orderData);
  }
  
  if (intent.requiresPolicySearch) {
    const policyChunks = await hybridRetriever.search(currentMessage, { topK: 3 });
    dynamicContext += formatPolicyMarkdown(policyChunks);
  }
  
  return assemblePromptPayload({ system: session.systemPrompt, context: dynamicContext, history: session.prunedHistory, input: currentMessage });
}

Craft Move 02Strict Token Quotas with a Deterministic Shedding Hierarchy

Explicit Budgeting & Eviction Priority

The Rule: Define an explicit token ceiling for each partition (System, RAG, History, Output). When total tokens exceed budget, execute deterministic eviction in prioritized order.

✓ Failure Prevented: Prevents arbitrary truncation by LLM APIs that blindly cut off system instructions or active user constraints.

Deterministic Eviction Controller (TypeScript)

export function enforceTokenBudget(payload: ContextPayload, maxLimit = 8192): ContextPayload {
  let total = countTokens(payload);
  if (total <= maxLimit) return payload;

  // Eviction Priority 1: Truncate raw tool outputs / verbose logs
  if (payload.toolOutputs && total > maxLimit) {
    payload.toolOutputs = summarizeToolOutputs(payload.toolOutputs);
    total = countTokens(payload);
  }

  // Eviction Priority 2: Drop oldest conversation turns (>3 turns old)
  while (payload.history.length > 2 && total > maxLimit) {
    payload.history.shift(); // Remove oldest turn
    total = countTokens(payload);
  }

  // Eviction Priority 3: Drop lowest-ranked RAG chunks (below 0.75 score)
  if (payload.ragChunks.length > 2 && total > maxLimit) {
    payload.ragChunks.pop(); // Remove lowest scoring chunk
  }

  return payload; // System instructions and active input are NEVER evicted
}

Craft Move 03Rolling Turn Compression on a Defined Cadence

Scheduled Compaction & Summarization

The Rule: Do not let multi-turn conversation logs grow unbounded. After every 4 turns or when history exceeds 2,000 tokens, trigger background LLM compaction into an immutable summary card.

✓ Failure Prevented: Prevents long-chat attention collapse, contradictory advice, and exponential token billing.

Rolling Conversation Compactor (TypeScript)

export async function compactSessionHistory(session: ChatSession): Promise<ChatSession> {
  const UNCOMPACTED_THRESHOLD = 4; // turns

  if (session.rawTurns.length >= UNCOMPACTED_THRESHOLD) {
    const summaryCard = await llm.summarize({
      prompt: "Extract key customer constraints, confirmed order IDs, and resolved dispute facts into concise bullet points.",
      turns: session.rawTurns
    });

    session.compactedSummary = mergeSummaries(session.compactedSummary, summaryCard);
    session.rawTurns = session.rawTurns.slice(-2); // Keep only last 2 turns raw
  }

  return session;
}

Craft Move 04Dedicated Working Memory Block for Intermediate Reasoning

Working Scratchpad & Ephemeral State

The Rule: Separate intermediate reasoning state from permanent dialogue logs. Use a dedicated structured JSON scratchpad that updates across turns and is cleared upon task resolution.

✓ Failure Prevented: Prevents the model from having to re-derive multi-step calculations or customer eligibility from scratch on every turn.

Working Scratchpad State Schema (JSON)

{
  "scratchpad": {
    "activeOrderId": "88412",
    "customerIntent": "RETURN_ITEM",
    "verifiedCriteria": {
      "within30Days": true,
      "unopenedBox": true,
      "receiptAttached": true
    },
    "calculatedRefund": {
      "itemPrice": 42.50,
      "restockingFee": 0.00,
      "netRefundAmount": 42.50
    },
    "readinessStatus": "READY_FOR_HARNESS_EXECUTION"
  }
}

Working Memory

Interactive Scratchpad State Simulator

See how a structured working scratchpad carries intermediate reasoning across three turns of the Customer Refund Assistant without bloating the permanent conversation log:

CUSTOMER DIALOGUE (TURN 1)

Customer:

“Hi, I need to return order #88412. I bought the BT Headset last Tuesday.”

Assistant Response:

I found your order #88412 for the BT Headset ($42.50) purchased 12 days ago. To verify eligibility, is the product packaging unopened?

💡 Turn 1 initializes verified order facts without polluting dialogue memory with raw database records.

ACTIVE WORKING SCRATCHPAD (JSON)480 Tokens Total

{
  "orderId": "88412",
  "itemSku": "HEADSET-PRO-BT",
  "price": 42.50,
  "purchaseDaysAgo": 12,
  "policy30DaysMet": true,
  "conditionVerified": null,
  "readyForRefund": false
}

Figure 2.4 · Memory Architecture

Permanent Dialogue Log vs. Ephemeral Working Scratchpad

Section 508 Accessible

Diagnostics

3 Recognizable Symptoms of Layer 2 Failures

When debugging unexpected model behavior, use this checklist to differentiate context failures from prompt defects:

Symptom 1: Long-Session Quality Collapse

Observable Symptom: The model gives crisp, accurate answers on Turns 1–3, but becomes evasive, repetitive, or contradictory by Turn 7.

False Attribution: "The model got dumber" or "The model context window is unreliable."

Layer 2 Remedy: Implement Scheduled Compaction (Move 3) and Eviction Priority Tier 2 to roll older turns into an immutable summary card.

Symptom 2: Overlooked Injected Facts

Observable Symptom: The customer order ID or return policy clause was explicitly injected in the prompt, but the model still claims it does not know or hallucinates a generic answer.

False Attribution: "The model ignored my prompt" or "Need to write a stronger prompt command."

Layer 2 Remedy: Apply Cross-Encoder Reranking to prune top-k to top-3 and place critical dynamic context at the primacy (beginning) or recency (end) of the window.

Symptom 3: Runaway Token Cost & Latency Inflation

Observable Symptom: API billing and time-to-first-token (TTFT) latency climb exponentially with each conversation turn without any improvement in response quality.

False Attribution: "LLMs are inherently too slow and expensive for multi-turn production."

Layer 2 Remedy: Enforce Explicit Token Budgeting (Move 2) with fixed 25% output buffers and Just-in-Time lazy context retrieval.

Running Case Study

Refund Assistant: Layer 2 in Action

In Topic 01, our prompt-only assistant failed because it hallucinated order details it had never seen. With Layer 2 in place, when Sarah Connor messages: “Can you refund order #88412? I bought it last Tuesday,” the Context Engine executes three parallel lookups:

1. Alphanumeric SQL Query (Order Database)

Executes parameterized lookup for order 88412 linked to authenticated customer cust_9921, pulling item SKU, price ($42.50), purchase date (12 days ago), and return eligibility.

2. Semantic RAG Search (Acme Return Policy Manual)

Searches store policy embeddings for “headphones electronics return window”, retrieving the 30-day undamaged goods clause and $50 automated authorization threshold.

3. Sliding-Window Session Memory

Includes the previous 2 messages where Sarah verified her email and stated that the headset arrived in an unopened box.

Category Boundary

Boundary with Trust & Retrieval Engineering

Context Curation vs. Vector Indexing Mechanics

This topic focuses on the runtime discipline of context curation and attention budgeting inside the active window. The deep mathematical mechanics of enterprise document chunking, sparse BM25 inverted indexes, vector embedding benchmarks, and hybrid reranking architectures are comprehensively covered in the Trust & Retrieval Engineering category:

Explore Trust & Retrieval Engineering (RAG Systems Sub-Track)

The Next Boundary: Context Ingestion ≠ Execution Authority

When Sarah says “Awesome, please process that refund now,” the model cannot simply generate text saying “I have refunded $42.50.” Language models must not execute banking APIs directly via natural language. To safely intercept tool calls, validate payloads with Zod schemas, and enforce $50 hard authorization limits, we must step into Layer 3: Harness Engineering.

Continue to Topic 03: Harness Engineering & Schema Contracts

Try This with AI: Context Eviction & Working Memory Architect

Copy this prompt to architect your production context budgeting, eviction hierarchy, and working scratchpad memory.

You are a Principal AI Systems Engineer specializing in Enterprise Context Curation and RAG architectures.

I am designing the Layer 2 Context Management architecture for an AI application:
- Application Domain: [e.g., Enterprise Customer Return & Dispute Assistant]
- Context Window Limit: [e.g., 32,768 tokens]
- Multi-Turn Nature: [e.g., 5 to 15 conversational turns per dispute session]

Please design:
1. An explicit 5-tier Eviction Priority Hierarchy detailing what gets shed when token budgets are exceeded.
2. A Scheduled Compaction Policy (turn frequency and trigger token threshold).
3. A structured JSON Working Scratchpad Schema to carry intermediate decision state without bloating dialogue history.
4. Safeguards to prevent "Lost in the Middle" attention degradation.

Prompt Engineering & Instruction Design

The Four Layers of LLM Engineering

Harness Engineering & Schema Contracts

The Four Layers of LLM Engineering

Community Discussion & Feedback

Attributed peer feedback and official Netspective architecture notes.

Was this documentation helpful?(100% found this helpful • 0 ratings)

Leave Feedback or Question

○ Loading user info...

Discussion (0)

Loading discussion thread...