Context Engineering for AI Agents: Memory and Tools

Viacheslav Dubrov

2025-10-05 · Updated 2026-08-16 · 14 min · AI Agents

Context engineering is the pipeline that selects what a model sees before each decision: instructions, examples, knowledge, memory, tool definitions, observations, and guardrails. An agent does not act on everything the system knows. It acts on the working set assembled for the next model call.

That selection is where many failures begin. A stale preference looks current. Retrieved text contains an instruction. A long tool result hides the failed precondition. A summary remembers the decision but loses which file changed.

For engineers building or operating AI agents, retrieval systems, and tool-using applications, the practical job is to assemble the smallest sufficient working set for each agent decision while preserving scope, provenance, and permissions. This article shows how to design a per-decision context pipeline, find its trust boundaries, test whether its working set supports the task, and make failures inspectable rather than blaming a model that only sees the assembled input.

TL;DR. Treat context as a typed, provenance-carrying runtime artifact. Select it per step, enforce tenant and permission scope before retrieval, and separate trusted instructions from untrusted data. Budget by utility, validate actions outside the model, and evaluate task outcomes rather than context length alone.

Context is a decision input, not memory

The context window is the model’s current input plus generated tokens. It may contain conversation turns, but it is not a durable memory system. Long-term memory, document indexes, databases, and artifact stores live outside the window. A context pipeline chooses what to copy in.

Window size is a capacity limit, not a quality guarantee. Lost in the Middle and RULER show that retrieval and reasoning can vary with position, task, model, and sequence length. The operational lesson is not that middle tokens are always ignored. It is that adding relevant-looking tokens can still reduce task performance.

The diagram shows the study’s qualitative result, not a universal attention curve. Liu et al. tested multi-document question answering and key-value retrieval. They often observed better performance when the relevant item appeared near the start or end. The size and shape of the effect changed across models, tasks, and context lengths. Test position as a variable in your own evaluation set instead of assuming a fixed middle penalty.

The following is an illustrative Pydantic schema, not a tested repository implementation. It validates field types and literal values only. It does not authorize the tenant, keep total_input_tokens consistent with items, persist the manifest, or enforce model behavior.

from datetime import datetime
from typing import Literal

from pydantic import BaseModel

class ContextItem(BaseModel):
    item_id: str
    kind: Literal["instruction", "task_state", "evidence", "memory", "tool"]
    source: str
    trust: Literal["trusted", "untrusted"]
    retrieved_at: datetime | None = None
    version: str | None = None
    token_count: int

class ContextManifest(BaseModel):
    request_id: str
    tenant_id: str
    items: list[ContextItem]
    total_input_tokens: int

The manifest makes a failure reproducible. “The model hallucinated” becomes a testable question: which evidence, version, permission scope, and tool schema did it actually receive?

The assembly lifecycle

A reliable pipeline performs these operations in order:

1. Resolve trusted request context. Authenticate the actor, tenant, locale, time, and current task state outside the model.
2. Choose the next decision. A planning step, evidence lookup, tool selection, and final response need different context.
3. Retrieve within scope. Apply authorization and tenant filters before semantic ranking, not after documents enter the candidate set.
4. Rank and budget. Select items for utility, freshness, authority, and diversity under an input budget.
5. Assemble with trust boundaries. Keep policy in instruction channels and quote retrieved content as data. Retrieved instructions do not become system policy.
6. Generate a typed proposal. Constrained output can enforce supported syntax and shape. It cannot make values correct.
7. Validate and execute. Application code checks authorization, business rules, tool arguments, and postconditions.
8. Record provenance and outcome. Save the manifest, selected source IDs, tool result references, validation result, and task outcome.

The lifecycle is per decision. Reusing one large context for an entire agent run creates stale evidence and gives every step access to material it does not need.

Give each context source one job

Instructions define durable behavior

Instructions state role, policy, output contract, and escalation behavior. Keep stable content stable to improve prefix caching, but do not freeze values that change at runtime. A refund threshold belongs in a policy service or versioned data record, not copied into a prompt indefinitely.

Instruction hierarchy is a control boundary, not a security sandbox. A model can still follow malicious text in a retrieved document. Delimit untrusted content, state that it is evidence rather than instruction, restrict tools independently, and test prompt-injection cases.

Do not copy one provider’s role names into a universal hierarchy. The OpenAI Model Spec defines authority levels for platform or system, developer, and user instructions. The Anthropic Messages API uses a top-level system parameter and user and assistant messages. It does not expose an equivalent developer role in that API. Other runtimes make different choices. Map your policy to the provider’s documented channels, then enforce permissions and side effects in application code.

Task state records commitments

Conversation prose is a poor source of truth for multi-step work. Keep explicit state for objective, current phase, completed actions, pending approvals, artifact references, and test status. The model may summarize that state for narration. Application code owns the canonical version.

Examples demonstrate edge decisions

Few-shot examples are useful when they clarify a hard boundary, not merely repeat the schema. Select examples that match the current decision and include consequential edge cases. Evaluate example retrieval like document retrieval: a superficially similar but policy-incompatible example can be worse than none.

Knowledge supplies evidence

Retrieval is appropriate for fresh, private, or citable facts. There is no universal best top_k, chunk size, hybrid weight, or reranker cutoff. Tune the whole path against questions with known supporting evidence.

A useful evidence item includes:

• source and stable document identifier
• version or effective date
• permission scope
• quoted span and location
• retrieval and reranking scores for debugging

Do not ask the model to cite a URL it never received. Do not log full private documents merely to debug selection.

Memory supplies scoped continuity

Memory is retrieved data with additional lifecycle risk. As design metadata, track the subject, provenance, purpose, applicable consent or legal-basis decision, creation time, expiry or review policy, and deletion path. Applicable privacy law and product counsel determine the legal basis for a product and its data, not this checklist.

Fixed rules such as “preferences live 365 days” are not portable policy. Retention follows product need, user expectation, and law. Before loading a memory, check:

1. Is it for this authenticated subject and tenant?
2. Is its purpose relevant to this decision?
3. Is it current enough to use?
4. Is its source authoritative or merely model-inferred?

Treat inferred memories as hypotheses. Do not silently turn one model response into a permanent user fact.

Tool contracts expose capabilities

Tool descriptions should explain preconditions, effects, authorization scope, input schema, output schema, idempotency, and meaningful failures. A schema-valid call can still be unauthorized or unsafe.

After execution, replace verbose raw output with a typed observation that preserves the decision-relevant result and a reference to the full artifact:

{
    "tool": "check_api_key_status",
    "status": "revoked",
    "checked_at": "2026-07-15T09:30:00Z",
    "account_id": "A-123",
    "artifact_ref": "toolrun://req-42/step-3"
}

The MCP specification standardizes how clients interact with tools, resources, and prompts. It does not authorize their use or make returned content trustworthy. Keep gateway, credential, and policy checks outside the model and protocol description.

Load detail when a decision needs it

Progressive disclosure is a context-assembly pattern: keep identifiers and trusted task state available, then fetch detail for the current decision. It is not a promise that a specific token budget will work for every model.

The router must apply authorization and tenant scope before retrieval. It should return evidence spans, memory records, and tool schemas that serve the current decision. The model then proposes an action. Application code validates that proposal before execution. Anthropic’s tool-context guidance uses the same selective principle for large toolsets. Tool search keeps definitions outside the context window until the model asks for them. Measure whether the extra routing step improves task success, latency, and tokens per successful task in your system.

How context degrades

Poor context does not fail in only one way. The following five-pattern degradation model is my synthesis for debugging, informed by the long-context evaluations above and prompt-injection research:

• Position loss: the required evidence is present, but the model uses it inconsistently because of its position and the surrounding sequence. Long-context evaluations show that the effect varies by model and task. There is no universal “bad middle” range.
• Poisoning: an incorrect memory, stale document, malicious instruction, or bad tool observation enters the working set and influences later decisions.
• Distraction: relevant evidence competes with material that is recent or semantically similar but unnecessary for the current step.
• Confusion: overlapping instructions, examples, or tool descriptions leave the model with several plausible interpretations of the task.
• Clash: two authoritative-looking items disagree about a value, policy, or next action and the assembler does not expose their versions or precedence.

These failures require different fixes. Better ranking may help distraction but cannot repair a stale source. Delimiters may help separate data from instructions but cannot authorize a tool. A larger context window can preserve more conflicting material without resolving the conflict.

When a run fails, inspect the manifest and ask which pattern occurred before changing the prompt or adding another retrieval stage.

Budget by utility, not by component quota

A context budget reserves room for output and then allocates input to the current decision. Start from the model’s supported window, subtract the maximum output and protocol overhead, and pack candidates into what remains.

Score candidates with features your evaluations can challenge:

utility = relevance × authority × freshness × scope_match
          − redundancy_penalty − injection_risk

Treat the formula as a design prompt. In a policy answer, authority may dominate semantic similarity. In debugging, a recent failing log may outrank general documentation.

Order also matters. Keep stable trusted instructions first when the provider’s cache semantics benefit from a shared prefix. Put the current task and decision close to the evidence it references. Avoid timestamps or request IDs in stable prefixes when they are not needed there.

Compress without losing state

Compression is lossy unless the original remains addressable. Factory Research’s evaluation of three compression approaches over long-running agent sessions frames the target as tokens per task, not tokens per request. I adapt that into an engineering recommendation: optimize tokens per successful task, not tokens per request, under comparable task conditions—not as a universal law. Factory Research evaluation

Use separate mechanisms for separate material:

• Conversation: summarize decisions, unresolved questions, and commitments.
• Tool observations: retain typed findings and artifact references. Remove boilerplate and repeated payloads.
• Retrieved evidence: keep source IDs, supporting spans, and effective dates so the system can re-fetch.
• Task state: store canonically outside the summary.
• File or artifact trail: maintain an explicit index of reads, writes, hashes, and test results.

Trigger compaction from measured degradation or a budget threshold chosen for the model and task. Do not publish a generic “compress at 70%” rule as if all models fail at the same point.

Evaluate compression with probes that require continuation, not lexical overlap:

• What is the current objective and next action?
• Which files or records changed?
• Which decision was rejected and why?
• Which source supports the current claim?
• What approval is still pending?

Run the same task with and without compression. Compare success, incorrect actions, re-fetches, latency, and total tokens.

Optimize the context path

Optimization should preserve the decision contract. Four techniques are useful once you have a measured bottleneck:

These techniques solve different problems. Compaction and context editing can reduce the content sent to the model. Selective tool loading avoids sending unused schemas. Prompt caching can reduce the cost of repeated prefixes, but it does not reduce the number of tokens in the context window. Partitioning changes which capabilities and evidence each decision receives. The Anthropic tool-context guide makes these distinctions explicit, and its context-editing documentation exposes configurable triggers rather than a universal threshold.

Compact completed history

Replace old conversation turns with a structured handoff that records decisions, unresolved questions, changed artifacts, and test state. Keep the original transcript or artifacts addressable when review or recovery requires them.

Mask verbose observations

A tool may return pages of logs when the next step needs a status, error code, and artifact reference. Convert the raw result into a typed observation after validating it, and keep the full payload outside the prompt. Do not let the model summarize away the only evidence of failure.

Preserve cacheable prefixes

Providers and runtimes can reuse work when the beginning of a request remains stable. Keep durable instructions and tool schemas in a consistent order, and move timestamps, request IDs, retrieved evidence, and current state into the dynamic suffix. Confirm the provider’s cache semantics before designing around them.

Partition by decision

A planner, retriever, tool caller, and final-answer step do not need the same material. Give each step the minimum trusted instructions, state, evidence, and tools it needs. Partitioning reduces token use and capability exposure at the same time, but only task-level evaluations can show whether it removed necessary information.

Secure the context supply chain

Prompt-injection research treats untrusted external content as an attack surface (paper). For this article’s operational model, context poisoning can arrive through documents, memories, tool results, skills, or prior assistant messages. Marking text “untrusted” helps the model, but enforcement must be architectural.

The following supply-chain checklist is my engineering recommendation. It turns protocol and threat-model boundaries into application controls. Neither the MCP specification nor the prompt-injection paper enforces this complete checklist.

Use these boundaries:

1. Authorize before retrieval and tool execution.
2. Separate data from instruction channels and delimit external text.
3. Allowlist tools per step and actor. Default to no side-effect capability.
4. Validate resource identifiers rather than letting the model invent tenant keys or file paths.
5. Require confirmation for high-impact operations based on policy, not model confidence.
6. Scan and review executable skills or connectors before installation.
7. Prevent secrets and raw sensitive context from entering logs and long-term memory.

Automatic “repair” is appropriate only for changes that preserve meaning, such as parsing a known date format. Filling missing tool arguments with “sensible defaults” can change the operation. Ask for clarification or reject when meaning is uncertain.

Worked example: an API-key support request

For “Why is my API key not working?”, the next decision is diagnostic evidence collection, not final answer generation. The assembler might include:

• trusted support policy and response contract
• authenticated account ID and plan from application state
• the current ticket objective and actions already attempted
• two current runbook spans selected within the product/version scope
• a scoped memory that the key was created three days ago, with provenance
• check_api_key_status and search_incidents, but not key deletion or rotation tools

The model proposes a read-only status check. Application code authorizes the account, calls the tool, and records a typed observation. A second model call receives the relevant runbook spans plus that observation. The final response cites the runbook version, never prints the key, and offers rotation only as a separately authorized action.

Notice what stays out: unrelated ticket history, every support example, raw account dumps, mutation tools, and memories for other tenants.

Anti-patterns to test explicitly

• Stuff the window: loading all retrieved documents, history, memories, and tools because capacity remains.
• RAG everywhere: using semantic retrieval for values that belong in a database, policy service, or authenticated application state.
• Unbounded memory: retaining inferred facts without scope, expiry, correction, or deletion semantics.
• One call for every stage: asking one prompt to retrieve, reason, authorize, mutate, and explain without observable boundaries.
• Schema equals correctness: treating valid JSON as proof that values, permissions, or business decisions are valid.
• No component evaluations: grading only the final prose and missing retrieval, context selection, or tool failures.

Turn each anti-pattern into a counterexample in the evaluation set. A guideline that is never exercised by a task or trace is easy to violate without noticing.

Evaluate the assembler, not just the answer

Create a fixed task set with evidence labels, permission boundaries, required tool calls, and forbidden actions. For every context-policy change, measure:

Use ablations to find causal value: remove memory, reranking, examples, or compression one at a time. A component that adds tokens without improving the relevant slice should not load by default.

Conclusion

Good context engineering is selective and accountable. It does not fill a large window because capacity is available. It constructs a step-specific working set from trusted instructions, canonical task state, scoped evidence, reviewed memory, and permitted tools.

The loop is short: assemble, manifest, propose, validate, execute, and evaluate. When a decision fails, that loop tells you whether the missing piece was evidence, freshness, authority, state, or policy. It also gives you a test for the next change.

References

• Lost in the Middle — position-dependent use of long context
• RULER — multi-task evaluation of effective context length
• OpenAI Model Spec — provider-specific instruction authority levels
• Anthropic Messages API — system parameter and message-role structure
• Manage tool context — selective tool loading, caching, and context editing
• Anthropic context editing — configurable tool-result clearing and compaction
• Model Context Protocol specification — protocol concepts and current specification
• Evaluating Context Compression for AI Agents — tokens-per-task framing and probe-based evaluation
• Formalizing and Benchmarking Prompt Injection Attacks and Defenses — threat taxonomy and defenses