Harness Engineering for AI Agents: Designing Control Loops

Viacheslav Dubrov

2026-07-22 · Updated 2026-08-01 · 24 min · AI Agents

An agent’s reasoning loop chooses the next action. Its harness supplies context, validates proposed tool calls, authorizes them, dispatches what it accepts, records the results, and decides whether the task is complete.

The first three of those jobs already have an article. Memory supplies the context, tool use defines what can be proposed, and security decides what runs. They ran as separate posts because they are separate engineering problems. Choosing between Qdrant and pgvector has nothing to do with writing a PreToolUse deny rule.

They also share a moment: the gap between the model naming an action and the machine performing it. Each answers a question about that gap, and the harness is the code that holds it open long enough to ask all three.

For engineers building or reviewing coding-agent harnesses, the remaining job is to decide whether finished work is actually complete and to prove that each control in the loop earns what it costs. Part 5 covered the runtime that keeps the process alive underneath.

Beyond the first explicit acceptance check, every added retry, handoff, or evaluator is a hypothesis about an observed failure. It earns a place only when a controlled comparison shows that it helps.

The simplest acceptance check is easy to write for a small research agent like the one this series has been building — a LangGraph agent that fetches market data and writes an analyst report. A hook outside the model validates the report against a schema and checks that it actually contains stock tickers; a malformed report keeps the run open. Twelve lines of ordinary code, and the model does not get to declare its own output well-formed. The repo’s own gate is softer: a fresh-context evaluator that votes, then a human. (Part 4 sketches the deterministic version.)

What that example cannot show is the interesting part: what happens when the evidence is ambiguous, when a retry might double-charge someone, or when the work outlives the session that started it. Those need a task with a sharper pass/fail boundary than a research report has. The research agent stays as the acceptance-check example; a small fictional store repository joins it for the retry and handoff cases. The coding task is to lower the threshold for a 10% automatic discount from $100 to $75 in src/checkout.py. The repository has two required checks:

• pytest tests/test_checkout.py verifies the discount calculation.
• pnpm playwright test tests/checkout_discount.spec.ts adds an $80 item in a local test store and checks that the checkout page shows an $8 discount.

The example is a teaching fixture, not a real application or benchmark. Every attempt starts from the same commit and seeded test data. The harness may accept the change only when both commands pass and the trace ties those results to the tested commit.

The diagram follows the discount change from proposal to evidence. The harness supplies the task and files, checks the proposed edit_file arguments and permissions, and dispatches the accepted call. After the runtime applies the edit, the harness runs the named unit and browser acceptance tests. A failed command goes back to the model as evidence for another turn; two passing commands make the change eligible for acceptance.

What the harness owns

OpenAI’s Codex loop walkthrough describes the basic cycle. The harness assembles a prompt, asks the model for the next action, sends an accepted tool call to the runtime, and appends the result. Then it asks again. That repeats until the harness accepts the result or hands control back to the user.

Implementations may merge several responsibilities into one process. The failure boundaries are still different:

The runtime row covers four things: session, sandbox, checkpoint, and trace. All four store state or confine execution. The model proposes the action, and the reasoning loop chooses the next move. The harness decides whether a proposed call may run and whether the evidence is sufficient to finish, which is why it gets its own article. Part 5 counts the harness alongside those four as one of five primitives you have to place before shipping; this article splits it back out.

When a failure appears, diagnose the boundary that should respond. A poor plan may need better instructions or model reasoning. If edit_file targets a path outside src/, the harness should reject it. A sandbox process that dies before the edit runs belongs to the runtime, which must restart the worker or report the crash.

Where the earlier parts land

The harness row above is doing most of the work in that table, and it is where Parts 2, 3, and 4 end up. Each of them decides one thing about a single turn:

Parts 3 and 4 share step 3, and that overlap is the whole argument for treating them as one program. The same layer of harness code that rejects a malformed argument also rejects a call that is permitted but not yet approved. Split across two services, those two rejections drift apart, and a call that passes schema validation in one place gets authorized somewhere that never saw the schema.

The split still matters for debugging: a wrong-file edit is a Part 4 path rule, not a Part 2 retrieval problem. A section near the end of this article turns that into a routing table.

OpenAI’s own harness-engineering case study describes a bootable application instance for each worktree. The team also wired browser automation into the agent environment and exposed logs, metrics, and traces.

A task such as “no span in these four critical user journeys exceeds two seconds” became testable because the agent could run the application and query the same signals an engineer would inspect. The case study is product-specific. What transfers is the condition behind the result: the application and its performance signals had to be available inside the agent environment.

Lopopolo, the author of that case study, maintains a field guide for harness engineering. It names the two levers this article pulls: hold the model and coding agent fixed as a black box, and engineer the context and tools around them. His framing also explains why so much of the harness ends up as ordinary code.

An organization’s quality bar, procedures, exception history, and authority relationships sit outside what a general model can know. The harness surfaces them as repository instructions, permission rules, and acceptance checks. Each accepted run can feed its lessons back into those artifacts instead of relying on the next session to rediscover them.

Follow the discount change from proposal to acceptance

For the discount task defined above, the model proposes changing calculate_discount in src/checkout.py. Several things happen before that edit counts as progress:

1. The context builder supplies the task, repository instructions, relevant files, prior tool results, and the current plan.
2. The model proposes an edit_file call with a path and replacement text.
3. The tool boundary (the harness code between proposal and execution) validates the arguments, checks the path against the allowed scope, and asks for approval if the operation needs it.
4. The runtime applies the edit in the sandbox and returns a structured result.
5. The harness runs pytest tests/test_checkout.py, followed by pnpm playwright test tests/checkout_discount.spec.ts, and reads both exit codes. The browser test checks the visible $8 discount on the seeded $80 cart.
6. The harness decides what the results mean. A failed check becomes new context for the next model turn, and a passing run makes the task a completion candidate.
7. A passing result becomes completion evidence only after the harness records the command, exit code, and tested artifact version in the trace.

After step 2, no file has changed. The harness can reject ../../secrets.env, require approval for a destructive command, or stop a run that has exhausted its budget. That is the last cheap moment you get. After the tests run, the harness reads their exit codes itself. The model cannot mark its own edit as passing.

The trace should show the proposed path and replacement text, the permission decision, the files that changed, the tested commit, and both command results. A final done message without those records does not prove that this change passed its required checks.

Decide where each rule is enforced

The requirement that tests/checkout_discount.spec.ts passes belongs in deterministic code, not in the prompt. The harness dispatches the Playwright command to the runtime, reads its exit code, and refuses to end the run while it fails. A prompt can remind the model to run the test. It cannot stop the model from declaring success without evidence.

Other rules fit different layers:

Tool contracts separate proposal from permission

The discount task only needs file edits and test commands. A state-changing API has a different failure mode, so switch examples for this section. Suppose the agent can call create_test_order against a staging order service while setting up test data. This tool is not one of the discount task’s acceptance checks. It is useful here because a timeout can hide whether the service created an order.

The tool boundary needs more than a natural-language description. It needs an explicit tool contract. Part 3 argued for one from the model’s side: clear actions, compact feedback, recoverable errors. The harness needs the same contract for a different reason. It has to decide, without asking the model, whether a call may run and whether a failed call may be repeated. For create_test_order, that means a contract with:

• validated arguments, so malformed input is rejected before execution
• a structured result such as { "order_id": "123", "created": true }, so later checks do not have to parse free-form text
• an effect category that records whether the call only fetches information or changes a file, database record, or external service. It also records whether repeating the call is safe. This label tells the harness whether an automatic retry could duplicate work. The harness can retry get_order_status when the service defines that lookup as read-only. It must not blindly retry create_test_order, because the first call may already have created the order
• a timeout and retry policy, so a lost response does not trigger an unlimited sequence of calls
• a permission rule that says what approval is required. Reading order status may run automatically, while creating an order may require confirmation

The natural-language description is text shown to the model. It might say, “Create a test order for checkout verification.” That sentence helps the model decide when to propose create_test_order. It does not authorize the call. In this example, the harness’s Model Context Protocol (MCP) client validates the arguments, applies its own rules, and checks server trust, approval requirements, and retry safety before dispatching anything. That check is the ordered deny/ask/hook/allow ladder from Part 4, with one question added: whether a call that already failed may be sent again.

An MCP server publishes tool descriptions and optional behavior annotations to the client. A faulty or malicious server could describe a state-changing tool as harmless. A client that accepted that claim automatically might run or retry create_test_order without approval and create a duplicate. The MCP specification therefore requires clients to treat tool annotations as untrusted unless the server itself is trusted.

The specification does not prescribe one universal trust setting, so you need an explicit trust policy for your deployment; a server cannot make its own annotations trustworthy. That policy decides which metadata may influence permission or retry decisions and which annotations remain advisory only.

Retrying a state-changing call needs replay protection

Part 5 puts an idempotency key on every side-effecting tool call. The harness is what decides when that key has to carry the weight. create_test_order creates the order but its HTTP response is lost. The harness sees a timeout and cannot tell whether the server completed the request. Repeating the call may create a second order.

A status lookup can be retried when the service defines it as read-only. A creation call needs the key: the client attaches a unique request identifier, and the service returns the first result instead of creating another order when it sees that identifier again. Without this protection, the harness should check whether the order exists or ask for a human decision before another attempt. AWS documents this pattern in its idempotent API guidance.

Acceptance needs independent evidence

A successful create_test_order response establishes only that the tool returned data. It does not prove that a coding task passed its tests. If a later browser test depends on the staged order, the harness must validate the response schema and still run that test before accepting the code change.

Some criteria cannot be reduced to an exit code. For a separate visual-design task, a fresh-context evaluator can compare a rendered page or diagram with a written rubric — “fresh-context” meaning a second model session that starts with no history of the run and reads the produced artifacts rather than the transcript. Check that evaluator against human reviews before you let it gate completion.

A payment-adapter migration needs a handoff

Switch tasks again, but stay in the fictional store repository. The agent now has to migrate checkout from payment adapter v1 to v2. The work spans the checkout handler, payment client, configuration, and tests, so it may outlast one model session — one continuous stretch of model context, ended by a restart or by a deliberate fresh start rather than carried forward.

Before the first session reaches its context limit, it has modified several files, started a local payment sandbox, and left tests/payment_migration.spec.ts failing. That browser acceptance test completes one payment through adapter v2 and verifies the recorded provider ID. A conversation summary can orient the next model session, but it cannot restart the sandbox or prove which files are currently modified.

The next session has to recover three things:

Compaction replaces older messages with a shorter summary so the current session can continue. A progress handoff records what the next session needs: the current branch, changed files, last test command and output, and the next unresolved step.

A handoff file is document memory written for one reader, the next model session, and it is a different artifact from the checkpoint the runtime restores. The checkpoint answers where execution stopped. The handoff answers what the work means and what remains. Restore a checkpoint with no handoff and the next session gets a resumable process and no idea which of the four touched areas — handler, client, configuration, tests — it already finished. That is what produces duplicate work.

If the old conversation contains stale assumptions, the harness can start a fresh model session with that handoff and the current workspace. Replacing a crashed worker and restoring its processes is a separate runtime recovery job.

A small documentation edit may need none of these mechanisms. The payment migration needs a handoff once work crosses sessions because the next model session must reconstruct both the workspace and the task status.

Anthropic’s experiments with long-running coding agents used git history and a progress file between sessions. Anthropic’s later harness-design report separates compaction from a fresh-context handoff and reports that handoffs add orchestration, token use, and wall time, without publishing figures that attribute any of that overhead to the handoff itself.

Use traces to distinguish three failures

The next three rows are illustrative trace sketches, not measured runs or output from the companion lab. Each row shows a different failure and therefore a different harness response.

A transient-looking failure does not make every call safe to retry. The first row is a read-only lookup. The second row is a state-changing request, so the idempotency key and server-side status decide whether another creation attempt is allowed. The third row is not a tool failure at all; the harness has not yet collected the evidence required to accept the discount change.

A chat transcript records what the model saw. It cannot prove whether the order service committed a request before the response disappeared. The transcript is the agent’s account of events; the trace is what the machine actually did. When the two disagree, believe the trace. The trace must join the client call, approval decision, idempotency key, server result or status lookup, tested commit, and acceptance-test result. Those fields tell the harness which of the three paths it is on.

Before adding a component, name the repeated failure it should reduce and the number you will track. Remove the component if a controlled comparison does not move that number enough to repay its cost. Most harnesses I have seen grow the other way: someone hits a bad run, adds a guard, and the guard stays forever because nobody can prove it is safe to delete. That is how you end up with a loop nobody wants to touch.

Turning those repeated failures into a versioned regression suite is its own job. I wrote that up separately in AI Agent Evaluation in Production.

Measure one change at a time

An ablation measures whether a harness component causes the expected effect by changing or removing that component while the rest of the experiment stays fixed. For example, does editor linting help this model on this task suite?

Use the following protocol:

1. Freeze the model version, task instances, environment, grader, and prompts outside the component under test.
2. Give both variants the same total token, time, and dollar budget.
3. Choose the number of trials or stopping rule before running the comparison.
4. Run the same task instances in both variants. Because model outputs vary, repeat each task several times.
5. Report the mean together with the spread or confidence interval.
6. Count every started trial, including timeouts, policy stops, harness crashes, and evaluator failures.

Success rate alone can hide an expensive component. At minimum, track broken tasks accepted as complete, cost and wall time per completed task, tool errors, duplicate orders, review minutes, and manual permission overrides. Choose the metric that carries the real cost for your product. A two-point rise in completed tasks is a bad trade if it doubles your review queue.

A paired payment-migration experiment makes the progress handoff measurable. Each control/treatment pair starts from the same repository commit and seeded checkpoint, with the same model, task, grader, and total budget. The handoff is the only switch. The primary metric counts duplicate tool actions after resume: an action is duplicate when its operation and artifact match a step that the previous session had already completed.

The SWE-agent paper fixes GPT-4 Turbo on the 300-task SWE-bench Lite split and reports 18.0% resolved with its full interface, compared with 11.0% for a shell-only agent given a worked demonstration and 7.3% for the same agent without one. The paper’s headline 10.7-point gap is measured against that 7.3% baseline; Part 3 works the same three numbers from the interface-design side. The paper also changed individual interface features:

These numbers belong to that model, benchmark, and $4 per-task cap. The three rows below the reference are the useful one-feature tests: each changed an interface feature while the model and evaluation setup stayed fixed.

LangChain published a broader fixed-model comparison for deepagents-cli. It reports a Terminal-Bench 2.0 increase from 52.8% to 66.5% with gpt-5.2-codex fixed while its team changed the system prompt, tools, and middleware. The post bundles several changes and omits a confidence interval, fixed-total-budget comparison, and per-change ablation table. That result cannot identify which change helped. Model names in this section are the ones each study fixed at the time it ran; the protocol is what transfers, not the model list.

Anthropic’s long-running application report is a qualitative, product-specific case study rather than a controlled benchmark. The application is RetroForge, a 2D retro game maker; in Sprint 3 the harness evaluator checked 27 criteria covering its level editor. The work began on earlier Opus models, and when Opus 4.6 shipped the team removed harness components one at a time to see which ones the newer model had made redundant. It reports that evaluator calls became overhead on tasks Opus 4.6 could complete reliably alone, but still helped near the model’s edge. The example is a reason to revalidate old scaffolding when the model changes; it does not estimate a general effect size.

Keep the harness editable after it earns its place

Ablation keeps a harness small, but its code can still outlive the model it was tuned for. A request such as “mask secrets in every capture path” names a behavior, not a file. In a production harness, that behavior may span execution stages and shared state. Before you can change it safely, you have to find every implementation site — and so does the coding agent you delegate it to.

A 2026 preprint from Wang et al., the Harness Handbook, calls that search behavior localization. The handbook builds a behavior-centric map from the harness codebase. Static analysis, which needs no model calls, extracts a program graph, and an LLM then organizes its units into execution stages.

The maintainer or coding agent starts with a system overview, opens the relevant execution stage, and descends to source-grounded entries for a function or file. A state register records where shared state is written and read between stages. This hierarchy keeps the overview small while preserving a path to source.

Freshness is a separate rule. Every locator must resolve against the live repository. The handbook freezes stale entries instead of guessing, and each non-empty diff resynchronizes the entries it affects.

The diagram compresses the modification loop: a behavior-only request descends the handbook’s levels, every candidate locator is verified against the live repository before the plan is written, and each applied diff resynchronizes the map.

The Handbook evaluation follows the protocol this article argues for. It covers two open-source harnesses: Terminus-2 (six Python files) and the Codex monorepo (2,267 Rust files). In each, a read-only planner powered by DeepSeek-V4-Pro either explored the repository directly or routed through the handbook. Requests, repository, tool permissions, and decoding were identical in both arms. Three judges (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro) scored each edit plan on localization, scope control, and reasoning — note that one judge is the same model that produced the plans:

The handbook-assisted planner won more often and used fewer planner tokens in both repositories. The conditions stay attached to that result: three LLM judges scored edit plans produced by one planner model on two harnesses. The study evaluated plans, not executed diffs or production defect rates.

Try the method in the companion lab

The harness-demo project at commit 517353f3 is a small, deterministic exercise with 12 generic synthetic tasks covering code changes such as fix-parser-edge-case, split-large-module, and wire-browser-test. It does not implement the fictional store repository.

Each task fixture declares a difficulty plus four boolean conditions: a flaky tool, lost progress, a missed implementation gap, and ambiguous completion. The simulator derives a fifth condition for difficult tasks that also need a progress file: without context_reset, compaction preserves stale assumptions. A deterministic grader marks a task as passed only when the selected configuration handles every applicable condition. No model or external service runs.

The commands answer different questions:

• make check runs Ruff and seven unit tests, including the validator that rejects any ablation pair that changes more than one component.
• make run prints a cumulative teaching matrix, then five valid leave-one-component-out comparisons.
• make failures names the unhandled condition for every failed task. The full harness should end with all synthetic tasks pass.

make check
make run
make failures

The causal section of make run looks like this:

component                 control  treatment  delta
retry_policy              8/12     12/12       +4
progress_handoff          7/12     12/12       +5
evaluator                 8/12     12/12       +4
fail_closed_acceptance    7/12     12/12       +5
context_reset            10/12     12/12       +2

For each row, the control is the full configuration with one component removed; the treatment restores only that component. The earlier cumulative matrix is useful for orientation, but some of its adjacent rows add several components at once and therefore cannot identify a cause.

The lab validates every declared pair before running it. Its regression tests also include an intentionally invalid pair that changes retry policy and evaluator together; the validator rejects it.

The lab compares all five component fields when it validates a pair. This runnable excerpt shows the same guard on one valid progress-handoff pair:

from dataclasses import dataclass, fields

@dataclass(frozen=True)
class Config:
    progress_handoff: bool = False
    evaluator: bool = False
    retry_policy: bool = False
    fail_closed_acceptance: bool = False
    context_reset: bool = False

def changed_components(control: Config, treatment: Config) -> tuple[str, ...]:
    return tuple(
        field.name
        for field in fields(control)
        if getattr(control, field.name) != getattr(treatment, field.name)
    )

control = Config(progress_handoff=False, evaluator=True, retry_policy=True)
treatment = Config(progress_handoff=True, evaluator=True, retry_policy=True)
assert changed_components(control, treatment) == ("progress_handoff",)

Which layer to open when a run goes wrong

The series ran inner to outer, and this is where that ordering pays. A failing agent run usually has one owner:

Four of those six rows are harness code. Row 5 is the runtime underneath it, and row 1 is the only one a prompt can still move.

Start with one loop and one acceptance check

I would start a coding-agent harness with one capable model, repository instructions, a few narrow tools, a sandbox, and one explicit acceptance test. I would record tool calls, results, costs, and that final test in one trace so the first useful failures are visible without reconstructing them from terminal logs and chat transcripts. This is a proposed baseline, not evidence from a deployed system.

From there, add only what a trace justifies. Record who maintains each component, how many tokens or seconds it adds, and which regression test would justify removing it after a model upgrade.

Six months later, someone who sees progress_handoff=True should be able to find the failed traces that justified it and the regression cases that still keep it there. The traces explain why the component exists; a current behavior map explains where to touch it.

If you arrived here from a search, the five preceding articles built a system around a reasoning loop:

1. The loop chooses the next move.
2. Memory supplies context, and a real Postgres checkpoint store preserves it.
3. Tool contracts define actions and the result shapes that later checks can read.
4. Security adds the deny hook and stop-hook validator. Both remain sketches in the example, but they mark the control points.
5. The runtime keeps the process alive across sessions and failures.

The series also added an MCP sidecar to show where data-provider tokens belong and an evaluator node that checks the draft report before a human sees it. Those are ordinary pieces of code around a model call. The router is harness code for the same reason: it chooses the reasoning pattern before the reasoning loop starts.

Your next step is to instrument one small loop. Record tool calls, results, costs, and one explicit acceptance test. Add one control only after a trace shows the failure it addresses. Compare it with a fixed control, then remove it when the measured benefit disappears.

References

• OpenAI, Unrolling the Codex agent loop.
• OpenAI, Harness engineering: leveraging Codex in an agent-first world.
• Lopopolo, Harness engineering: anthology, field guide, and agent context bundle.
• Anthropic Engineering, Effective harnesses for long-running agents.
• Anthropic Engineering, Harness design for long-running application development.
• LangChain, Improving Deep Agents with harness engineering.
• Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.
• Wang et al., Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable, arXiv:2607.13285, 2026.
• AWS, Making retries safe with idempotent APIs.
• Model Context Protocol, Tools specification.
• Market Analyst Agent Repository

The Market Analyst Agent code is on GitHub.