Harness engineering: the 7 layers around an LLM agent

Key takeaways

A reliable agent is built from seven layers: tools, context, memory, permissions, the loop, evaluation and observability.
Each layer fails in its own way, so name the layer before you try to fix a bug.
Permissions and observability are the layers teams skip most often, and the ones that hurt most in production.
Add layers in order of risk, not novelty: limits and logs before memory and multi-agent designs.

Why think in layers at all#

Agent bugs are slippery because a single bad outcome can have several plausible causes. The agent sent the wrong email. Was the tool description ambiguous? Did the right recipient fall out of the context? Did a memory from last week override the instruction? Did anyone approve it? Without a map, you tweak the prompt and hope.

Layers give you the map. Each one has a clear job and a clear interface to its neighbours. When something goes wrong you can ask which layer should have prevented it, then fix that layer rather than patching the symptoms in the prompt.

The seven below are not a standard. They are a practical grouping that covers what we see in agents that run for real people: tools, context, memory, permissions, the loop, evaluation and observability.

The seven layers of an agent harness, from tools to observability.

Layer 1: Tools#

Tools are how the agent touches the world. Each tool has a name, a description and an argument schema, and the model chooses among them based on that text alone.

Responsibility: expose a small set of well-named actions whose results the model can understand and act on.

Typical failure: overlapping tools, vague descriptions, arguments the model has to guess, and results that are huge or cryptic.

Design rule: write tool descriptions as if they were prompts, because they are. State when to use the tool, when not to, and what comes back. We go deep on this in our post on tool schemas and MCP, but the short version is that a clear tool beats a clever prompt.

Layer 2: Context#

The context window is the only thing the model can see. Layer two decides what goes in it on every call: the system prompt, tool definitions, conversation history, tool results and any retrieved documents.

Responsibility: keep the window relevant, ordered and within budget.

Typical failure: old tool output piling up, irrelevant documents crowding out the instruction, or a summary that quietly dropped the one fact that mattered.

Design rule: treat every token as something you chose to include. Trim tool results at the source, summarize old turns deliberately, and keep stable content at the front so it stays cache-friendly. Context is large enough to deserve its own treatment, so see our separate post on context engineering.

Layer 3: Memory#

Memory is information that outlives a single conversation: user preferences, project facts, the result of earlier runs. It sits outside the window and is pulled in when needed.

Responsibility: store the right things, retrieve them at the right moment, and let them be corrected or deleted.

Typical failure: stale facts treated as current, memories that contradict the user's latest request, and storing sensitive data nobody meant to keep.

Design rule: make memory inspectable and editable by the user, store the source and date with each item, and prefer the live state of a system over a remembered copy of it. If the agent can ask GitHub whether the issue is still open, it should not trust a note saying it was open last Tuesday.

Layer 4: Permissions#

This layer sits between what the model asks for and what actually runs. It answers three questions for every tool call: is this allowed, does a human need to confirm it, and with whose credentials does it run?

Responsibility: enforce least privilege and put people in the loop for risky actions.

Typical failure: one all-powerful token shared by every tool, no distinction between reading and deleting, and approvals that exist only as a line in the prompt.

Design rule: classify actions by risk (read, write, delete) and set a rule for each: go ahead, ask, or never. Default to asking for anything that changes or removes data. Enforce it in code at execution time. Our security overview describes how Setu applies this.

Layer 5: The loop and control flow#

The loop calls the model, runs tools, feeds results back and decides when to stop. Around it sit the control-flow features: step limits, timeouts, retries, planning steps, and sub-agents for isolating work.

Responsibility: make runs finish, and make them finish predictably.

Typical failure: infinite or near-infinite loops, retries that repeat a non-idempotent action, and a planner that produces a plan nobody checks.

Design rule: every run gets a hard step ceiling and a time limit. Retries apply only to calls that are safe to repeat. When the agent is stuck, it should return a clear failure with what it tried, not keep spending. Prefer the simplest control flow that works; add planning or sub-agents when a measured failure demands them.

Layer 6: Evaluation#

Evaluation is how you know a change helped. It is a set of tasks with known good outcomes that you re-run whenever you touch a prompt, a tool or the model.

Responsibility: turn "it feels better" into a repeatable check.

Typical failure: no tests at all, tests that only cover the happy path, or judging results by reading a couple of transcripts.

Design rule: start with ten or so real tasks, including ones that should end in a refusal or a question. Check the end state (was the right issue created?) as well as the wording of the answer. Add a new case every time you find a bug in production, so the same mistake cannot return unnoticed. Run the suite against fakes of your tools where you can, so it is fast and safe.

Layer 7: Observability#

Observability is the record of what the agent did and why: each model call, each tool call with its arguments and result, timings, errors and the approvals that were granted or refused.

Responsibility: make any run explainable after the fact.

Typical failure: logs that capture the final answer but not the steps, or logs that capture everything including secrets.

Design rule: log the sequence of decisions in a form a person can read, and keep secrets and tokens out of it. If a non-engineer cannot tell from the log what the agent did with their account, the log is not finished.

How Setu approaches these layers#

Setu covers the layers that touch real accounts so you do not have to build them. Tools arrive through a private MCP URL for each user. Permissions are per app and per risk level, with rules of go ahead, ask or never, and new accounts ask before writes and deletes. Connections can be read-only. Observability shows up as an activity log written in plain language, and some reversible actions can be undone for 24 hours.

Setu is not an evaluation tool or a memory store, so those layers remain yours. Workflows with schedules, webhooks and recipes cover the loop for recurring jobs. Browse the integrations to see which apps are available.

Frequently asked questions#

What are the main layers of an LLM agent harness?

A practical breakdown is tools, context, memory, permissions, the loop and control flow, evaluation and observability. Each layer has its own responsibilities and its own typical failure modes.

Which harness layer should I build first?

Start with a small set of clear tools and a bounded loop with step and time limits, then add permission checks and logging. Memory and multi-agent designs come later, once the basics are measurable.

How is harness engineering different from prompt engineering?

Prompt engineering shapes the text the model sees. Harness engineering also covers the code around it: running tools, enforcing permissions, managing context and memory, limiting the loop, testing and logging.

How do I test an AI agent?

Keep a fixed set of realistic tasks with known good end states, run them after every change, and check the actions taken as well as the final text. Add a case for every production bug you fix.