Testing Harness Components Independently

Over the weekend I read An Empirical Study of Harness Design for Coding Agents. There are papers that compare coding harnessesharnessManages the agentic loop, tools, context, and permissions for coding agents. Often wraps an agent and/or model provider sdk. with one another and test them against different models. The difference with this paper is that it evaluates three components of a harness: planningPlanningNot to be confused with plan mode, this is a harness feature that lets the agent create and update its own task plan while it works. The plan is stored separately from the conversation history and supplied to the LLM on every turn., action spaceaction spaceThe ways an agent can interact with its environment. The paper evaluated two: One with filesystem tools including a bash tool, and one with only the bash tool., and context managementcontext managementPrevents exceeding the model's context window. The paper evaluated four context management strategies and the absence of context management altogether., against different models.

The takeaway is that harnessharnessManages the agentic loop, tools, context, and permissions for coding agents. Often wraps an agent and/or model provider sdk. features behave differently across models and sizes, so you should test them independently against your chosen model and version. Makes sense, but there are some related implications that stood out to me.

The paper found that large models performed worse when provided filesystem tools like read_file, glob_files, grep, and so on. Large models did better in terminal tasks when these tools were removed, leaving only the Bash tool. Weaker models performed better with filesystem tools because they attempted to invoke tools like list_files even when they didn't exist! Nemotron-3 30B suffered failures of 66% Bash-only Terminal-Bench trajectories due to this.

The same large frozen model scored better on one benchmark and worse on the other when filesystem tools were removed. To understand why, consider the tasks involved. SWE-Bench Verified tasks are code fixes where the agent is given an issue and a codebase to fix it in. On the other hand, Terminal-Bench 2.1 tasks are terminal workflows, like booting Windows 3.11 in a QEMU virtual machine and configuring VNC on it.

As the authors point out, the larger models tend to compose operations into single commands with Bash. I see this evidence myself as it becomes more challenging to follow agent session traces, since you end up scrolling through this kind of output:

sed -i \
  's/cright\[-right.shape\[0\]:, -right.shape\[1\]:] = 1/cright[-right.shape[0]:, -right.shape[1]:] = right/' \
  astropy/modeling/separable.py &&
python - <<'PY'
from astropy.modeling import models as m
from astropy.modeling.separable import separability_matrix

cm = m.Linear1D(10) & m.Linear1D(5)
print("cm:")
print(separability_matrix(cm))

print("\nPix2Sky_TAN() & Linear1D(10) & Linear1D(5):")
print(separability_matrix(m.Pix2Sky_TAN() & m.Linear1D(10) & m.Linear1D(5)))

print("\nPix2Sky_TAN() & cm:")
print(separability_matrix(m.Pix2Sky_TAN() & cm))
PY

Relatedly, there is an issue on the Claude Code project, unconfirmed by maintainers, that suggests Anthropic instructs the model to avoid fs tools its own harnessharnessManages the agentic loop, tools, context, and permissions for coding agents. Often wraps an agent and/or model provider sdk. provides when running in bypass-permissions mode, and instead to "Do your work through the Bash tool wherever it can accomplish the job".

Next up: Similar to how the Linux kernel is split into subsystems, agent harnessesharnessManages the agentic loop, tools, context, and permissions for coding agents. Often wraps an agent and/or model provider sdk. are complex enough, at this point, that components benefit from separation and optimization. I'm looking forward to reading Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design next, which proposes an evolution agent that evolves specialized harnesses for different tasks. More on that soon.