From a Single LLM Call to a Reliable Agent: Building an Advanced Harness
Building an Advanced Agentic Harness
This post upgrades a basic agentic harness into a production-grade system using composition: typed tools with Pydantic validation, a plan DAG with parallel execution, tiered memory under a retrieval budget, a verification hierarchy, and multi-dimensional budgeting. The author builds a city comparison agent to illustrate each primitive, addressing specific failure modes like invalid tool arguments, sequential bottlenecks, context bloat, and silent errors. The result is a fast, safe, debuggable, and measurable system that turns one LLM call into a reliable agent.
The guiding question for the whole exercise is a simple one: How do you turn a single LLM call into a reliable system that can plan, act, recover, and prove it did the right thing?
- hanneshdc
Any benchmarks showing if this actually improves problem solving? Or reduces errors?
The idea is cool, but from own experience in harness engineering, lots of cool sounding ideas can have a negative impact on performance due to emergent and confounding effects.
So I'm a bit skeptical!
- ilaksh
So if I understand correctly, one of the agents is creating the workflow as a DAG dynamically for each new job it's given? That's the most interesting part to me. The rest I was already pretty familiar with.
So maybe that's a big chunk of what you need for an 'AI Company': an agent that manages the goals and hierarchies. Although of course the DAG and agent hierarchy is not quite the same thing. But maybe the workflows and subworkflows are what matter.
- bryan0
I (like presumably many others) have built something similar. My main difference though is that the critics operate on each stage of development before it can move onto the next. The stages are defined by deliverable artifacts: issue, plan, pull request. Critics must approve each artifact before you can move onto the next. So the process is defined by a DAG which defines how to transition successfully from one artifact to the next. It's been fun to work on and I would like to open source it soon, but I assume many others are working on similar systems.
- DerrickDevo1
A good tutorial. Generally speaking, the harness is the environment layer between a language model and its task, such as the action set it can call, the state, the context it can see and the memory etc.
However, currently the bigger question comes to my experience during harness is actually not where we use LLM in the system, but where we do NOT use LLM in the system. And the validation of the results becomes more and more important. Any thoughts on this?
- budududuroiu
> The plan is a graph
I much prefer giving the LLM a REPL loop, and injecting all the tools as functions inside the REPL loop.
That means that the LLM isn't constrained to writing a DAG, it can write code that loops, exits early, etc.