A few days ago, I read The Agent Evals Handbook and shared some notes. Then I built mini-agent to make the agent loop concrete: ask the model, run its tools, feed the results back, and repeat.
The next step felt natural: build a small coding-agent eval too.
Once an agent can read files, change code, and run tests, how do we tell whether it actually did a good job? And when we change its instructions, how do we know whether it got better?
I wanted a system small enough to read from end to end. Here is the idea, following a single bug-fixing task.
1. Decide what “done” means
Give an agent a buggy program and ask it to fix it. It reads the code, changes a file, runs a test, and replies:
Fixed. Tests pass.
Did it succeed?
The file may have changed while the bug remains. One example may pass while other behavior breaks. The agent may not have run the checks it claims to have run.
Before the attempt starts, we need acceptance criteria.
In this project, primary success means that the final program passes independent checks of the requested behavior and regressions, and the agent completes normally. Passing those checks is evidence within their coverage, not proof that the program is correct for every possible input.
2. Let it work—and keep the evidence
The agent has three tools: read, write, and test. It can run public examples or supply a few small tests of its own.
That feedback helps it choose the next step. For evaluation, I also keep the execution trace, final code, test results, and termination state.
These records make failures explainable. An attempt might fail because the repair is wrong, the response is cut short, a request raises an exception, or the turn budget runs out. All affect delivery, but they call for different changes.
The artifact tells us what was delivered. The trace helps explain how we got there.
3. Check the result independently
Agent-authored tests are useful, but the agent can get their expected answers wrong too. A misunderstanding can appear in both the code and its tests.
So I separate feedback during execution from final acceptance. The agent uses tests to work on the problem; the evaluator checks the final program against cases prepared independently.
Consider a knapsack problem that allows several items to be selected. A wrong repair that picks only the most valuable single item can still pass some examples. Acceptance needs cases where combining items is necessary.
I also inspect the grader with the buggy starter, a reference repair, and a deliberately wrong repair. Can the checks distinguish them?
Before trusting a score, check whether your grading method catches the mistakes you care about.
4. Correct code and accurate claims are separate
One recorded attempt illustrates this well.
The agent ran public tests before editing. After editing, it ran six tests it had written itself. Its final reply claimed that both the public and additional tests passed.
The final program passed independent acceptance. But public tests on the old code did not establish that the new code passed them too.
A successful repair can still come with an unsupported verification claim.
I use an auxiliary LLM judge to examine those claims and retain its reasoning and errors. It sometimes returns no valid verdict, and it has not been calibrated against independent human labels. I treat its assessments as diagnostic evidence, not a dependable final authority.
5. Use the eval to test a change
With these pieces in place, I could ask a concrete question: would instructions to reproduce the bug, test after editing, check regressions, and report verification honestly improve repairs?
Version A had basic repair instructions. Version B added those steps. Both used the same model, tools, and budgets, with fresh starting files for every attempt. A/B runs were paired by task and repeat, with their order randomized and interleaved.
The final comparison used five small algorithm-repair tasks, three attempts per version on each task: thirty attempts in total.
A succeeded in 15/15 attempts. B succeeded in 13/15.
In this experiment, the additional instructions produced no observed repair benefit.
One unsuccessful B attempt hit the response limit before writing a repair. Another encountered an HTTP exception on its first request. These records do not establish that B was worse at solving algorithms, or that detailed instructions generally hurt.
They describe delivery under these tasks and conditions. Five tasks are still a small sample; repeating each one does not create new kinds of tasks. The problems came from a public algorithm benchmark, so their presence in model training is unknown.
6. Keep building
The experiment did not give me a “better prompt wins” result. It gave me more concrete next steps.
One idea is to bind verification to code versions. After an edit, earlier results remain in the record but no longer imply that the current code is verified. The agent could see: “authored tests passed on this version; public tests have not run yet.”
For response truncation, hold the prompt fixed and compare response budgets. For missing judge verdicts, investigate its budget and calibrate its judgments with human examples. These are hypotheses to test, not improvements already demonstrated.
Building mini-agent made the execution loop concrete. Building a small eval made success, evidence, and reliability concrete too.
For me, the useful question is: what should I change next, and what evidence would show that the change worked?
The code, English and Chinese guides, and experiment records are here. Take a look, try your own experiments, and tell me what you find:
https://github.com/caulif/mini--coding-agent-eval
Originally published on X.