Every code agent demos well. That's the problem.
You wire one up, give it a clean task on a clean repo, and it produces a working feature while you watch. Everyone nods. Then you point it at your actual codebase, and the hit rate quietly falls off a cliff. The demo wasn't dishonest, it was just measuring the easiest possible case and calling it representative.
If you're deciding whether to trust an agent with real work, the demo tells you almost nothing. What tells you something is an eval: a repeatable measurement on tasks that look like your tasks. Here's how I think about building one, from doing it for LaraCopilot's pipeline.
The one number that matters most
If I could track a single metric, it would be merge-without-edit rate: of the changes an agent produces, what fraction ship with zero human edits.
I like it because it's honest in a way most agent metrics aren't. It can't be gamed by a good demo. It captures correctness, style-fit, and "did it actually understand the task" in one number, because a human editing the output is the ground truth that something fell short. And it maps directly to the thing you care about: how much work the agent actually took off your plate versus how much it just relocated into review.
Watch it as a trend, not a trophy. A pipeline that goes from 30% to 55% merge-without-edit over a month is winning, even if 55% sounds unimpressive in a launch post.
Measure per lane, never in aggregate
A single site-wide number hides everything useful. An agent might merge 80% clean on CRUD endpoints and 15% on core billing logic, and the average, 47%, describes neither and misleads you about both.
So we measure by lane: category of task. CRUD, test backfills, refactors, bug fixes, new domain logic. Each lane gets its own merge-without-edit rate tracked over time. This is what lets autonomy be a dial instead of a switch. A lane that climbs past a threshold and holds there earns more automation; a lane that stays low keeps a human firmly in the loop. You're not deciding "do I trust agents," you're deciding "do I trust agents with this," which is the only version of the question that has a real answer.
Build the eval set out of your own history
The trap is evaluating on toy tasks. If your eval set is "build a todo app," you're measuring the demo again.
Your git history is a better source. Take real merged PRs, revert them to their starting state, and use the original ticket as the prompt. Now you have tasks with known-good outcomes drawn from your actual codebase, conventions, and messiness. Run the agent against them and compare. Twenty to thirty of these, spread across your lanes, is enough to catch regressions and tell whether a prompt or model change actually helped or just felt like it did.
Keep the set frozen between changes so you're comparing like with like, and refresh it occasionally so the agent can't overfit to a stale snapshot.
Grade on outcomes, not vibes
"The code looks good" is not a measurement. For each eval task, decide up front what passing means and check it mechanically where you can:
- Does it build and pass the existing tests? Non-negotiable, and free to check.
- Does it pass tests written for that specific task? The strongest signal, since it encodes the actual intent.
- Does it stay in scope? An agent that fixes the bug and also "helpfully" refactors three unrelated files failed, even if everything's green. Diff size against the reference PR is a decent proxy.
Where a check has to be subjective, an LLM-as-judge with a tight rubric is fine for triage, as long as you spot-check its grades against your own. Don't let the thing you're evaluating also be the sole judge of whether it passed.
The metrics that flatter you, and what to watch instead
A few numbers look like progress and aren't:
- Lines of code generated. Measures verbosity, not value. A good agent often deletes.
- Task completion rate, if "completed" just means "returned something." An agent that always produces a diff scores 100% and tells you nothing.
- First-pass speed. Fun in a demo, irrelevant if the output needs three rounds of fixes. Time-to-merged, including review, is the real cost.
The two I'd add alongside merge-without-edit: review rounds per accepted change (how many back-and-forths before it's mergeable) and escaped-defect rate (agent-written changes that shipped and later caused an incident). The first tells you the true cost; the second tells you whether your gates are actually holding.
Where I've landed
An eval isn't a certificate you earn once. It's a dashboard you keep, because the model changes under you, your codebase drifts, and last month's numbers stop describing this month's reality. The teams that adopt agents well aren't the ones that found a magic prompt. They're the ones who can answer, with a number, "is this actually working," and who re-ask it every week.
The demo will always look great. Build the thing that tells you the truth after the demo ends.
I'm building LaraCopilot, an AI development platform for Laravel, and writing about agentic systems and MCP as I go. If you're building evals for your own agents, email me. I usually reply within a day.