The eval wasn't about the model.

I wanted a plain answer to a practical question: could a general coding agent contend with Archibot on real Archibus-style work? I built a small benchmark to find out, and the answer ended up being less about a single model than about what the eval was actually allowed to prove.

The first version was too easy. The agents edited files, ran checks, and passed the public tests, which felt useful for about a day. Then the score stopped meaning much. If every serious lane scores 8/8, the eval is mostly telling me the runner works and the examples are easy to satisfy; it is not telling me whether the agent understands the parts of the work that usually bite later.

Why a benchmark at all.

This started with Cursor Composer 2.5. I wanted to know whether it could replace or at least pressure the managed Archibot path for Archibus coding tasks, and the only honest way to answer that was to stop comparing chat output. I needed a harness that forced each agent to write code, run in a clean task copy, and survive a verifier that did not care how confident the final answer sounded.

The harness used fresh task copies, deterministic verifiers, and sanitized Archibus-style fixtures. None of the tasks were customer systems; they were small, public-safe versions of work I already care about: Archibus metadata compared to SQL table definitions, AXVW view bindings, WebCentral runtime profiles, SQL safety checks, and WebCentral startup remediation plans.

The bench also mapped 71 Archibus skills across 14 task families. That mattered because I didn't want to test a blank model guessing at a private domain. The product question is harder and more useful: if the agent has context, tools, and a plausible path to the answer, does it still keep the rule that the support person actually needed preserved?

A coding agent with tools is like a junior engineer with shell access. The shell access matters, but it does not mean they understand your production rules.

The visible tests lied.

The visible tests were useful smoke tests. They proved the agent wrote files in the right place and produced output in the right shape, but they did not prove much more than that. They were checks for basic cooperation, not evidence that the answer would hold up when the easy examples ran out.

So I added hidden verifier overlays. The agent finished first, then the harness copied in harder checks and ran them, which made the eval behave more like review. The agent could still use the normal tools for the visible task, but it could not tune every branch of the implementation to the hidden cases while it was writing.

That is where the interesting failures showed up, and they were not dramatic. They were boring in the way enterprise software is usually boring: quoted identifiers, inline primary keys, JavaScript handlers written as quoted object properties, XML namespaces, uppercase database names, and comments that mentioned a scoped column without actually scoping the SQL.

The 5/5 that was really a 0/5.

Cursor Composer 2.5 first looked fixable. Copied skills moved it a little; mandatory Cursor rules moved it a lot; and a deterministic preflight tool closed the last gap on the tuned hidden set. The lane went from 1/5 to 4/5 to 5/5, which is exactly the kind of progress that can make you believe the eval is teaching the right lesson.

That sounded like the product answer: give the agent rules and a local checker. It is still a useful product pattern, but it was not proof of generalization.

I froze the setup and made a new holdout with different edge cases. Composer with copied skills scored 0/5. Composer with frozen rules scored 0/5. Composer with frozen rules and the old preflight tool also scored 0/5. That old 5/5 was not fake in the sense that the harness lied; it was fake in the more dangerous sense that I had tuned the support around the last exam and almost mistook that for capability.

That was the most useful result in the whole run. It kept me from shipping myself a fake conclusion.

What the larger sweep showed.

I don't use the old 5-task table as the main read anymore. It was useful as a warning, but it was too small for the article. The better count set is the 18-task run, and the cleaner baseline is medium across the board.

On that view, Archibot medium was the strongest lane I reran cleanly. It passed 17 of 18 tasks with the hidden checks held back until verification. Cursor GPT-5.5 medium-fast passed 12 of 18 after I stripped the bench copy down and cleaned Cursor's project cache between tasks. Claude Sonnet medium passed 9 of 18 in Claude Code.

That table is not a clean model leaderboard. Cursor and Claude Code are agent harnesses, not bare models. They can search, edit, run tests, and use their normal local tools. For the stricter Cursor row, I removed copied benchmark context, prior reports, docs, hidden verifier directories, and eval-specific Cursor project cache before each task. Archibot has its own endpoint context, so the fair reading is narrower than "model X is smarter than model Y." With medium effort and useful tools, the managed Archibot path kept more of the Archibus-specific rules.

The higher-reasoning runs are still notable, but I wouldn't mix them into the medium baseline. Archibot high finished the full 18-task set at 18/18. Claude Opus xhigh finished 12/18, and Cursor GPT-5.3 Codex high-fast finished 13/18. Those runs are useful ceiling checks, but they answer a different question than the default product tradeoff.

Composer also needs a separate asterisk. A later Cursor Composer run showed 16/18 in the agent harness, which looks impressive at first, but the isolation was not as strict as the label sounded. It disabled copied add-on context; it did not remove normal Cursor tools, shell search, or ambient workspace state. After checking the sessions, I treat that 16/18 as a Cursor harness event, not proof that the base Composer model learned Archibus.

Where the agents actually broke.

The misses were all the boring parts of enterprise software. SQL table definitions had to normalize quoted names, inline primary keys, NVARCHAR, default values with commas, and type equivalence; AXVW view files had handlers in shapes the easy tests didn't cover; runtime selection had to handle uppercase database names and numeric Java/Tomcat inputs. None of that sounds impressive in a demo, but those are the details that make an answer usable.

The SQL safety task was the best example. A bind placeholder is not tenant scope by itself; the scoped column has to appear in the SQL. A comment saying the word doesn't count, and an alias like token_masked doesn't count either. The agent has to preserve the actual safety rule, not the surface text around it.

The remediation task had the same shape. A WebCentral startup failure can have several ugly clues in the log, and the answer has to sort them instead of flattening everything into a generic checklist. A database connection failure beats a later login failure. A license-cache file can be part of the fix. The wording matters because the output is supposed to guide an operator, not just satisfy a parser.

Boundary

This is not the cost post yet.

The money question matters, but the billing basis is not the same across lanes.

The managed Archibot path has token accounting. Claude can report provider cost in the right mode. Cursor has quota-backed product behavior plus modeled token cost. Mixing those together into one neat dollar table would make the chart look more precise than the evidence.

The right cost metric is cost per finished task, including failed runs and retries. That needs a separate instrumented run and a dated pricing snapshot. This post is about the eval shape.

What I'd tell anyone building an agent eval.

Don't stop at visible tests. They are necessary, but they are not enough. If the agent can see every case that matters, you are mostly measuring how well it fits examples, and that is a different skill from preserving a rule under review pressure.

Freeze the rules and freeze the tools, then run a fresh holdout with new edge cases. If the score collapses, you didn't build general capability; you built a better answer key for the last exam.

Also separate setup failures from quality failures. The Cursor Claude Opus lanes failed at provider capacity. That says something about product reliability, but it does not say much about Claude's coding ability. The direct Claude Code run was the cleaner quality check, and it still missed hidden Archibus edges.

The point is narrow, and that is why I trust it. In this harness, as refreshed on May 29, 2026, across this Archibus-style task family, hidden verifiers changed the conclusion. Without them, I would have believed a neat score that mostly proved the runner worked, and that is exactly the mistake I was trying to avoid.

An agent benchmark without hidden verifiers mostly measures how well the agent can satisfy the examples it can already see.