The last article ended with a claim: we spent the year giving the loop permission to tell the truth. Two weeks later, the loop used that permission on me.
On Friday, the newest skill in the Agentic QE Fleet — an adversarial review court where independent AI reviewers from different vendors prosecute every change — heard its first case. The case was its own release. The verdict was BLOCK. Three data-loss and crash bugs were sitting behind a fully green unit suite, and the court found all three before a single user could.
This is the article about why that verdict is the best thing that happened to me in two weeks, and about the machinery that made it possible.
Munich, in Two Paragraphs
Last week I was in Munich, at Accenture’s internal Quality Finale event, invited by their Quality Matters team. The invitation had a nice arc to it: first I was a guest on one of their podcasts, and the conversation went well enough that they asked me to come tell the golden-age-of-QA story in person. So I did, with the updated version: generation is solved, verification and validation are the new craft, and the people who have spent careers designing oracles and reasoning about risk are holding exactly the skills this moment is pricing up.
I spent some time with my friend Lalit, the same Lalit whose QCSD swarms now live in the fleet, and I listened to several of their presentations. The honest take: they are working seriously on AI-assisted tools and developing internal solutions, and they are still somewhere I was about a year ago, some teams maybe six to nine months. That is not a criticism; it is the same measurement I took in Madrid and in Budapest, and it keeps coming back roughly the same. It is also the whole reason I keep showing up and sharing: not to be ahead, but so that fewer people have to spend a year discovering what is already known and published.
What a Harness Actually Is, and Why QA/QE People Should Care
I promised myself this article would slow down and explain the thing I keep mentioning, because the word is suddenly everywhere. Ruv’s MetaHarness passed 100,000 npm downloads in its first two weeks. When 100,000 developers download a harness generator, the concept has left the laboratory. So here is the QE version, in plain terms.
When people say “an AI agent,” they picture the model. But the model is only the engine. The harness is the car built around it: the planner that breaks a task into steps, the context builder that decides which files the model actually sees, the reviewer that checks output before it is trusted, the retry policy, the tool policy, the memory, and the scoring policy that defines “did this go well?” Two systems using the exact same model get wildly different results purely because one has a better harness. If you have ever watched a good tester and a poor tester use the same tool, you already understand this.
A metaharness is one level up: a system whose product is the harness itself. It routes each request to the cheapest model that is good enough, evolves the harness one mutation at a time and keeps only changes that measurably help, orchestrates the agents and tools on top, and verifies every change with a frozen scorer and signed provenance. Ruv’s phrase for the strategy is the one I would put on a poster: freeze the model, evolve the harness. His flywheel is a QE loop wearing a different jacket: run, measure, learn, mutate, verify, promote. His plain-language explainer is worth your time, and his gists are where the honest numbers live, methodology caveats included.
Now the part I care about. Every pillar of a metaharness is a quality discipline that our community has practiced for decades, renamed:
Routing is risk-based resource allocation. Deciding which model handles which task, based on measured evidence of what each tier can do, is the same judgment call as deciding where to spend limited testing effort. Measured on real workloads, cost-optimal routing runs about 28.5% cheaper than always using the top model while ~98% of answers still clear the quality bar.
Evolution needs an oracle. A harness that mutates itself is only safe if something decides which mutations survive, and that something is a test. In the fleet, the oracle is mutation kill, real coverage, and suite cost. In Ruv’s Darwin work, it is the repo’s own tests. No oracle, no evolution, only drift.
Verification is provenance. Signed agent configurations, drift detection against the shipped source, receipts for every promotion. This is the audit trail regulated industries have always demanded, now applied to agent behavior.
The practical takeaway for a quality team: you do not need to build any of this from scratch. The fleet gives you the QE-specific harness, MetaHarness generates repo-specific harnesses for any agent host, and the Cognitum meta-proxy gateway gives you the billing layer that makes it affordable: authoritative per-request cost receipts, server-side spend caps that a key physically cannot exceed, and scope-enforced keys, so a worker key can be locked out of the frontier tier as a server-side guarantee, not a policy hope. More of the factory is coming to public soon, including meta-LLM. Concrete use cases we shipped this cycle in the Agentic QE Fleet: run QE analysis on the Claude subscription you already pay for instead of a metered API key, run it on your ChatGPT subscription through the new Codex provider for cross-vendor diversity, cap total spend per fleet with an enforced budget, and prosecute a release in the QE-Court before it ships.
Quality Nets Around a Moving Platform
Most of my time over the past two weeks did not go into the fleet. It went into the Cognitum platform, which is scheduled for release next week. Ruv keeps pushing the boundaries of what is possible, and my job, the fun part of it, is to keep up by wrapping multiple layers of quality and security nets around what he builds: CI/CD pipelines, release procedures, and an agentic approach to quality engineering across the full SDLC, using Ruflo, the AQE fleet, MetaHarness, and our internal Cognitum tools together.
The most interesting piece was setting up 24/7 agents that monitor the repos: monitoring for new CVEs, triaging open issues, proposing solutions, and escalating to a human when confidence is lacking. Setting up agents to guard other agents’ work with our own agentic tools is exactly as recursive as it sounds. I am still testing it to validate the claims and whether it will provide proper value to our pipeline without adding noise.
And the platform work handed me two more entries for the collection I have been building all year: completion theater at the infrastructure level. A CI job that looked green for a long time while the quality gate was not actually running: the lint step failed early, so typecheck and tests never executed, and the green checkmark measured nothing. And the recurring deploy-parity trap: a GitHub issue marked closed is not proof the fix is live. We caught a merged hard-cap fix whose function was still the old one in production.
The lesson generalizes to everything in this article: a claim of done, whether from an agent, a pipeline, or an issue tracker, is a claim, not evidence.
The Court That Blocked My Release
The fleet had a quieter cycle by count, v3.11.5 through v3.13.0, but the arc of those releases is the whole thesis of this blog compressed into two weeks.
It started with a system-integrity audit of my own self-learning loop, and the findings were humbling. Dream-cycle insights were piling up without ever becoming reusable patterns because the fix that should have promoted them only ran in manual tests, never in scheduled cycles. Pattern usage was double-counted. And the vector store had a failure mode where an interrupted export left it permanently unusable, silently: this repository’s own pattern store had been degraded for nine days without a single visible symptom. Worst of all, aqe eval run had been defaulting to a mock executor, scoring canned responses instead of real model output. Fabricated eval results, in a quality tool, from the person who writes articles about completion theater. I fixed it, published the fix, and I am writing it here because the Integrity Rule does not have an exception for my own embarrassment.
The response to all of that was a learning-integrity layer: a frozen, hash-pinned oracle benchmark so quality regressions cannot hide behind a moving target; provenance tiers on every promotion, where real test execution outranks an LLM judge, which outranks a structural heuristic; Ed25519-signed, replayable receipts for every change the learning system makes to itself; and a significance gate, so the flywheel no longer promotes improvements that are inside the noise. A designed experiment with real statistics emerged from the same work, and its first screen produced my favorite counterintuitive finding of the month: heavy scaffolding measurably lowers reliability, and retrieval adds no quality to tasks the model already knows. More harness is not automatically better harness. You have to measure.
And then, on Friday, I added a new skill to the fleet — QE-Court. Every change gets prosecuted by independent AI reviewers from different vendors, GPT through the Codex provider, Cognitum, and Claude, each with their own probe sets. A SHIP verdict must survive an escalating overturn round before it stands. Verdicts are three-valued: SHIP, REMAND, or BLOCK, recorded in a signed court record, and the human stays the final judge.
Its first case was the v3.13.0 release itself. The court said BLOCK. Behind a fully green unit suite, it found three real bugs: a subprocess exit that could crash the caller, an auto-restore path that could tear the database under concurrency, and a failed ledger migration that could silently stop recording spend and bypass the budget cap. All three fixed before shipping.
That is completion theater caught in the act, and this time not because I happened to be watching, but because the system is designed so that somebody is always prosecuting. My agents will keep telling me they are done. After v3.13.0, they have to prove it in court.
The Frontier Never Won a Task
On Saturday, Ruv asked me to run benchmarks against api.cognitum.one and analyze them from a QE perspective: the cost-Pareto question, what is the cheapest way to achieve each quality-oriented goal. This is the same question as in the darwin-qe work last month, but this time against our own gateway, using authoritative cost receipts instead of estimates.
Seven QE tasks, spanning the oracle ladder from execution-based (test generation scored by mutation kill) through labeled (security triage, PR severity, flaky diagnosis, adversarial review) to structural (coverage gaps). Three policies per task: always-cheap, always-frontier, and cheap-with-escalation. The tiers resolved to GLM 5.2 as the cheap tier and Claude Fable 5 as the frontier. Total spend for the entire session: 42 cents.
The result was not close. The cheap tier matched or beat the frontier tier on every single task, at 3 to 4 times less cost. The always-frontier policy never won a task. During test generation, the frontier model could not even produce runnable output; it kept explaining rather than emitting code.
I want to be precise here, the way I would want anyone benchmarking my tools to be precise: that zero is a real, reproduced instruction-following gap on this specific code-only task, not a blanket capability claim, and the samples are small, five or six items per task, so the shape is indicative rather than statistically tight. The methodology mattered more than the headline: the test-generation oracle went through four measured iterations, because the first version had an over-specified assertion that gave the frontier tier a false zero, and a false zero is exactly the kind of lie a benchmark must not tell.
Two design details I would encourage any team to steal. First, the token model: the worker key is scope-limited to the cheap tiers on the server side, so it physically cannot spend frontier money, and the judge key that can is isolated on its own account with its own hard cap. Guardrails as infrastructure, not as instructions. Second, every row in the run record carries a request ID that reconciles one-to-one against the billing ledger, and the client and server totals agreed within about 1 or 2 percent. Cost is ground truth here, never an estimate.
The conclusion feeds directly into the QE-Court routing table, replacing a hand-authored guess with measured data: reserve frontier models as occasional advisors; do not default to them. Measurement beats vibes, and this time the measurement cost less than a coffee.
The Craft Already Has a Name
On Wednesday, we held the first event of the Ministry of Testing AI Chapter. Together with Oleksandr Romanov, hosted by Ujjwal Kumar Singh, we discussed Google’s whitepaper on the new SDLC with vibe coding, the one that ends with the line: “Generation is solved. Verification, judgment, and direction are the new craft.”
That craft is not new, and it already has a name: quality engineering. The questions the audience brought were the ones I hear everywhere now. Will we work less or more? My short answer: the work moves toward verification and validation, which is where we already live. Should testers still learn to code? The question is no longer whether you can write code. It is whether you can judge code you did not write. Reading, specifying, and judging are the skills the whitepaper is pointing at, and they land squarely on the tester’s home turf. If you missed it, the recording is available through the chapter, and if you have not joined the AI Chapter yet, you are welcome.
What Is Ahead
Next week, the Cognitum platform release, the thing most of these quality nets were built for. The court, the receipts, the sentinel agents, all of it goes to work on a real launch.
And on July 30, something with a texture I love: Adam is hosting a one-off Agentics Foundation meetup in Vienna, with five practitioners from London, Seattle, Novi Sad, and Kansas delivering lightning talks in a city where none of us live. Bence, Klara, Adam, Christopher, and me. My ten minutes have a title that will surprise no reader of this blog: “Completion Theater: When Your Agent Says ‘Done’, and Lies.” After this month, I have fresh material: a court transcript in which the defendant was my own release.
Two weeks ago I wrote that we gave the loop permission to tell the truth. This is what it looks like when it does. The truth was a blocked release, a nine-day silent degradation, a benchmark where the expensive model lost every round, and a roomful of testers discovering that the industry has finally arrived at their doorstep.
Build the nets. Give them jurisdiction over your own work first. And when the verdict goes against you, say thank you, fix the bugs, and ship.
Stay curious. Keep learning. Keep sharing. Knowledge is power.
This is the thirty-third article in The Quality Forge series. Previous: “The Loop Came Home” described the darwin-qe benchmarks, our first panel meetup, and the week Adam and Klara came to Novi Sad. This one describes the two weeks after: quality nets around the Cognitum platform ahead of its release, Munich’s Quality Finale, the QE-Court blocking its own first release in v3.13.0, and the cost-Pareto benchmark where the frontier model never won a task. Cognitum is at cognitum.one.
Dragan Spiridonov is the Head of Agentic Quality Engineering @ Cognitum One, Founder of Quantum Quality Engineering, Secretary of the Agentics Foundation BoD, Chair of the Agentics Foundation Training Committee, and one of the AI Chapter leads for the Ministry of Testing. He is currently building the Serbian Agentic Foundation Chapter in partnership with StartIt centers across Serbia.