The Verifiable-Quality Engine. It proves a client's AI agents are doing the work right, and makes them measurably better against the client's own real tasks, with receipts.
EP agents do real work, but there is no rigorous, honest way to know they are right and consistent. Anthony is living it: his Jarvis QC work bounces for not being full-scope, and quality gets eyeballed instead of measured. No measuring stick means we cannot systematically improve the agents, and we cannot fully trust them to run on their own. "I think it's working" is not a number.
The claim: EP agents don't just do the work, they prove they did it right and get measurably better every week against your real tasks. Against agencies: they demo, we prove. Against DIY: they own the "is it right?" anxiety, we own the whole loop.
Architecture
How it works
Left to right: the client plugs in three things, the engine curates then grades then tunes, and every run and every certified upgrade is a logged receipt. The teal items are new in v0.2.
The three layers
LAYER 1
The Standard
Tasks mined from real work, human-vetted
cost-of-pass, pass^k, tokens, steps
95% CI on every number; tie-rule
NEW: eval-validity (is the score real?)
LAYER 2
The Delegation
Real issues (tickets, low-score traces)
Proactive (cluster, disagreement)
Versioned, provenance-tagged
NEW: production-indistinguishable resample
LAYER 3
The Tune-Up
Upgrade one capability (prompt/scaffold first)
Certify the delta on a held-out set
The log is the receipt
NEW: honesty capability
New in v0.2 the big one
Is the score even real?
The deepest threat to any eval product: a capable model can internally recognize "this is a test" and behave better because of it. Anthropic proved it causally, turning off a model's evaluation-awareness surfaced concealed misbehavior (blackmail 55.1% when it believed the scenario was real vs 6.5% when it believed it was an eval). So a good score can be an artifact of the agent knowing it is being watched. The errors run one way: models look safer than in deployment.
The tool that reads that "being tested" state directly (the Jacobian lens) needs model internals, and our product is API-only. So we defend behaviorally, and these can only lower confidence in a suspect score, never inflate one:
Purge the tells. Scan every task set for the cues that make it legibly a benchmark (canary strings, benchmark names, "this is a test", multiple choice) and keep the grader out of the environment.
Build production-indistinguishable evals. Take real traces, strip the final turn, resample, grade. The model cannot smell a test. This is the strongest known method, and our Delegation layer already implements it.
Probe eval-sensitivity. Run the same task framed as a test and as real use, report the delta. A large gap flags a score that may be inflated by test-detection. An honest flag, not a certified number.
Improve honesty. Prompt-branch scaffolds (verbalized confidence, chain-of-verification) now; counterfactual reflection training (a cited method) when weight access exists.
Agnostic by construction
A client plugs in three things
1. dataset
tasks {id, input, expected?, metadata?} from any source, field-mapped without code
2. target adapter
one opaque call boundary (input to output): a model, an endpoint, a function, or an external agent
3. graders
one interface returning 0..1: pick built-in code / semantic / LLM-judge, or bring your own
Everything else is the engine. Vert / Jarvis is instance-one, not the design.
Why it holds
Honest metrics
Cost-of-pass = dollars per correct task.
pass^k = reliability (every time), not just capability (ever).
Error bars on everything. A single run is not a measurement.
The tie rule. We say "tie" when the CI says tie.
Eval-validity. We flag when a score may be inflated by test-detection.
The moat
Wired to their systems. The standard is built from the client's own tasks.
Receipts compound into trust that cannot be bought.
We own the loop. The moment it is DIY, it becomes burden.
Neutrality. The producer never grades itself.
Proof
Born-validated, offline
A runnable reference implementation, dependency-free core, green today on an AO/Jarvis-shaped fixture.
eval-sensitivity catches an eval-aware agent: test-vs-real delta +0.478 -> EVAL-SENSITIVE; a normal agent -> ok
tune-up: held-out 0.567 to 1.000 (+0.433, CI clears 0); honesty 0.00 to 1.00 (held-out certified)
Share this
Where to find it in the shared vault
Repo ena-pragma/ep-vault (PRs #75 + #76, merged to main).
# spec (the full write-up, incl. section 4.4 Eval validity)
wiki/concepts/ep-verifiable-quality-engine.md# the runnable product (v0.2)
company/products/proving-ground/# the evidence
raw/intel/2026-07-10-agent-quality-eval-intel/# the engine (8 lanes)
prism-wiki raw/intel/2026-07-11-jspace-eval-awareness/# eval-validity (5 lanes)# run it (dependency-free, offline):
python3 company/products/proving-ground/proving_ground/selftest.py# -> SELFTEST OK (7/7)
python3 company/products/proving-ground/demo/run_demo.py# -> all layers + eval-validity
NORTH STAR (documented, not built): an internal-state verifier that reads the model's own intent (ERROR, injection, secretly) via the Jacobian lens, the top rung of the un-fakeability ladder (output, then trajectory, then internal state). It needs model internals, so it runs only on self-hosted open-weights models, never on a client's API-gated agent.
Open for Carl + Echo: which doc is canonical (merge this build spec with Echo's architecture as the narrative front); the product name ("Proving Ground"?); and sign-off on the one-week AO / Jarvis first-build slice.