EP Product · internal breakdown

Proving Ground

The Verifiable-Quality Engine. It proves a client's AI agents are doing the work right, and makes them measurably better against the client's own real tasks, with receipts.

For: Carl + Echo · Built + shipped: 2026-07-10 · ena-pragma/ep-vault PR #75 (merged)

The problem

We can't measure whether the agents are right

EP agents do real work, but there is no rigorous, honest way to know they are right and consistent. Anthony is living it now: his Jarvis QC work bounces for not being full-scope, and quality gets eyeballed instead of measured. No measuring stick means we cannot systematically improve the agents, and we cannot fully trust them to run on their own. "I think it's working" is not a number.

The claim: EP agents don't just do the work, they prove they did it right and get measurably better every week against your real tasks. Against agencies: they demo, we prove. Against DIY: they own the "is it right?" anxiety, we own the whole loop.

Architecture

How it works

The flow reads left to right: the client plugs in three things, the engine curates then grades then tunes, and every run and every certified upgrade is a logged receipt. Production traces feed back into task selection, so the standard sharpens itself over time.

The three layers

LAYER 1

The Standard

Grade the client's agent on their real tasks, honestly.

  • Tasks mined from real work, human-vetted (a "Verified" gate is not optional)
  • cost-of-pass, pass^k, tokens, steps
  • 95% CI on every number
  • Declare a winner only when the CI clears zero, else say tie
LAYER 2

The Delegation

Pick what to grade. Never random.

  • Real issues: tickets, support, low-scoring or thumbs-down traces, ranked by impact
  • Proactive: cluster to find unknown failure modes, active learning on human vs model disagreement
  • Versioned, every task carries its provenance
LAYER 3

The Tune-Up

Upgrade one capability, and prove it.

  • Revise the prompt/scaffold (carries most of the gain, needs no weight access)
  • Certify the delta on a held-out split the optimizer never saw
  • The log is the receipt that makes "it got better" verifiable

Agnostic by construction

A client plugs in three things

Everything else is the engine. That is what makes it plug-and-play for any client. Vert / Jarvis is instance-one, not the design.

1. dataset
tasks {id, input, expected?, metadata?} from any source, with a field-map so their column names bind without code
2. target adapter
one opaque call boundary (input to output): a model, an endpoint, a function, or an external agent
3. graders
one interface returning 0..1: pick built-in code / semantic / LLM-judge, or bring your own

Why it holds

Honest metrics

  • Cost-of-pass = dollars per correct task. Exposes the needlessly expensive agent that accuracy-only grading hides.
  • pass^k measures reliability (every time), not just capability (ever).
  • Error bars on everything. A single run is not a measurement.
  • The tie rule. We say "tie" when the confidence interval says tie. We refuse to fake a winner.

The moat

  • Wired to their systems. The standard is built from the client's own tasks. It is not portable.
  • Receipts compound into trust that cannot be bought.
  • We own the loop. The moment it is DIY, it becomes burden.
  • Neutrality. The producer never grades itself. Cursor structurally cannot offer this. It grades its own models on its own private bench.

Grounded in an 8-lane external intel run (every claim live-cited): Cursor evals, SWE-bench Verified, grading statistics, eval-set curation, prompt optimization (GEPA), cost metrics, agnostic harness design, capability evals. e.g. OpenAI filtered 68.3% of raw real tasks as broken, and a broken harness alone moved a score 16% to 33.2%; prompt optimization carries most of the gain, weights-alone loses by up to 60%.

Proof

Born-validated, offline

A runnable reference implementation, dependency-free core, green today on an AO/Jarvis-shaped fixture.

PASS metrics · runner + compare · delegation · tuneup = SELFTEST OK
tune-up demo: held-out baseline 0.567 to 1.000, delta +0.433 (95% CI clears 0), verdict improved
honest tie demo: equivalent prompts return tie; better prompt returns winner

Share this

Where to find it in the shared vault

Repo ena-pragma/ep-vault (PR #75, merged to main).

# spec (the full write-up)
wiki/concepts/ep-verifiable-quality-engine.md

# the runnable product
company/products/proving-ground/

# the evidence (8 lanes, every claim cited)
raw/intel/2026-07-10-agent-quality-eval-intel/

# run it (dependency-free, offline):
python3 company/products/proving-ground/proving_ground/selftest.py   # -> SELFTEST OK
python3 company/products/proving-ground/demo/run_demo.py            # -> all 3 layers

Open for Carl + Echo: which doc is canonical (recommend merging this build spec with Echo's ep-verifiable-quality-architecture.md as the narrative front); the product name ("Proving Ground"?); and sign-off on the one-week AO / Jarvis first-build slice.

Prepared by Prism · 2026-07-10 · EP