The grading loop

Amdahl gives you two primitives. This is the loop you build around them: write a prompt, ground it in real customer evidence, draft, grade against your own conversations, fix, rerun.

Amdahl gives you two things to build with, and you assemble everything else.

The division of labour

Amdahl

Search — ask your customer conversations a question, get rows back - Eval — grade a prompt or a message against what those customers actually said

You

the prompts - your own canon: ICP, product truth, brand voice - the agents and the orchestration - the scheduling and the approval flow

That split is deliberate. Amdahl is infrastructure, not a solution to one team's workflow, which means the assembly work lands on you. This guide is the shape that assembly takes: the loop itself, what to measure while you run it, and the handful of things that will quietly give you a wrong answer if you do not know about them.

Organizing your bench is the companion — where the prompts, evidence and scorecards live once you are running this dozens of times a week.

The two primitives

Search (search.query) takes a plain-language question about your customer conversations and returns rows. By default it blocks, and that synchronous path is sized to stay under a 15-second ceiling by construction. The natural-language lane also takes async: true, which hands back a job id you poll rather than blocking and runs under a ~180-second budget — reach for it when the question is broad enough that the fast path would cut it short. There is no web or news leg on Search — market signal and prose synthesis over results are both Chat's job.

Eval (evals.run) takes a prompt, a message, or both, and grades them against your own customer conversations. It returns a run id you poll. The default eval is prompt-and-message-eval.

Two layers come back from a grade, and you read both:

A rubric score. Every rubric line is a binary pass/fail verdict with a sentence of reasoning behind it. The headline number is derived from those: score_15 = 1 + 4 × (passed / total). The message rubric has five lines, so the dial takes exactly six values — 1, 1.8, 2.6, 3.4, 4.2, 5. The response ships checks_passed and checks_total next to score_15 so you can read "4 of 5 checks passed" instead of a 4.2 that looks like a rating. Read the fraction.
Cited customer quotes. Each carries a stance: supports, contradicts, or neutral. Quote text is hydrated server-side from an id, so the model cannot invent or paraphrase one. A contradicts is a belief you hold that a customer argued against, and it is the highest-value thing the loop produces.

Grounding is an input to drafting; the eval is a post-hoc check on whether the draft used it. The order in the loop is load-bearing, not stylistic — Search pulls the evidence before anything is written, and Eval then asks whether what you wrote reaches for it.

Grade copy that was drafted without an evidence pull and the low grounding score is correct and uninformative: it is telling you the draft never had the evidence, which you already knew. It is not a verdict on your writing, and reading it as one is the most common way an honest "your copy did not reach for the evidence" gets heard as "the eval is broken". If you skipped the ground step, go back and do it rather than arguing with the number.

How the grade is produced

A score is only worth reading if it could have come out the other way. Three properties of the grading step are what make that true.

Both candidates are judged blind, side by side. What you submitted and what Eval wrote arrive at the judge as candidate_1 and candidate_2 — one call, one rubric, one frozen evidence set. Which one is shown first is decided by a hash of your own submitted text: deterministic, so an identical rerun reproduces it, and uncorrelated with which candidate is ours. A judge that knows which draft it just wrote grades it kindly. This one cannot tell.

One bar, and both sides are held to it. The default eval sets it at 4.2, which on a five-line binary rubric means four of the five checks. The same threshold decides both candidates, so a run lands in one of four places:

Outcome	What it tells you
Both clear the bar	What you sent was already good enough. Ship it.
Yours misses, the improvement clears	The gap is the finding. Read what changed.
Both miss	The evidence did not support a stronger version either.
Yours clears, the improvement does not	Keep what you sent.

A miss is reported, not hidden. If the improvement misses the bar, the loop revises against the judge's own critique and re-grades — two rounds is the ceiling. If it still misses, the report says so, with the score it reached, the bar it was measured against, and what held it back.

The judge writes a critique for both candidates before anything is un-blinded. That is what keeps the revision round honest: it cannot go easy on the draft it is about to improve, because it does not know which one that is.

The two rubrics — five lines for the message, five for the prompt — are on the Prompt and Message Eval page.

The loop

One pass

Write
The prompt. "Draft the homepage hero for our ICP."
you
Ground
Pull the customer evidence that should inform it.
search
Draft
Produce the copy.
your model
Grade
Send the prompt and its output together.
eval
Fix
Apply the suggestions to the output and to the prompt.
you

then fix the output and the prompt, and rerun

You will run this dozens of times a week. The repetition is where the useful signal comes from — one graded artifact tells you about one artifact, twenty tell you whether your instincts about your own market survive contact with what your customers said.

Split it into two skills, not one

If you drive this from an agent harness, the obvious move is one skill for the whole loop. Resist it.

Skill	Does	Hands back
`ground-and-draft`	Pulls evidence, freezes it, drafts, creates the run folder	Drafts for you to read
`grade-and-report`	Submits to Eval, polls the run, renders the report card	The scorecard

Both ship with Amdahl. They live in the public Amdahl cookbook, under skills/ — drop them into your agent harness (.claude/skills/ for Claude Code), set AMDAHL_API_KEY, and they work as-is. Each carries the scripts described below: ground.sh, and grade.sh / ab.sh / repeat.sh. The same cookbook carries the copy-paste recipes and the Claude Code plugin.

The gap between the two is where you look at a draft before it gets graded, and where you decide whether you agree with the verdict. Automate that gap away and the exercise stops producing anything you could not get from a load test.

grade-and-report polls: Eval is a queued run, so the skill submits, waits, and then fetches GET /eval-runs/{id}/report — the server-authored card. Paste that block rather than summarising it. The eval grades two artifacts, your draft and a rewrite it produced, and the card is the one place both are stated with their labels intact; a summary written in your own words is precisely where the two merge into a confident sentence about your copy that the numbers do not support. (Both run-level readings describe your draft: overall_score from eval_version 2.14.0 onward, and the run's pass / fail bucket from 2.16.0, which buckets on input_passed ?? passed. The per-case passed still follows the rewrite, with input_passed, improved_passed and transition sitting beside it — two sides on one payload, which is exactly the pair a hand-written summary collapses. A verdict stored before 2.16.0 answers the other question, so do not pool it with a newer one.) It also records reused from the submit response — the next section explains why that one field is what catches a silent no-op.

Once you are running batches, add a third skill that chains both. Keep it away from anything you are actually learning from.

What to measure

Two readings per run. Record both, every time.

The score. Record passed and total, not just the rolled-up score_15. The mean is a rendering of the fraction, and the fraction is the measurement. With five checks the dial has six positions, so a real improvement is invisible until it flips a whole check. If you iterate against the number and it does not move, you have learned that you have not yet flipped a check, and nothing else.

The claims. How many drew a supports quote, how many drew a contradicts, how many drew nothing at all. A score that rises while the supported-claim count stays flat is worth a second look.

Live evidence versus frozen evidence

Use live search when you are producing work. Freeze the evidence when you are comparing two versions of a prompt.

A second search returns different quotes, so any change you see could be your edit or could be retrieval. For any prompt-improvement pass: pull the evidence once, write it to a file, and feed that same file to every attempt.

Inside an eval run this is built in. Pass evidence_from_run: "<run id>" to evals.run and the new run reuses that run's frozen quotes instead of retrieving its own — no file to manage. Its sibling candidate_from_run re-grades a prior run's improved version without writing a new one, so you can hold the evidence still or hold the writing still and see which of the two your change actually moved. Pass reuse: "force" when you sample either: two identical pinned requests share one content address, so on the default the second is served the first and an N-sample spread reads as a perfect zero.

And when you have already run the pair, GET /eval-runs/{id}/compare/{other_id} tells you how much evidence they actually shared, and withholds the score delta when they did not share enough for it to mean anything. See Comparing two runs.

evidence_from_run is message-derived: it freezes the pool a particular draft retrieved. Pinning draft v2 to draft v1's run therefore grades v2 on the slice v1 happened to select. That is fine for iterating one draft and wrong for comparing two positionings, which retrieve differently by construction.

scope narrows the problem rather than erasing it. It lets you hold the cohort constant with typed filters instead of with whichever draft ran first, so a comparison finally has a control set that is neither draft's — but retrieval inside the slice is still seeded by the message. Run the A/B under one scope and say plainly which tier of the evidence was held constant and which was not.

Two caches will fool you, and they are different caches

A call that succeeds is not a call that did work

Both caches below return a valid, complete, well-formed response. Neither returns an error. The only thing that distinguishes a measurement from a replay is a flag you have to go and read.

Search results. Query results are cached for a short window, keyed on the compiled SQL rather than on your English. Two differently-worded questions that compile to the same SQL will hit the cache; the same question that compiles differently on a second pass will miss it. Check the cached flag on detail.internal before you conclude anything about consistency.

Eval runs. This one is bigger. evals.run defaults to reuse: 'cached', and the run fingerprint content-addresses the business, the eval slug and version, the inputs, and every one of the server-resolved controls that sit beside them — the evidence pin, the candidate pin, and scope. That last group matters more than it looks: a scoped request and an unscoped one carry identical inputs and differ only in which conversations the judge sees, so without a dimension of its own the default reuse: 'cached' would hand you a whole-corpus verdict for a cohort question. Filter order is canonicalized, so two orderings of the same scope share one address rather than each paying for a run. Submit the same prompt and message twice and you get the same run id back — an in-flight run holds an at-most-once claim on the fingerprint and the second caller joins it, and a run that already completed inside the reuse window (15 minutes by default) is returned as-is. Five back-to-back submissions of identical input are one run, so the score is arithmetically incapable of moving.

Pass reuse: "force" whenever you want a fresh grade. It is one word, and it is the difference between a measurement and a replay. The response tells you which happened: reused: true means no new grading ran.

Grade the voice separately

Eval grades claims against customer conversations. Brand voice is not a claim about a customer, so the graded rubric has nothing to say about it.

The mechanism for voice is the rule grader: deterministic, no model call, scores the fraction of checks that passed, and takes length bounds, must_contain, must_not_contain for banned claims, and a call-to-action check.

You already get one. prompt-and-message-eval runs a basic-hygiene rule grader over every message — length between 30 and 3000 characters, a call-to-action check, and a banned-phrase list (revolutionary, game-changer / game changer, best-in-class, world-class, cutting-edge, industry-leading, synergy). It abstains on a prompt-only run rather than failing.

The upper bound is an outlier guard, not a style target. It is there to catch "this is a document, not an email". We measured length against real reply outcomes on 127,872 labelled outbound emails and the effect reverses sign between workspaces: in some, longer openers reply materially better; in others, slightly worse. So a pass on this check is not a claim that your message is well-sized, and there is no length we can tell you to write to. If you want a house style bar, that is exactly what your own length check is for — set it where your own data says, not where ours does.

Authoring your own eval is in limited beta: evals.create / update / delete refuse today. evals.validate still works, so you can author a definition and dry-run it without storing it. Until authoring opens, compile your brand-voice rules into checks anyway and run them as a local deterministic check alongside the Amdahl grade.

Sequence

Run the loop by hand, three times

No repo, no layout, no skills. Write a prompt, pull evidence with Search, draft, submit to Eval, read the quote verdicts. Keep the files wherever.

Pass reuse: "force" from the very first run so you never build an intuition on replayed output.

Doing this first means the layout you build next has a shape you observed rather than one you guessed at.

Create the repo and the canon

Build the bench layout, moving your three hand runs in as the first run folders. Write the canon files — product truth, ICP, brand voice — each with an owner and a review date at the top. Compile the voice rules into checks. Seed your coverage table.

Still no skills. Automating before you have felt the manual version encodes a guess.

Write the two skills, then run a cold-versus-assisted comparison

Write the skills based on what the first week was actually like, and get an API key for repeat runs.

Then pick five real artifacts. Write each one cold first, from your own read of the market, and save it. Run the loop on the same brief. Grade both against the same frozen evidence and compare on both readings: which passed more checks, which drew more supports, and whether either drew a contradicts.

The distance between your instinct and the evidence is the thing worth measuring.

Add volume

Point the bench at outbound, call prep, one-pagers and deal notes, where the volume is. Wire grading in ahead of anything that publishes. Outbound is also where account and audience start earning their keep, so start passing them and start reading the abstain state.

Tests worth running early

Run the first one before anything else — everything below it assumes an answer.

Before any of these

Pass reuse: "force" and check that reused came back false. With the default, identical input returns the identical run and every stability test in this table reports a perfect result it never measured.

Test	Method	Reads on
Does the score discriminate	Grade three artifacts you would rank confidently yourself, best to worst, against frozen evidence. Read `passed`/`total`, not the mean.	Whether a score delta means anything at all. Separating in your order is a usable dial; flat across all three is a scale that cannot discriminate.
Quote stability	Same input three times, frozen evidence, `reuse: "force"`	Whether the same claims draw the same stances
Pretty lies	A well-written false claim against an awkward true one	How the rubric weights tone against grounding, and whether the false one draws a contradicting quote
Grade the rewrite	Feed Eval's own suggested rewrite back in	Whether it passes more checks and draws more supporting quotes, or whether the judge just likes its own prose
Latency	Time ten runs end to end, including the poll	Whether grading can sit inline in a human workflow
Suggestion quality	Follow one, rerun, note the check that flipped and your own read of the copy	Whether following the guidance improves the work or just the number

Two bounds we have already measured

Treat these as bounds rather than as your numbers — they come from our own judge harness rather than from prompt-and-message-eval directly.

The run-to-run noise floor is around 7 preference points at n=30. A 0.55 preference rate is undetectable in a 30-pair run. Size any A/B against that.
Turning the temperature down does not stabilize a judge. The variance lives in rubric lines that admit two coherent readings of the same text, and it moves when you make a line decidable — not when you change a sampling parameter. On the current models temperature is rejected outright anyway, so the knob is not available to you.

So run a short replay to confirm your own harness behaves, and spend the time on discrimination instead. Whether the score separates quality you can rank yourself is the property every downstream use of the number depends on.

One run is not a measurement, and this is not a small effect. A single draft graded 15 times on byte-identical input spanned a submitted-side grade of 0.0 to 1.0 and produced all four transitions; across 18 repeat groups the submitted draft's pass/fail flipped 22% of the time. The mechanism is quantization — five binary rubric lines put score_15 on six possible values with the bar at 4.2, so one line flipping flips the verdict. grade-and-report's repeat.sh takes a median of N and drops degenerate runs; take that median before you act on a change, and before you tell anyone a draft got better.

And A/B on the submitted side. compare.score_delta differences overall_score, whose meaning changed at eval_version 2.14.0: before it, that field blended your draft with the rewrite, and on a live pinned pair whose true draft delta was −0.8 it reported −0.167. A pair that SPANS that version differences two different quantities — which the endpoint now checks, as of 2026-08-10: it reads both rows' eval_version, withholds the delta the crossing invalidates under delta_withheld_reason: "eval_version_boundary", and names the transitions crossed on version_boundary. What survives is submitted_score_delta, which is derived from the improvement report rather than from overall_score and so means one thing on both sides of the line: a payload-meaning crossing withholds score_delta alone and still ships it, while a grading-work crossing takes both, because there the two runs came off two different instruments. Read submitted_score_delta — and where you want the operands, report.headline.submitted.score_15 on each run is the same reading on the axis a human quotes, which is what ab.sh uses. The compare endpoint's evidence_overlap is correct and still worth reading: it is what tells you the pin held. See Comparing across an eval version.

Decide before you start

These are calls, not lookups. Leave them open and every verdict gets measured against a different bar, which makes them incomparable.

What counts as evidence. One customer once? Three across accounts? Weighted by deal size? Eval has already made this call — at least 3 distinct external speakers, 25 utterances, and 2 distinct companies. The same three numbers gate both scoping paths, audience and scope, off one shared set of constants, so the two cannot drift apart on what "enough" means. Adopting those floors as your own convention costs nothing and makes your rows comparable to the report's.

What counts as a claim. Marketing copy is mostly sentences that are not assertions about customers. Which ones are in scope for grading? This is load-bearing once the claim is your unit of measurement.

Whether you pass audience, account and scope, and on which artifacts. These change what the report is allowed to say about who the copy is for. A homepage hero probably has no single audience and should run unscoped; an outbound sequence has all three available, and running it unscoped throws away the scoping the grader offers.

scope is the one that makes this a real decision rather than a formality. audience resolves free text into one closed shape: a role_level seniority cohort, plus an optional buying-role refinement (economic_buyer or champion) when you named one. Two dimensions, both inferred from prose, and no others. scope is the general form: typed filters over the same field vocabulary Search advertises, so a segment, a stage, a time window or a tier of account is a cut you state rather than one you describe in prose and hope the grader honoured. Decide the convention now, and record the abstain state whenever you pass one — a run that abstained to whole-corpus grading and a run that graded the slice you asked for are not comparable, and the score looks identical in both cases. Recording only what you sent leaves you unable to tell them apart later.

Do not scope everything tightly on day one

A narrow slice that keeps abstaining on thin_evidence is telling you something real about your own corpus. Reaching for allow_thin_evidence: true to make the abstain go away converts that finding into a confident number computed over four utterances — the run will label it below_floors: true, and nothing downstream of you will read that label unless you put it in the scorecard.

What success looks like

"The website got better" is the work. Here is the read:

Runs completed through the full loop
Claims graded, split by supported / contradicted / ungrounded
Checks passed out of total, and whether that tracked your own read of the copy
Disagreements logged, split by cause
Whether the suggestions improved the copy by your own judgment
At least one belief you held that the evidence killed

Going further

Everything above is v1: a manual loop and one readable folder per run. An open-source prompt-eval harness (Promptfoo and friends) is the v2 — a grid of prompts against inputs with results side by side, with an Amdahl grade plugged in as a custom scorer.

Do not start there. It earns its place when you are running one prompt across many accounts and hand-comparing has got old, or when you have changed a prompt and need to know which past cases got worse. None of v1 is wasted if you go that route: prompts, canon and evidence already live as separate addressable files, which is most of why the layout looks the way it does.

If you do build the adapter, put reuse: "force" in it. A grid that replays cached runs produces a very clean and very fake table.