Evals

Grade a prompt or a message against your own customer conversations before you send it. /evals hands back a graded report card over content, scored against your workspace's own customer data. An eval is a configured grading pipeline: the inputs it accepts, the cases it grades, and the graders that score each one. The flagship built-in, prompt-and-message-eval, takes a prompt AND/OR a drafted message (send one or both) and hands back a before→after improvement report built around relevant positioning for the specific customer: it grades what you sent, then produces AND grades an improved, reusable prompt (a template that uses Amdahl to position, verify, and keep the messaging relevant to that customer) plus a relevantly positioned message — with per-dimension reasoning, verbatim customer quotes that back (or contradict) each side, a score lift, and a one-line "what changed." So "is this message any good, and how do I make it better?" becomes evidence you can act on. You can also author your own eval.

Two shapes, one call: at review time a person hands over a draft and gets a report card back, and in a loop an agent grades what it just generated and reads gate.passed before it acts (see Use it as a gate). A run is read-pure, so it is always safe to put in that path.

Two ways to grade

Every case declares a subject — the thing it grades — in one of two modes:

• provided — you PASS IN the content to grade. This is prompt-and-message-eval: you send a prompt, an message, or both (at least one), and it grades what you sent against real customer evidence, then improves it toward relevant positioning for the specific customer. The flagship, and the default.
• generated — the eval PRODUCES the content itself by running a question through the fast-search lane, then grades the answer. This is the regression-harness mode: grade whether your answer pipeline still answers well. Needs no inputs. No built-in ships in this mode today - author one if you want it.

One surface, one run contract, two subject modes.

What a run does

• It grades — it never changes your data. A run is READ-PURE by construction: it only ever dispatches read ops (retrieving customer quotes, a system-of-record data.query, running a generated case's question). Firing an eval can never mutate tenant state, so it is always safe to repeat.
• v1 grades CURRENT data, internal-only. Evidence and answers come from your own warehouse and cluster themes with no paid web fan-out (the run's context is clamped to reads, which drops external search). There is no as-of knob in v1 — as-of ("grade against a past snapshot") is a v1.1 capability.
• It is async — you get a handle, then poll. evals.run validates the inputs, starts the grading job, and returns a run id immediately. You read the verdict from eval_run://<id> once the run completes. The call never blocks on the grading.
• It verifies independently — it is not the generator grading itself. Retrieved quotes are FROZEN for the whole run, so every candidate within that run is judged against the same evidence, and the improvement report's judge scores both candidates in ONE BLINDED call (unlabelled, ordered by a content hash) so it cannot tell which one it wrote. That is the difference between an eval and asking the drafting model whether it did a good job. Across two SEPARATE runs the evidence is not held fixed unless you pin it — see Comparing two runs.

Quick start — grade + improve a message

Send a prompt, a drafted message, or both — at least one. The default eval is prompt-and-message-eval.

curl -X POST https://app.amdahl.ai/api/platform/v1/evals/run \
  -H "X-API-Key: $AMDAHL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "eval": "prompt-and-message-eval",
    "inputs": {
      "prompt": "Write a short cold nudge to a RevOps leader at a mid-market SaaS company.",
      "message": "Hi Sam - noticed your team just doubled headcount. Most RevOps leaders we talk to hit a wall on pipeline hygiene right around that inflection. Worth a quick 15 minutes next week?"
    }
  }'

Send one or the other:

• Message only → it grades your message, then suggests an improved rewrite AND an example reusable prompt (a template that tells the writer to use Amdahl to position the offer for the specific customer — research their situation, position, verify each claim, keep it relevant to them), explains why the prompt helps, and grades that too.
• Prompt only → it simulates the draft your prompt would produce and grades THAT as the "before," then produces an improved prompt + message and grades the "after."
• Both → the message you sent is the "before"; the improved prompt + message is the "after."

The improved message is always a suggestion — you are never required to send it verbatim.

Parameters

Inputs are validated against the eval's declared input_schema before anything runs: a missing required field, a wrong type, an out-of-enum value, or a violated "at least one of" constraint (send prompt-and-message-eval neither a prompt nor a message and it refuses) comes back as one invalid_argument error carrying the per-field problems (details.input_errors). An unknown eval slug is the other invalid_argument shape; past that the run never returns success: false.

The regression-harness mode needs an eval you author

A generated eval takes no inputs — it produces the answer it grades. No built-in ships in that mode today, so there is nothing to point eval at until you author one; eval://list is where you find the slugs your workspace actually has.

prompt-and-message-eval is not one of them. It is a provided eval, and its schema requires at least one of prompt / message, so a run with no inputs is refused up front with invalid_argument and a details.input_errors entry rather than grading anything.

The run handle

{
  "data": {
    "success": true,
    "run_id": "b1c2d3e4-...",
    "reused": false,
    "status": "queued",
    "eval_slug": "prompt-and-message-eval",
    "eval_version": "2.33.0",
    "resource": "eval-run://b1c2d3e4-...",
    "poll": {
      "action": "status",
      "run_id": "b1c2d3e4-...",
      "resource": "eval-run://b1c2d3e4-...",
      "report_resource": "eval-run://b1c2d3e4-.../report",
      "wait_ms_max": 30000,
      "hint": "Read the run back with the status action ..."
    },
    "console_url": "https://console.amdahl.ai/..."
  }
}

• run_id / resource are the handle — poll resource (or GET /eval-runs/<run_id>) for the verdict. The scheme on the wire is hyphenated (eval-run://), because an underscore is illegal in a URI scheme and new URL("eval_run://…") throws in a standards-compliant MCP client. Both spellings resolve on every read surface, so an existing integration is unaffected — but read the resource the response hands you rather than re-spelling it.
• poll.wait_ms_max is the long-poll ceiling (30 s). Pass wait_ms up to it and the read blocks server-side until the run settles, so a full grade is a handful of blocking reads rather than a tight loop. Prefer this over hand-rolling a poller.
• eval_version is part of the run fingerprint, so it is what a cache or a comparison keys off. It moves whenever the eval's grading changes; treat the value here as the shape, not a constant to hardcode.
• reused is true when a still-active run for the same inputs already existed and was handed back instead of starting a new one (see Reuse).
• status is the run's lifecycle state at hand-off: queued, then running, then a terminal complete / failed / canceled. It is spelled complete, never completed.
• console_url is the human-readable report; hand it to the user. Omitted, never guessed, when the slug does not resolve.

Poll for the verdict

curl https://app.amdahl.ai/api/platform/v1/eval-runs/b1c2d3e4-... \
  -H "X-API-Key: $AMDAHL_KEY"

{
  "data": {
    "run": {
      "id": "b1c2d3e4-...",
      "eval_slug": "prompt-and-message-eval",
      "eval_version": "2.33.0",
      "status": "complete",
      "verdict": { "...": "the graded report card (below)" },
      "created_at": "2026-08-07T19:25:04.276Z",
      "finished_at": "2026-08-07T19:28:35.016Z"
    }
  }
}

Read data.run.status; when it is complete, data.run.verdict carries the graded report card (below). A missing / cross-tenant id returns null.

curl "https://app.amdahl.ai/api/platform/v1/eval-runs/b1c2d3e4-...?wait_ms=30000" \
  -H "X-API-Key: $AMDAHL_KEY"

The report card — what to show a person

GET /eval-runs/{id}/report returns the run already written up:

curl https://app.amdahl.ai/api/platform/v1/eval-runs/b1c2d3e4-.../report \
  -H "X-API-Key: $AMDAHL_KEY"

{
  "data": {
    "report": {
      "run_id": "b1c2d3e4-...",
      "status": "complete",
      "markdown": "## Eval report — prompt-and-message-eval 2.5.0\n\nYour draft did not pass...",
      "headline": {
        "submitted": {
          "score_15": 2.6,
          "checks": { "passed": 2, "total": 5 },
          "usage": "as_provided"
        },
        "improved": {
          "score_15": 5,
          "checks": { "passed": 5, "total": 5 },
          "usage": "illustration_only"
        },
        "threshold": 4.2,
        "transition": "fail_to_pass",
        "sentence": "Your draft did not pass - the improved version does.",
        "lift": 0.6,
        "lift_reportable": true,
        "lift_badge": "+2.4 points",
        "confidence": { "level": "high", "reasons": [], "caveat": null },
        "stop_reason": "threshold_met",
        "quotes_cited": 6,
        "quotes_cited_submitted": 0,
        "quotes_cited_improved": 6,
        "what_changed": "Tied the opening claim to what two customers actually said.",
        "input_simulated": false,
        "facet": "message"
      }
    }
  }
}

markdown is meant to be pasted, not summarised. It is the one endpoint on this API that returns prose, and the reason is narrow: the eval grades two different artifacts, so a caller who writes their own summary is exactly where the rewrite's score becomes a claim about the user's copy. A block you paste cannot be reshaped. It renders on every run state — a failure, an abstain and an in-flight run each get an honest card rather than an empty body.

headline is the same derivation in machine form, so a script that needs a number and the block a person reads cannot disagree. Three fields there are worth knowing:

• lift_badge is null unless lift_reportable is true. Below the measured run-to-run noise floor the card says "no measurable change" instead of printing a small confident number. lift still carries the raw value.
• input_simulated: true means you submitted no draft. On a prompt-only run the grader writes a specimen so there is something to score, and without this flag a failing "before" reads as your writing failing.
• quotes_cited is blended; the per-side counts are the honest read. A run grades two artifacts, so "6 customer quotes" does not mean six pieces of customer evidence were weighed against YOUR copy. quotes_cited_submitted and quotes_cited_improved say which side cited what — and on most runs the first is 0. Do not add them: an utterance cited on both sides is one quote in quotes_cited and appears in both counts, and a quote carried only by a suggestion is in quotes_cited and in neither.

A run with no before/after — an abstain, a failure, an eval whose graders produce no improvement report — returns headline: null and a card that says so.

The two drafts, side by side

GET /eval-runs/{id}/drafts (MCP: the drafts action) is the only read that carries the texts — the prompt and message you submitted, the improved versions the eval wrote, each with its score, its checks fraction, and its per-rubric-line reasoning. Read it when you want the line-by-line comparison rather than the written-up card.

curl https://app.amdahl.ai/api/platform/v1/eval-runs/b1c2d3e4-.../drafts \
  -H "Authorization: Bearer $AMDAHL_KEY"

{
  "data": {
    "run_id": "b1c2d3e4-...",
    "status": "complete",
    "submitted": { "prompt": null, "message": { "text": "Hi Dana, most VPs I talk to say ramp takes months.", "source": "submitted_inputs", "score_15": 1, "checks": { "passed": 1, "total": 5 }, "dimensions": [{ "name": "Grounding", "passed": false, "reasoning": "No quote backs the ramp claim." }] } },
    "improved": { "prompt": { "…": "…" }, "message": { "text": "…", "source": "candidate_set", "score_15": 5, "checks": { "passed": 5, "total": 5 } } },
    "simulated_before": false,
    "absent_reason": null
  }
}

This read returns status flat (data.status), the same root spelling every eval read answers to — see the polling warning. It carries no improvement block, though, so the before/after facets are not readable here — see the facets note.

Three fields exist so you can tell what you are holding:

• source names which carrier answered — submitted_inputs (your own bytes, untouched), candidate_set (the clean copy frozen at generate time), verdict_facet, or grader_rewrite. They do not all carry the same guarantee, so the read says which one it used rather than presenting them as interchangeable.
• annotated: true means the stored copy carried <unverified> markers that were stripped to produce text. The judge never saw the markers, so text is what was graded — but it is not byte-identical to the stored facet, and a re-derivation has to know that.
• absent_reason is why a side is empty: not_produced (a refusal — roughly 9% of completed runs write no rewrite, correctly), run_not_complete, no_improvement_report, or not_submitted. A missing side is null with a reason, never an empty string and never a zero score.

Cited quotes stay on GET /eval-runs/{id}/evidence — folding them in here would rebuild the oversized payload this read exists to replace.

What the score is, and what it is not

Four things hold for every run, on every workspace, regardless of how much data you have. They are properties of how the grade is built, not observations about one corpus — and each one is a way the headline number gets misread.

overall_score and the run verdict both describe your copy; the case-level passed does not. overall_score is the grade of the copy you SUBMITTED, and since eval_version 2.16.0 the run verdict buckets on the submitted side too (input_passed ?? passed). The case-level passed still follows the IMPROVED version the eval produced, so a case can read passed: true beside a low overall_score: the rewrite cleared the bar and your draft did not, which is the finding, not a contradiction. Never read a case passed — or a verdict stored before 2.16.0 — as "my draft is fine".

Read verdict.headline instead. It is the SAME object the report card returns, from the same derivation, so a script reading the verdict and a person reading the card cannot disagree:

"headline": {
  "submitted": { "score_15": 2.6, "checks": { "passed": 2, "total": 5 }, "usage": "as_provided" },
  "improved": { "score_15": 5, "checks": { "passed": 5, "total": 5 }, "usage": "illustration_only" },
  "threshold": 4.2,
  "transition": "fail_to_pass",
  "sentence": "Your draft did not pass - the improved version does.",
  "lift": 0.6,
  "lift_reportable": true,
  "lift_badge": "+2.4 points",
  "confidence": { "level": "high", "reasons": [], "caveat": null },
  "stop_reason": "threshold_met",
  "quotes_cited": 6,
  "quotes_cited_submitted": 0,
  "quotes_cited_improved": 6,
  "what_changed": "Tied the opening claim to what two customers actually said.",
  "input_simulated": false,
  "facet": "message"
}

submitted.score_15 is the one number to quote. It is on the same [1, 5] axis as threshold, so 2.6 against a bar of 4.2 reads directly. The run LIST carries the same two scalars as verdict_summary.input_score_15 / input_score, so a history view does not need a detail read per row, and each case carries them as input_score_15 / input_score. If you are reporting a run to a person, paste the report card's markdown — it already makes this split correctly.

headline is null when no case produced a before→after report (a pure rule eval, an abstained run), and absent on runs graded before it shipped.

Grounding rewards quote-traceable copy and penalizes abstract category copy. The grounding dimension asks whether each claim traces to something a customer actually said, so writing that stays at the category level ("streamline your workflows", "enterprise-grade reliability") scores low even on a workspace with thousands of conversations. A low grounding score is not a verdict on how much data you have — it is a verdict on whether the copy reaches for it. That is the point of the dimension rather than a limitation of it: the fix is naming the specific situation the specific buyer described.

The number is a coaching read, not a measurement. The judge returns a binary verdict per rubric line and score_15 is 1 + 4 * (passed / total) over those verdicts — a structured second opinion on writing, grounded in cited quotes. It does not predict reply rate, and lift is the coach's own before/after read, never a measured conversion lift. Treat it the way you would treat a colleague's review with receipts attached: worth acting on, not worth reporting as a metric.

lift mostly measures how low YOUR draft scored, not how much the rewrite added. The improved side is written against the same rubric it is then graded on, so it lands near the bar on most runs — which makes it close to a constant, and lift is improved − submitted. Two independent readings, one tenant each: at eval_version 2.9.0 the improved side cleared the 4.2 bar on 92% of runs (n = 89 runs / 40 distinct messages) and lift was −0.90 correlated with the submitted score on that same set, still −0.88 when restricted to runs where lift_reportable is true; a 90-day window at eval_version 2.15.0 (410 scored runs) reads improved_mean_15 4.24 against the same 4.2 bar, with submitted_mean_15 2.38.

The consequence to design around: a large lift means your draft scored low, and the largest lifts come from the weakest submissions. transition inherits it — its first half is the submitted verdict by definition and its second half is near-constant — and the noise floor does not fix it, since filtering to reportable lifts barely moves the correlation. Do not build a dashboard, a target or a quality gate on lift, lift_badge or transition. Use headline.submitted.checks — the fraction. It is the only number on the card that is about your writing. How the submitted side is scored is under active revision; the fraction is the field that will keep meaning what it says.

Scope the grade to a slice

By default a run is graded against your whole customer corpus. Pass scope and it is graded against a named slice of it — what executives said, what people who raised a pricing objection said, what buyers at closed-won accounts said:

{
  "eval": "prompt-and-message-eval",
  "inputs": { "message": "<your draft>" },
  "scope": {
    "audience": "customer_voice",
    "filters": [
      { "field": "role_level", "op": "eq", "value": "executive" },
      { "surface": "deals", "field": "deal_stage_status", "op": "eq", "value": "won" }
    ]
  }
}

Each filter is { surface?, field, op, value? } — the same shape Search's filter lane takes, plus a per-filter surface. They are ANDed, capped at 25 per run, and surface defaults to interactions.

Three surfaces, and you may mix them in one scope:

Mixing is the whole point: "executives at closed-won accounts" is one utterance-grain cut and one deal-grain cut, and search.query — which takes a single surface per call — cannot express it in one go.

Never guess at field names. The catalog lists every filterable field per surface with its type, description and admitted operators, and is derived from the same schema the eval's compiler validates against:

curl "https://app.amdahl.ai/api/platform/v1/search/fields" \
  -H "X-API-Key: $AMDAHL_KEY"

Over MCP, the search tool's fields action, or the search_field://list resource. See Search → Step 0 — in particular why you should read sample_values before filtering a coded field.

A slice has to clear an evidence floor before it is used — 3+ distinct external speakers, 25+ utterances, 2+ distinct companies, all three. A search hands you four rows and you judge them yourself; an eval hands you a grade with the slice's name stapled to it, so the same four rows become "this is what enterprise champions say" with a score behind it. The grade is real and the framing is a lie.

When a scope cannot be applied the run still completes and still grades your prompt and message, and names which of seven things happened: not_provided · invalid · no_evidence · thin_evidence · lookup_failed · no_matching_accounts · too_many_accounts. It lands on the run's progress trail — progress.steps[] on eval_run://<id>, as a scope_resolved step carrying the reason, a written message you can show a person, and detail where there is something actionable to say. It is not on the verdict or the report card, so read the trail rather than inferring the slice from the score. lookup_failed means our check broke, never "you have no data."

What the run falls back TO depends on what else scoped it: with no inputs.audience it is your whole corpus, but a resolved inputs.audience is gated on its own path and survives an abstained scope — so the run is graded against that cohort, not corpus-wide. When both resolve they COMPOSE: the slice is their intersection, not one of them winning.

Pass allow_thin_evidence: true when the narrow cut is the question. The run then grades the slice and the scope_resolved step carries below_floors: true beside the real counts — a thin grade that is labelled as one, rather than a thin grade that reads like a cohort finding.

Two consequences worth knowing:

• A scope changes the evidence, not just the framing. A cleared slice has its own utterances drawn and tagged segment — the tier that licenses "teams like yours" — spread across companies so one talkative account cannot speak for the slice.
• scope is part of the run fingerprint. A filtered request and an unfiltered one are different runs under the default reuse: "cached"; otherwise the second would be served the first one's corpus-wide verdict under a slice's name. Filter ORDER is canonicalized here, so re-ordering the same filters does not fork the address — though the evidence-pin check compares a different, order-sensitive rendering, so a re-ordered pin is refused rather than reused.

A misspelled key (operator for op, column for field) is refused with invalid_argument at the run call rather than quietly dropped — a dropped filter returns a well-formed report about a wider slice than you asked for, and nothing in it says so. A scope that narrows nothing — {}, {"filters": []}, {"audience": "all"}, a bare {"allow_thin_evidence": true} — is accepted, and all four mean the same thing: graded unscoped, with the outcome recorded as not_provided. Nothing is hidden, so there is nothing to refuse.

Full rules, including the deal-grain account bound and the per-reason remedies, in Filter scoping.

Wiring the eval into a pipeline

The report card is built for a person. A pipeline — a LangSmith evaluator, a CI check, an outbound send/hold gate — needs one boolean and one fraction about the copy it already has, and the fields a reasonable integrator reaches for first are the wrong ones:

• improvement.transition.improved_verdict follows the IMPROVED version the eval wrote, which clears its bar on ~92% of runs — wired as a gate, it passes everything.
• verdict reads the submitted side today, but was re-meant at eval_version 2.16.0 (before that it bucketed on the improved side), so a gate over stored rows mixes two questions — and it is a four-way bucket, not a send/hold bit.
• lift mostly measures how weak your draft was, not quality (see above). Do not threshold it.
• overall_score is right on current runs but re-meant at eval_version 2.14.0, so a naive read over stored rows mixes two definitions.

Two pieces exist so none of that has to be re-derived:

mode: "gate" — grade what you sent and stop

Pass inputs.mode: "gate" and the run grades ONLY the submitted side: the same viability gates, the same retrieved-and-frozen customer evidence (pins included), the same binary per-dimension judge — then it stops. No rewrite, no improved-side grade, no suggestions. One judge call instead of several, so a gate run is markedly cheaper and faster than a full one. With a message and a prompt both submitted it gates the message; prompt-only gates the prompt, and nothing is ever simulated.

GET /eval-runs/{id}/gate — the machine read

Works on every run (gate-mode or full; historical rows are derived on read). Supports the same wait_ms long-poll (max 30000) as the detail read, and rides the MCP evals tool as the gate action.

{
  "data": {
    "run_id": "9c2f…",
    "eval_slug": "prompt-and-message-eval",
    "status": "complete",
    "gate": {
      "passed": false,
      "checks_passed": 2,
      "checks_total": 5,
      "score_15": 2.6,
      "threshold": 3.5,
      "basis": "message",
      "simulated": false,
      "dimensions_failed": ["Verified specifics", "CTA clarity"]
    }
  }
}

• passed is the send/hold bit: did the SUBMITTED copy clear the bar.
• checks_passed / checks_total is the score as a fraction — the right numeric to report upstream (LangSmith score = checks_passed / checks_total).
• denominator says what checks_total counted, and you should read it before comparing two runs. applicable_only (the default since eval 2.33.0) means a rubric line your kind of artifact could not satisfy — declared via artifact_type — left both the numerator and the denominator, so checks_total can be smaller than the rubric. all_dimensions counts every line, which is what every run before 2.33.0 did; null on those older runs means the same thing. The exclusion is symmetric — an inapplicable line is dropped whether it passed or failed — so switching a workspace between the two can move an individual score either way. Do not infer this from eval_version: it is a per-workspace setting, so two runs on the same version can have been divided differently. Trend checks_passed / checks_total only across runs whose denominator matches.
• gate is null while the run is in flight, and null when the run refused (status is complete but not_applicable_reason rides beside it — not_outreach, empty_corpus, report_unavailable, …). A refusal is not a fail: nothing was graded, so a pipeline should skip or surface it, never hold the send on it.
• simulated is true only on a full prompt-only run, where the graded "submitted" side is a specimen the eval wrote — a pipeline gating real sends should treat that as not-my-copy.
• On a FULL run read through this endpoint, threshold is stamped for new runs and null on historical rows (the stored bar there belongs to the improved side and is not republished).

A minimal poll loop: POST /evals/run with {"inputs": {"message": …, "mode": "gate"}}, then GET /eval-runs/{run_id}/gate?wait_ms=30000 until status is terminal. See the cookbook's LangSmith recipe for a complete evaluator.

Re-grading an edit: pin the evidence

Two plain runs each retrieve their own customer quotes, so the score delta between a draft and its edit mixes your change with a retrieval change. Hold the evidence fixed by passing evidence_from_run (the first run's id) on the second run — then the delta is attributable to the edit.

You do not have to remember that: every settled run whose quote pool was recorded carries a regrade block on the detail read and the gate read, with the call ready to paste:

{
  "regrade": {
    "suggested_params": { "eval": "prompt-and-message-eval", "evidence_from_run": "9c2f…" },
    "frozen_at": "2026-08-05T21:02:00Z",
    "quotes": 14,
    "evidence_resource": "eval_run://9c2f…/evidence"
  }
}

Merge suggested_params into your next POST /evals/run alongside the edited inputs. The block is absent while the run is in flight and absent when no pool was recorded — whatever it advertises, the pin will accept.

To draft from the graded quotes before re-running, read the pool itself: GET /eval-runs/{id}/evidence (MCP: the evidence action) returns every frozen quote with its id, tier and provenance (who said it, when, at what deal stage), exactly as a pinned run would be graded against it. evidence is null when the run never recorded a pool — a fact about the run, not an error.

Reading a verdict

Once status is complete, run.verdict is an EvalVerdict. For a message grade the meaty grader (improvement_loop) carries the full before→after report on improvement, plus the AFTER side surfaced on the back-compat dimensions / quotes / rewrite fields:

{
  "eval_slug": "prompt-and-message-eval",
  "eval_version": "2.33.0",
  "overall_score": 0.4,
  "headline": {
    "submitted": { "score_15": 2.6, "checks": { "passed": 2, "total": 5 }, "usage": "as_provided" },
    "improved": {
      "score_15": 4.2,
      "checks": { "passed": 4, "total": 5 },
      "usage": "illustration_only"
    },
    "threshold": 4.2,
    "transition": "fail_to_pass",
    "sentence": "Your draft did not pass - the improved version does.",
    "lift": 0.4,
    "lift_reportable": false,
    "lift_badge": null,
    "confidence": {
      "level": "moderate",
      "reasons": ["Only 2 customer quotes were available to ground against."],
      "caveat": "Read the scores as directional - some of the inputs were thinner than usual."
    },
    "stop_reason": "threshold_met",
    "quotes_cited": 2,
    "quotes_cited_submitted": 0,
    "quotes_cited_improved": 2,
    "what_changed": "Tied the opening claim to what two customers actually said.",
    "input_simulated": false,
    "facet": "message"
  },
  "overall_reasoning": "Your submitted draft did not hold up against the evidence; the improved version does. See the before/after report for what changed. The gaps are grounding, specificity - each one is explained line by line below. Grounded in 2 customer quotes from your data. Your draft cited none; the rewrite cited 2.",
  "verdict": "pass",
  "cases": [
    {
      "case_id": "message-vs-voice",
      "label": "Relevant positioning vs. customer evidence",
      "answered": true,
      "applicable": true,
      "score": 0.9,
      "passed": true,
      "input_passed": false,
      "input_score_15": 2.6,
      "input_score": 0.4,
      "improved_passed": true,
      "transition": "fail_to_pass",
      "latency_ms": 8340,
      "graders": [
        {
          "grader_id": "basic-hygiene",
          "kind": "rule",
          "score": 1.0,
          "passed": true,
          "rationale": "All 3 rule check(s) passed."
        },
        {
          "grader_id": "grounded-improvement",
          "kind": "improvement_loop",
          "score": 0.8,
          "passed": true,
          "rationale": "Your version passed 2/5 rubric dimensions; the improved version passed 4/5 — each judged pass/fail blind against the same customer quotes (score 4.2/5, bar 4.2/5). Grounded the claims in customer quotes.",
          "improvement": {
            "before": {
              "label": "before",
              "prompt": "Write a short cold nudge to a RevOps leader at a mid-market SaaS company.",
              "prompt_note": "as provided",
              "prompt_reasoning": "The prompt asks for a generic nudge with no grounding in real customer language.",
              "message": "Hi Sam - noticed your team just doubled headcount. Most RevOps leaders we talk to hit a wall on pipeline hygiene right around that inflection. Worth a quick 15 minutes next week?",
              "message_note": "as provided",
              "score": 0.4,
              "score_15": 2.6,
              "checks_passed": 2,
              "checks_total": 5,
              "dimensions": [
                {
                  "name": "Grounding",
                  "pass": false,
                  "score": 1,
                  "reasoning": "The headcount hook is generic; only the hygiene wall is real."
                },
                {
                  "name": "Specificity",
                  "pass": false,
                  "score": 1,
                  "reasoning": "Broad claims, no concrete customer language."
                },
                {
                  "...": "abridged - 3 of the 5 rubric lines are omitted here; checks_passed / checks_total count all five"
                }
              ],
              "quotes": [
                {
                  "text": "the headcount thing every vendor leads with never lands",
                  "source": "Cold outreach reactions",
                  "stance": "contradicts"
                }
              ],
              "reasoning": "The draft leans on a hook customers say they tune out."
            },
            "after": {
              "label": "after",
              "prompt": "Draft a cold nudge to a RevOps leader at a mid-market SaaS company. Use Amdahl to pull the specific pain points our own customers describe around scaling their sales team, and lead with the one that resonates most — not a generic headcount hook.",
              "prompt_note": "improved — reusable",
              "message": "Hi Sam - a few RevOps leads we work with said their pipeline data started breaking down right past ~40 reps. If that rings true, worth 15 minutes on how they got ahead of it?",
              "message_note": "suggested rewrite — not required to send verbatim",
              "score": 0.8,
              "score_15": 4.2,
              "checks_passed": 4,
              "checks_total": 5,
              "dimensions": [
                {
                  "name": "Grounding",
                  "pass": true,
                  "score": 5,
                  "reasoning": "Anchors on the verbatim ~40-rep breaking point customers named."
                },
                {
                  "name": "Specificity",
                  "pass": true,
                  "score": 5,
                  "reasoning": "One concrete, customer-sourced number instead of a vague inflection."
                },
                {
                  "...": "abridged - 3 of the 5 rubric lines are omitted here; checks_passed / checks_total count all five"
                }
              ],
              "quotes": [
                {
                  "text": "once we grew past ~40 reps our pipeline data just fell apart",
                  "source": "Pipeline hygiene",
                  "stance": "supports"
                }
              ],
              "reasoning": "The rewrite anchors on real customer language and drops the hook the evidence contradicts."
            },
            "lift": 0.4,
            "what_changed": "Grounded the claims in customer quotes."
          },
          "dimensions": [
            {
              "name": "Grounding",
              "pass": true,
              "score": 5,
              "reasoning": "Anchors on the verbatim ~40-rep breaking point customers named."
            },
            {
              "name": "Specificity",
              "pass": true,
              "score": 5,
              "reasoning": "One concrete, customer-sourced number instead of a vague inflection."
            },
            {
              "...": "abridged - 3 of the 5 rubric lines are omitted here; the improved side passed 4 of the 5, see improvement.after"
            }
          ],
          "quotes": [
            {
              "text": "once we grew past ~40 reps our pipeline data just fell apart",
              "source": "Pipeline hygiene",
              "stance": "supports"
            }
          ],
          "rewrite": "Hi Sam - a few RevOps leads we work with said their pipeline data started breaking down right past ~40 reps. If that rings true, worth 15 minutes on how they got ahead of it?",
          "evidence": {
            "before_score": 2.6,
            "after_score": 4.2,
            "lift": 0.4,
            "message_simulated": false,
            "retrieved_quotes": 11,
            "retrieval_status": "ok"
          }
        }
      ]
    }
  ],
  "summary": { "total_cases": 1, "passed_cases": 1, "failed_cases": 0, "not_applicable_cases": 0 }
}

• overall_score is the grade of the copy you submitted, in [0, 1] — the mean, over applicable cases, of each case's submitted side. It is the same number as verdict.headline.submitted.score_15 rendered on the [0, 1] axis instead of [1, 5], read through the same derivation, so the two cannot disagree. null when no case applied — see Not-applicable; it is never 0 for a run that graded nothing. On runs stamped with an eval_version before 2.14.0 this field is a mean across BOTH artifacts and is a grade on neither.
• overall_reasoning is a short plain-language summary of WHY the run landed where it did — assembled deterministically from the graders' own reasoning + rule failures + the count of quotes it grounded against. No extra LLM call.
• verdict is the bucket: pass = every applicable case passed, fail = none passed, partial = some, not_applicable = no case could be applied (see below). Derived from the per-case pass count, not from overall_score, and since eval_version 2.16.0 that count reads the SUBMITTED side (input_passed ?? passed) — so a pass says YOUR draft cleared the bar. A verdict stored before 2.16.0 bucketed on the case-level passed, which follows the IMPROVED side, and there a pass says only that the eval's rewrite cleared the bar. For a pipeline read gate.passed instead of either — see Wiring the eval into a pipeline.
• cases[] — one per subject. answered is whether the subject carried gradable content; applicable is whether the case counted toward the verdict; latency_ms is the wall-clock (0 for a provided message, the question's latency for a generated answer). passed follows the IMPROVED side whenever the case ran an improvement grader — so it says whether the eval's rewrite cleared the bar, not whether yours did. The two reads are on the case beside it: input_passed (what you submitted), improved_passed (what the eval wrote), and transition (pass_to_pass / fail_to_pass / pass_to_fail / fail_to_fail). input_score_15 / input_score are the numbers behind input_passed rather than the bit it collapses to, and input_simulated is present (and always true) when the graded "submitted" artifact was a specimen the eval wrote because you sent a prompt and no message. A case with no improvement grader keeps the plain "every applicable grader passed" meaning.
• graders[] — one per grader. score is normalized to [0, 1], rationale is the plain-language explanation, and depending on the kind: improvement (the full before→after report, below), dimensions (per-rubric-dimension 1-5 score + a sentence of reasoning), quotes (the verbatim customer utterances the grader cited, each with a source theme and a supports / contradicts / neutral stance), rewrite (a grounded better version), and evidence (the receipt — SoR value vs answer value for sor_anchored, before/after scores + retrieval counts for improvement_loop).

The before→after report (improvement)

Reading the facets

The improvement block is nested inside the grader that produced it. From the body of GET /eval-runs/{id}, the full path is:

data.run.verdict.cases[0].graders[1].improvement.facets[]

Two things routinely go wrong here, and neither one fails loudly:

• If the detail payload is too big for your transport, read GET /eval-runs/{id}/improvement (resource eval_run://<id>/improvement). It returns this same block in a bounded shape — every facet with its scores, per-rubric-line verdicts, cited quotes and counters, plus lift, transition, suggestions, prompt_patch, confidence and coverage.

  It omits exactly two fields, and says so on every response in an omitted[] naming each one and where it still lives: grader_meta (the blinding and evidence-provenance disclosure) and stage_trace (per-stage timings). Those describe how the run executed rather than what it found, and they are ~38% of the block's bytes — which is what lets the rest fit.

  The other narrow reads — /drafts, /report, /gate, /evidence — carry no improvement block at all. Before this read existed, a payload_note on an oversized detail body pointed only at those four, so following it cost you the facets. The note now names what each read carries; pick the one holding the field you need.

• The not-applicable cap counters are on GRADED blocks only. A facet can ship UNGRADED — the reusable prompt on a message-only run, or the before side when the unpaired absolute grade is unavailable. Those carry dimensions: [] and no truncated_not_applicable / discarded_not_applicable, because no rubric ran, so neither cap could have bound and a 0 there would assert a measurement nobody took. On a graded block both are emitted even when zero, so absence there does mean the run predates the counters. To take a rate, count over facets with a non-empty dimensions array; that selects the graded blocks and keeps both cases out of the denominator.

• An absent parent reads as an empty list. facets is never returned empty: a message facet is always seeded. So improvement.facets coming back empty means the whole improvement object was absent, which happens on mode: "gate" (no rewrite is performed by design) and on any refusal or unavailability — empty_corpus, evidence_unavailable, report_unavailable, prompt_too_large, not_gradeable, not_outreach. Read not_applicable_reason beside it to tell which; do not infer "the two sides tied" or "the lift was below the floor", as neither removes a facet. An engine failure or a loop timeout reports report_unavailable — generate_failed, grade_failed and loop_deadline are internal loop-failure values that never appear in not_applicable_reason; the distinguishing detail rides on evidence.reason.

improvement_loop returns one improvement object with two graded sides:

• before — the message you sent (message_note: "as provided"), OR the draft simulated from your prompt-only run (message_note: "simulated from your prompt") — graded on the rubric with per-dimension reasoning and cited quotes. prompt echoes the prompt you sent (or null when you sent none), with prompt_reasoning on what the prompt did well or poorly.
• after — an improved, REUSABLE prompt (a template that tells the writer to use Amdahl to nail the relevant positioning for the specific customer — research their situation, position the offer for their case, verify each claim, keep it relevant; prompt_note: "improved — reusable" when you supplied a prompt, or "example — reuse this" when you didn't) + a relevantly positioned improved message (always message_note: "suggested rewrite — not required to send verbatim") — graded the same way.
• lift — the after score minus the submitted score, in [-1, 1]. It is not after.score - before.score, and subtracting the two published blocks will not reproduce it. Since eval_version 2.19.0 the submitted side is read TWICE: before is an UNPAIRED grade of your copy (the absolute level, and the side overall_score and headline.submitted project from), while lift is taken against the BLINDED PAIRED grade of the same copy, published beside it as before_paired on the message facet. Both blocks of the subtraction then come from one judge call, which is what pairing buys. When before_paired is absent — the prompt facet, and any run graded before 2.19.0 — the two reads are the same block and lift is after.score - before.score after all.
• lift_reportable — whether that lift is bigger than the grader's own run-to-run noise. Read this before you read lift. A judge scoring the same text twice does not return exactly the same number, so a small lift can be the instrument moving rather than the writing improving. When this is false, lift is still returned for analysis but should not be quoted as a figure.
• what_changed — a one-line summary of the difference.
• regressions[] — every rubric dimension the rewrite scored LOWER than your original, with before_score / after_score, the after-side reasoning, and flipped_to_fail when the binary verdict flipped (both sides recorded one). Absent, not empty, when the rewrite cost nothing. A lift is a mean, so a rewrite can gain three points on grounding while dropping positioning from 5 to 1 and still report a positive number; nothing else on the report surfaces that, because what_changed is written by the model describing its own rewrite. Read this before you adopt a suggested message.

The noise threshold behind lift_reportable is not one fixed number. A side's score is the mean of its per-dimension scores, so a rubric with more lines produces a steadier mean, and the threshold scales as 1/sqrt(n) — a measured relationship, not an assumed one. Two consequences worth knowing:

• An eval you author with a wider rubric can resolve a smaller lift than one with a narrow rubric. That is real, not a formatting artifact.
• If the judge returns fewer dimensions than the rubric declares (visible as dimensions_scored below dimensions.length), the score rests on fewer real judgements than it appears to, and the threshold widens to match. A lift that would be reportable on a fully-scored run may not be on a short-scored one.

The AFTER side is ALSO surfaced on the grader's top-level dimensions / quotes / rewrite fields, so a reader that only knows the older shape still renders the improved message.

Alongside those, the report carries:

• facets[] — the before/after pair for the prompt and the message separately. They are graded on different rubrics, so they never share a score, reasoning, quotes, or worked examples. Each side carries score_15, its dimensions, its quotes, and its good_examples. before / after above remain as a back-compat mirror of the message facet.

• checks_passed / checks_total on every graded side — what score_15 actually IS. The judge returns a binary verdict per rubric line, and the headline is 1 + 4 * (checks_passed / checks_total) rendered on a five-point axis. A score_15 of 5 means "cleared all five checks", not "perfect" — quote the fraction rather than the rating. (These also ship under the older names passed / total, carrying identical values. Prefer checks_*: passed here is an integer COUNT while passed on a case, a grader, or a dimension is a boolean VERDICT, and reading one as the other is a mistake that has already been made in the field.)

• passed on a dimension — the boolean verdict, alongside the older pass. Both always carry the same value, so a consumer that reaches for either gets a real answer instead of undefined.

• usage on every graded artifact — what the thing IS: as_provided (untouched), simulated_specimen (a draft written so there was something to score), reusable_prompt (the takeaway — a template you keep), illustration_only (an example produced so the score difference could be measured — not a message to send).

• audience — a discriminated union on status. resolved carries the seniority cohort the run was scoped to plus the evidence counts behind the decision, and that cohort's own quotes are retrieved and tagged segment, so a scoped run grades against a different pool than an unscoped one rather than only being told to assume a cohort. abstained carries one of five reason values (not_provided / unresolvable / no_evidence / thin_evidence / lookup_failed) and a server-authored message. Narrow on status before reading dimensions — an abstained run was graded against your whole corpus, and presenting it as cohort-scoped makes every score under it read wrong. lookup_failed means our check broke, never "you have no data." Full rules in Audience scoping.

• tool_kit — { callable, out_of_scope }. research_steps are bounded by what YOUR key can actually run, so a step you would get a 403 for is never proposed. out_of_scope counts the calls that exist but your scopes do not cover; when it is 0 there is nothing to caveat.

• suggestions[] — anchored, surgical edits (keep / add / strengthen / remove / reorder). anchor_quote is verified server-side to be a literal substring of your text; a paraphrase is dropped rather than shown, so a suggestion can never quote a line you did not write.

• transition — the verdict SPLIT. input_verdict and improved_verdict are recorded separately, because your draft failing is the finding you ran the eval to get, not a failure of the eval. explanation is populated only when the improved side missed the bar.

  stop_reason says why the revision loop stopped, and is the field to branch on: threshold_met (the rewrite cleared the bar), no_critique (nothing further to change), deadline (a wall-clock budget ran out), model_call_cap, or max_iterations. deadline is expected, not exceptional — at the shipped configuration a run whose first round misses the bar and still has a critique to act on will usually report it. It costs you a further revision pass and nothing else: the reported grade is the FIRST round either way, and overall_score is the submitted side alone, so a cut loop cannot move a number in either direction. The matching reader-facing sentence rides confidence.reasons[]. Treat an absent transition as unknown rather than as converged — the whole block is omitted when the unpaired absolute grade is unavailable, and iterations / stop_reason go with it.

  The two sides are judged asymmetrically, on purpose. input_verdict is the bar and nothing else — your draft is yours to write however you like. improved_verdict additionally requires the copy to be fully rendered: if the improved message still carries an unfilled placeholder ({{token}}, [Insert the metric], [Company]) it does not read as passing whatever it scored, and explanation quotes the spans so you can see what matched. A message with a slot in it is a template, not something you can send. The count rides the improved message facet as placeholders_unresolved, and no number moves — it is taken after grading, so score_15, the dimensions and the lift all still describe the copy as written. The improved prompt is deliberately exempt: a reusable prompt is supposed to carry slots.

• coverage — how much of a long submitted prompt was actually graded (total_chars / graded_chars / truncated / per-section included). Past 12,000 characters a prompt is sectioned on its own headings; past 250,000 the run refuses rather than grading a sliver.

• research_steps[] — runnable Amdahl calls that would close the evidence gaps found, each validated against the live operation registry (unknown op, write-shaped op, or SQL the query gate refuses → dropped before you see it).

• prompt_patch — the prompt-facet suggestions assembled into ONE pasteable block (text to paste, lines[] structured, applies_to_submitted_prompt). Composed server-side from the same suggestions[] the report itemizes, so it can never claim an edit the list does not. Absent — not an empty block — when there are no prompt suggestions.

• confidence — { level, reasons }. How much weight the numbers can carry: high (the quiet common case, empty reasons), else moderate / low with reader-facing reasons (thin evidence, an errored retrieval leg, a run that stopped on a budget rather than the threshold, a truncated prompt, a side scored on fewer rubric lines than the rubric declares). It never changes a score — it says how firmly to hold one.

• grader_meta — model_calls, blinded, evidence_quotes, evidence_scope (plus the deprecated evidence_frozen), and two disclosure blocks. blinded: true means both candidates were scored in ONE call, unlabelled, with the order derived from a content hash, so the judge could not favour its own draft; blinding carries the MECHANICS that claim rests on (paired, order_shuffled, improved_shown_first for this run, separate_generate_and_grade, judge_model, generator_model) — facts the run recorded while executing, not a restatement of the prompt. blinding.sampling reports what the two stages actually sampled at, per stage — {"temperature": null, "reason": "model_rejects_temperature"} means the configured temperature: 0 never reached the model (the 5-series rejects the parameter outright), so that call ran at the API default. configured echoes what was asked for, so the gap between intent and effect is visible rather than inferred. evidence_provenance labels the different quote counts a report carries — pool (retrieved and frozen), in_context (how many of the pool reached the judge's context), rounds (retrieval rounds run), cited (distinct ids the judge cited across every graded block), resolved (how many of those resolved to a real quote), errored_legs. All are real and different; unlabelled they read as the report contradicting itself. When the run was pinned it also carries pinned_from (run_id, frozen_at, as_of), so you can see you are grading against an older view of your data.

evidence_scope is the field to read when you are comparing two runs. {"kind": "run"} means the evidence was retrieved for this run and held fixed across its revision rounds — which says nothing about any other run. {"kind": "pinned", "from_run_id": ..., "frozen_at": ...} means it was reused from that run, which is what makes two runs comparable. The older evidence_frozen: true only ever meant the first of those; it is deprecated and kept for compatibility.

mode: "rewrite" is the default: you get a full improved prompt and message, each graded against what you sent. Send mode: "advisory" when your prompt is a living document you are not going to replace — you then get anchored suggestions[] against what you already have instead of a rewrite, and the prompt facet has no after side.

Every field above is optional on the wire, so a verdict stored before any of them existed still parses.

Quotes are retrieved, never written by the model

The grader cannot invent a customer quote. Quotes are RETRIEVED from your cluster/theme index first, each tagged with an id; the LLM may only cite those ids per side; the server hydrates the cited ids back to the verbatim text. A quote you see in either side of the report is a real utterance from your data — fabrication is structurally impossible, not merely discouraged.

Every quote carries a tier, and the tier is a licence. It says whose voice the quote is, which bounds what a claim built on it may say:

• account — one named company said it. The ONLY tier that backs "you told us…", and the licence is per company: the quote carries the company it came from, and it backs a claim about that company and no other. Both the account you are writing to and every reference_accounts comparable land here, each tagged with its own name.
• segment — the cohort the run was scoped to said it. Backs a claim at the cohort's own level of aggregation: "teams like yours…", "some of these teams…".
• corpus — a recurring pattern across your conversations. Backs "the RevOps leaders we talk to say…".

The segment licence is a band, and both edges are graded. Widening a cohort quote to the named account — "you", "your team" — fails grounding, because that needs account. So does narrowing it to a single speaker — "one lead told us…" — because a pattern several companies stated is not one anecdote. Match the cohort's level and you are inside it.

segment is emitted as of eval version 2.6.0. Before that it was a declared value with no producer, so a run came back account or corpus only and "teams like yours" copy had no tier that could back it.

A quote with no tier predates tiering and reads as corpus — the weakest standing, never the strongest. Reaching past a quote's tier is graded as a grounding failure, not a style note, because naming the wrong scope turns a grounded line into a claim of a conversation that never happened.

Citing a comparable customer

Naming a customer you already did the work for is the ordinary sales move, and until eval version 2.28.0 the eval could not grade it. account was single-valued and did two jobs at once — who the draft is going to AND whose evidence to retrieve — so a message to a prospect came back with every quote at corpus tier, and "11x scales its AI-driven onboarding" was correctly marked as an account-specific claim with no account-tier support. There was none, because nothing had asked for 11x's.

The two jobs are now two inputs. account stays the recipient; reference_accounts names the comparables, and each one's own quotes are drawn at account tier tagged to that company:

{
  "inputs": {
    "message": "<draft to a prospect>",
    "account": "Artisan",
    "reference_accounts": "11x, Degreed"
  }
}

The licence is per company, and that is the whole point. An account-tier quote from 11x backs a claim about 11x — and about no other company, including the recipient. So "11x scales its AI-driven onboarding" is groundable once 11x is a reference account, while "you are scaling onboarding" about the prospect still needs the prospect's own evidence and nothing else will do. Citing one company's quote to assert something about another is graded as reaching past the tier, exactly like citing a corpus pattern for "your situation". When the recipient has no evidence, the honest moves are to make the claim at cohort level or to attribute it to the reference customer whose quote backs it.

What to expect from a run that names comparables:

• Up to three names, comma-separated (semicolons and newlines also split; case-insensitive duplicates spend one slot). A fourth is reported, not honoured.
• Up to two quotes each, six across the leg. A per-company cap is what stops the most talkative comparable filling the roster's budget — name three proof points and you get three.
• The account tier does not grow. Its ten slots are shared: what the comparables take, the recipient's own draw gives up, floored so the recipient always keeps at least one. That is deliberate — the recipient's quotes are the only evidence in the pool that can license a claim about them.
• Only their side of the conversation. The recipient's draw includes a few of your own recent messages to them, so the draft does not contradict what you last said; a comparable's does not, because your words about 11x are not 11x's.
• Every name is answered. The account_resolved step carries reference_accounts (the canonical names that resolved and contributed quotes), reference_quotes (how many landed), and reference_outcomes — one entry per name you typed, either ok with the canonical name and its quote + utterance counts, or abstained with a reason: the same not_in_corpus / no_quotable_utterances / unresolvable / lookup_failed vocabulary the recipient uses, plus over_limit (past the third name) and is_recipient (you listed the company you are writing to among its own comparables). A name your workspace has no conversations under is reported rather than dropped, so you are never left looking for missing data that is not the problem.

A pinned run (evidence_from_run) draws nothing here: the pool comes from the pin. It must name the same set of comparables the pinned run was drawn under (order does not matter) or the run is refused rather than graded, for the same reason a mismatched account is — grading anyway would put one company's words behind another company's proof point, at the tier that licenses naming them.

What the run was missing — requires_input

A run given no account grades against corpus-wide evidence, and corpus tier cannot license a claim about what this company said. When that happens the verdict says so, and — when it can verify a value — what to send:

"requires_input": [
  {
    "field": "account",
    "why": "Scoping to an account draws that account's own quotes, the only evidence tier that licenses a claim about what THEY said.",
    "suggested": "Acme Corp",   // resolved against YOUR company_name rows
    "utterances": 47,           // so thin coverage is visible before you re-run
    "matched_on": "Acme"        // the phrase in your draft it came from
  }
]

Re-run with inputs.account: "Acme Corp" and the report comes back with account-tier evidence.

It is reported, never applied. Nothing on the payload was drawn using a suggestion — the run graded exactly what you sent. A wrong account would attribute one company's words to another, so the value is offered and never assumed; only an account you send changes what gets retrieved.

suggested is absent when no company in your draft could be verified against your own data. The gap alone is honest; a guessed company name is not. requires_input is absent entirely when the run had what it needed.

This is also why the eval never stops to ask you a question: a tool that blocks deadlocks every headless caller, so the gap ships as data on the response and your agent decides whether to re-run, prompt a human, or accept corpus tier.

Re-run the evidence yourself — evidence_provenance.requests

The report tells you which quotes backed a claim. It also tells you the calls that produced them, so you can run them yourself, change them, and put them in a prompt.

"grader_meta": {
  "evidence_provenance": {
    "pool": 60,
    "in_context": 20,
    "requests": [
      {
        "op": "data.query",
        "params": {
          "sql": "SELECT content, speaker_type, timestamp FROM interactions WHERE company_name = 'Acme' AND speaker_type = 'external' ...",
          "allow_truncation": true
        }
      },
      { "op": "data.cluster_search", "params": { "query": "deal health scoring", "limit": 12 } }
    ]
  }
}

Each entry is a registry operation id and its params — the same strings the REST API, the MCP tools, and research_steps all take. Nothing here is a private label; you can send any of them as-is.

Why this is the field to build on. Every other number in evidence_provenance is a count — how much evidence there was, how much reached the model. You cannot act on "we retrieved 60 quotes". You can act on "run this query and quote the result". If you are using the eval to improve a prompt, this is the part that belongs in the prompt: a rule that says "ground your claims in customer quotes" is unactionable, and one that names the call is a procedure your writer can follow every time.

Ordering is narrowest-first — account, then cohort, then corpus — so requests[0] is the most specific call the run made, and usually the one most worth reusing.

What the params deliberately omit. No business_id and no user id. Your own key supplies tenancy, and echoing an id would give you a request that looks copy-pasteable while naming the wrong workspace. Everything present is a parameter you control.

Absent when there was nothing to record. A pinned run reuses a source run's evidence instead of issuing calls of its own, so it carries no requests; so do runs from before the field shipped. Absent means "no calls were made here", never "no calls were needed".

How deep retrieval goes, and what the judge actually reads

Retrieval runs in rounds. The run plans a set of search queries from your submission — the intent behind it, plus each specific claim it makes — and searches your conversations with the first batch. If the pool of distinct quotes comes back under the target, it goes back for the rest of the plan. It stops on whichever comes first: the pool is deep enough, a round returns nothing the pool does not already hold, the retrieval budget is spent, or your workspace has no themes at all. Duplicate quotes collapse across rounds, so a second round that re-finds the same material ends the loop rather than padding the count.

The pool and the prompt are two different budgets, and the report states both. evidence_provenance.pool is how many distinct quotes were retrieved and frozen — everything the report can show you. evidence_provenance.in_context is how many of those were placed in the judge's context for a single graded call, which stays bounded so a deep pool cannot push one grading call past its token ceiling. When the pool is deeper than the prompt budget, the quotes that reach the judge are sampled evenly across the search queries, so a wider search buys broader coverage of your draft's claims rather than more of whatever the first query happened to match. evidence_provenance.rounds says how many rounds it took: consistently needing more than one is a signal that the workspace's corpus is thin for the questions being asked.

Account-tier evidence exists only when you pass an account AND that account is in your data with buyer-side conversation. Otherwise the run says which of those was missing rather than grading you on cohort evidence under an account heading. See the worked example for the rule applied end to end. The same tier also carries any reference customers you named, each tagged with its own company — so account-tier quotes in one pool can belong to several companies, and each licenses claims about its own.

Segment-tier evidence works the same way one rung out: pass an audience, and if the cohort clears the evidence floors the run draws that cohort's own utterances and tags them segment. The draw is spread across companies — no single account may contribute more than a couple of quotes — because a cohort can clear the distinct-companies floor and still have its most recent utterances come from one talkative customer, and a "teams like yours" licence over one company's words is the same over-claim one rung down.

External-tier evidence is the opt-in fourth rung, and the only one that is not customer voice. Pass include_external: true and one web fan-out runs — the query is planned by a lightweight model from your submission, with a deterministic fallback so a model hiccup never silently produces an internal-only run — and its ranked snippets join the pool tagged external, each labelled with its publication or domain. The tier is a licence, stated to the writer and the judge alike: an external quote backs a claim about the market or the public record ("analysts report", "public benchmarks show") and can never back "our customers say" — citing one for a customer-voice claim is the same grounding failure as reaching past a customer tier. A bounded slice of the external quotes reaches the judge's context; the rest land in the pool for you to inspect, so raising external_cap widens what you can read without re-tuning the judge. The run's progress trail reports the searched query, the quote count and the distinct-source spread as external_resolved — or a typed reason it could not (no_scope, no_results, lookup_failed), never a silent skip.

Which rungs a run actually stood on

The three tiers together decide what the graded copy was allowed to claim, so every run reports its coverage as structured fields rather than only in prose:

corpus_only is the one to alert on. It is the tier-overreach condition stated as a boolean, so "was this graded on evidence that could back what it says?" is a query rather than a close reading.

Quote provenance — who said it, when, and at what stage

A quote carries optional provenance so you can weigh it, not just read it. "A VP at an account in Negotiation said this in May" and "an IC said it two years ago" are very different pieces of evidence, and the report used to present them identically.

The split is by SOURCE, not by tier: account and segment quotes are read straight off your conversations, where those columns live, while corpus quotes come from theme rows whose payload carries none of them.

Three rules govern all five, and they are the reason you can trust what you see:

1. They come from the warehouse row, never the model. The grader's citation carries an id and a stance and nothing else, so there is no channel through which it could assert who spoke or when — the server resolves the id back to the retrieved row and copies from there. This is the same structural guarantee that makes the quote text itself unfabricable.
2. An absent field is absent. If the warehouse cannot answer, the field is simply not on the response — there is no "Unknown", no empty string, no zero. A missing deal_stage_at_time means we do not know, and it is the common case.
3. deal_stage_at_time is the stage THEN, never the stage now. It is derived from the interaction's own deal against recorded stage history, and where that history does not answer, the warehouse abstains rather than extrapolating (measured wrong 45.9% of the time on one CRM). It is never back-filled from the deal's current stage, which would fill nearly every gap with an answer to a different question.

Two deliberate omissions. Speaker names are not surfaced — attaching a named individual buyer to a verbatim utterance inside a report built to be shared is a privacy decision we do not make on your behalf; speaker_side and role_level answer what a reader is weighing without naming a person. Corpus quotes carry no date or stage — the theme index the corpus leg reads does not store either per quote, so they are absent rather than approximated.

speaker_side is worth reading closely: retrieval does not filter on speaker, so some quotes in a report are your own team rather than a customer. The badge tells you which.

Your own side of the thread counts as context

Account-tier retrieval draws both sides: the account's own words, plus up to three of your team's most recent messages to them, ranked by recency rather than by relevance to the draft. That applies to the RECIPIENT only — a reference customer contributes their side and nothing else, because what you last said to 11x is not evidence about 11x.

This is deliberate, and it fixes a specific failure. On a relationship carried by email between calls, every recent touch is yours — so a buyer-side-only pool froze at the last thing the customer happened to say, and the eval graded a draft as though nothing had happened since. Measured on one workspace before the change: 37 of 95 accounts had no visible evidence at all, and another 30 were stale by an average of 61 days. Drafts opened with "last we talked" while months of follow-up sat unread in the corpus.

The rule that internal words never back a customer claim is unchanged — it is now enforced by labelling rather than by exclusion. A quote from your team carries speaker_side: "internal", the grader sees it tagged OURS, and citing it as evidence of what the customer thinks is a grounding failure. It is there so the draft does not repeat what you already sent, contradict it, or write as though it never happened.

Email is read as a message, not as a thread

Email bodies carry the chain they are replying to, plus envelope headers, plus (from some connectors) raw HTML. Quotes are bounded by length, and that bound is measured on the message — quoted chains and headers are stripped first, and the quote you see is the stripped text.

Without this, a two-sentence reply carrying six replies of history was excluded for being too long. That bound was originally sized against spoken turns, whose median is around 200 characters; email runs roughly six times longer, so it silently dropped about a third of inbound customer email on packaging rather than substance. Call transcripts have no chain to strip and are unaffected.

Not-applicable — never a false fail

A case that cannot be graded honestly comes back not_applicable and is bucketed OUT of pass/fail, rather than scored zero. A run whose every case is not-applicable has verdict: "not_applicable".

Read not_applicable_reason — the reasons ask different things of you, and overall_reasoning states the one that applied:

A refused run reports no score at all. overall_score is null, never 0, and the cause rides beside it as a top-level not_applicable_reason — on the detail read and on the run-list verdict_summary, so a monitor never has to parse prose to tell a refusal from a bad grade:

{
  "verdict": "not_applicable",
  "overall_score": null,
  "not_applicable_reason": "evidence_unavailable"
}

The refused case also carries no input_passed, improved_passed or transition. Those answer "did your copy clear the bar", and in a refusal nothing cleared or missed anything — a refused run used to ship input_passed: true off the hygiene checks that happened to still run, which read as a pass for a run that graded nothing.

What not_outreach means. The rubric scores commercial writing — positioning against a customer problem, grounding in what customers said, verified specifics, differentiation, the ask. It applies at any stage of a commercial relationship: cold prospecting, following up an open deal, chasing a proposal or signature, negotiating a pilot, expanding or renewing, re-engaging a quiet account. It does not apply to correspondence with no persuasive job — scheduling and calendar coordination, delivery or appointment notices, receipts and billing, support replies, internal notes — and sitting inside an active deal does not change that. Scoring those against a sales rubric produces a number that looks like a grade and means nothing.

A single grader whose system-of-record surface is unavailable only drops that one grader; the case still grades on the rest. An empty corpus is a reason to abstain, never to score a message zero.

Dimensions that could never apply

The section above is about a whole case abstaining. A narrower version happens inside a graded case: a single rubric line that the kind of writing you sent could never satisfy, however well it is written.

The clearest example is website or landing copy. The message rubric leads with "is the offer positioned for THIS customer's specific situation" — and landing copy is addressed to a market, so there is no "this customer" for it to be positioned for. It fails that line on every run. Three of the five prompt rubric lines have the same problem: they presuppose one named account to research and a decision about whether to send.

Every graded dimension now says so. A line that is structurally inapplicable carries:

{
  "name": "Relevant positioning: is the offer positioned for THIS customer's specific situation, not a generic pitch?",
  "pass": false,
  "score": 1,
  "reasoning": "The copy addresses a category of buyer rather than one situation.",
  "not_applicable": true,
  "not_applicable_reason": "You declared this as landing copy, which is written for a market rather than one named recipient, so this check has nothing to hold it to.",
  "not_applicable_source": "declared_type"
}

not_applicable_source says who made the call: declared_type is a deterministic consequence of the artifact_type you sent, judge is the grader's own reading of the artifact in front of it. A dimension that is merely unmet, weak or unsupported is a fail, not inapplicable — and at least two dimensions are always scored, whatever the grader returns.

When the flag hits a cap — or contradicts the score

Two independent caps bound how much of a rubric can be excluded, and a third check rejects a claim the grader's own scores disprove. Each graded facet reports what each one threw away:

All three are emitted even when zero, which is the ordinary answer. That is deliberate: they exist to be counted across runs, and a rate needs to tell "nothing was dropped on this run" apart from "this run predates the counters", which is what an absent field means on a graded facet. They are also absent on a facet that was never graded — a generated prompt shipped for its text carries no rubric and no dimensions, so no cap could have applied. Counting over facets whose dimensions array is non-empty keeps all three out of your denominator. None of the three is read by anything — no score, verdict, lift or overall_score is computed from them, and a dimension that loses its flag to the contradiction guard is graded exactly as it always was.

Telling it what you sent (artifact_type)

Optional, and worth setting when what you sent is not a message to a named person:

Declaring it does two things: the grader is told what it is reading, and the dimensions that kind of artifact cannot satisfy are flagged deterministically. Today only landing_copy excludes any — the other three are accepted so you can state what you sent (and so the run addresses distinctly for reuse), but they exclude nothing until there is a measurement behind doing so.

Exactly which dimensions landing_copy drops, so you can tell an exclusion from a fail:

Everything else still grades, including CTA clarity — a landing page can have a clear ask, so the check is not inapplicable to the kind. Across declared landing_copy runs it passes on 8 of 14 submitted facet sides and 20 of 28 gate-mode grades; a dimension that most of the kind satisfies is a real signal, and dropping it would excuse landing pages that bury their ask.

Every run reports what it resolved to, whether you declared one or not:

"artifact_type": { "value": "landing_copy", "source": "explicit" }

source is explicit when you declared it, default when nothing was established, and inferred when the run's own classifier read one. An inferred value is a label and nothing more — it does not select a rubric, exclude a dimension, or reach the grader. It stays that way until its accuracy has been measured in both directions; the last unmeasured classifier on this pipeline turned out to be refusing 24% of its refusals wrongly.

Use it as a gate

Most of this guide reads as a coach for a person. The same call is also a quality gate an agent runs on its own output before it acts — validate the message against the customer's own conversation history, and only then send it.

research the account
        ↓
generate the message
        ↓
POST /evals/run  ──→  GET /eval-runs/{id}/gate?wait_ms=30000
        ↓                    (poll until status is terminal)
   gate.passed
    ↙          ↘
  pass         fail
   ↓             ↓
  send      regenerate with the improved PROMPT
                  ↓
              re-grade → send

The run is async by design, so the gate is a step in your pipeline with a wait in it, not an inline function call — build it as a state the message sits in, not as a blocking call inside a request handler.

Five rules separate a gate that works from one that quietly does the wrong thing:

• Gate on gate.passed, and never on the improved side. gate.passed is the send/hold bit: did the copy YOU submitted clear its bar. Report checks_passed / checks_total as the numeric — the one fraction on the payload that is about your writing — and read gate.threshold for the bar it was decided against. Do not gate on improvement.transition.improved_verdict: it follows the version the eval WROTE, which is graded against the rubric it was written to and clears the bar on ~92% of runs, so wired as a gate it passes almost everything. Do not gate on run.verdict either — it is a four-way bucket whose meaning moved at eval_version 2.16.0, so a window of stored rows mixes two questions. And do not re-threshold overall_score yourself: that re-implements a decision the eval already made, with worse information.
• not_applicable must not block — and neither does a null passed. A workspace with no customer-conversation data yet returns not_applicable, not fail — there was nothing to ground against. The whole gate block is null in that case (and while the run is still in flight), with not_applicable_reason beside it saying why. gate.passed is itself boolean | null, so a block that IS present can still carry passed: null: the run recorded a checks fraction but no pass/fail on the submitted side. Both nulls read the same way — abstain and pass through, never as passed: false. A guard that only tests for a missing block treats the second one as a fail and holds the send. Blocking on either means a new workspace can never send anything.
• Never auto-send the after message. It carries usage: "illustration_only" — a specimen produced so the score difference could be measured, written by a model that has never met the recipient. The durable artifact is the improved prompt (usage: "reusable_prompt"); on a failed gate, regenerate with it and re-grade rather than shipping the illustration.
• Know which retry you are building. The eval already runs ONE bounded revision internally when its improved side misses the bar, reported as transition.iterations with an explanation when it still could not clear. Your loop's retry is the OUTER one, over a freshly generated message. Two nested unbounded retries is how a gate turns into a spend.
• The reported grade is the FIRST round. transition.reported_round says so, and it is 1 regardless of what iterations says. How many revision rounds fit is partly a function of wall clock, so reporting the best of them made the same submission score higher on a faster host. Reporting the first round makes the number one draw on every host. The later rounds still run — you still get the revision work, and iterations / stop_reason still say how much search happened — they just no longer move the score.

On repeats, reuse: "cached" (the default) returns the last active run for the same inputs, so re-grading an unchanged message costs nothing and returns the same verdict; a changed message is different inputs and grades fresh. If you author your own eval, pass_threshold on an evidence_judge or improvement_loop grader is where you set the bar the gate reads.

A passing verdict means the content is grounded in what your customers actually said. It is not a prediction of reply rate or conversion — gate on it for consistency and defensibility, and say which it is when you report the number.

The eight grader kinds

A case passes only when all of its applicable graders pass. Browse them via grader_kind://list; each carries a uses_llm flag.

• rule — deterministic checks over the message TEXT, no LLM: length bounds (chars or words), must_contain / must_not_contain (e.g. banned hype phrases), and a has_cta call-to-action heuristic. Passes when every check passes. Abstains (not-applicable) on a prompt-only run, when there is no message to check.
  • The built-in prompt-and-message-eval bounds a message at 30–3000 characters, and that upper bound is an outlier guard, not a style target — it is there to catch "this is a document, not an email". We measured length against real reply outcomes on 127,872 labelled outbound emails and the effect reverses sign between workspaces: in some, longer openers reply materially better; in others, slightly worse. So there is no length we can tell you to write to, and a pass on this check is not a claim that your message is well-sized. If you want a house style bar, fork the eval and set your own length check — that is what the tighter values in the authoring example are for.
• improvement_loop — the meaty one, and what prompt-and-message-eval leans on. ONE LLM pass over retrieved customer quotes that grades a BEFORE (your message, or a draft simulated from your prompt), then produces AND grades an improved reusable prompt (a template that uses Amdahl to position for the specific customer — position, verify, keep it relevant, not generic and not just voice) + a relevantly positioned message, and reports the lift + what changed — the full before→after report. Cites verbatim quotes per side. Returns not-applicable when the workspace has no customer data yet, and also when a report cannot be produced (report_unavailable) — it never manufactures a neutral score to stand in for a grade it did not make.
• evidence_judge — the grade-only sibling. Retrieves real customer quotes, then an LLM scores the message against them on each rubric dimension WITH per-dimension reasoning, cites the verbatim quotes (supporting or contradicting), and proposes a grounded rewrite — but no before/after. Same not-applicable semantics.
• deterministic — rule checks over a GENERATED answer's stats (did it answer, in time, without erroring, with enough rows / citations). The generated-mode sibling of rule.
• sor_anchored — a system-of-record anchor. Runs sor_sql (a data.query over your warehouse) for a ground-truth scalar, then compares an answer figure against it within a relative tolerance. Catches silent under-return — an answer that looks fine but omits most of the true population. No LLM.
• judge — an LLM scores an answer 1-5 on each rubric dimension; the case passes when the mean meets the pass_threshold. Degrades to a neutral, non-passing score if the judge is unavailable.
• figure_anchored — re-runs every figure in a report against its OWN system-of-record query and checks each one reconciles to the warehouse within tolerance. No LLM. Makes "every number is a receipt" structural — a figure is ungradeable without the query that reproduces it — and fails the report if any number diverges or its query is refused.
• structure — checks a report is complete and honest against a structured record set: every must-cover record is mentioned, and every record that raises a risk also states a next action. No LLM. Catches the silent omission — a dropped open deal, a flagged risk with no response — that a trustworthy brief cannot make.

prompt-and-message-eval uses rule + improvement_loop. The deterministic / sor_anchored / judge / figure_anchored / structure kinds are available to evals you author. Read the exact cases with eval://<slug>.

Comparing two runs

Every run retrieves its own customer evidence, seeded from what you submitted. So if you edit a draft and re-grade it, you changed two things: the copy, and the quotes it was measured against. The score difference mixes them.

This is not a small effect. On a real pair we measured — the same cold email, edited once — the two runs shared none of their four retrieval queries and only 5 of 12 recorded quotes. The second run scored worse, and none of that difference was attributable to the edit.

Two things fix it:

Pin the evidence. Grade v1 normally, then grade v2 against v1's quotes:

{ "inputs": { "message": "<v2>" }, "evidence_from_run": "<v1 run id>" }

The second run retrieves nothing and reuses the first run's frozen pool, so the only thing that changed is your copy. The response echoes an evidence_pin block with the source run and when those quotes were frozen; the report carries the same under grader_meta.evidence_scope.

Re-running the judge over fixed text

Pinning the evidence holds the RETRIEVAL still, so a difference between two runs is attributable to your copy. It does not hold the WRITER still: each run generates its own improved version, so a difference between two runs of the same submission still mixes the judge with the generator.

candidate_from_run closes that. It grades a prior run's improved version verbatim — no generate call is made at all:

{
  "inputs": { "message": "<the same message>" },
  "candidate_from_run": "<run id>",
  "reuse": "force"
}

Everything the judge reads is now fixed, so a score spread across N such runs is judge variance and nothing else. Three things to know:

• It pins that run's quotes too, automatically. The frozen text's citations (q1, q2, …) address the pool its own run froze, so grading it against a fresh pool would silently re-point every one of them. Pass the same id as evidence_from_run or leave it out; a different id is refused.
• Pass reuse: "force" when sampling. Two identical pinned requests share a content address, so the default cached joins them into one run — which is correct for a re-request and useless for an N-sample measurement.
• The request must match what the candidate was written for. Send different copy, or a different mode, and the run is refused rather than graded: the judge scores your submission and the candidate as a PAIR, so pairing a frozen rewrite with a draft it was not written from reports a difference between two unrelated texts as a lift.

The response echoes a candidate_pin block (source run, when the text was frozen, the mode it was written under), and the report carries grader_meta.candidate_scope. Read it beside evidence_scope: a spread is attributable to the judge only when both say pinned.

Check before you trust a comparison. GET /eval-runs/{id}/compare/{other_id} (eval_run://<id>/compare/<other_id>) returns both verdicts, how much evidence the two runs actually shared, and the deltas only when they were graded on the same basis:

{
  "data": {
    "comparison": {
      "evidence_overlap": {
        "shared": 5,
        "only_a": 7,
        "only_b": 7,
        "jaccard": 0.263,
        "measured": true
      },
      "delta_attributable": false,
      "caveat": "These two runs were graded against DIFFERENT customer evidence …",
      "remedy": "Re-run the second version with evidence_from_run set to … "
    }
  }
}

When delta_attributable is false there is no delta in the response at all. That is deliberate: a number next to a warning still gets quoted, and the warning does not travel with it.

Read submitted_score_delta, not score_delta. An A/B is asking "did MY copy get better". score_delta differences overall_score, whose meaning changed at eval_version 2.14.0 — so on a pair of newer runs it now agrees with the submitted delta, while a pair that SPANS that version differences two different quantities and is not a comparison at all. The endpoint now refuses that pair rather than emitting it (see Comparing across an eval version), and submitted_score_delta is derived from the improvement report rather than from overall_score, so it means one thing on both sides of the line. An attributable comparison also returns:

{
  "a": { "submitted_score_15": 1.8, "submitted_score": 0.2, "input_simulated": false },
  "b": { "submitted_score_15": 5.0, "submitted_score": 1.0, "input_simulated": false },
  "submitted_score_15_delta": 3.2,
  "submitted_score_delta": 0.8
}

submitted_score_15_delta is on the [1, 5] axis your dials and threshold use; submitted_score_delta is the same movement on score_delta's [0, 1] axis. Comparisons of runs graded before this shipped get them too — the numbers are derived when you read, not required to have been written.

Both are also withheld when either run graded no submitted artifact — a subtraction against a missing operand is not a weaker finding, it is not one.

delta_withheld_reason says which control failed — they need different things from you, and only one of them is fixed by pinning evidence:

Comparing across an eval version

Pinning the evidence controls the quotes and candidate_from_run controls the writer. Neither controls the instrument: two runs graded under different eval_versions were read by different judges, or store fields that mean different things. The endpoint reads both rows' eval_version and refuses what it cannot support — and which delta it refuses depends on what the boundary moved:

A payload_meaning bump did not move a single model call, so the two runs were graded by the same instrument — only the FIELD overall_score came to denote something else, which makes score_delta a subtraction of two quantities under one name. submitted_score_delta is derived from the improvement report rather than read off overall_score, so it means the same thing on both sides and is the delta to read; suppressing it would cost you the one measurement that survived. A grading_work or both bump moved the judge, the prompt or the evidence draw, so both numbers are readings of two instruments and the difference between them is partly the instrument.

An unrecorded version — one the ledger has never seen, or two runs of two different evals — is treated as the widest case. That is "cannot say", not "nothing crossed", and the safe direction is to withhold.

Whenever the two versions differ, the crossed transitions ride the payload on version_boundary, and they do so even when delta_withheld_reason says something else (a refused run reports not_scored, which is the plainer fact, and the pair is still not poolable):

{
  "delta_withheld_reason": "eval_version_boundary",
  "version_boundary": {
    "a": "2.27.0",
    "b": "2.28.0",
    "recorded": true,
    "crossed": [{ "at": "2026-08-10", "from": "2.27.0", "to": "2.28.0", "kind": "both" }]
  },
  "caveat": "These two runs were graded under different eval versions (2.27.0 and 2.28.0), crossing 2.27.0 -> 2.28.0 on 2026-08-10 (both). …",
  "remedy": "Re-run the baseline under the same eval_version as the candidate …"
}

recorded: false means the ledger has no entry for one of the two versions. Read crossed[].kind rather than collapsing it to a boolean — it is the only thing that tells a re-meant field from a moved instrument. The caveat names the transitions in prose, capped at three with a count of the rest; the full list is always on crossed.

A run stored before eval_version existed reports null and is not refused on this axis: the test is a mismatch, never an absence, and refusing an unknown would retire every comparison against your own history to prove something not in evidence.

Reuse: cached vs force

An eval run is content-addressed by a fingerprint over (workspace, eval slug, eval version, inputs, the judge + generator models, the pinned evidence run) — everything that fully determines the graded work. Because the inputs are folded in, a different message re-grades and the same message reuses; because the models and the evidence pin are folded in, a model upgrade or a pinned run cannot be served a verdict produced under different conditions.

• reuse: cached (default) — served from an existing run in two cases. In flight, at any age: a run for the same fingerprint that is still queued or running is handed back with reused: true instead of starting a duplicate. Recently completed: a run that reached complete within the last 15 minutes is handed back verbatim, on the reasoning that the customer corpus cannot have moved in that time (the window is a server setting; reused: true and a terminal status on the ack are how you tell). A run that failed or was canceled is never reused. Two simultaneous cached re-requests for the same inputs resolve to one run (an at-most-once dedup guarantee).
• reuse: force — always starts a fresh run, and consults neither path.

Caching is opt-out, not a freshness guarantee, and it is not a pinning mechanism either. Inside the 15-minute window a cached re-request returns the earlier verdict rather than grading against anything that has landed since; outside it, the same request grades fresh, so an unchanged message re-submitted an hour later can legitimately score differently. To guarantee a fresh grade, pass reuse: force and check that reused came back false. Note that the window matches on the fingerprint alone: a run you started with reuse: force is itself reusable by a later cached request for the same content, so if you are sampling the spread with repeated force draws, keep reading with force — a cached read will hand you back one arbitrary draw from your own sample.

The KPI trend

One run is a report card; the KPI is the trend over all of them — "is our writing improving against our own customer evidence, week over week".

GET /api/platform/v1/evals/prompt-and-message-eval/kpi?window_days=90&granularity=week

Also readable as the resource eval://<slug>/kpi, and over MCP as the evals tool's kpi action (which defaults the slug, so { "action": "kpi" } alone answers the trend ask). Scope: evals:read.

The payload carries a series of buckets (day / week / month), current + prior window stats, and delta_15 — the current-vs-prior movement.

Four honesty rules, so the number means what it looks like it means:

• The trend tracks the SUBMITTED side (verdict.headline.submitted), read from the headline rather than from overall_score. Before eval_version 2.14.0 that field blended your draft with the eval's rewrite, so a trend over it would have measured the rewriter; it now reports the submitted side too, but the headline stays the source here because it means the same thing on both sides of that version boundary — and a trend window is exactly the query that spans one.
• Thin windows abstain. A window with fewer than 3 scored runs returns null means and a named abstain_reason (no_runs / thin_window) instead of asserting a mean over noise. Series points always render with their own counts, so a one-run week reads as exactly that.
• Each mean carries the count it was taken over. n_scored is submitted_mean_15's denominator and n_scored_improved is improved_mean_15's. The two differ whenever a run graded one side and not the other, and each window mean abstains below 3 against its own count. The two sides therefore abstain independently, in both directions: the improved-side ceiling can be null in a window where the KPI is asserted, and improved_mean_15 can carry a number in a window where the KPI abstained. abstain_reason answers for the submitted side, which is the KPI. Whenever it is present submitted_mean_15 is null, however improved_mean_15 reads beside it — a window of 2 submitted grades and 4 improved ones returns abstain_reason: "thin_window", submitted_mean_15: null, and an asserted improved_mean_15 over n_scored_improved: 4.
• Refusals count as refusals. A not_applicable run raises refusal_rate but never drags a mean toward zero.

Feed it to a weekly Routine ("report the messaging KPI movement to the team"), an external dashboard over REST, or just open the eval in the console — the band above the compose panel renders the same numbers.

The eval builder

The built-ins not fitting is exactly what the builder is for: configure your own grading pipeline — the input fields it accepts, the cases, and the graders — and store it for your workspace. Every write is validated against the same moat first, so you get author-time feedback instead of a broken run.

A minimal message-style eval:

curl -X POST https://app.amdahl.ai/api/platform/v1/evals \
  -H "X-API-Key: $AMDAHL_KEY" -H "Content-Type: application/json" \
  -d '{
    "slug": "my-cold-open-check",
    "name": "Cold-open check",
    "input_schema": { "fields": [
      { "key": "prompt", "type": "long_text", "label": "Prompt", "required": true },
      { "key": "message", "type": "long_text", "label": "Message", "required": true }
    ]},
    "cases": [{
      "id": "grade",
      "label": "Grade the opener",
      "subject": { "mode": "provided", "input_field": "message" },
      "graders": [
        { "kind": "rule", "id": "shape", "checks": [
          { "type": "length", "min": 20, "max": 90, "unit": "words" },
          { "type": "has_cta" },
          { "type": "must_not_contain", "phrases": ["revolutionary", "game-changer"] }
        ]},
        { "kind": "evidence_judge", "id": "voice", "retrieval": { "seed": "both", "sources": ["clusters"] },
          "rubric": ["grounding", "specificity", "tone_fit"], "pass_threshold": 3.5 }
      ]
    }]
  }'

The moat enforces the structural rules: a kebab-case slug that is not a reserved built-in (prompt-and-message-eval, plus retired slugs kept resolvable), well-formed input fields (including a valid require_at_least_one referencing declared fields), a provided subject whose input_field references a DECLARED field, a generated target that uses search.query, and coherent grader configs (a rule's checks, an evidence_judge's or improvement_loop's rubric + pass_threshold between 1 and 5 + retrieval sources). Anything malformed comes back as { valid: false, errors: [...] } (validate) or a validation_failed envelope (create / update).

retrieval.sources — which corpus a grader reads

Two values are accepted, and they read different things:

Omit sources and you get ["clusters"], which is what every built-in eval uses.

We measured search as a grounding source and do not recommend it. It reaches far more of your corpus — on one workspace the cluster index surfaces 5,090 distinct quotes against 442,893 citable utterances — so it looks like the richer choice. Two paired A/Bs (72 drafts each, same drafts both arms) say otherwise: swapping half the pool for raw utterances moved checkable specifics per rewrite down (8.24 → 7.92) and unsupported specifics up (2.56 → 2.90), and both effects scaled with how much was swapped. The reason is that cluster representative quotes are selected for quality and raw utterances are not, so a bigger pool is also more material to over-read. The harm was largest on the workspace with the biggest corpus — the one where search reaches most.

Use clusters to ground claims. search remains available for a grader that genuinely wants raw breadth over curated relevance, and it is worth knowing that this is the trade you are making.

Browse evals and grader kinds

Everything the run surface consumes is readable, over both the MCP resource schemes and their REST twins (scope evals:read):

Over MCP

Evals ride the Amdahl MCP server's evals coarse tool. run grades content, status reads the result back, drafts returns the two graded texts side by side, improvement returns the before/after report in a bounded shape (the one to use when status warns that the payload is oversized — it is the only narrow read carrying the facets), list_runs / list / get browse, and create / update / delete / validate author your own eval:

{
  "action": "run",
  "eval": "prompt-and-message-eval",
  "inputs": { "prompt": "Write a cold nudge...", "message": "Hi Sam - ..." }
}

run returns the run handle immediately — it never blocks a full grading in one call. Take the run_id from that response and read the verdict back with status:

{
  "action": "status",
  "run_id": "b1c2d3e4-...",
  "wait_ms": 30000
}

wait_ms is optional and capped at 30000: the call blocks until the run finishes, so a grade is a run plus a couple of waiting status calls rather than a poll loop. Omit it for an immediate read.

Every result is data only — no directive text.

Scopes

evals:execute and evals:write are on the MCP customer-agent key bundle (and inherited up the ladder), so a tenant key can run AND author evals without any extra grant; evals:read is on the read-only base, so a viewer / read-only key can browse, validate, and poll but not fire or author.

Not yet

Being honest about the v1 edges:

• v1 grades current data, internal-only. Evidence and answers read your warehouse + cluster themes with no paid web fan-out, and there is no way to move off "now." As-of grading ("how good were our answers as of date X") is a v1.1 capability — it needs the answer/evidence lane to time-travel alongside the system-of-record anchor, otherwise a current answer would be graded against a stale ground truth. The eval_runs.as_of column and the fingerprint's as-of dimension are dormant scaffolding for it (always null in v1).
• Reuse is run-grained. The fingerprint is per run (over the inputs), not per case, and it does not pin to a pipeline generation — so cached is a dedup, not a freshness signal. Per-cell caching and generation-pinned as-of are v1.1 refinements.