Evals API

Evals grade generated content against your workspace's own customer conversations. Not "is this well written" — does it say something these buyers actually care about, and can we prove it? The evidence is verbatim quotes retrieved from your own calls and emails, so what comes back is your customers' words rather than a model's taste.

Two shapes, one call:

• At review time — a person hands over a draft and the prompt behind it, and gets a report card: the lines that are not landing, the quotes that prove it, and a better reusable prompt to keep.
• In a loop — an agent grades what it just generated and reads the verdict before it acts. See Use it as a gate. A run is read-pure by construction, so it is always safe to put in that path.

Why this is not "just prompt the model better." Generating and verifying are different jobs, and here they are done by different passes over evidence the generator does not control. Retrieved quotes are frozen for the whole run, so every candidate within that run is judged against the same evidence. Across two runs they are not: each retrieves its own, seeded from what you submitted. To compare two drafts, pin the second to the first with evidence_from_run — otherwise the score difference mixes your edit with a different set of quotes. For the improvement report the judge then scores both candidates in ONE blinded call — unlabelled, ordered by a content hash — so it cannot tell which one it wrote and cannot be kind to itself. What you get back is an explainable score with citations, not a second opinion from the model that drafted the thing.

One eval ships today, and it needs no setup:

Read Prompt and Message Eval for what it does and how it works. You can also author your own eval (see Evals) — during the current beta, authoring is limited; contact the Amdahl team for access.

Reach for it when you have a drafted message and want it checked - and improved - against how your buyers actually talk, and when you are tuning a prompt or a slice and need a number to move. It is the "backtest a claim against an INDEPENDENT source of truth" discipline, applied to a piece of content.

Two subject modes, one run contract. Every eval case declares a subject in one of two modes: provided (you pass in the content to grade - this is prompt-and-message-eval, and the default) or generated (the eval runs a question through the fast-search lane and grades the answer it produces - the regression-harness mode, available to evals you author).

Three things hold for every run:

• It grades - it never changes your data. A run is READ-PURE by construction: it only ever dispatches read ops (retrieving customer quotes, a system-of-record data.query, running a generated case's question), so firing an eval can never mutate tenant state and is always safe to repeat.
• It is async. evals.run validates the inputs, starts the grading job, and returns a run id immediately. You read the verdict from eval_run://<id> once the run completes - the call never blocks on the grading.
• v1 grades current data, internal-only. Evidence and answers come from your own warehouse and cluster themes with no paid web fan-out (the run's context is clamped to reads). There is no as-of knob in v1: the answer/evidence lane reads current data, so grading it against a past snapshot would compare current content to a stale ground truth. As-of grading is a v1.1 capability.

Run it - grade + improve a message

Send a prompt, a drafted message, or both - at least one. The default eval is prompt-and-message-eval. Message only → it also suggests a reusable prompt; prompt only → it writes a specimen draft to grade; both → your message is the "before."

mode: "rewrite" is the default: the eval returns a full improved prompt and message, each graded against what you sent. The improved prompt is the durable takeaway — a fixed message helps one send, a fixed prompt helps every send after it. Send mode: "advisory" when your prompt is a living document you are not going to replace; the eval then leaves your writing alone and returns anchored suggestions[] against it, and the prompt facet has no after side.

curl -X POST "https://app.amdahl.ai/api/platform/v1/evals/run" \
  -H "X-API-Key: $AMDAHL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "eval": "prompt-and-message-eval",
    "inputs": {
      "prompt": "Write a short cold nudge to a RevOps leader at a mid-market SaaS company.",
      "message": "Hi Sam - noticed your team just doubled headcount. Most RevOps leaders we talk to hit a wall on pipeline hygiene right around that inflection. Worth a quick 15 minutes next week?"
    }
  }'

Inputs are validated against the eval's input_schema before anything runs: a missing required field, a wrong type, an out-of-enum value, or a violated "at least one of" constraint (send prompt-and-message-eval neither a prompt nor a message and it refuses) comes back as one invalid_argument with the per-field problems on details.input_errors. An unknown eval slug is the other invalid_argument shape; past that the run never returns success: false.

A generated eval takes no inputs — it produces the answer it grades. No built-in ships in that mode, so there is nothing to point eval at until you author one (eval://list has the slugs your workspace actually has). prompt-and-message-eval is not one: it is provided, and a run that sends it no prompt and no message is refused with invalid_argument rather than grading anything.

The response is the async handle, not a verdict:

{
  "data": {
    "success": true,
    "run_id": "b1c2d3e4-...",
    "reused": false,
    "status": "queued",
    "eval_slug": "prompt-and-message-eval",
    "eval_version": "2.33.0",
    "resource": "eval-run://b1c2d3e4-..."
  }
}

• run_id / resource are the handle - poll resource (or GET /eval-runs/<run_id>) for the verdict. The scheme on the wire is hyphenated (eval-run://): an underscore is illegal in a URI scheme, so new URL("eval_run://…") throws in a standards-compliant MCP client. Both spellings resolve on every read surface — but read the resource you were handed rather than re-spelling it.
• eval_version rides the run fingerprint, so it is what a cache or a comparison keys off. It moves whenever the eval's grading changes or a stored field comes to mean something else — the current value is 2.28.0, and the value here is the shape, not a constant to hardcode. Scores from two versions are not poolable: comparing two runs refuses a pair that spans a boundary, and an export spanning more than one names them so you can filter.
• reused is true when a still-active run for the same inputs already existed and was handed back instead of starting a new one (see Reuse).
• status is the run's lifecycle state at hand-off: queued, then running, then a terminal complete / failed / canceled.

Poll for the verdict

curl "https://app.amdahl.ai/api/platform/v1/eval-runs/b1c2d3e4-..." \
  -H "X-API-Key: $AMDAHL_KEY"

Read data.run.status; when it is complete, data.run.verdict carries the report card. A missing / cross-tenant id returns null.

The body is { "data": { "status": …, "run": { … } } }. The block below is the data.run.verdict fragment — the report card itself, printed on its own:

{
  "eval_slug": "prompt-and-message-eval",
  "eval_version": "2.33.0",
  "overall_score": 0.4,
  "overall_reasoning": "Your submitted draft did not hold up against the evidence; the improved version does. See the before/after report for what changed. The gaps are grounding, specificity - each one is explained line by line below. Grounded in 2 customer quotes from your data. Your draft cited none; the rewrite cited 2.",
  "verdict": "pass",
  "cases": [
    {
      "case_id": "message-vs-voice",
      "label": "Relevant positioning vs. customer evidence",
      "answered": true,
      "applicable": true,
      "score": 0.9,
      "passed": true,
      "latency_ms": 8340,
      "graders": [
        {
          "grader_id": "basic-hygiene",
          "kind": "rule",
          "score": 1.0,
          "passed": true,
          "rationale": "All 3 rule check(s) passed."
        },
        {
          "grader_id": "grounded-improvement",
          "kind": "improvement_loop",
          "score": 0.8,
          "passed": true,
          "rationale": "Your version passed 2/5 rubric dimensions; the improved version passed 4/5 — each judged pass/fail blind against the same customer quotes (score 4.2/5, bar 4.2/5). Grounded the claims in customer quotes.",
          "improvement": {
            "before": {
              "label": "before",
              "prompt": "Write a short cold nudge to a RevOps leader at a mid-market SaaS company.",
              "prompt_note": "as provided",
              "message": "Hi Sam - noticed your team just doubled headcount. Most RevOps leaders we talk to hit a wall on pipeline hygiene right around that inflection. Worth a quick 15 minutes next week?",
              "message_note": "as provided",
              "score": 0.4,
              "score_15": 2.6,
              "checks_passed": 2,
              "checks_total": 5,
              "dimensions": [
                {
                  "name": "Grounding",
                  "pass": false,
                  "score": 1,
                  "reasoning": "The headcount hook is generic; only the hygiene wall is real."
                },
                {
                  "...": "abridged - 4 of the 5 rubric lines are omitted here; checks_passed / checks_total count all five"
                }
              ],
              "quotes": [
                {
                  "text": "the headcount thing every vendor leads with never lands",
                  "source": "Cold outreach reactions",
                  "stance": "contradicts"
                }
              ],
              "reasoning": "The draft leans on a hook customers say they tune out."
            },
            "after": {
              "label": "after",
              "prompt": "Draft a cold nudge to a RevOps leader at a mid-market SaaS company. Use Amdahl to pull the specific pain points our own customers describe around scaling their sales team, and lead with the one that resonates most.",
              "prompt_note": "improved — reusable",
              "message": "Hi Sam - a few RevOps leads we work with said their pipeline data started breaking down right past ~40 reps. If that rings true, worth 15 minutes on how they got ahead of it?",
              "message_note": "suggested rewrite — not required to send verbatim",
              "score": 0.8,
              "score_15": 4.2,
              "checks_passed": 4,
              "checks_total": 5,
              "dimensions": [
                {
                  "name": "Grounding",
                  "pass": true,
                  "score": 5,
                  "reasoning": "Anchors on the verbatim ~40-rep breaking point customers named."
                },
                {
                  "...": "abridged - 4 of the 5 rubric lines are omitted here; checks_passed / checks_total count all five"
                }
              ],
              "quotes": [
                {
                  "text": "once we grew past ~40 reps our pipeline data just fell apart",
                  "source": "Pipeline hygiene",
                  "stance": "supports"
                }
              ],
              "reasoning": "The rewrite anchors on real customer language and drops the hook the evidence contradicts."
            },
            "lift": 0.4,
            "what_changed": "Grounded the claims in customer quotes."
          },
          "dimensions": [
            {
              "name": "Grounding",
              "pass": true,
              "score": 5,
              "reasoning": "Anchors on the verbatim ~40-rep breaking point customers named."
            },
            {
              "...": "abridged - 4 of the 5 rubric lines are omitted here; the improved side passed 4 of the 5, see improvement.after"
            }
          ],
          "quotes": [
            {
              "text": "once we grew past ~40 reps our pipeline data just fell apart",
              "source": "Pipeline hygiene",
              "stance": "supports"
            }
          ],
          "rewrite": "Hi Sam - a few RevOps leads we work with said their pipeline data started breaking down right past ~40 reps. If that rings true, worth 15 minutes on how they got ahead of it?",
          "evidence": {
            "before_score": 2.6,
            "after_score": 4.2,
            "lift": 0.4,
            "message_simulated": false,
            "retrieved_quotes": 11,
            "retrieval_status": "ok"
          }
        }
      ]
    }
  ],
  "summary": { "total_cases": 1, "passed_cases": 1, "failed_cases": 0, "not_applicable_cases": 0 }
}

Reading it

• overall_score is the grade of the copy you submitted, in [0, 1] — the mean, over applicable cases, of each case's submitted side. It is verdict.headline.submitted.score_15 on the [0, 1] axis instead of [1, 5], read through the same derivation, so the two cannot disagree. ⚠️ Before eval_version 2.14.0 it was the mean of each case's applicable graders, which judge two different artifacts (the hygiene rules read what you submitted, improvement_loop scores the version the eval wrote) — a grade on neither. Check eval_version before quoting it on a stored run, and never average it across runs on different versions; verdict.headline.submitted means the same thing on both sides of that line.
• overall_reasoning is a short plain-language summary of WHY the run landed where it did - assembled deterministically from the graders' own reasoning + rule failures + the cited-quote count. No extra LLM call.
• verdict is the bucket: pass = every applicable case passed, fail = none passed, partial = some, not_applicable = no case could be applied (see below). Derived from the per-case pass count, not from overall_score. ⚠️ Since eval_version 2.16.0 that count reads the SUBMITTED side (input_passed ?? passed), so a pass says your draft cleared the bar; before 2.16.0 it read the improved side, so a stored pass from that window says only that the eval's rewrite did. The two answer different questions — do not pool verdicts across that line, and note the case-level passed in the next bullet still follows the improved side.
• cases[] - one per subject. answered is whether the subject carried gradable content; applicable is whether the case counted toward the verdict. passed follows the IMPROVED side whenever the case ran an improvement grader — it says whether the eval's rewrite cleared the bar, not whether yours did, so a case can read passed: true while the report says your draft is unusable. That is the two reads doing their jobs, not a contradiction. Both reads are on the case beside it: input_passed (what you submitted), improved_passed (what the eval wrote), and transition (pass_to_pass / fail_to_pass / pass_to_fail / fail_to_fail) — read input_passed for a verdict on your own copy, and see Use it as a gate for which one a gate should branch on. A case with no improvement grader keeps the plain "every applicable grader passed" meaning.
• graders[] - one per grader. score is normalized to [0, 1], rationale is the plain-language explanation, and depending on the kind: improvement (the before→after report, below), dimensions (per-rubric 1-5 score + a sentence of reasoning), quotes (the verbatim customer utterances cited, each with a source theme and a supports / contradicts / neutral stance), rewrite (a grounded better version), and evidence (the receipt). Store the evidence with the score; a score without its evidence cannot be audited later.

The before→after report (improvement)

improvement_loop returns one improvement object. before is the message you sent (message_note: "as provided") or the specimen draft written for a prompt-only run ("simulated from your prompt"); after is an improved, REUSABLE prompt plus an illustrative message. lift is after minus before; what_changed is a one-line summary. The AFTER side is ALSO surfaced on the grader's top-level dimensions / quotes / rewrite, so a reader that only knows the older shape still renders.

Alongside those, the report carries:

• facets[] — the before/after pair for the prompt and the message separately. They are different artifacts graded on different rubrics, so they never share a score, reasoning, quotes, or worked examples. Each side carries score_15, its dimensions, its quotes, and its good_examples.
• usage on every graded artifact — the field that says what the thing IS. as_provided (untouched), simulated_specimen (a draft we wrote so there was something to score), reusable_prompt (the takeaway — a template you keep), illustration_only (an example produced so the score difference could be measured — not a message to send).
• audience — a discriminated union on status. resolved names the seniority cohort the run was scoped to and carries the evidence counts behind that decision; abstained carries one of five reason values (not_provided / unresolvable / no_evidence / thin_evidence / lookup_failed) plus a server-authored message. Narrow on status before reading dimensions — an abstained run was graded against the whole corpus, and rendering it as cohort-scoped makes every score under it read wrong. lookup_failed is our check failing, never "you have no data." See Audience scoping.
• tool_kit — { callable, out_of_scope }. The suggested calls are bounded by what YOUR key can run, so a step you would get a 403 for is never proposed; out_of_scope counts what exists beyond your scopes, and 0 means there is nothing to caveat.
• suggestions[] — anchored, surgical edits (keep / add / strengthen / remove / reorder). anchor_quote is verified server-side to be a literal substring of your text; a paraphrase is dropped rather than shown, so a suggestion can never quote a line you did not write.
• transition — the verdict SPLIT. input_verdict and improved_verdict are recorded separately, because your draft failing is the finding you ran the eval to get, not a failure of the eval. explanation is populated only when the improved side missed the bar.
• coverage — how much of a long submitted prompt was actually graded (total_chars / graded_chars / truncated / per-section included). Past 12,000 characters a prompt is sectioned on its own headings; past 250,000 the run refuses rather than grading a sliver.
• research_steps[] — runnable Amdahl calls that would close the evidence gaps found, each validated against the live operation registry (unknown op, write-shaped op, or SQL the query gate refuses → dropped before you see it).
• grader_meta — model_calls, blinded, evidence_quotes, evidence_scope. blinded: true means both candidates were scored in ONE call, unlabelled, with the order derived from a content hash — the judge could not favour its own draft. evidence_scope says what the evidence was held fixed across: {"kind": "run"} (retrieved for this run) or {"kind": "pinned", …} (reused from a prior run, which is what makes two runs comparable). The older evidence_frozen: true meant only the first of those and is deprecated.

Every field above is optional on the wire, so a pre-v2 stored verdict still parses.

Quotes are retrieved, never written by the model

The grader cannot invent a customer quote. Quotes are RETRIEVED from your cluster/theme index first, each tagged with an id; the LLM may only cite those ids per side; the server hydrates the cited ids back to the verbatim text. A quote you see in either side of the report is a real utterance from your data - fabrication is structurally impossible.

Every quote carries a tier, and the tier is what a claim built on it may say. Read it before you attribute anything:

segment is emitted as of eval version 2.6.0. Before that it was a declared value with no producer — a run came back account or corpus only. A resolved audience (or a resolved scope) now draws that cohort's own utterances, spread across companies so one talkative account cannot stand in for the group, and they arrive tagged segment. So the absence of segment quotes is now a statement rather than a gap in the instrument: either you scoped no cohort, or the one you named did not clear the floors. segment_status and segment_quotes on the improvement report say which — see Evidence tiers.

Account-tier evidence exists only when you pass an account AND that company is in your data with buyer-side conversation. Pass one and it is retrieved and tagged; on a cold account — most first-touch outbound — there is nothing to retrieve, so what comes back is cohort evidence and the report says which of not_provided / unresolvable / not_in_corpus / no_quotable_utterances / lookup_failed applied rather than filing corpus quotes under an account heading. not_in_corpus ("no company matched that name") and no_quotable_utterances ("we hold this account, and nothing it said clears the citable band — every utterance is internal, under 40 characters, or over 2,000") split what shipped through eval version 2.7.0 as a single no_conversations — they ask opposite things of you, so branch on them separately; no_conversations is no longer emitted. Branch on the reason; read account_abstain_detail, which carries the specific half — the name that found nothing, or the account's real utterance count. A quote with no tier predates tiering and reads as corpus, the weakest standing.

Reaching past a quote's tier — attributing a corpus quote to the recipient — is graded as a grounding failure, not a style note: naming the wrong scope turns a grounded line into a claim of a conversation that never happened. The worked example applies the rule end to end.

Citing a comparable customer

Pointing at a customer you already did the work for is the ordinary sales move, and until eval version 2.28.0 the eval could not grade it: account was single-valued and did two jobs at once — who the draft is going to AND whose evidence to retrieve — so a message to a prospect came back with every quote at corpus tier, and "11x scales its AI-driven onboarding" was correctly marked as an account-specific claim with no account-tier support.

The two jobs are now two inputs. account is the recipient; reference_accounts names the comparables, and each one's quotes are drawn at account tier tagged to that company:

{
  "inputs": {
    "message": "<draft to a prospect>",
    "account": "Artisan",
    "reference_accounts": "11x, Degreed"
  }
}

The account licence is per company. A quote from 11x backs a claim about 11x and about no other company, including the recipient — so naming the proof point is groundable while a claim about the prospect still needs the prospect's own evidence. Citing one company's account quote to assert something about another is graded as reaching past the tier, exactly like citing corpus for "your situation".

Mechanics, all bounded: up to three names (comma-separated), two quotes each, six across the leg — and the account tier does not grow, so what the comparables take the recipient's own draw gives up, floored so the recipient always keeps at least one slot. Only the comparable's own side is drawn; your outbound to them is not evidence about them.

Every name you type is answered. The account_resolved step carries reference_accounts (canonical names that resolved and contributed quotes), reference_quotes, and reference_outcomes — one entry per name, either ok with the canonical name and its quote + utterance counts, or abstained with a reason: the recipient's own not_in_corpus / no_quotable_utterances / unresolvable / lookup_failed vocabulary plus over_limit (past the third name) and is_recipient (you listed the company you are writing to among its own comparables). Nothing is silently dropped.

A pinned run (evidence_from_run) draws nothing here and must name the same set of comparables the pinned run was drawn under, or it is refused rather than graded — grading anyway would put one company's words behind another company's proof point, at the tier that licenses naming them.

Not-applicable - never a false fail

A case that cannot be graded honestly comes back not_applicable and is bucketed OUT of pass/fail rather than scored zero. A run whose every case is not-applicable has verdict: "not_applicable", and overall_reasoning states which reason applied.

Branch on not_applicable_reason, never assume an empty corpus. Nine values ship — empty_corpus, evidence_unavailable, not_outreach, not_gradeable, prompt_too_large, report_unavailable, no_subject_text, no_decidable_checks, surface_unavailable — and they ask different things of the caller (connect data / re-run / send different content / split the submission). The Evals guide has the full table with what each one means.

not_outreach is the one most likely to surprise: the rubric scores commercial writing at any stage of a relationship, but not correspondence with no persuasive job (scheduling, receipts, support replies, internal notes), even inside an active deal. In every not-applicable case the submission was never assessed, so there is no score to read into it.

A single grader whose system-of-record surface is unavailable only drops that one grader. An empty corpus is a reason to abstain, never to score a message zero.

Use it as a gate

Everything above reads as a coach for a person. The same call is also a quality gate an agent runs on its own output before it acts — validate the message against the customer's own conversation history, and only then send it.

Two pieces exist for exactly this, so a pipeline never has to re-derive the decision out of the coaching payload:

• inputs.mode: "gate" grades ONLY the copy you submitted — same viability gates, same retrieved-and-frozen customer evidence, same binary per-dimension judge — then stops. No rewrite, no improved-side grade, no suggestions. One judge call instead of several, so a gate run is markedly cheaper and faster than a full one.
• GET /eval-runs/{id}/gate is the machine read. It works on every run (gate-mode or full; historical rows are derived on read), takes the same wait_ms long-poll (max 30000) as the detail read, and rides the MCP evals tool as the gate action.

research the account
        ↓
generate the message
        ↓
POST /evals/run  {"inputs": {"message": …, "mode": "gate"}}
        ↓
GET /eval-runs/{id}/gate?wait_ms=30000  ──→  poll until status is terminal
        ↓
   gate.passed
    ↙          ↘
  pass         fail
   ↓             ↓
  send      regenerate, then re-grade → send

Use mode: "gate" when all you need is the send/hold bit. Run a FULL eval when you also want the improved prompt to drive the regeneration — the /gate read still gives you the same decision bit on that run.

The run is async by design: evals.run validates the inputs, starts the job, and hands back an id you poll. So the gate is a step in your pipeline with a wait in it, not an inline function call — build it as a state your message sits in, not as a blocking call inside a request handler.

Four rules make the difference between a gate that works and one that quietly does the wrong thing:

• Gate on gate.passed, and never on the improved side. gate.passed is the send/hold bit: did the copy YOU submitted clear its bar. Report checks_passed / checks_total as the numeric — the one fraction on the payload that is about your writing — and read gate.threshold for the bar it was decided against. Do not gate on improvement.transition.improved_verdict: it follows the version the eval WROTE, which is graded against the rubric it was written to and clears the bar on ~92% of runs, so wired as a gate it passes almost everything. run.verdict and overall_score are a different mistake — both describe the SUBMITTED side on current runs, but both were re-meant on a stored window (verdict at eval_version 2.16.0, where the bucket moved off the improved side; overall_score at 2.14.0 — see Reading it), so a gate over stored rows mixes two questions, and verdict is a four-way bucket rather than a send/hold bit. Do not threshold lift either: it mostly measures how low your draft scored, not how good the copy is. The Evals guide carries the measurements behind that.
• not_applicable must not block. A workspace with no customer-conversation data yet returns not_applicable, not fail — there was nothing to ground against. gate is null in that case (and while the run is still in flight), with not_applicable_reason beside it saying why. Treat a refusal as abstain and pass through; blocking on it means a new workspace can never send anything.
• Never auto-send the after message. On a FULL run it carries usage: "illustration_only" — a specimen produced so the score difference could be measured, written by a model that has never met the recipient. The durable artifact is the improved prompt (usage: "reusable_prompt"). On a failed gate, regenerate with that prompt and re-grade; do not ship the illustration. A mode: "gate" run writes neither, so there is nothing to mis-send.
• Know which retry you are building. A FULL run already performs ONE bounded revision internally when its own improved side misses the bar, reported as transition.iterations with an explanation when it still could not clear (a gate-mode run does none of this — it grades once and stops). Your loop's retry is the OUTER one, over a freshly generated message. Two nested unbounded retries is how a gate turns into a spend.

On repeats: reuse: "cached" (the default) returns the last active run for the same inputs, so re-grading an unchanged message costs nothing and returns the same verdict. A changed message is different inputs and grades fresh. Pass reuse: "force" only when you want a new grade of identical content.

If you author your own eval, pass_threshold on an evidence_judge or improvement_loop grader is where you set the bar the gate reads.

The eight grader kinds

A case passes only when all of its applicable graders pass. Browse them via grader_kind://list; each carries a uses_llm flag.

• rule - deterministic checks over the message TEXT, no LLM: length bounds, must_contain / must_not_contain (e.g. banned hype phrases), and a has_cta heuristic. Abstains (not-applicable) on a prompt-only run.
• improvement_loop - the meaty one, and what prompt-and-message-eval leans on. Five stages: extract what the sender is TRYING to do; fan out several concurrent retrieval passes (one per intent seed, one per specific claim) and FREEZE the resulting evidence for the whole run; generate the improvement (this pass scores nothing); grade both candidates in one BLINDED, PAIRED call so the judge cannot favour the draft it just wrote; and revise once if the improvement missed the bar. mode: "rewrite" (the default) replaces the prompt you sent; mode: "advisory" returns anchored suggestions against it instead. Returns not-applicable on an empty corpus, and also when a report cannot be produced (report_unavailable) — it never manufactures a neutral score to stand in for a grade it did not make.
• evidence_judge - the grade-only sibling. Retrieves real customer quotes, then an LLM scores the message against them on each rubric dimension WITH reasoning, cites the verbatim quotes, and proposes a grounded rewrite - but no before/after. Same not-applicable semantics.
• deterministic - rule checks over a GENERATED answer's stats (did it answer, in time, without erroring, with enough rows / citations). The generated-mode sibling of rule.
• sor_anchored - runs a data.query for a ground-truth scalar, then compares an answer figure against it within a relative tolerance. Catches silent under-return / truncation. No LLM.
• judge - an LLM scores an answer 1-5 on each rubric dimension; the case passes when the mean meets the threshold. Degrades to a neutral, non-passing score if the judge is unavailable.
• figure_anchored - re-runs every figure in a report against its OWN system-of-record query and checks each reconciles to the warehouse within tolerance. No LLM. Makes "every number is a receipt" structural, and fails the report if any number diverges or its query is refused.
• structure - checks a report is complete and honest against a structured record set: every must-cover record is mentioned, and every record that raises a risk also states a next action. No LLM. Catches the silent omission a trustworthy brief cannot make.

prompt-and-message-eval uses rule + improvement_loop. The other six kinds (deterministic, sor_anchored, judge, evidence_judge, figure_anchored, structure) are available to tenant-authored evals. Read the exact cases of any eval with eval://<slug>.

Reuse: cached vs force

A run is content-addressed by a fingerprint over (workspace, eval slug, eval version, inputs, the judge + generator models, the pinned evidence run) - everything that fully determines the graded work. Because the inputs are folded in, a different message re-grades and the same message reuses; because the models and the evidence pin are folded in, a model upgrade or a pinned run is never served a verdict produced under different conditions:

• reuse: cached (default) - served from an existing run in two cases. In flight, at any age: a run for the same fingerprint that is still queued or running is handed back with reused: true instead of starting a duplicate. Recently completed: a run that reached complete within the last 15 minutes is handed back verbatim, on the reasoning that the customer corpus cannot have moved in that time (the window is a server setting; reused: true and a terminal status on the ack are how you tell). A run that failed or was canceled is never reused. Two simultaneous cached re-requests for the same inputs resolve to one run (an at-most-once dedup).
• reuse: force - always starts a fresh run, and consults neither path.

Caching is opt-out, not a freshness guarantee, and it is not a pinning mechanism either. Inside the 15-minute window a cached re-request returns the earlier verdict rather than grading against anything that has landed since; outside it, the same request grades fresh, so an unchanged message re-submitted an hour later can legitimately score differently. To guarantee a fresh grade, pass reuse: force and check that reused came back false. Note that the window matches on the fingerprint alone: a run you started with reuse: force is itself reusable by a later cached request for the same content, so if you are sampling the spread with repeated force draws, keep reading with force - a cached read will hand you back one arbitrary draw from your own sample.

Comparing two runs

Every run retrieves its own evidence, seeded from what you submitted. Edit a draft and re-grade it and you changed two things: the copy, and the quotes it was measured against.

Pin the second run to the first so only the copy moved:

{ "inputs": { "message": "<v2>" }, "evidence_from_run": "<v1 run id>" }

And check any comparison before you act on it:

curl https://app.amdahl.ai/api/platform/v1/eval-runs/$A/compare/$B \
  -H "X-API-Key: $AMDAHL_KEY"

It returns both verdicts, evidence_overlap (shared / only-a / only-b / jaccard), and a score_delta only when the two were graded on the same basis. Otherwise it returns delta_attributable: false, a caveat saying what was not controlled, a typed delta_withheld_reason, and a remedy naming the fix — and no delta at all. Needs only evals:read. Full walkthrough: Comparing two versions.

Pinning the evidence does not control the instrument. Two runs graded under different eval_versions were read by different judges, or store fields that mean different things, so the endpoint reads both rows' version and withholds with delta_withheld_reason: "eval_version_boundary". Which delta it withholds depends on what the boundary moved, and version_boundary carries the crossed transitions ({ at, from, to, kind }) so you can tell:

• payload_meaning only — the graded work did not move; only overall_score came to denote something else. score_delta is withheld and delta_attributable stays true, because submitted_score_delta is derived from the improvement report rather than read off overall_score and means the same thing on both sides. Read that one.
• grading_work, or both — the judge, the prompt or the evidence draw moved, so both numbers are readings of two instruments. Attribution is blocked and neither delta ships. The 2.27.0 → 2.28.0 transition is both.
• A version the ledger has no record of (including two runs of two different evals) — recorded: false, treated as the widest case. That is "cannot say", not "nothing crossed".

version_boundary rides the payload whenever the two versions differ, even when delta_withheld_reason reports something else — the pair is still not poolable. A run stored before eval_version existed reports null and is not refused on this axis: the test is a mismatch, never an absence.

Author your own eval

The built-ins not fitting is what the builder is for: configure your own grading pipeline - the input fields it accepts, the cases, and the graders - and store it for your workspace. Every write goes through the same moat first, so you get author-time feedback instead of a broken run.

The moat enforces the structural rules: a kebab-case slug that is not a reserved built-in (prompt-and-message-eval, or its retired outreach-eval / message-grader aliases), well-formed input fields (including a valid require_at_least_one referencing declared fields), a provided subject whose input_field references a DECLARED field, a generated target that uses search.query, and coherent grader configs (an improvement_loop / evidence_judge rubric + pass_threshold 1-5 + retrieval sources). Anything malformed comes back as { valid: false, errors: [...] } (validate) or a validation_failed envelope (create / update). Full walkthrough in the Evals guide.

Browse evals and grader kinds

Everything the run surface consumes is readable over both the MCP resource schemes and their REST twins (scope evals:read):

• eval://list (GET /evals) - every eval available to your workspace (built-ins + your authored ones), with input_schema on the summary.
• eval://<slug> (GET /evals/:slug) - one eval with its input_schema and full cases (each case subject + its grader specs).
• eval_run://list (GET /eval-runs) - this workspace's runs, newest first (filter ?eval_slug= / ?status=).
• grader_kind://list (GET /grader-kinds) - the eight grader kinds, each with an uses_llm flag.

Export runs

Pull many graded runs at once - for a spreadsheet, a notebook, or an analysis script. Three doors over one filter (scope evals:read):

Filter by eval (default prompt-and-message-eval), eval_versions, from / to on created_at, statuses, verdicts, and limit. The file doors take the same filters as query params.

Ask before you pull. count_only (or GET /evals/export/count) answers over the identical filter, so the number you see is the file you get:

curl "https://app.amdahl.ai/api/platform/v1/evals/export/count?from=2026-07-01" \
  -H "X-API-Key: $AMDAHL_KEY"

{
  "data": {
    "eval_slug": "prompt-and-message-eval",
    "scope": "self",
    "count": {
      "total": 412,
      "truncated": false,
      "max_rows": 5000,
      "versions": ["2.12.0", "2.17.0", "2.19.0"],
      "first_run_at": "2026-07-01T09:14:22.108Z",
      "last_run_at": "2026-08-05T16:02:51.774Z",
      "verdict_counts": {
        "buckets": { "pass": 240, "partial": 88, "fail": 51, "not_applicable": 0 },
        "ungraded": 33
      }
    }
  }
}

versions is complete, never a sample - it is paged, so a version that only appears deep in the set still shows up.

What a row promises

Every number that needs a qualifier ships beside it, because the flat row is the shape most likely to be read by someone who will never open the nested verdict:

cases_total says how many cases the run graded; the detail columns describe the first.

Size

The file doors have no row cap. GET /evals/export.csv and .jsonl page through the match set with a cursor and write each page to the socket as it arrives, so a 40,000-run export costs the server the same memory as a 400-run one. Your workspace's own runs are yours; there is no threshold at which you stop being allowed to download them.

They also compress. Send Accept-Encoding: gzip - every browser and HTTP library does by default - and the body arrives gzipped with Content-Encoding: gzip. A verdict is JSON, so this is roughly a tenth of the bytes; your client decompresses it transparently and saves the file under its normal name. curl needs --compressed to do the same.

curl --compressed -OJ "https://app.amdahl.ai/api/platform/v1/evals/export.jsonl?from=2026-01-01" \
  -H "X-API-Key: $AMDAHL_KEY"

POST /evals/export does cap at 5,000 rows, because it holds every row in one JSON response body rather than streaming. Over that it is refused, not cut - a partial payload is indistinguishable from the population once it is in a spreadsheet:

{
  "error": {
    "code": "over_cap",
    "message": "8,431 runs match; this response returns at most 5000. Narrow the date range, set limit to take the most recent N, or download the file (GET /evals/export.csv), which has no row cap.",
    "details": { "total": 8431, "max_rows": 5000 }
  }
}

Three remedies, and the last is usually the right one: narrow the range, set limit, or take the file.

limit is not truncation

limit takes the most recent N runs. It is a deliberate top-N, so it never needs allow_truncation and is never refused for matching a larger set - the distinction being that truncation is the export cutting a set you asked for in full, while a limit is you asking for a slice. It applies to every door; on the file doors it is the only thing that bounds the row set at all.

allow_truncation remains for the JSON op: it accepts the most recent 5,000 of an over-cap set deliberately.

Scope

You export your own runs; a workspace admin exports the workspace. Pass redact_quotes on the JSONL door to blank the verbatim customer quotes (they are emptied, never removed, so the absence is legible); CSV never carries quote text at all.

The one-a-minute limit

File downloads are limited to one per minute per person. The count is not - a UI calls it on every filter change.

A 429 tells you when it lifts:

{
  "error": {
    "code": "rate_limited",
    "message": "Export rate-limited. Try again in 43 seconds.",
    "details": { "retry_at": "2026-08-07T19:04:12.000Z", "retry_after_seconds": 43 }
  }
}

Use retry_at, not the seconds, if you are drawing a timer. It is an absolute instant, so it survives a page reload and a different server answering the next request; a countdown cannot be reconstructed after either. GET /evals/export/count reports the same cooldown object (or null), so a dialog can show the correct remaining time the moment it opens.

Export runs as training data

GET /evals/export.jsonl?format=pairs|sft|judge reshapes each line into a training record instead of the stored verdict.

curl --compressed -OJ \
  "https://app.amdahl.ai/api/platform/v1/evals/export.jsonl?format=pairs&eval_versions=2.25.0" \
  -H "Authorization: Bearer $AMDAHL_KEY"

Read this before you train on it

Both sides are model-generated, and the same rubric wrote and graded them. The improved copy was written by Claude under the eval's own playbook, and the judge that scored it composes that same evidence standard. So these are distillation examples - a smaller model learning to imitate this pipeline - and they are not:

• ground truth for what a human expert would have written, or
• evidence that the improved copy performs better with real buyers.

Every record states this on its own face:

{
  "format": "pairs",
  "chosen": "Dana, your team flagged SSO as the blocker last quarter…",
  "rejected": "Hi Dana, saw your rollout - wanted to connect…",
  "provenance": {
    "both_sides_model_generated": true,
    "rejected_is_human_written": true,
    "eval_version": "2.33.0",
    "lift": 1.6,
    "lift_reportable": true,
    "run_id": "…",
    "generator": "amdahl-eval"
  }
}

It is on every record rather than in a header because training pipelines shuffle, split into train/val, and concatenate with other corpora - a header is lost on the first of those, and the record then travels with no statement of what it is.

One instrument version per file

A training file pins one eval_version, and a filter spanning more is refused with the versions named so you can pick:

{
  "error": {
    "code": "mixed_eval_versions",
    "message": "This filter spans 2 instrument versions, and a training file must pin one. Re-run with eval_versions=<one of: 2.12.0, 2.20.0>.",
    "details": { "versions": ["2.12.0", "2.20.0"] }
  }
}

The rubric and the generation prompt both changed across versions, so "chosen" does not mean the same thing in an early version as in the current one. A mixed file trains toward the average of two standards while looking like one dataset. The CSV and raw JSONL doors are not pinned - they carry eval_version on every row, so you can group by it, which a loss function cannot.

Runs that produce no record

The file is smaller than your run count, by design. A row is left out when we cannot honestly label it:

sft and judge are looser than pairs on purpose: sft asserts "this is good output" rather than "this is better than that", so the noise floor does not bear on it, and judge wants low-scoring artifacts - a grader trained only on passes has never seen a fail.

Prerequisites

Scopes: evals:execute to run and evals:write to author (both editor-tier, on the customer-agent key bundle), evals:read to browse, validate, poll, and export (a viewer-tier scope, on the read-only bundle) - so a read-only key can browse, validate, poll, and export but not fire a run or author.

A prompt to hand an agent

Grade + improve this outbound message for relevant positioning against our
customer evidence - use the evals tool.

Run the prompt-and-message-eval eval with the prompt and the drafted message as
inputs (reuse: force), then poll the resource the run handed back
(eval-run://<id>) until it is complete.

Report the overall score and the pass/partial/fail verdict, then the
improvement_loop result: the before→after report - the before score and
reasoning, the improved reusable prompt and grounded message (note it is a
suggestion), the after score, every cited customer quote (with its
supports/contradicts stance), and the score lift + what changed.

See also

• Evals guide - the full consumer walkthrough (subject modes, the verdict shape, retrieve-then-cite, not-applicable, reuse semantics, the eval builder).
• Comparing two versions - pinning evidence with evidence_from_run, and the overlap check that tells you whether a score difference is attributable to your edit.
• Search - the lane a generated eval runs its questions through.
• Endpoints overview - the whole surface and prerequisites.