Evals

Evals grade generated content against your workspace's own customer conversations. Not "is this well written" — does it say something these buyers actually care about, and can we prove it? The evidence is verbatim quotes retrieved from your own calls and emails, so what comes back is your customers' words rather than a model's taste.

Two shapes, one call:

• At review time — a person hands over a draft and the prompt behind it, and gets a report card: the lines that are not landing, the quotes that prove it, and a better reusable prompt to keep.
• In a loop — an agent grades what it just generated and reads the verdict before it acts. See Use it as a gate. A run is read-pure by construction, so it is always safe to put in that path.

Why this is not "just prompt the model better." Generating and verifying are different jobs, and here they are done by different passes over evidence the generator does not control. Retrieved quotes are frozen for the whole run, so every candidate is judged against the same evidence. For the improvement report the judge then scores both candidates in ONE blinded call — unlabelled, ordered by a content hash — so it cannot tell which one it wrote and cannot be kind to itself. What you get back is an explainable score with citations, not a second opinion from the model that drafted the thing.

One eval ships today, and it needs no setup:

Read Prompt and Message Eval for what it does and how it works. You can also author your own eval (see Evals) — during the current beta, authoring is limited; contact the Amdahl team for access.

Reach for it when you have a drafted message and want it checked - and improved - against how your buyers actually talk, and when you are tuning a prompt or a slice and need a number to move. It is the "backtest a claim against an INDEPENDENT source of truth" discipline, applied to a piece of content.

Two subject modes, one run contract. Every eval case declares a subject in one of two modes: provided (you pass in the content to grade - this is prompt-and-message-eval, and the default) or generated (the eval runs a question through the fast-search lane and grades the answer it produces - the regression-harness mode, available to evals you author).

Three things hold for every run:

• It grades - it never changes your data. A run is READ-PURE by construction: it only ever dispatches read ops (retrieving customer quotes, a system-of-record data.query, running a generated case's question), so firing an eval can never mutate tenant state and is always safe to repeat.
• It is async. evals.run validates the inputs, starts the grading job, and returns a run id immediately. You read the verdict from eval_run://<id> once the run completes - the call never blocks on the grading.
• v1 grades current data, internal-only. Evidence and answers come from your own warehouse and cluster themes with no paid web fan-out (the run's context is clamped to reads). There is no as-of knob in v1: the answer/evidence lane reads current data, so grading it against a past snapshot would compare current content to a stale ground truth. As-of grading is a v1.1 capability.

Run it - grade + improve a message

Send a prompt, a drafted message, or both - at least one. The default eval is prompt-and-message-eval. Message only → it also suggests a reusable prompt; prompt only → it writes a specimen draft to grade; both → your message is the "before."

The improved prompt is the durable takeaway. The improved message is an example produced so the score difference could be measured (usage: "illustration_only") — evidence that the prompt is better, not a message to send. Send mode: "advisory" when your prompt is a large living document you are not going to replace, and you get anchored suggestions against it instead of a rewrite.

curl -X POST "https://app.amdahl.co/api/platform/v1/evals/run" \
  -H "X-API-Key: $AMDAHL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "eval": "prompt-and-message-eval",
    "inputs": {
      "prompt": "Write a short cold nudge to a RevOps leader at a mid-market SaaS company.",
      "message": "Hi Sam - noticed your team just doubled headcount. Most RevOps leaders we talk to hit a wall on pipeline hygiene right around that inflection. Worth a quick 15 minutes next week?"
    }
  }'

Inputs are validated against the eval's input_schema before anything runs: a missing required field, a wrong type, an out-of-enum value, or a violated "at least one of" constraint (send prompt-and-message-eval neither a prompt nor a message and it refuses) comes back as one invalid_argument with the per-field problems on details.input_errors. An unknown eval slug is the other invalid_argument shape; past that the run never returns success: false.

Point eval at a generated eval (with no inputs) to grade an answer it produces itself:

{ "action": "run", "eval": "prompt-and-message-eval" }

The response is the async handle, not a verdict:

{
  "success": true,
  "run_id": "b1c2d3e4-...",
  "reused": false,
  "status": "queued",
  "eval_slug": "prompt-and-message-eval",
  "eval_version": "1.2.0",
  "resource": "eval_run://b1c2d3e4-..."
}

• run_id / resource are the handle - poll resource (or GET /eval-runs/<run_id>) for the verdict.
• reused is true when a still-active run for the same inputs already existed and was handed back instead of starting a new one (see Reuse).
• status is the run's lifecycle state at hand-off: queued, then running, then a terminal complete / failed / canceled.

Poll for the verdict

curl "https://app.amdahl.co/api/platform/v1/eval-runs/b1c2d3e4-..." \
  -H "X-API-Key: $AMDAHL_KEY"

Read run.status; when it is complete, run.verdict carries the report card. A missing / cross-tenant id returns null.

{
  "eval_slug": "prompt-and-message-eval",
  "eval_version": "1.2.0",
  "overall_score": 0.75,
  "overall_reasoning": "The improved version lifts grounding and specificity. Before 2/5 → after 4/5. Grounded against 2 customer quotes from your data.",
  "verdict": "pass",
  "cases": [
    {
      "case_id": "message-vs-voice",
      "label": "Relevant positioning vs. customer evidence",
      "answered": true,
      "applicable": true,
      "score": 0.75,
      "passed": true,
      "latency_ms": 8340,
      "graders": [
        { "grader_id": "basic-hygiene", "kind": "rule", "score": 1.0, "passed": true, "rationale": "All 3 rule check(s) passed." },
        {
          "grader_id": "grounded-improvement",
          "kind": "improvement_loop",
          "score": 0.75,
          "passed": true,
          "rationale": "Before 2/5 → after 4/5 (threshold 3.5). Grounded the claims in customer quotes.",
          "improvement": {
            "before": {
              "label": "before",
              "prompt": "Write a short cold nudge to a RevOps leader at a mid-market SaaS company.",
              "prompt_note": "as provided",
              "message": "Hi Sam - noticed your team just doubled headcount. Most RevOps leaders we talk to hit a wall on pipeline hygiene right around that inflection. Worth a quick 15 minutes next week?",
              "message_note": "as provided",
              "score": 0.25,
              "dimensions": [{ "name": "Grounding", "score": 2, "reasoning": "The headcount hook is generic; only the hygiene wall is real." }],
              "quotes": [{ "text": "the headcount thing every vendor leads with never lands", "source": "Cold outreach reactions", "stance": "contradicts" }],
              "reasoning": "The draft leans on a hook customers say they tune out."
            },
            "after": {
              "label": "after",
              "prompt": "Draft a cold nudge to a RevOps leader at a mid-market SaaS company. Use Amdahl to pull the specific pain points our own customers describe around scaling their sales team, and lead with the one that resonates most.",
              "prompt_note": "improved — reusable",
              "message": "Hi Sam - a few RevOps leads we work with said their pipeline data started breaking down right past ~40 reps. If that rings true, worth 15 minutes on how they got ahead of it?",
              "message_note": "suggested rewrite — not required to send verbatim",
              "score": 0.75,
              "dimensions": [{ "name": "Grounding", "score": 4, "reasoning": "Anchors on the verbatim ~40-rep breaking point customers named." }],
              "quotes": [{ "text": "once we grew past ~40 reps our pipeline data just fell apart", "source": "Pipeline hygiene", "stance": "supports" }],
              "reasoning": "The rewrite anchors on real customer language and drops the hook the evidence contradicts."
            },
            "lift": 0.5,
            "what_changed": "Grounded the claims in customer quotes."
          },
          "dimensions": [{ "name": "Grounding", "score": 4, "reasoning": "Anchors on the verbatim ~40-rep breaking point customers named." }],
          "quotes": [{ "text": "once we grew past ~40 reps our pipeline data just fell apart", "source": "Pipeline hygiene", "stance": "supports" }],
          "rewrite": "Hi Sam - a few RevOps leads we work with said their pipeline data started breaking down right past ~40 reps. If that rings true, worth 15 minutes on how they got ahead of it?",
          "evidence": { "before_score": 2, "after_score": 4, "lift": 0.5, "message_simulated": false, "retrieved_quotes": 11, "retrieval_status": "ok" }
        }
      ]
    }
  ],
  "summary": { "total_cases": 1, "passed_cases": 1, "failed_cases": 0, "not_applicable_cases": 0 }
}

Reading it

• overall_score is the mean case score in [0, 1] over applicable cases; each case score is the mean of its applicable graders' scores. For improvement_loop, the case score is the AFTER side's score.
• overall_reasoning is a short plain-language summary of WHY the run landed where it did - assembled deterministically from the graders' own reasoning + rule failures + the cited-quote count. No extra LLM call.
• verdict is the bucket: pass = every applicable case passed, fail = none passed, partial = some, not_applicable = no case could be applied (see below). Derived from the per-case pass count, not from overall_score.
• cases[] - one per subject. answered is whether the subject carried gradable content; applicable is whether the case counted toward the verdict; passed is whether every applicable grader passed.
• graders[] - one per grader. score is normalized to [0, 1], rationale is the plain-language explanation, and depending on the kind: improvement (the before→after report, below), dimensions (per-rubric 1-5 score + a sentence of reasoning), quotes (the verbatim customer utterances cited, each with a source theme and a supports / contradicts / neutral stance), rewrite (a grounded better version), and evidence (the receipt). Store the evidence with the score; a score without its evidence cannot be audited later.

The before→after report (improvement)

improvement_loop returns one improvement object. before is the message you sent (message_note: "as provided") or the specimen draft written for a prompt-only run ("simulated from your prompt"); after is an improved, REUSABLE prompt plus an illustrative message. lift is after minus before; what_changed is a one-line summary. The AFTER side is ALSO surfaced on the grader's top-level dimensions / quotes / rewrite, so a reader that only knows the older shape still renders.

Alongside those, the report carries:

• facets[] — the before/after pair for the prompt and the message separately. They are different artifacts graded on different rubrics, so they never share a score, reasoning, quotes, or worked examples. Each side carries score_15, its dimensions, its quotes, and its good_examples.
• usage on every graded artifact — the field that says what the thing IS. as_provided (untouched), simulated_specimen (a draft we wrote so there was something to score), reusable_prompt (the takeaway — a template you keep), illustration_only (an example produced so the score difference could be measured — not a message to send).
• suggestions[] — anchored, surgical edits (keep / add / strengthen / remove / reorder). anchor_quote is verified server-side to be a literal substring of your text; a paraphrase is dropped rather than shown, so a suggestion can never quote a line you did not write.
• transition — the verdict SPLIT. input_verdict and improved_verdict are recorded separately, because your draft failing is the finding you ran the eval to get, not a failure of the eval. explanation is populated only when the improved side missed the bar.
• coverage — how much of a long submitted prompt was actually graded (total_chars / graded_chars / truncated / per-section included). Past 12,000 characters a prompt is sectioned on its own headings; past 250,000 the run refuses rather than grading a sliver.
• research_steps[] — runnable Amdahl calls that would close the evidence gaps found, each validated against the live operation registry (unknown op, write-shaped op, or SQL the query gate refuses → dropped before you see it).
• grader_meta — model_calls, blinded, evidence_quotes, evidence_frozen. blinded: true means both candidates were scored in ONE call, unlabelled, with the order derived from a content hash — the judge could not favour its own draft.

Every field above is optional on the wire, so a pre-v2 stored verdict still parses.

Quotes are retrieved, never written by the model

The grader cannot invent a customer quote. Quotes are RETRIEVED from your cluster/theme index first, each tagged with an id; the LLM may only cite those ids per side; the server hydrates the cited ids back to the verbatim text. A quote you see in either side of the report is a real utterance from your data - fabrication is structurally impossible.

They are cohort quotes, not the recipient's. Retrieval searches your whole corpus for the themes the message is about; it does not filter to the account being written to, and on a cold account it could not. So the evidence supports "the VPs of Engineering we talk to say…" and never "your engineers said…" - naming the wrong scope turns a grounded line into a claim of a conversation that never happened. The worked example applies the rule end to end, including what changes once an account is in your data.

Not-applicable - never a false fail

If your workspace has no customer-conversation data yet, the grounding grader cannot ground anything. Rather than fail the message, that case comes back not_applicable (not_applicable_reason: "empty_corpus") and is bucketed OUT of pass/fail. A run whose every case is not-applicable has verdict: "not_applicable" and an honest overall_reasoning. A single grader whose system-of-record surface is unavailable only drops that one grader. An empty corpus is a reason to abstain, never to score a message zero.

Use it as a gate

Everything above reads as a coach for a person. The same call is also a quality gate an agent runs on its own output before it acts — validate the message against the customer's own conversation history, and only then send it.

research the account
        ↓
generate the message
        ↓
POST /evals/run  ──→  poll eval_run://<id>
        ↓
   improved_verdict
    ↙          ↘
  pass         fail
   ↓             ↓
  send      regenerate with the improved PROMPT
                  ↓
              re-grade → send

The run is async by design: evals.run validates the inputs, starts the job, and hands back an id you poll. So the gate is a step in your pipeline with a wait in it, not an inline function call — build it as a state your message sits in, not as a blocking call inside a request handler.

Four rules make the difference between a gate that works and one that quietly does the wrong thing:

• Gate on a verdict, not on a score. run.verdict is derived from the per-case pass count, and improvement.transition.improved_verdict is the one you want for prompt-and-message-eval — it is recorded separately from input_verdict precisely so "the draft I asked you to diagnose was bad" never reads as "the gate failed." transition.threshold tells you the bar that verdict was measured against. Thresholding overall_score yourself re-implements a decision the eval already made, with worse information.
• not_applicable must not block. A workspace with no customer-conversation data yet returns not_applicable, not fail — there was nothing to ground against. Treat it as abstain and pass through; blocking on it means a new workspace can never send anything.
• Never auto-send the after message. It carries usage: "illustration_only" — a specimen produced so the score difference could be measured, written by a model that has never met the recipient. The durable artifact is the improved prompt (usage: "reusable_prompt"). On a failed gate, regenerate with that prompt and re-grade; do not ship the illustration.
• Know which retry you are building. The eval already runs ONE bounded revision internally when its own improved side misses the bar, and reports it as transition.iterations with an explanation when it still could not clear. Your loop's retry is the OUTER one, over a freshly generated message. Two nested unbounded retries is how a gate turns into a spend.

On repeats: reuse: "cached" (the default) returns the last active run for the same inputs, so re-grading an unchanged message costs nothing and returns the same verdict. A changed message is different inputs and grades fresh. Pass reuse: "force" only when you want a new grade of identical content.

If you author your own eval, pass_threshold on an evidence_judge or improvement_loop grader is where you set the bar the gate reads.

The six grader kinds

A case passes only when all of its applicable graders pass. Browse them via grader_kind://list; each carries a uses_llm flag.

• rule - deterministic checks over the message TEXT, no LLM: length bounds, must_contain / must_not_contain (e.g. banned hype phrases), and a has_cta heuristic. Abstains (not-applicable) on a prompt-only run.
• improvement_loop - the meaty one, and what prompt-and-message-eval leans on. Five stages: extract what the sender is TRYING to do; fan out several concurrent retrieval passes (one per intent seed, one per specific claim) and FREEZE the resulting evidence for the whole run; generate the improvement (this pass scores nothing); grade both candidates in one BLINDED, PAIRED call so the judge cannot favour the draft it just wrote; and revise once if the improvement missed the bar. Set mode: "advisory" to get anchored suggestions against the prompt you already have instead of a rewrite. Returns not-applicable on an empty corpus; degrades to a neutral, non-passing score if the model is unavailable.
• evidence_judge - the grade-only sibling. Retrieves real customer quotes, then an LLM scores the message against them on each rubric dimension WITH reasoning, cites the verbatim quotes, and proposes a grounded rewrite - but no before/after. Same not-applicable + degrade semantics.
• deterministic - rule checks over a GENERATED answer's stats (did it answer, in time, without erroring, with enough rows / citations). The generated-mode sibling of rule.
• sor_anchored - runs a data.query for a ground-truth scalar, then compares an answer figure against it within a relative tolerance. Catches silent under-return / truncation. No LLM.
• judge - an LLM scores an answer 1-5 on each rubric dimension; the case passes when the mean meets the threshold. Degrades to a neutral, non-passing score if the judge is unavailable.

prompt-and-message-eval uses rule + improvement_loop. The other four kinds (deterministic, sor_anchored, judge, evidence_judge) are available to tenant-authored evals. Read the exact cases of any eval with eval://<slug>.

Reuse: cached vs force

A run is content-addressed by a fingerprint over (workspace, eval slug, eval version, inputs) - the inputs that fully determine the graded work. Because the inputs are folded in, a different message re-grades and the same message reuses:

• reuse: cached (default) - if a run for the same fingerprint is still active (not failed / canceled), evals.run hands it back with reused: true instead of starting a duplicate. Two simultaneous cached re-requests for the same inputs resolve to one run (an at-most-once dedup).
• reuse: force - always starts a fresh run.

Caching is opt-out, not a freshness guarantee: v1 always grades current data, so a cached re-request returns the last run for the same inputs regardless of how much time has passed. To re-grade the same message against the latest data, pass reuse: force.

Author your own eval

The built-ins not fitting is what the builder is for: configure your own grading pipeline - the input fields it accepts, the cases, and the graders - and store it for your workspace. Every write goes through the same moat first, so you get author-time feedback instead of a broken run.

The moat enforces the structural rules: a kebab-case slug that is not a reserved built-in (prompt-and-message-eval, or its retired outreach-eval / message-grader aliases), well-formed input fields (including a valid require_at_least_one referencing declared fields), a provided subject whose input_field references a DECLARED field, a generated target that uses search.run, and coherent grader configs (an improvement_loop / evidence_judge rubric + pass_threshold 1-5 + retrieval sources). Anything malformed comes back as { valid: false, errors: [...] } (validate) or a validation_failed envelope (create / update). Full walkthrough in the Evals guide.

Browse evals and grader kinds

Everything the run surface consumes is readable over both the MCP resource schemes and their REST twins (scope evals:read):

• eval://list (GET /evals) - every eval available to your workspace (built-ins + your authored ones), with input_schema on the summary.
• eval://<slug> (GET /evals/:slug) - one eval with its input_schema and full cases (each case subject + its grader specs).
• eval_run://list (GET /eval-runs) - this workspace's runs, newest first (filter ?eval_slug= / ?status=).
• grader_kind://list (GET /grader-kinds) - the six grader kinds, each with an uses_llm flag.

Prerequisites

Evals are part of Agent Platform v2, gated per workspace by the agent_v2 flag: on a flag-off workspace every eval surface refuses with 403 feature_disabled, and the MCP evals tool is not registered. Scopes: evals:execute to run and evals:write to author (both editor-tier, on the customer-agent key bundle), evals:read to browse, validate, and poll (a viewer-tier scope, on the read-only bundle) - so a read-only key can browse, validate, and poll but not fire a run or author.

A prompt to hand an agent

Grade + improve this outbound message for relevant positioning against our
customer evidence - use the evals tool.

Run the prompt-and-message-eval eval with the prompt and the drafted message as
inputs (reuse: force), then poll eval_run://<id> until it is complete.

Report the overall score and the pass/partial/fail verdict, then the
improvement_loop result: the before→after report - the before score and
reasoning, the improved reusable prompt and grounded message (note it is a
suggestion), the after score, every cited customer quote (with its
supports/contradicts stance), and the score lift + what changed.

See also

• Evals guide - the full consumer walkthrough (subject modes, the verdict shape, retrieve-then-cite, not-applicable, reuse semantics, the eval builder).
• Search (fast lane) - the lane a generated eval runs its questions through.
• Endpoints overview - the whole surface and prerequisites.