Prompt and Message Eval

Slug: prompt-and-message-eval. The eval Amdahl ships. It grades a drafted message and/or the prompt behind it against verbatim quotes from your own calls and emails, then writes and grades an improved version. It needs no setup, and a run that names no eval uses it.

This page is the instrument's specification. For a worked run, read Grade a cold email. For the request and response contract shared by every eval (polling, the report card, gates, comparing runs, export), see the Evals API.

The rubrics

The message and the prompt are graded separately, each on its own five binary lines. They never share a score.

Each line is pass or fail, so a side's score_15 is 1 + 4 × (passed / total) and takes six values: 1, 1.8, 2.6, 3.4, 4.2, 5. A full run reads both sides against a bar of 4.2; mode: "gate" compares against the eval's pass_threshold of 3.5. On a five-line rubric the two select the same drafts, because nothing lands between 3.5 and 4.2.

A rule grader also checks the message text: 30 to 3,000 characters, and a call to action. The upper bound is an outlier guard ("this is a document, not an email"), not a style target: measured against reply outcomes, the effect of length reverses between workspaces.

Inputs

These go inside inputs:

These go beside inputs, at the top level of the request:

At least one of prompt / message must be supplied. Send both when you have both — grading the pair is what produces a useful improved prompt, because the eval can see what the prompt asked for and what it actually got.

Modes

mode goes inside inputs. Omit it and you get rewrite.

Advisory: when your prompt is a living document

Plenty of teams do not have a one-line prompt. They have a living rules document — 40 pages of positioning guidance, objection handling, and things legal will not let you say. Nobody is going to replace that because an eval suggested a new one. advisory is the mode for exactly that case.

mode: "advisory" leaves your document alone. Instead of a rewrite you get anchored, surgical suggestions:

{
  "kind": "add",
  "facet": "prompt",
  "anchor_quote": "Always mention our SOC 2 certification in the first email.",
  "anchor_offset": 8214,
  "section_id": "s6",
  "title": "Gate the SOC 2 line on the buyer actually asking",
  "detail": "Change this to: mention SOC 2 only when the account has raised security, compliance, or procurement. Otherwise cut it.",
  "why": "Across the retrieved quotes, security comes up from procurement and never from the engineering buyer this sequence targets. Leading with it reads as boilerplate to the person receiving it.",
  "dimension": "Relevant positioning",
  "quotes": [
    {
      "text": "Honestly the SOC 2 stuff is procurement's problem, not mine.",
      "stance": "contradicts"
    }
  ]
}

Five kinds of suggestion come back:

Long documents are sectioned, never quietly truncated

Past 12,000 characters your prompt is split on your own headings and packed head-and-tail (the opening frames the task; the closing usually carries the hard rules). Every run reports coverage:

"coverage": {
  "total_chars": 48219,
  "graded_chars": 23904,
  "truncated": true,
  "strategy": "sectioned",
  "sections": [
    { "id": "s1", "heading": "Who we sell to", "chars": 1840, "included": true },
    { "id": "s2", "heading": "Positioning by segment", "chars": 6210, "included": true },
    { "id": "s7", "heading": "Legacy objection scripts", "chars": 9022, "included": false }
  ]
}

You always know what was read. Past 250,000 characters the run refuses rather than returning a confident number computed over a sliver of your document.

Advisory mode, on a document you are keeping

rewrite is the default, so advisory has to be named explicitly:

curl -X POST https://app.amdahl.ai/api/platform/v1/evals/run \
  -H "X-API-Key: $AMDAHL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "eval": "prompt-and-message-eval",
    "inputs": {
      "mode": "advisory",
      "prompt": "<your 40-page outbound rules document>",
      "message": "<a message it produced>"
    }
  }'

How it works under the hood

Five stages. Four of them exist to make the number trustworthy rather than merely produced:

1. Read the intent. One cheap pass extracts what you are trying to do — the offer, the ask, and each specific claim your draft makes. Retrieval is seeded from that, never from your draft. (Seeding on a weak draft returns generic themes, and those generic themes then become the evidence the draft is held to. That is circular.)
2. Gather evidence. Several retrieval passes run at once — one per intent seed, one per specific claim — so "cuts onboarding from six weeks to two" is checked against the themes that speak to that, not against the average of your whole message. The set is then frozen for the entire run — every candidate, including a revision round, is judged against the same quotes. Across two separate runs it is not frozen: retrieval is seeded from what you submitted, so editing your draft also changes what gets searched for. That is what evidence_from_run is for.
3. Write. A model produces the improvement, the suggestions, and the examples. It scores nothing.
4. Grade, blind. A separate pass scores both candidates in one call, unlabelled, with the presentation order derived from a content hash. The judge does not know which one it wrote, so it cannot be kind to itself.
5. Revise, if needed. If the improved version misses the bar (4.2/5), it gets exactly one more attempt, guided by the judge's own critique. On a full run that same 4.2 is the bar BOTH sides are read against: one threshold decides the submitted verdict and the improved one. The eval's other bar, pass_threshold 3.5, is what mode: "gate" compares against, and on this rubric the two select the same drafts anyway — five binary lines put score_15 on six values (1, 1.8, 2.6, 3.4, 4.2, 5), so nothing lands between 3.5 and 4.2. Then it stops and tells you honestly why it could not do better.

When it does not grade

Nothing to grade against? The run comes back not_applicable with empty_corpus, never a false fail.

Not the kind of writing this reads? Also not_applicable, with not_outreach. The rubric scores commercial writing at any stage (prospecting, an open deal, a proposal or signature, a pilot, a renewal, a re-engagement) but not correspondence with no persuasive job (scheduling, receipts, support replies, internal notes), even inside an active deal. Your work was not assessed, so there is no score to read into it. The reason table says what each reason asks of you.

The improvement did not clear the bar? You get the score, the transition, and an explanation naming what still holds it back.

Audience scoping

If you pass an audience, the run does two things with it before grading anything.

First it resolves it to a seniority — executive, manager, or individual contributor. Titles resolve directly ("VP of Sales", "Head of L&D", "SDRs", "Sr. Mgr., Talent"), and so do the bare words. Something that names a function, a vertical, or a company size ("marketing", "healthcare", "mid-market logistics") names no seniority, and the run says so rather than guessing one.

Then it checks your data for that cohort. The report is only scoped to an audience once your workspace has enough recorded conversations with them to be worth grading positioning against:

All three must pass. If they do not, the run still completes and still grades your prompt and message — it just grades them against your whole customer corpus, and the report says which of five things happened:

A scoped run carries the counts it decided on, so you can see the evidence behind the framing rather than taking it on trust.

And a cleared cohort changes the evidence, not just the framing. The run draws that cohort's own utterances and tags them segment — the tier that licenses "teams like yours". The draw is spread across companies, no more than a couple from any one of them, because the distinct-companies floor above is about the cohort and says nothing about which quotes come back: a cohort can clear it and still have its most recent conversations come from one talkative customer, and a cohort-wide licence over one company's words is the same over-claim one rung down. segment_companies on the report says how many the draw actually spanned.

On runs graded before eval version 2.6.0, audience only told the judge to assume a cohort while retrieval stayed corpus-wide, so do not compare scoped runs across that version.

Filter scoping

audience cuts on one axis — seniority — and takes free text to do it. scope is the general form: typed predicates over the same field vocabulary Search advertises, so "grade this against what people at enterprise closed-won accounts actually said" is expressible without a bespoke input for every axis someone might want.

It sits beside inputs, not inside it — same level as reuse and evidence_from_run:

curl -X POST https://app.amdahl.ai/api/platform/v1/evals/run \
  -H "X-API-Key: $AMDAHL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "eval": "prompt-and-message-eval",
    "inputs": { "message": "<your draft>" },
    "scope": {
      "audience": "customer_voice",
      "filters": [
        { "field": "role_level", "op": "eq", "value": "executive" },
        { "field": "pushback_type", "op": "in", "value": ["pricing", "build_vs_buy"] },
        { "surface": "deals", "field": "deal_stage_status", "op": "eq", "value": "won" }
      ]
    }
  }'

A filter is the same shape Search's filter lane takes, plus a surface per filter. op is one of eq · neq · in · not_in · contains · gt · gte · lt · lte · between · is_null · not_null — but which operators a given field admits is a property of that field, so read the catalog rather than assuming.

The three surfaces

surface defaults to interactions. Naming it on a filter is how you predicate on a fact that lives at a different grain than the quotes do:

Mixing surfaces in one scope is the point, and it is the one deliberate difference from search.query, which takes a single surface per call. "Executives at closed-won accounts" is one utterance-grain cut AND one deal-grain cut; forcing a choice would make the most useful scope inexpressible.

Deal-grain filters cannot slice utterances directly, so the run resolves them to the accounts they match and scopes the quote draw to those accounts. That resolution is bounded: a deal-grain filter matching 500 accounts or more abstains with too_many_accounts rather than carrying a clipped set and calling the result "enterprise closed-won accounts" — the grade would be real and the label would be a lie. The bound is on the read, not on the count, so exactly 500 is on the abstaining side of it.

Discover the field names — never guess

Field names, types, admitted operators and (for coded fields) the exact spellings live in the Search field catalog, derived from the same schema the eval's filter compiler validates against, so the two cannot drift:

curl "https://app.amdahl.ai/api/platform/v1/search/fields" \
  -H "X-API-Key: $AMDAHL_KEY"

Over MCP it is the list_search_fields tool, or the search-field://list resource. Full walkthrough, including why sample_values matters before you filter a coded field, in Search → Step 0.

The evidence floors

A resolved scope is checked against your corpus before it is used, against the same three floors audience applies — imported from it rather than restated, so the two can never disagree about what "enough evidence" means:

All three must pass. The floors exist because of what an eval returns that a search does not: a search hands you four rows and you judge them; an eval hands you a grade with the slice's name stapled to it. The same four rows become "this is what enterprise champions say", asserted with a score behind it. The grade is real and the framing is a lie, and the framing is the part people quote.

What is refused before the run starts, and what is not

Two SHAPE problems come back as invalid_argument from the run call itself, before a run id is minted. They are refusals rather than in-run abstains because at dispatch you can still be told and can still fix it:

• A key we do not recognise. scope and each filter are strict: operator for op, column for field, a stray values is refused, not ignored. The usual default is to strip an unknown key, and a stripped filter widens the slice you thought you had asked for while returning a perfectly well-formed report about it.
• More than 25 filters, for the same reason: narrowing is still available to you at this point.

A scope that narrows nothing is accepted, not refused. {}, { "filters": [] }, { "audience": "all" } and a bare { "allow_thin_evidence": true } — an override with nothing to override — all pass validation, and all mean the same thing: the run is graded unscoped and the scope outcome is not_provided, whose line ("No filters were supplied…") is true of every one of them. Nothing is hidden by accepting them, so refusing would buy no honesty, and it would break a UI that sends an empty array the moment someone clears the filter panel.

Field and operator SEMANTICS are not checked at dispatch either. A misspelled field, an operator the field does not admit, an unknown surface value — the compiler that owns the field catalog sees those inside the run, and they surface as the invalid abstain below rather than as a dispatch-time error. The dispatch gate checks the shape; the run checks the vocabulary.

When a scope is not used, the run still completes

A scope is an enrichment of the report, never a precondition for it. If it cannot be applied the run still grades your prompt and message, and names which of seven things happened:

What it falls back TO is not always your whole corpus. With no other scoping input, it is: the segment tier draws nothing and the run is graded corpus-wide. But inputs.audience is gated separately and independently, so if you sent one and it cleared its own floors, an abstained scope leaves THAT cohort in place — the filters are gone, the run is not unscoped. Read the two together before quoting what a run spoke for.

Where the outcome shows up

The decision is made once, before any grading, and lands on the run's progress trail — progress.steps[] on eval_run://<id> (GET /eval-runs/<id>), as one scope_resolved step. It is emitted on every run, including one that passed no scope at all, so "this run stood on no filter slice" is a recorded fact rather than an absence you have to infer:

Branch on reason; show a reader message; read detail when you need to fix the filters.

Grading a deliberately narrow slice

allow_thin_evidence: true is the escape hatch for the case where the narrow cut is the question — a new segment with four recorded calls, and you want to know how the message reads against those four.

The run then proceeds scoped, and the scope_resolved step carries below_floors: true beside the real evidence counts. Honest by default, overridable on purpose, never silent either way: what the flag buys you is a thin grade that is labelled as one, not a thin grade that reads like a cohort finding. (Labelled on the progress trail — see the caveat above about the report card.)

A scope changes the evidence, not just the framing

Same rule as a cleared audience: the run draws that slice's own utterances and tags them segment — the tier that licenses "teams like yours". The draw is spread across companies, no more than a couple from any one of them, because the distinct-companies floor is about the slice and says nothing about which quotes come back.

Filters and inputs.audience COMPOSE — they do not race. When both resolve, the slice is their INTERSECTION: the cohort predicate is ANDed into the filters before the floors are measured, so the counts describe the same cut the quotes are drawn from, and the slice name carries both halves. Naming only the filters would describe a wider slice than the one that was graded — the same class of lie as widening, pointed the other way.

The composition is why an inputs.audience that resolved is still standing when a scope abstains: it was gated on its own path, so the run falls back to that cohort rather than to your whole corpus.

Scope is part of the run's identity

scope is a named dimension of the run fingerprint, so a filtered request and an unfiltered one are different runs under the default reuse: "cached". Without that they would be identical in content and the second would be served the first one's verdict — a corpus-wide grade returned under a slice's name, which is the same lie the abstain rules exist to prevent, arriving by a different door.

Pinning evidence also compares the slice — but on a different rendering of it, and the difference is worth knowing before you script an A/B. The fingerprint uses a canonical key: the filters are sorted, so re-ordering them does not fork the address. The pin check compares the DISPLAY LABEL — the line the report shows, rendered in the order you listed the filters in.

The consequence: grade v2 against v1's quotes under a contradicting scope and the run fails rather than grading, which is the point — a pinned pool drawn from one slice cannot honestly answer a question asked about another. But grade against the SAME filters listed in a different order and it fails too, on a pin that was in fact compatible. That is the safe direction to be wrong in — you re-run and pay for a fresh draw, rather than one slice's words being reported under another slice's name — so if you are pinning across runs, build the filter array once and reuse it rather than rebuilding it per call.

Runnable calls in the report

research_steps are real Amdahl calls you can paste to close the evidence gaps the report found. Every one is validated before it ships — an operation that does not exist, one that would mutate your workspace, or SQL our query gate would refuse is dropped rather than shown.

They are also bounded by what your key can actually run: a step you would get a 403 for is not a useful suggestion, so the report only proposes calls covered by your own scopes. The tool_kit field carries the arithmetic — callable is how many operations your key could run, out_of_scope how many exist beyond it. When out_of_scope is 0 there is nothing to caveat; when it is not, the report says how many further calls exist without listing them.

The response, annotated

The report lives at verdict.cases[].graders[].improvement. Trimmed to the fields worth knowing:

{
  "mode": "rewrite",             // the default; "advisory" and "gate" are the alternatives

  "transition": {
    "input_verdict": "fail",        // your draft did not clear the bar - the FINDING
    "improved_verdict": "pass",     // the improvement did
    "transition": "fail_to_pass",
    "threshold": 4.2,               // the bar, config-driven and seeded by Amdahl
    "iterations": 1                 // no revision round was needed
  },

  "facets": [                       // prompt and message NEVER share a score
    {
      "facet": "message",
      "before": {
        "usage": "as_provided",     // exactly what you sent, untouched
        "score_15": 1.8,            // 1 + 4*(1/5) - one rubric line passed
        "checks_passed": 1,
        "checks_total": 5,
        "dimensions": [ { "name": "Relevant positioning", "pass": false, "score": 1, "reasoning": "..." } ],
        //            ^ abridged - 4 of the 5 lines omitted; 1 of the 5 passed
        "quotes":     [ { "text": "Speed honestly isn't our problem...", "stance": "contradicts" } ],
        "good_examples": []
      },
      "after": {
        "usage": "illustration_only",   // <- a specimen, NOT a message to send
        "score_15": 5,                  // all five passed - NOT "perfect"
        "checks_passed": 5,
        "checks_total": 5,
        "dimensions": [ ... ],
        "quotes":     [ ... ],
        "good_examples": [
          { "label": "Opening line", "text": "...", "why": "...", "quotes": [ ... ] }
        ]
      },
      "lift": 0.8                   // the [0,1] gap: 1.0 - 0.2
    },
    {
      "facet": "prompt",
      "before": { "usage": "as_provided",    "score_15": 1.8, "dimensions": [ ... ] },
      "after":  { "usage": "reusable_prompt", "score_15": 4.2, "dimensions": [ ... ] },
      "lift": 0.6
    }
  ],

  "suggestions": [                  // anchored, surgical - see Advisory mode above
    { "kind": "keep", "facet": "message", "title": "Keep the low-friction ask", "...": "..." }
  ],

  "research_steps": [               // validated against the live op registry before you see them
    {
      "op": "data.cluster_search",
      "params": { "query": "release approval friction" },
      "purpose": "Pull the approval theme so the next email can lead with the cohort's own language"
    }
  ],

  "coverage": { "total_chars": 118, "graded_chars": 118, "truncated": false, "strategy": "verbatim" },

  "audience": {                     // discriminated on `status` - narrow before reading `dimensions`
    "status": "resolved",
    "dimensions": { "role_level": "executive", "raw": "VP Engineering at a 500-person SaaS company" },
    "evidence": { "distinct_speakers": 7, "utterances": 210, "distinct_companies": 4 }
  },
  // ...or, when it could not be scoped:
  // "audience": { "status": "abstained", "reason": "thin_evidence", "message": "..." }

  "tool_kit": { "callable": 12, "out_of_scope": 5 },   // research_steps are bounded by YOUR scopes

  "confidence": {                   // how much weight the numbers carry, and why
    "level": "moderate",
    "reasons": ["Only 4 customer quotes backed this grade."]
  },

  "prompt_patch": {                 // the prompt edits above, as one pasteable block
    "heading": "Prompt patch - apply these to your reusable prompt",
    "applies_to_submitted_prompt": true,
    "lines": [ { "kind": "add", "verb": "ADD", "instruction": "...", "why": "..." } ],
    "text": "# Prompt patch - apply these...\n- ADD: ... (...)"
  },

  "grader_meta": {
    "model_calls": 3,
    "blinded": true,                // both candidates scored unlabelled, in one call
    "blinding": {                   // ...and the recorded mechanics behind that claim
      "paired": true,
      "order_shuffled": true,
      "improved_shown_first": false,      // which way it landed on THIS run
      "separate_generate_and_grade": true,
      "generator_model": "claude-sonnet-5",
      "judge_model": "claude-sonnet-5"
    },
    "evidence_quotes": 14,
    "evidence_frozen": true,        // DEPRECATED - within-run only; read evidence_scope
    "evidence_scope": {             // what the evidence was held fixed ACROSS
      "kind": "run"                 // retrieved for this run; another run has its own
      // when pinned: { "kind": "pinned", "from_run_id": "...", "frozen_at": "...", "as_of": null }
    },
    "evidence_provenance": {        // the three counts, labelled
      "pool": 14,                   // retrieved and frozen for the run
      "cited": 3,                   // distinct ids the judge cited across every graded block
      "resolved": 3,                // of those, how many resolved to a real quote
      "errored_legs": 0
      // when pinned, also: "pinned_from": { "run_id", "frozen_at", "as_of" }
    }
  },

  "before": { "...": "back-compat mirror of the message facet" },
  "after":  { "...": "back-compat mirror of the message facet" },
  "lift": 0.8,
  "what_changed": "Biggest gain: Relevant positioning (1/5 -> 5/5)."
}

What the score counts

Each graded side carries checks_passed and checks_total next to score_15, because they are what the score IS — the fraction, not a rating. The worked example at the top of this page has the arithmetic and the six values the dial takes; the short version is that a score_15 of 5 means "cleared all five checks", not "perfect", which is how a bare 5 beside a threshold reads.

"before": { "score_15": 1.8, "checks_passed": 1, "checks_total": 5 },
"after":  { "score_15": 5,   "checks_passed": 5, "checks_total": 5 }

The older spelling passed / total still ships alongside and carries the same integers. Prefer checks_*: passed one level out — on a case, on a grader result — is a boolean verdict, and a shipped grading script that reached for the obvious key on a dimension got undefined and printed every dimension as a failure.

Quote the fraction rather than the rating. Both are the same number; only one says what it measured.

How much the numbers are worth

confidence is a derived read on the run itself — level (high / moderate / low) plus the reasons behind anything below high. It never changes a score; it tells you how firmly to hold one. A run drops below high when the evidence pool was thin, a retrieval leg errored, the run stopped on a budget rather than on the threshold, a long prompt was truncated, or a side was scored on fewer rubric lines than the rubric declares. high carries an empty reasons and is the quiet common case.

Evidence tiers

Every retrieved quote carries a tier, and the tier is a licence for what a claim built on it may say:

A resolved audience draws that cohort's own utterances, spread across companies so one talkative account cannot stand in for the group, and they arrive tagged segment. Runs graded before eval version 2.6.0 carry no segment quotes at all.

segment_status and segment_quotes on the improvement report tell you which happened. A run that scoped to a cohort your workspace has too little of comes back segment_status: "abstained" with the reason, and the absence of segment quotes then IS a statement about your data.

A quote with no tier predates tiering and reads as corpus — the weakest standing, never the strongest. Reaching past a quote's tier (attributing a corpus quote to the account) is graded as a grounding failure, not a style note.

Account-tier evidence only exists when you pass an account AND that account is in your data with buyer-side conversation. Otherwise the run says which of those was missing, on account_abstain_reason, rather than quietly grading you on cohort evidence under an account heading:

The citable band has three exclusions, not two, and the third is the one people do not expect: a buyer-side utterance longer than 2,000 characters is dropped as well, because an un-segmented transcript block is not a quotable line. So an account whose conversations are held as long, unbroken turns can abstain no_quotable_utterances while holding thousands of utterances — the count in account_abstain_detail is what makes that visible.

Every abstain that has something specific to say carries it on account_abstain_detail, beside the reason: which name found nothing, or the account's real utterance count when nothing it said was citable. The reason is the branch to code against; the detail is the sentence to read. The same text rides the live account_resolved progress step, so a caller watching a run sees it at the moment it is decided.

not_in_corpus and no_quotable_utterances split what shipped through eval version 2.7.0 as one no_conversations label. They mean opposite things and ask opposite things of you — "we hold nothing under that name" versus "we hold this account and it has never said anything we can quote" — and the collapsed label read as the second when it was usually the first. Matching is a case-insensitive substring on the company name as your records spell it, so a brand name that differs from the CRM's (LlamaIndex where the record says Runllama) lands on not_in_corpus with a full conversation history sitting under the canonical name. If you switch on this field, add the two new values; no_conversations is no longer emitted.

"Scored blind", checkable

blinded: true on its own is an assertion. grader_meta.blinding is the set of facts the run recorded while executing, which is what makes it a disclosure: both candidates scored in one call against one rubric and one evidence set (paired), presentation order taken from a content hash of the text you submitted (order_shuffled), which way that landed on this run (improved_shown_first), and whether writing and scoring were separate calls (separate_generate_and_grade) rather than one model marking its own homework. The two model ids are recorded too.

The prompt patch

prompt_patch is the prompt-facet suggestions assembled into ONE pasteable block. It is composed server-side from the same suggestions[] the report itemizes, so it can never claim an edit the itemized list does not; text is the plain-ASCII body to paste, lines[] the same edits structured. Absent when there are no prompt suggestions — you get null, never an empty block. applies_to_submitted_prompt: false means you sent no prompt, so the lines seed one rather than edit one.

The prompt half is the half that compounds: a fixed message helps one send, a fixed prompt helps every send after it.

usage is the field to read. It tells you what each artifact is, so you never present a specimen as a ready-to-send email:

Check lift_reportable before you quote the lift

A judge scoring the same text twice does not return exactly the same number, so a small lift can be the instrument moving rather than the writing improving. Every report carries a lift_reportable boolean answering the only question that matters here: is this difference bigger than the grader's own run-to-run noise?

"lift": 0.7,
"lift_reportable": true

When it is false, the lift is still returned — it is useful in aggregate across many runs — but it should not be quoted as a figure for a single message. The per-dimension reasoning and the cited quotes are still fully valid either way; it is only the single headline number that is unresolvable.

The threshold is not one fixed value. A side's score is the mean of its per-dimension scores, so more rubric lines make a steadier mean, and the threshold scales as 1/sqrt(n) — a measured relationship on this grader, not an assumed one. Two things follow:

• An eval you author with a wider rubric can legitimately resolve a smaller lift than one with a narrow rubric.
• If the judge returned fewer dimensions than the rubric declares — visible whenever dimensions_scored is below the length of dimensions — the score rests on fewer real judgements than it looks like it does, and the threshold widens to match. The same lift can be reportable on a fully-scored run and not on a short-scored one.

Older slugs

This eval shipped first as message-grader, then as outreach-eval. Both old slugs still resolve, and the outbound_message input key is still accepted as an alias for message. New code should use prompt-and-message-eval with message.

See also

• Evals API: polling, the report card, gates, comparing runs, feedback and export.
• Grade a cold email: this instrument run on one email, read end to end.
• How grading works: what a run does and what its score is not.