How grading works

An eval grades a piece of writing against your own customer conversations. You send a prompt, a drafted message, or both. You get back a pass or fail on each rubric line, the customer quotes each verdict rests on, a send/hold bit, and an improved version of what you sent.

The built-in eval is prompt-and-message-eval. It needs no setup, and a run that names no eval uses it. You can also author your own.

This page explains how a run works. The Evals API is the field reference, and Grade a cold email walks through one real run.

Two ways to use it

• At review time. A person hands over a draft and reads the report card: the lines that are not landing, the quotes that show why, and a better reusable prompt.
• In a loop. An agent grades what it just wrote and reads gate.passed before it acts. See Gate before send.

A run never changes your data, so it is always safe to put in either path.

Eval or the Optimizer

They answer different questions. The Optimizer writes a message from what you know about the recipient, in the sender's voice, on the same call, with no connected data needed. A draft is optional: sent with context, it is the starting point. The eval grades it: a pass or fail per rubric line with the customer quotes behind each verdict, and a better reusable prompt, which needs your conversations. Use the Optimizer to write a message; use the eval to see why a message, or the prompt behind it, is not landing. Optimizer or eval puts them side by side.

Two subject modes

Every case in an eval declares the subject it grades:

• provided: you send the content. prompt-and-message-eval works this way: send a prompt, a message, or both (at least one).
• generated: the eval produces the content itself by running a question through Search, then grades the answer. Use it to regression-test an answer pipeline. No built-in eval uses this mode; author one if you want it.

What a run does

• It only reads. A run dispatches read operations: retrieving quotes, querying the warehouse, running a generated case's question. It cannot change your workspace.
• It grades current data. Evidence comes from your warehouse and themes as they stand now. There is no option to grade against a past snapshot; to hold evidence fixed while you iterate, pin it.
• Outside evidence is opt-in. By default every quote is your customers' own words. Pass include_external: true and one web search adds public quotes, tagged external, which can back a claim about the market but never a claim about what your customers say.
• Who it is for is beta. On a workspace with recipient evidence turned on, a run also takes recipient, recipient_notes and message_type (see Recipient fields). In this release they address the run, are echoed back as names and counts, and a recipient is resolved into the profile, the brief and (for a cold message) the thesis that the run read carries as run.recipient_profile, run.brief and run.thesis; they do not change the grade yet.
• It is async. POST /evals/run returns a run id at once. You read the result with GET /eval-runs/{id}?wait_ms=30000, which blocks until the run finishes or 30 seconds pass. A full run takes minutes.
• The judge is not the writer. One call writes the improvement and scores nothing. A separate call scores your draft and the improvement together, unlabelled, in an order taken from a content hash, so the judge cannot tell which one it wrote.

Quotes are retrieved, never written

The model cannot invent a customer quote. Quotes are retrieved from your corpus first, each with an id. The judge may cite only those ids, and the server swaps each cited id back to the stored text, dropping any it does not recognise. A quote in a report is a real utterance from your data.

Every quote carries a tier, and the tier limits what a claim built on it may say:

Reaching past a quote's tier is graded as a grounding failure. Writing "your engineers told us" when the evidence is a corpus pattern turns a grounded line into a claim of a conversation that never happened.

Some quotes are your own team's messages to the account, tagged speaker_side: "internal". They are there so a draft does not contradict what you last sent; citing one as customer evidence is a grounding failure.

What the score is, and what it is not

Each rubric line is pass or fail. The headline is the pass fraction on a five-point axis, score_15 = 1 + 4 × (passed / total). On a five-line rubric it takes exactly six values: 1, 1.8, 2.6, 3.4, 4.2 and 5. A 5 means "passed all five checks", not "perfect". Quote the fraction.

A run grades two things: your copy and the improved version. headline.submitted is your copy; headline.improved is the one the eval wrote. The improved side is written against the same rubric it is graded on, so it clears the bar on most runs (92% of runs in one measurement at eval_version 2.9.0, n = 89). That makes lift, the difference between the two, mostly a measure of how low your draft scored. headline.submitted.checks is the only number on the report about your writing. Do not build a dashboard, target or gate on lift or transition.

Here the submitted copy passed two of five checks and the improved version passed four. The lift is not reportable, and the number to quote is submitted.checks:

"headline": {
  "submitted": { "score_15": 2.6, "checks": { "passed": 2, "total": 5 }, "usage": "as_provided" },
  "improved": { "score_15": 4.2, "checks": { "passed": 4, "total": 5 }, "usage": "illustration_only" },
  "threshold": 4.2,
  "transition": "fail_to_pass",
  "sentence": "Your draft did not pass - the improved version does.",
  "explanation": null,
  "lift": 0.4,
  "lift_reportable": false,
  "lift_badge": null,
  "confidence": { "level": "high", "reasons": [], "caveat": null },
  "stop_reason": "threshold_met",
  "quotes_cited": 3,
  "quotes_cited_submitted": 0,
  "quotes_cited_improved": 3,
  "what_changed": "Tied the opening claim to what two customers actually said.",
  "input_simulated": false,
  "facet": "message"
}

It is a coaching read, not a measurement. A grade is a structured second opinion grounded in cited quotes. It does not predict reply rate or conversion. Say so when you report it.

One run is one draw. Grading the same text twice does not always return the same score. lift_reportable says whether a before/after difference is larger than that spread, and comparing two runs withholds a delta that is not. The measured spread is on Models and limits.

Grounding rewards copy that reaches for evidence. Category copy ("streamline your workflows") scores low on grounding even on a workspace with thousands of conversations. A low grounding score says the copy did not use the evidence, not that you have too little of it.

When a run refuses to grade

A case that cannot be graded honestly comes back not_applicable with a reason, and is left out of pass and fail. A run where no case applied reports verdict: "not_applicable" and overall_score: null, never 0.

The common reasons: empty_corpus (no customer conversations yet), not_outreach (the writing has no persuasive job, such as a scheduling note or a receipt), evidence_unavailable and report_unavailable (our failure; re-run). The full table is in the Evals API.

Connect a source before the first eval. An eval grades against your customer conversations. On a workspace with no synced conversations every run returns not_applicable with empty_corpus: it grades nothing, and it still counts as a run. The setup check (GET /setup/status, or the MCP get_setup_status tool) reports how many sources are connected. Until one has synced, use the Message Optimizer, which needs no data and then checks style only.

evals.run says this when you start the run: on a workspace with no connected source its response carries setup_needed, with a message and a connect_url for the user. When the workspace also holds no conversation themes and the run could only end empty_corpus, it is refused with 409 (setup_needed) before it counts as a run. include_external, a pinned run, or an eval with a case of only rule or structure graders still runs. Otherwise the run is still queued. Show the user message and connect_url, and do not retry until a source is connected. See A workspace with nothing connected, the setup_needed handling in Errors, and the FAQ.

A refusal is not a fail. Nothing was assessed. A gate must pass through on a refusal, and an average must drop refused runs rather than count them as zero.

Comparing runs

Each run retrieves its own evidence, seeded from what you sent. Edit a draft and re-grade it and both the copy and the quotes change, so the score difference mixes the two. Pass evidence_from_run with the first run's id and the second run reuses its quotes, so only the copy changed. The compare read then tells you whether the difference is attributable, and withholds it when it is not.

Where to go next

• Evals API: every request field, response field, read and export.
• Prompt and Message Eval: the built-in instrument's rubric, inputs, modes and scoping.
• Grade a cold email: one run, read end to end.
• Evidence and trust: the fields to branch on before you act.