Evidence and trust
The fields that say how far a Search result or an evaluation from Amdahl can be trusted, what each one means when it is null, and a policy for acting on them in code.
Every Search result and every evaluation carries fields that say how much it can be trusted. They are the part of a response your code should branch on before it uses the rest. GTM engineers whose agents act on Amdahl's results without a person in the loop start here. This page collects the fields in one place; the Search and Evaluate a message references carry the full detail.
Three rules hold across all of them:
- Null is not zero. A field that could not be measured comes back
nullor with anunavailablestatus, never as0. A zero is a measurement. - An abstention is not a fail. When a criterion could not be evaluated honestly, it says
applicable: falseand gives anot_applicable_reason. Nothing was assessed, so there is nothing to read into, and the criterion is out of the score. - Quotes are retrieved, never written. Every customer quote in a Search response is a real utterance from your customers' own conversations, read from the store, not written by a model.
On a Search result
| Field | What it tells you | Act on it |
|---|---|---|
mode_ran | Which of the four search modes answered. Modes can read different stores, and what one row counts can differ. | Compare two results only when the same mode answered both. |
corpus.store, corpus.grain | The physical store (warehouse, pgvector, theme_index) and what one row counts. | A theme is a cluster, not an utterance. Do not quote it as one person's words. |
corpus.empty_reason | Why an empty result is empty. | read_failed is an error: retry. no_match means nothing was said. weak_matches_only means rephrase. |
corpus.speaker_scope | Whose speech the store can hold: all, or external_only (customer side only). | Do not report an internal-vs-external split off an external_only store. |
corpus.coverage.basis | Whether the time-range numbers describe the whole match set, a truncated slice, or nothing (unavailable). | Read this before any date or recency claim. |
corpus.truncated | A cap cut the result. | Do not treat the rows as the population. |
compiled.sql | The query as you asked for it, before the gate added your tenant filter. | Keep it as a record of intent. |
detail.internal.status | On a plain-language search, whether the SQL ran (ok) or not (failed). | failed during a burst can mean the workspace's read budget ran out. Back off before rewording. |
On an evaluation
An evaluation is messages.optimize with mode: "grade", and its result comes back in grade. With evidence: "workspace", Amdahl evaluates the message against your customers' own conversations. A prompt run's emails carry the same grade block.
| Field | What it tells you | Act on it |
|---|---|---|
grade.passed, grade.total | How many criteria the draft met, out of the criteria that were evaluated. | Report this fraction, not only the score. |
grade.passes_bar | Whether the score met the bar (4.2). boolean or null. | false holds. null means nothing was evaluated: pass through, do not block. |
grade.lines[].applicable, not_applicable_reason | Whether a criterion was evaluated, and why not (corpus_empty, evidence_read_failed, …). | A criterion with pass: null is unknown, never a fail. evidence_read_failed is Amdahl's outage: evaluate again later. |
grade.score | 1 + 4 × passed / total, or null when no criterion was evaluated. Never 0. | Drop null scores from averages. |
evidence.used | Whether the four evidence criteria read your customers' conversations. | Say a draft was evaluated against your customers only when it is true. |
disclosures.single_run | Always true: one evaluation of one text. | Evaluate a draft several times before you report that a change helped. |
disclosures.repeat_spread | How often the evaluation model's score moved by more than 0.2 on a repeat, in Amdahl's measurement. | Treat a difference of one criterion as inside that spread. |
optimize_id | The handle of this evaluation. | Two responses with the same optimize_id are one evaluation, not two. |
lift_reportable (prompt runs) | Whether the held-out difference between the original and the revised prompt can be reported. | Do not quote lift when this is false. The train scores are never lift. |
What an evaluation is, and is not
An evaluation is structured: each criterion is a pass or a fail with a reason. So on five evaluated criteria, the score takes exactly six values: 1, 1.8, 2.6, 3.4, 4.2 and 5. For what the score measures, see What an evaluation measures.
One evaluation is one reading. In Amdahl's measurement, the evaluation model (Claude Sonnet 5.5 with thinking off) moved its score by more than 0.2 on about 15% of texts when it evaluated them again. To measure a small change, evaluate each version several times and compare medians. The current models and limits are on Models and limits.
A policy for acting on a result
A starting point for code that acts without a person in the loop. Tighten it for anything with real cost.
| Situation | Do |
|---|---|
Search: empty_reason is read_failed, or detail.internal.status is failed | Retry with backoff. Do not report "nothing found". |
Search: coverage.basis is unavailable, or truncated is true | Use the rows, but do not state a population, a date range or a recency. |
Search: speaker_scope is external_only | Quote it as customer voice only. |
Evaluation: passes_bar is true | Proceed. |
Evaluation: passes_bar is false | Hold. Fix the failed criteria from their reason, or have Amdahl write the message from your context, then evaluate again. |
Evaluation: passes_bar is null | Abstain and pass through. Surface not_applicable_reason to a person. |
Evaluation: evidence.used is false, or only cta_clarity was evaluated | No claim was evaluated against your customers' conversations. Proceed only when a person reads the copy first. |
The patterns that use these fields end to end are on Patterns.