The learning loop
How evaluations feed the Amdahl learning model and your agent: ground a message in your customer conversations with Search, write it, evaluate it, fix it, and improve the prompt that writes it. What to measure while you run it, and what gives you a wrong answer.
Amdahl is an applied machine learning company for GTM agents. Your agent writes, and Amdahl optimizes it to win. This guide is the loop that makes that happen run after run.
Amdahl
- The learning model: processes your customer data, models what wins with your buyers, and evaluates and optimizes your agent's output against it
- Search: ask your customer conversations a question, get rows and quotes back
- The optimize call: write a message, evaluate a message, and evaluate or improve the prompt that writes your messages
You
- the prompts
- your own canon: ICP, product truth, brand voice
- the agents and the orchestration
- the scheduling and the approval flow
Amdahl works with any agent, so the agent and its orchestration stay yours. This guide covers the loop itself, how each pass feeds back, what to measure while you run it, and the few things that quietly give you a wrong answer if you do not know about them.
Build your bench is the companion page. It says where the prompts, evidence and evaluations live once you run this loop dozens of times a week.
How evaluations feed the model and your agent
Each pass of the loop sends something back in two directions.
- Into your agent. An evaluation names each criterion that failed and why. You fix the draft and the prompt behind it. When the same failure keeps coming back,
optimize_promptevaluates the prompt itself, and inimprovemode returns a revision only when it wrote better email on recipients held out from the revision. - Into the learning model. Every optimize run is kept with your workspace: the request, the versions it tried and how each scored. When the sender uses a message,
record_message_feedbackrecords what was actually sent and what happened. Feedback markeddid_it_myselforreported_to_metrains that sender's preferences. Aninferrededit is stored and scored, but does not.
The calls
Search (search.query) takes a question about your
customer conversations and returns rows. By default it blocks, and that
synchronous path stays under a 15-second ceiling by construction. A
plain-language question also takes async: true, which returns a job id that
you poll, and runs under a budget of about 180 seconds. Use it when the question
is broad enough that the fast path would cut it short. Search has no web or news
leg, and no call on the API has one. Written synthesis over your own data is
Research's job.
Evaluate a message (messages.optimize with mode: "grade",
MCP optimize_message) evaluates one draft and writes nothing. It returns on
the same call. Send "evidence": "workspace" to evaluate the draft against your
own customer conversations. What you know first-hand that no conversation holds
(what a buyer told you in person) goes in facts, inside context. A fact
licenses the claim it states, never an embellishment of it, and it is never
counted as a customer quote.
Evaluate or improve a prompt (messages.optimize_prompt,
MCP optimize_prompt) takes the reusable prompt your team or agent writes from.
It writes one email per sample recipient and evaluates each one. In improve
mode (the default) it also tries revisions of the prompt and returns one only
when it wrote better email on recipients held out from the revision. A run takes
minutes, so the call returns a run_id that you read with get_prompt_run.
Record what happened (messages.record_feedback,
MCP record_message_feedback) records what the sender did with a message a run
returned, with the text they actually sent.
What an evaluation is
An evaluation has five criteria: relevant_positioning, grounding,
verified_specifics, differentiation and cta_clarity. Each one is a pass or
a fail with one sentence of reason (reason).
Read the fraction, not the score. The response sends passed and total
next to score, so you can read "4 of 5 criteria passed" instead of a 4.2 that
looks like a rating. passes_bar says whether the draft met the bar. The score
formula and the bar are in the API reference: Evaluate a message.
The four evidence criteria measure the draft against your customers' words.
They abstain unless the call sends evidence: "workspace" on a workspace with
synced conversations. An abstaining criterion has pass: null,
applicable: false and a not_applicable_reason (for example corpus_empty).
It is out of both sides of the fraction: unknown is never zero. So without
evidence, only cta_clarity is evaluated and the score is 1 or 5. When no
criterion is applicable, score and passes_bar are null, never 0. See
What an evaluation measures.
One evaluation is one reading. Every evaluation says so in
disclosures.single_run: true. In Amdahl's measurement, the evaluation model
(Claude Sonnet 5.5 with thinking off) moved its score by more than 0.2 on about
15% of texts when it evaluated them again. disclosures.repeat_spread carries
that figure when it is available. Because the score moves one criterion at a
time, one criterion that changes its verdict can move a draft across the bar.
Grounding is an input to drafting. The evaluation is a reading, after drafting, of whether the draft used it. The order in the loop matters: Search pulls the evidence before anything is written, and the evaluation then asks whether what you wrote reaches for it.
If you evaluate copy that was drafted without an evidence pull, a failed
grounding criterion is correct and tells you nothing new: the draft never had
the evidence. It is not a verdict on your writing. If you skipped the ground
step, go back and do it, rather than argue with the criterion.
The loop
Write
The prompt. "Write a first-touch email to a VP of Engineering at a mid-market SaaS company."
you
Ground
Pull the customer evidence that should inform it.
search
Draft
Produce the message.
your model or Amdahl
Evaluate
Evaluate the draft against your evidence.
optimize_message, mode grade
Fix
Fix the draft, and fix the prompt that wrote it.
you
then fix the draft and the prompt, and run it again
You will run this many times a week. The repetition is where the useful signal comes from. One evaluated draft tells you about one draft. Twenty tell you whether your instincts about your own market survive what your customers said.
When the same failure comes back across drafts, the problem is the prompt, not
the draft. Then evaluate the prompt itself: send it to optimize_prompt with
mode: "grade" to see how its emails score across sample recipients, or with
the default mode: "improve" to get a revision that wrote better email on the
held-out recipients.
Keep a person between drafting and evaluating
If you drive this from an agent harness, the obvious move is one skill that runs the whole loop. Do not do that. Split it in two:
| Step | Does | Hands back |
|---|---|---|
| Ground and draft | Pulls evidence, saves it, drafts, creates the run folder | Drafts for you to read |
| Evaluate and record | Evaluates the draft, saves the response, writes the manifest | The evaluation, criterion by criterion |
The gap between the two is where you read a draft before it is evaluated, and where you decide whether you agree with the verdict. If you automate that gap away, the exercise stops producing anything a load test could not give you.
When you report an evaluation, show each criterion with its verdict and its
reason, and the passed and total fraction. Do not rewrite the criteria
into one sentence of your own about the copy: a summary is where an abstained
criterion turns into a pass or a fail it never was.
Once you run batches, add a third step that chains both. Keep it away from any work you are learning from.
What to measure
Record these on every run.
The fraction. Record passed and total, not only score. The score is a
rendering of the fraction, and the fraction is the measurement. With five
criteria the score has six positions, so a real improvement is invisible until
it changes a whole criterion. If you iterate against the number and it does not
move, you know only that no criterion changed its verdict.
The criteria that abstained. Record which criteria were not applicable, and why. An evaluation with one applicable criterion and one with five can show the same score. They are not the same reading.
For a prompt, the held-out lift. lift is the held-out mean of the revised
prompt minus the held-out mean of the original. It is a number only when
lift_reportable is true. Otherwise it is null, and
lift_not_reportable_reason says why (for example held_out_under_3). The
train scores come from the in-loop scoring model that chose the revision,
so they are never lift, and disclosures.train_scores_are_not_lift says so.
Quote lift only.
Live evidence and frozen evidence
Use live search when you produce work. Freeze the evidence you draft from when you compare two versions of a prompt.
A second search can return different quotes, so a change you see could come from your edit or from retrieval. For any prompt-improvement pass, pull the evidence once, write it to a file, and give that same file to every attempt.
The evaluation's own evidence is live. With evidence: "workspace", each
evaluation reads quotes from your workspace's conversations at the time of the
call, and no call can pin the quotes an earlier evaluation read. So evaluate the
two versions you compare one after the other, against the same state of the
workspace. Do not compare an evaluation from this week with one from last month
as if both read the same conversations.
For a prompt, the sample recipients are the part you control. Send the same
recipients to every run you compare. If you leave them out, the run samples up
to six of your workspace's own accounts, and two runs can sample different ones.
Two kinds of repeat will fool you
Both repeats below return a valid, complete, well-formed response. Neither returns an error. Only a field tells a new measurement from a replay, and you have to read it.
Search results. Query results are cached for a short window, keyed on the
compiled SQL, not on your words. Two differently worded questions that compile
to the same SQL hit the cache. The same question that compiles differently on a
second pass misses it. Check the cached flag on detail.internal before you
conclude anything about consistency.
Optimize runs. The optimize call addresses a run by the content of the
request, so the same request again can return the result of the same run (this
is what lets a dropped connection recover its answer). Every evaluation carries
optimize_id, its handle. Two responses with the same optimize_id are one
evaluation, not two. A prompt run started with an idempotency_key that was
used in the last 24 hours returns the run the first call started, with
replayed: true. Read those fields before you count a repeat as a second
reading.
Evaluate the voice separately
The evaluation measures claims and the ask against your evidence. Brand voice is not a claim about a customer, so no criterion covers it.
For voice, use rules that code can enforce. When Amdahl writes a message,
context takes constraints that it enforces with code on every version:
max_words, banned_phrases, banned_punctuation and keep_cta: "strict" (see
Optimize a message). keep_cta: true only marks
the ask as the sender's choice and keeps none of its words. For a draft you evaluate,
run the same kind of rule yourself, next to the Amdahl evaluation, and report
the two verdicts separately.
Set a word limit from your own data, not ours. Amdahl measured length against real reply outcomes on 127,872 labelled outbound emails, and the effect reverses sign between workspaces: in some, longer openers got more replies; in others, slightly fewer. So there is no length we can tell you to write to. A word limit is a house style rule, and only your own outcomes can set it.
Record what happened
When the sender uses a message Amdahl returned, record it with
record_message_feedback. Send evidence (required): did_it_myself,
reported_to_me or inferred. These name who observed it, not how sure you
are. Then send at least one of sent_text (the text actually sent, edits
included), comment and outcome (used, used_with_edits, not_used or
unsure). With sent_text, the response's rubric_miss is true when Amdahl,
scoring as it does now, scored the sent text below the message the run
returned. An inferred edit is stored and scored, but it does not train the
sender's preferences. Each call stores one more event, so a retry stores it
twice.
Sequence
Run the loop by hand, three times
No repo, no layout, no skills. Write a prompt, pull evidence with Search, draft,
evaluate the draft with optimize_message in mode: "grade" and evidence: "workspace", and read each criterion and its reason. Keep the files anywhere.
Doing this first means the layout you build next has a shape you saw, not one you guessed.
Create the repo and the canon
Build the bench layout, and move your three hand runs in as the first run folders. Write the canon files (product truth, ICP, brand voice), each with an owner and a review date at the top. Turn the voice rules into code. Start your coverage table.
Still no skills. If you automate before you know the manual version, you encode a guess.
Write the two steps, then compare cold and assisted drafts
Write the two steps from what the first week was actually like, and get an API key for repeat runs.
Then pick five real messages. Write each one cold first, from your own read of the market, and save it. Run the loop on the same brief. Evaluate both, one after the other, and compare the fractions criterion by criterion: which criteria each one passed, and which abstained.
The distance between your instinct and the evidence is what is worth measuring.
Evaluate the prompt
Once the same criteria keep failing, send the prompt behind the drafts to
optimize_prompt. Send six recipients you choose, so that three are held out
and the lift can be reported. Read outcome first: improved returns a revised
prompt, and unchanged means no revision beat the original on the held-out
recipients.
Add volume
Point the bench at every sequence and every agent that writes outbound, where the volume is. Put an evaluation in front of anything that publishes, and record feedback on what was sent.
Tests worth running early
Run the first one before anything else. Everything below it assumes an answer.
Confirm that each repeat really evaluated again: a different optimize_id on each response. With the
same handle, every stability test below reports a perfect result it never measured.
| Test | Method | Reads on |
|---|---|---|
| Does the score discriminate | Evaluate three drafts you would rank yourself, best to worst, with evidence: "workspace". Read passed and total. | Whether a score difference means anything at all. A separation in your order is a usable dial. A flat result is a scale that cannot separate them. |
| Criterion stability | Evaluate the same draft three times, each a new evaluation | Which criteria change their verdict on a repeat |
| Pretty lies | A well-written false claim against an awkward true one | Whether grounding and verified_specifics fail the false one, whatever its tone |
| Evaluate the rewrite | Evaluate a message Amdahl wrote for you | Whether it passes more criteria, or only reads better |
| Latency | Time ten evaluations end to end | Whether evaluation can sit inline in a human workflow |
| Reason quality | Fix one failed criterion using its reason, evaluate again, and note which criterion changed and your own read of the copy | Whether following the reasons improves the work, or only the number |
One run is not a measurement. Because each criterion is a pass or a fail, a draft near the bar can pass on one evaluation and fail on the next. Before you act on a change, or tell anyone a draft got better, evaluate each version several times and compare the medians of the fractions.
Decide before you start
These are decisions, not lookups. If you leave them open, every verdict is measured against a different bar, and the verdicts cannot be compared.
Whether you evaluate with evidence, and on which drafts. An evaluation
without evidence: "workspace" reads the ask only. An evaluation with it
measures the draft's claims against your conversations. The two are different
readings, so record which one each evaluation was, and never pool them.
What counts as a claim. Most marketing copy is sentences that are not assertions about customers. Decide which ones are in scope, because the claim is your unit of measurement once you start counting them.
Which recipients a prompt is evaluated on. A prompt that writes for VPs of Engineering should be evaluated on recipients like them. Pick the six recipients once, write them down, and send the same ones to every run you compare. The sampled default is a fair start, but it is not a fixed control set.
What success looks like
"Our outbound got better" is the work. Here is the read:
- Runs completed through the full loop
- Criteria passed out of criteria applicable, and whether that matched your own read of the copy
- Criteria that abstained, and why
- For prompts, the held-out lift, and only when it is reportable
- Disagreements logged, split by cause
- Whether the reasons improved the copy by your own judgment
- At least one belief you held that the evidence killed
Going further
Everything above is version one: a manual loop and one readable folder per run. An open-source prompt-test harness (Promptfoo and similar tools) is version two: a grid of prompts against inputs with results side by side, with an Amdahl evaluation as a custom scorer.
Do not start there. It earns its place when you run one prompt across many accounts and comparing by hand has become slow, or when you changed a prompt and need to know which past cases got worse. None of version one is wasted if you go that way: prompts, canon and evidence already live as separate files, which is most of the reason for the layout.
If you build the adapter, have it store optimize_id with every evaluation. A
grid that counts one evaluation as many produces a clean table that measured
nothing.
See also
- Build your bench: the run folder, the manifest, the disagreement log
- Search: the grounding step
- Evaluate a message: the evaluation step, in full
- Improve a prompt: evaluate or improve the prompt behind your drafts
- Record feedback on a run: what happened to a message