Docs

The learning loop

How evaluations feed the Amdahl learning model and your agent: ground a message in your customer conversations with Search, write it, evaluate it, fix it, and improve the prompt that writes it. What to measure while you run it, and what gives you a wrong answer.

Amdahl is an applied machine learning company for GTM agents. Your agent writes, and Amdahl optimizes it to win. This guide is the loop that makes that happen run after run.

The division of labour

Amdahl

  • The learning model: processes your customer data, models what wins with your buyers, and evaluates and optimizes your agent's output against it
  • Search: ask your customer conversations a question, get rows and quotes back
  • The optimize call: write a message, evaluate a message, and evaluate or improve the prompt that writes your messages

You

  • the prompts
  • your own canon: ICP, product truth, brand voice
  • the agents and the orchestration
  • the scheduling and the approval flow

Amdahl works with any agent, so the agent and its orchestration stay yours. This guide covers the loop itself, how each pass feeds back, what to measure while you run it, and the few things that quietly give you a wrong answer if you do not know about them.

Build your bench is the companion page. It says where the prompts, evidence and evaluations live once you run this loop dozens of times a week.

How evaluations feed the model and your agent

Each pass of the loop sends something back in two directions.

  • Into your agent. An evaluation names each criterion that failed and why. You fix the draft and the prompt behind it. When the same failure keeps coming back, optimize_prompt evaluates the prompt itself, and in improve mode returns a revision only when it wrote better email on recipients held out from the revision.
  • Into the learning model. Every optimize run is kept with your workspace: the request, the versions it tried and how each scored. When the sender uses a message, record_message_feedback records what was actually sent and what happened. Feedback marked did_it_myself or reported_to_me trains that sender's preferences. An inferred edit is stored and scored, but does not.

The calls

Search (search.query) takes a question about your customer conversations and returns rows. By default it blocks, and that synchronous path stays under a 15-second ceiling by construction. A plain-language question also takes async: true, which returns a job id that you poll, and runs under a budget of about 180 seconds. Use it when the question is broad enough that the fast path would cut it short. Search has no web or news leg, and no call on the API has one. Written synthesis over your own data is Research's job.

Evaluate a message (messages.optimize with mode: "grade", MCP optimize_message) evaluates one draft and writes nothing. It returns on the same call. Send "evidence": "workspace" to evaluate the draft against your own customer conversations. What you know first-hand that no conversation holds (what a buyer told you in person) goes in facts, inside context. A fact licenses the claim it states, never an embellishment of it, and it is never counted as a customer quote.

Evaluate or improve a prompt (messages.optimize_prompt, MCP optimize_prompt) takes the reusable prompt your team or agent writes from. It writes one email per sample recipient and evaluates each one. In improve mode (the default) it also tries revisions of the prompt and returns one only when it wrote better email on recipients held out from the revision. A run takes minutes, so the call returns a run_id that you read with get_prompt_run.

Record what happened (messages.record_feedback, MCP record_message_feedback) records what the sender did with a message a run returned, with the text they actually sent.

What an evaluation is

An evaluation has five criteria: relevant_positioning, grounding, verified_specifics, differentiation and cta_clarity. Each one is a pass or a fail with one sentence of reason (reason).

Read the fraction, not the score. The response sends passed and total next to score, so you can read "4 of 5 criteria passed" instead of a 4.2 that looks like a rating. passes_bar says whether the draft met the bar. The score formula and the bar are in the API reference: Evaluate a message.

The four evidence criteria measure the draft against your customers' words. They abstain unless the call sends evidence: "workspace" on a workspace with synced conversations. An abstaining criterion has pass: null, applicable: false and a not_applicable_reason (for example corpus_empty). It is out of both sides of the fraction: unknown is never zero. So without evidence, only cta_clarity is evaluated and the score is 1 or 5. When no criterion is applicable, score and passes_bar are null, never 0. See What an evaluation measures.

One evaluation is one reading. Every evaluation says so in disclosures.single_run: true. In Amdahl's measurement, the evaluation model (Claude Sonnet 5.5 with thinking off) moved its score by more than 0.2 on about 15% of texts when it evaluated them again. disclosures.repeat_spread carries that figure when it is available. Because the score moves one criterion at a time, one criterion that changes its verdict can move a draft across the bar.

Grounding is an input to drafting. The evaluation is a reading, after drafting, of whether the draft used it. The order in the loop matters: Search pulls the evidence before anything is written, and the evaluation then asks whether what you wrote reaches for it.

If you evaluate copy that was drafted without an evidence pull, a failed grounding criterion is correct and tells you nothing new: the draft never had the evidence. It is not a verdict on your writing. If you skipped the ground step, go back and do it, rather than argue with the criterion.

The loop

One pass
  1. Write

    The prompt. "Write a first-touch email to a VP of Engineering at a mid-market SaaS company."

    you

  2. Ground

    Pull the customer evidence that should inform it.

    search

  3. Draft

    Produce the message.

    your model or Amdahl

  4. Evaluate

    Evaluate the draft against your evidence.

    optimize_message, mode grade

  5. Fix

    Fix the draft, and fix the prompt that wrote it.

    you

then fix the draft and the prompt, and run it again

You will run this many times a week. The repetition is where the useful signal comes from. One evaluated draft tells you about one draft. Twenty tell you whether your instincts about your own market survive what your customers said.

When the same failure comes back across drafts, the problem is the prompt, not the draft. Then evaluate the prompt itself: send it to optimize_prompt with mode: "grade" to see how its emails score across sample recipients, or with the default mode: "improve" to get a revision that wrote better email on the held-out recipients.

Keep a person between drafting and evaluating

If you drive this from an agent harness, the obvious move is one skill that runs the whole loop. Do not do that. Split it in two:

StepDoesHands back
Ground and draftPulls evidence, saves it, drafts, creates the run folderDrafts for you to read
Evaluate and recordEvaluates the draft, saves the response, writes the manifestThe evaluation, criterion by criterion

The gap between the two is where you read a draft before it is evaluated, and where you decide whether you agree with the verdict. If you automate that gap away, the exercise stops producing anything a load test could not give you.

When you report an evaluation, show each criterion with its verdict and its reason, and the passed and total fraction. Do not rewrite the criteria into one sentence of your own about the copy: a summary is where an abstained criterion turns into a pass or a fail it never was.

Once you run batches, add a third step that chains both. Keep it away from any work you are learning from.

What to measure

Record these on every run.

The fraction. Record passed and total, not only score. The score is a rendering of the fraction, and the fraction is the measurement. With five criteria the score has six positions, so a real improvement is invisible until it changes a whole criterion. If you iterate against the number and it does not move, you know only that no criterion changed its verdict.

The criteria that abstained. Record which criteria were not applicable, and why. An evaluation with one applicable criterion and one with five can show the same score. They are not the same reading.

For a prompt, the held-out lift. lift is the held-out mean of the revised prompt minus the held-out mean of the original. It is a number only when lift_reportable is true. Otherwise it is null, and lift_not_reportable_reason says why (for example held_out_under_3). The train scores come from the in-loop scoring model that chose the revision, so they are never lift, and disclosures.train_scores_are_not_lift says so. Quote lift only.

Live evidence and frozen evidence

Use live search when you produce work. Freeze the evidence you draft from when you compare two versions of a prompt.

A second search can return different quotes, so a change you see could come from your edit or from retrieval. For any prompt-improvement pass, pull the evidence once, write it to a file, and give that same file to every attempt.

The evaluation's own evidence is live. With evidence: "workspace", each evaluation reads quotes from your workspace's conversations at the time of the call, and no call can pin the quotes an earlier evaluation read. So evaluate the two versions you compare one after the other, against the same state of the workspace. Do not compare an evaluation from this week with one from last month as if both read the same conversations.

For a prompt, the sample recipients are the part you control. Send the same recipients to every run you compare. If you leave them out, the run samples up to six of your workspace's own accounts, and two runs can sample different ones.

Two kinds of repeat will fool you

A call that succeeds is not a call that did work

Both repeats below return a valid, complete, well-formed response. Neither returns an error. Only a field tells a new measurement from a replay, and you have to read it.

Search results. Query results are cached for a short window, keyed on the compiled SQL, not on your words. Two differently worded questions that compile to the same SQL hit the cache. The same question that compiles differently on a second pass misses it. Check the cached flag on detail.internal before you conclude anything about consistency.

Optimize runs. The optimize call addresses a run by the content of the request, so the same request again can return the result of the same run (this is what lets a dropped connection recover its answer). Every evaluation carries optimize_id, its handle. Two responses with the same optimize_id are one evaluation, not two. A prompt run started with an idempotency_key that was used in the last 24 hours returns the run the first call started, with replayed: true. Read those fields before you count a repeat as a second reading.

Evaluate the voice separately

The evaluation measures claims and the ask against your evidence. Brand voice is not a claim about a customer, so no criterion covers it.

For voice, use rules that code can enforce. When Amdahl writes a message, context takes constraints that it enforces with code on every version: max_words, banned_phrases, banned_punctuation and keep_cta: "strict" (see Optimize a message). keep_cta: true only marks the ask as the sender's choice and keeps none of its words. For a draft you evaluate, run the same kind of rule yourself, next to the Amdahl evaluation, and report the two verdicts separately.

Set a word limit from your own data, not ours. Amdahl measured length against real reply outcomes on 127,872 labelled outbound emails, and the effect reverses sign between workspaces: in some, longer openers got more replies; in others, slightly fewer. So there is no length we can tell you to write to. A word limit is a house style rule, and only your own outcomes can set it.

Record what happened

When the sender uses a message Amdahl returned, record it with record_message_feedback. Send evidence (required): did_it_myself, reported_to_me or inferred. These name who observed it, not how sure you are. Then send at least one of sent_text (the text actually sent, edits included), comment and outcome (used, used_with_edits, not_used or unsure). With sent_text, the response's rubric_miss is true when Amdahl, scoring as it does now, scored the sent text below the message the run returned. An inferred edit is stored and scored, but it does not train the sender's preferences. Each call stores one more event, so a retry stores it twice.

Sequence

Run the loop by hand, three times

No repo, no layout, no skills. Write a prompt, pull evidence with Search, draft, evaluate the draft with optimize_message in mode: "grade" and evidence: "workspace", and read each criterion and its reason. Keep the files anywhere.

Doing this first means the layout you build next has a shape you saw, not one you guessed.

Create the repo and the canon

Build the bench layout, and move your three hand runs in as the first run folders. Write the canon files (product truth, ICP, brand voice), each with an owner and a review date at the top. Turn the voice rules into code. Start your coverage table.

Still no skills. If you automate before you know the manual version, you encode a guess.

Write the two steps, then compare cold and assisted drafts

Write the two steps from what the first week was actually like, and get an API key for repeat runs.

Then pick five real messages. Write each one cold first, from your own read of the market, and save it. Run the loop on the same brief. Evaluate both, one after the other, and compare the fractions criterion by criterion: which criteria each one passed, and which abstained.

The distance between your instinct and the evidence is what is worth measuring.

Evaluate the prompt

Once the same criteria keep failing, send the prompt behind the drafts to optimize_prompt. Send six recipients you choose, so that three are held out and the lift can be reported. Read outcome first: improved returns a revised prompt, and unchanged means no revision beat the original on the held-out recipients.

Add volume

Point the bench at every sequence and every agent that writes outbound, where the volume is. Put an evaluation in front of anything that publishes, and record feedback on what was sent.

Tests worth running early

Run the first one before anything else. Everything below it assumes an answer.

Before any of these

Confirm that each repeat really evaluated again: a different optimize_id on each response. With the same handle, every stability test below reports a perfect result it never measured.

TestMethodReads on
Does the score discriminateEvaluate three drafts you would rank yourself, best to worst, with evidence: "workspace". Read passed and total.Whether a score difference means anything at all. A separation in your order is a usable dial. A flat result is a scale that cannot separate them.
Criterion stabilityEvaluate the same draft three times, each a new evaluationWhich criteria change their verdict on a repeat
Pretty liesA well-written false claim against an awkward true oneWhether grounding and verified_specifics fail the false one, whatever its tone
Evaluate the rewriteEvaluate a message Amdahl wrote for youWhether it passes more criteria, or only reads better
LatencyTime ten evaluations end to endWhether evaluation can sit inline in a human workflow
Reason qualityFix one failed criterion using its reason, evaluate again, and note which criterion changed and your own read of the copyWhether following the reasons improves the work, or only the number

One run is not a measurement. Because each criterion is a pass or a fail, a draft near the bar can pass on one evaluation and fail on the next. Before you act on a change, or tell anyone a draft got better, evaluate each version several times and compare the medians of the fractions.

Decide before you start

These are decisions, not lookups. If you leave them open, every verdict is measured against a different bar, and the verdicts cannot be compared.

Whether you evaluate with evidence, and on which drafts. An evaluation without evidence: "workspace" reads the ask only. An evaluation with it measures the draft's claims against your conversations. The two are different readings, so record which one each evaluation was, and never pool them.

What counts as a claim. Most marketing copy is sentences that are not assertions about customers. Decide which ones are in scope, because the claim is your unit of measurement once you start counting them.

Which recipients a prompt is evaluated on. A prompt that writes for VPs of Engineering should be evaluated on recipients like them. Pick the six recipients once, write them down, and send the same ones to every run you compare. The sampled default is a fair start, but it is not a fixed control set.

What success looks like

"Our outbound got better" is the work. Here is the read:

  • Runs completed through the full loop
  • Criteria passed out of criteria applicable, and whether that matched your own read of the copy
  • Criteria that abstained, and why
  • For prompts, the held-out lift, and only when it is reportable
  • Disagreements logged, split by cause
  • Whether the reasons improved the copy by your own judgment
  • At least one belief you held that the evidence killed

Going further

Everything above is version one: a manual loop and one readable folder per run. An open-source prompt-test harness (Promptfoo and similar tools) is version two: a grid of prompts against inputs with results side by side, with an Amdahl evaluation as a custom scorer.

Do not start there. It earns its place when you run one prompt across many accounts and comparing by hand has become slow, or when you changed a prompt and need to know which past cases got worse. None of version one is wasted if you go that way: prompts, canon and evidence already live as separate files, which is most of the reason for the layout.

If you build the adapter, have it store optimize_id with every evaluation. A grid that counts one evaluation as many produces a clean table that measured nothing.

See also