Evals

A score is a named evaluation result attached to a run - helpfulness = 0.9, grounded = true, tone = "friendly". Scores are how you measure output quality over time: a workflow can grade its own output inline, a person or an agent can grade a finished run after the fact, and you can filter runs by their scores.

ctx.score - score inline

Call ctx.score from workflow code to record a score on the current run. Like a step, it is durable and memoized: Duraton writes the score once, so a replay after a crash never records it twice. The score name is its identity, so you pass no id.

export const grader = defineWorkflow({
  name: "grade-reply",
  handler: async (ctx) => {
    const reply = await ctx.step.ai.generate("draft", { model: "claude-opus-4-8", prompt });

    await ctx.score({ name: "helpfulness", value: 0.9, comment: "clear and correct" });
    await ctx.score({ name: "grounded", value: 1, dataType: "boolean" });
    await ctx.score({ name: "tone", stringValue: "friendly", dataType: "categorical" });

    return reply.text;
  },
});

ctx.score also takes runId (to score a different run - see deferred scorers) and source, which is code on this path or llm-judge; human is the annotation path below and is rejected from ctx.score.

Score a finished run

Grade a run after it completes - a human review, or an agent judging another run's output - with the REST client (the raw endpoints are in the evals API reference). Each annotation is its own row (scoring the same name twice records both), and source names who produced it.

await duraton.runs.score(runId, {
  name: "human-review",
  value: 1,
  source: "human",
  comment: "looks right",
});

source is one of human (an annotation), code (a deterministic scorer - what ctx.score records), or llm-judge (an LLM-as-judge scorer). It defaults to human over the REST client.

Deferred scorers

A deferred scorer grades a run after it finishes, on its own - no code in the graded workflow. When any run reaches a terminal state, Duraton emits the reserved system event duraton/run.finished. Any workflow subscribed to it runs like a normal run (it shows up in the runs table) and can score the finished run by passing its runId to ctx.score.

import { defineWorkflow, RUN_FINISHED_EVENT, type RunFinishedEvent } from "@duraton/sdk";

export const gradeHelpfulness = defineWorkflow<RunFinishedEvent>({
  name: "grade-helpfulness",
  // Filter to the workflow you want to grade. This is what keeps the loop from feeding
  // itself: this scorer's own run.finished has a different workflow name, so it never matches.
  triggers: [{ event: RUN_FINISHED_EVENT, if: "event.data.workflow == 'draft-reply'" }],
  handler: async (ctx) => {
    const finished = ctx.event.data; // { runId, workflow, app, status, output?, error? }

    const judgment = await ctx.step.ai.generate<{ score: number }>("judge", {
      model: "claude-opus-4-8",
      prompt: `Rate the helpfulness 0..1 of: ${JSON.stringify(finished.output)}`,
      output: { type: "object", properties: { score: { type: "number" } }, required: ["score"] },
    });

    // Score the FINISHED run, not this one. Duraton stamps this workflow as the score's
    // scorer, so the graded run shows who judged it.
    await ctx.score({
      runId: finished.runId,
      name: "helpfulness",
      value: judgment.output?.score ?? 0,
      source: "llm-judge",
    });
  },
});

The duraton/run.finished payload (RunFinishedEvent) carries the finished run's runId, workflow, app, status (succeeded or failed), and its output (or error). A scorer must filter which runs it grades - a scorer with no filter would grade every finished run, including other scorers' runs. Because ctx.score is a durable step, a deferred score is written exactly once even if the scorer replays.

Read and filter by score

Read a run's scores, or filter the runs list to those carrying a named score in a value range.

const scores = await duraton.runs.scores(runId);

const good = await duraton.runs.list({ scoreName: "helpfulness", minScore: 0.5 });

In the console, a scored run shows a Scores count column in the runs table (the values open in the run inspector's Scores section), and the runs filter bar has a Score control that narrows the list to a named score within a min/max value range.

Fork-compare - re-run with one change

Fork-compare re-runs a finished run with one declared change - a different model, prompt, or params on a single AI step - so you can compare the outcome against the original. Duraton replays the run from the journal: the steps before the changed step are reused (their memoized output), and execution resumes at the change.

const fork = await duraton.runs.fork(baseRunId, {
  step: "draft", // the AI step to change (execution resumes here)
  model: "claude-haiku-4-5", // try a cheaper model...
  // prompt: "Be more concise.",    // ...or a different prompt
  // params: { temperature: 0.2 },  // ...or different params
});

The fork is a run like any other (it shows up in the runs table), linked back to the base by replayOf and carrying its declared change in fork. Because the change can't be applied to a stored prompt (Duraton never persists prompts), it is applied when the changed step re-executes - the workflow reruns and the override takes effect at that step.

A fork is a shadow run: its side effects render but do not deliver. A ctx.webhook.send or ctx.step.emit downstream of the change appears as a step in the fork but is not actually sent, so you can safely compare a change against a production run without firing its webhooks or events again.

To compare, read both runs' steps and scores and diff them - the base and the fork share step names, so aligning them is straightforward.

const [baseSteps, forkSteps] = await Promise.all([
  duraton.runs.steps(baseRunId),
  duraton.runs.steps(fork.id),
]);
const [baseScores, forkScores] = await Promise.all([
  duraton.runs.scores(baseRunId),
  duraton.runs.scores(fork.id),
]);

Datasets + eval runs

A dataset is a named collection of test cases - each an input your workflow runs against and an optional expected output to grade against. Fanning a dataset through a workflow creates an eval run-set: one run per item, all graded and compared together.

const ds = await duraton.datasets.create("qa-cases");
await duraton.datasets.addItem(ds.id, { input: { q: "2+2" }, expected: "4" });
await duraton.datasets.addItem(ds.id, { input: { q: "capital of france" }, expected: "Paris" });

// Fan the dataset through a workflow: one shadow run per item.
const { evalSet, runIds } = await duraton.datasets.runEval(ds.id, {
  workflow: "qa.answer",
  label: "baseline",
});

Each eval run is a shadow run - its side effects render but do not deliver (the same as a fork), so evaluating a workflow never fires its real webhooks or events. Inside the handler, the run knows it is an eval run and can grade itself against the item's expected output:

const answer = await ctx.step.ai.generate("answer", { model, prompt: `Question: ${ctx.event.data.q}` });

if (ctx.eval) {
  // ctx.eval is set only on an eval run; ctx.eval.expected is the item's expected output.
  await ctx.assert.equals(answer.text); // scores boolean "match" vs ctx.eval.expected
  await ctx.assert.usedStep("answer"); // scores boolean "used:answer" (tool-choice)
  // await ctx.assert.sequence(["retrieve", "answer"]); // step-sequence order
}

Each ctx.assert.* check records a boolean score with source: "code" and returns whether it passed, so they roll up into the run-set's aggregate alongside any ctx.score or llm-judge grade.

The assertions read the run's step journal so far, so a check only sees the steps dispatched before the call - place them after the steps they grade.

Read the run-set's aggregate to compare candidates:

const sets = await duraton.datasets.evalRuns(ds.id);
// each set: { label, runCount, scores: [{ name, avg, count }] }

The console's datasets view lists a dataset's eval sets with their per-name score averages and compares the two newest sets (base average → candidate average + delta) at run-set granularity. The dataset and eval-run endpoints behind all of this are in the evals API reference.

Agents grade runs too

Scoring, fork-compare, and datasets are all reachable from an AI agent as MCP tools - score_run, list_scores, fork_compare, create_dataset, add_dataset_item, run_eval, list_eval_runs. See the MCP reference.