Evals API

These endpoints are the HTTP surface of evals: score runs, author datasets of test cases, fan them through a workflow as eval run-sets, and fork a finished run with one declared change. Reads work with a public key; every POST needs a secret key.

Endpoints

Scores

GET /runs/{id}/scores returns a run's evaluation scores, newest first:

[
  {
    "id": "01JZR2Q8...", "runId": "01JZR2P4...",
    "name": "helpfulness", "value": 0.9,
    "dataType": "numeric", "source": "code",
    "comment": "clear and correct",
    "createdAt": "2026-07-01T10:00:05Z"
  },
  {
    "id": "01JZR2QX...", "runId": "01JZR2P4...",
    "name": "grounded", "value": 1,
    "dataType": "boolean", "source": "llm-judge",
    "scorer": "grade-helpfulness",
    "createdAt": "2026-07-01T10:00:07Z"
  }
]

POST /runs/{id}/scores records a score after the fact - a human review, or an agent grading another run's output. name is required; dataType defaults to numeric and source to human. It returns 201 with the recorded score:

{ "name": "human-review", "value": 1, "source": "human", "comment": "looks right" }

Scoring the same name twice records both rows - each annotation is its own entry.

The runs listing filters by score: GET /runs?scoreName=helpfulness&minScore=0.5 returns only runs carrying that named score, with minScore/maxScore bounding its value.

import { createClient } from "@duraton/sdk/client";

const duraton = createClient({ url: process.env.DURATON_URL! });
const scores = await duraton.runs.scores("01HXYZ...");
await duraton.runs.score("01HXYZ...", { name: "human-review", value: 1, source: "human" });

from duraton.client import AsyncDuratonClient, ScoreRequest

async with AsyncDuratonClient() as dx:
    scores = await dx.runs.scores("01HXYZ...")
    await dx.runs.score("01HXYZ...", ScoreRequest(name="human-review", value=1, source="human"))

import (
	"context"

	"duraton.dev/sdk-go/client"
)

dx := client.New(client.Options{})
ctx := context.Background()

scores, _ := dx.Runs.Scores(ctx, "01HXYZ...")
v := 1.0
dx.Runs.Score(ctx, "01HXYZ...", client.ScoreRequest{Name: "human-review", Value: &v, Source: "human"})

curl "$DURATON_URL/runs/01HXYZ.../scores"
curl -X POST "$DURATON_URL/runs/01HXYZ.../scores" -d '{"name":"human-review","value":1,"source":"human"}'

Datasets

A dataset is a named collection of test cases. POST /datasets creates one (name is required and unique per project; a taken name returns 409); GET /datasets lists them:

{ "id": "01JZR3AA...", "name": "qa-cases", "description": "regression questions", "itemCount": 12, "createdAt": "2026-07-01T09:00:00Z" }

POST /datasets/{id}/items appends one test case (input is required) and returns 201; GET /datasets/{id}/items lists them, excluding archived items unless ?archived=true:

{
  "id": "01JZR3AB...", "datasetId": "01JZR3AA...",
  "input": { "q": "capital of france" }, "expected": "Paris",
  "createdAt": "2026-07-01T09:01:00Z"
}

Eval run-sets

POST /datasets/{id}/eval fans the dataset through a workflow: one shadow run per non-archived item (side effects render but do not deliver), all linked to a new run-set. workflow is required; app disambiguates it when the same name exists in more than one app; label names the set for comparison. One fan-out runs at most 200 items; a larger dataset is cut to the first 200. It returns 201 with the set and the created run ids:

{
  "evalSet": {
    "id": "01JZR3B0...", "datasetId": "01JZR3AA...",
    "label": "baseline", "workflow": "qa.answer", "app": "assistant",
    "runCount": 12, "scores": [],
    "createdAt": "2026-07-01T09:05:00Z"
  },
  "runIds": ["01JZR3B1...", "01JZR3B2..."]
}

GET /datasets/{id}/eval-runs lists the dataset's run-sets, newest first, and GET /eval-sets/{id} returns one. scores aggregates each score name across the set's runs - the signal a baseline is compared to a candidate on:

"scores": [
  { "name": "match", "avg": 0.83, "count": 12 },
  { "name": "used:answer", "avg": 1, "count": 12 }
]

A dataset with no items to run returns 400; an eval on an unknown dataset returns 404.

const ds = await duraton.datasets.create("qa-cases");
await duraton.datasets.addItem(ds.id, { input: { q: "capital of france" }, expected: "Paris" });
const { evalSet, runIds } = await duraton.datasets.runEval(ds.id, { workflow: "qa.answer", label: "baseline" });
const sets = await duraton.datasets.evalRuns(ds.id);

from duraton.client import AsyncDuratonClient, DatasetItemInput, RunEvalInput

async with AsyncDuratonClient() as dx:
    ds = await dx.datasets.create("qa-cases")
    await dx.datasets.add_item(ds.id, DatasetItemInput(input={"q": "capital of france"}, expected="Paris"))
    result = await dx.datasets.run_eval(ds.id, RunEvalInput(workflow="qa.answer"))
    sets = await dx.datasets.eval_runs(ds.id)

curl -X POST "$DURATON_URL/datasets" -d '{"name":"qa-cases"}'
curl -X POST "$DURATON_URL/datasets/01JZR3AA.../items" -d '{"input":{"q":"capital of france"},"expected":"Paris"}'
curl -X POST "$DURATON_URL/datasets/01JZR3AA.../eval" -d '{"workflow":"qa.answer","label":"baseline"}'
curl "$DURATON_URL/datasets/01JZR3AA.../eval-runs"

Fork-compare

POST /runs/{id}/fork forks a finished run with one declared change - a different model, prompt, or params on a single AI step. The steps before the changed step are carried over and replay from their stored results; execution resumes at the change. The fork is a shadow run: its side effects render as steps but do not deliver, so comparing against a production run never fires its webhooks or events again. See fork-compare in the evals guide for the semantics.

The body names the step and at least one override:

{ "step": "draft", "model": "claude-haiku-4-5", "params": { "temperature": 0.2 } }

It returns the new run, linked back to the base by replayOf and carrying the declared change in fork:

{
  "id": "01JZR3C0...", "workflowName": "draft-reply", "app": "assistant",
  "status": "queued", "replayOf": "01JZR2P4...",
  "fork": { "step": "draft", "model": "claude-haiku-4-5", "params": { "temperature": 0.2 } },
  "triggerKind": "event", "startedAt": "2026-07-01T10:10:00Z"
}

const fork = await duraton.runs.fork("01HXYZ...", { step: "draft", model: "claude-haiku-4-5" });

from duraton.client import ForkInput

fork = await dx.runs.fork("01HXYZ...", ForkInput(step="draft", model="claude-haiku-4-5"))

fork, _ := dx.Runs.Fork(ctx, "01HXYZ...", client.ForkInput{Step: "draft", Model: "claude-haiku-4-5"})

curl -X POST "$DURATON_URL/runs/01HXYZ.../fork" -d '{"step":"draft","model":"claude-haiku-4-5"}'

The fork is recorded in the control-action audit log as fork_compare; an eval fan-out is recorded as run_eval.

Error codes