> ## Documentation Index
> Fetch the complete documentation index at: https://docs.duraton.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Production and operations
> Deploy new agent code without stranding runs in flight: how many runners to run, pin vs anycast routing, graceful shutdown, and the step-id footgun.
Duraton holds the durable state; your **runners** are stateless processes that execute steps. That
split is what makes production operations simple - a runner can crash, restart, or be redeployed and
the run resumes from its last checkpoint - but it leaves a few decisions to you: how many runners to
run, how to route to them, and how to roll out new code without stranding runs that are mid-flight.
## How many runners, and how to route
An app can register **many runners**; each is one process (one replica). A run is owned by an app and
executed by one of that app's live runners. The routing handle is the `runner` id on the event:
| | **Anycast** (no `runner` on the event) | **Pinned** (`runner` set on the event) |
| -------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------- |
| Where it runs | Any one live runner of the app, picked per invoke | Only the named runner id |
| Scaling | Horizontal - add replicas, Duraton spreads invokes across them | Bound to one instance |
| Survives a replica restart | Yes - the next invoke goes to another live replica | Only after that instance is back (parks meanwhile) |
| Use for | Stateless work behind a load balancer | Work bound to one instance - a local resource, a specific agent |
Because runners are stateless and Duraton **resends the full step memo on every invoke**, different
passes of the same anycast run may safely land on different replicas. Scale a stateless fleet
horizontally and route anycast; reach for a pin only when a run genuinely needs one specific instance.
A pinned run only makes progress while its instance is live. If that instance is down, the run
[parks and retries](/core/runners#when-no-runner-is-registered) up to the bounded no-runner
wait (default 5 minutes), then fails. Anycast has no such single point of failure as long as one
replica is live.
## Deploying new code while runs are in flight
A rolling deploy replaces runners one at a time. Because Duraton replays the full memo to whichever
runner answers, **a single run can take early passes on the old code and later passes on the new code**.
That is fine - and the reason durable execution survives deploys - *provided the step ids the run has
already completed still resolve to the same saved results on the new code.*
Safe to change in a rolling deploy:
* **Adding new steps after the current point.** A run that hasn't reached them yet just discovers them
on a later pass.
* **Changing the body of a step that hasn't run yet.** Only unexecuted steps pick up new logic.
Unsafe in a rolling deploy (see [the footgun](#step-ids-across-deploys) below):
* **Renaming, removing, or reordering steps that a run has already executed or is parked on.**
* **Changing a workflow's shape** so an already-completed step's id no longer appears.
For those, isolate the change - see [blue/green](#blue/green-for-incompatible-changes).
## Graceful shutdown and draining
There is no single "drain and exit" call: graceful shutdown means **stop being routed new work, let
in-flight work settle, then tear down**. How you do it depends on the transport.
`client.ready()` probes **Duraton's** `GET /readyz`, and the `draining: true` it can return
means **Duraton** is shutting down - it is not a readiness or drain signal for *your* runner. There
is no runner-side `draining` flag; you drive runner shutdown yourself as below.
### Connect runners
Call the handle's `close()` on your shutdown signal. It stops the socket's liveness pings and closes
the connection.
```ts theme={null}
const runner = connect({ app: "shop", runner: "agent-node-1", workflows });
process.on("SIGTERM", () => {
runner.close(); // stops liveness, closes the socket
});
```
A clean `close()` **deregisters the endpoint immediately**, so an orderly shutdown has no staleness
window to wait out. Any invoke that was in flight over that socket fails with a retriable transport
error, so Duraton re-dispatches it: if another replica is live it takes over at once; if this was the
last runner the run [parks and retries](/core/runners#when-no-runner-is-registered) until a
runner reappears.
That immediate eviction is the graceful path only. A connection lost abruptly instead of closed - a
hard kill, a network partition - leaves the endpoint listed until it ages out of the [staleness
window](/core/runners#liveness) (90 seconds), showing as **Stale** once its last-seen time
falls behind. An invoke that lands on it in the meantime fails as a retriable transport error and is
re-dispatched, so this costs a retry, not a run.
`close()` does **not** wait for in-flight step executions to finish. A step whose function had already
started but whose result had not yet been sent back is re-run on the runner that takes over - at-least
once for that one uncommitted step (already-completed steps are memoized and never re-run). If you need
in-flight work to finish on this instance, stop routing new runs to it and wait before calling
`close()`.
## Step ids across deploys
This is the classic footgun. A step id is hashed to the key Duraton stores its result under, and the
handler re-runs top to bottom on every pass, matching each step to its saved result by that key. **If a
step's id changes between the pass that saved its result and a later pass, the later pass finds no saved
result and treats the step as new.** The docs' "keep ids stable" rule is really about this: what
actually happens when you break it is silent, not a loud error.
Concretely, for a run that is in flight when you deploy renamed step ids:
* **A completed `step.run` you renamed** is a cache miss on the next pass - so its function **runs
again**, repeating its side effect (a second charge, a second email). The old saved result is
orphaned.
* **A parked `sleep` / `waitForEvent` / `approval` you renamed** is worse: the run is parked under the
old id, but the new code emits a step under the new id, so Duraton **parks again on the new step**
and the old park is orphaned. A `sleep` restarts its full duration; a `waitForEvent` waits again (and
an event that had already woken the old wait is discarded); an approval creates a fresh one. The run
doesn't crash - it silently re-does the wait, which looks like a hang until a fresh timeout fires.
* **Reordering or changing the count of same-id steps** (a loop) shifts the [positional
suffixes](/core/steps#reusing-an-id-loops) (`x`, `x:1`, `x:2`, ...), so every later
occurrence misses its saved result and re-runs.
None of these fail loudly; they duplicate side effects or resurrect waits. The rules that avoid them:
* Keep step ids **stable** across a deploy - never rename, remove, or reorder a step that in-flight runs
may have already executed or parked on.
* Derive loop step ids and iteration order from **already-durable data** so the positional suffixes stay
stable across passes.
* When a change to step ids or workflow shape is unavoidable, don't roll it out under the same app while
runs are in flight - isolate it.
## Blue/green for incompatible changes
For a change that would break in-flight runs - renamed or reordered steps, an incompatible workflow
shape - don't let old runs hop onto the new code. Isolate the two versions and let the old ones drain:
1. Deploy the new version as a **distinct app** (or, if you pin, a distinct `runner` id) so Duraton
treats it as a separate routing target.
2. Cut new events over to the new app/runner.
3. Leave the old fleet running until its in-flight runs reach a terminal state - each one finishes on
the same code it started on.
4. Retire the old fleet once it has drained.
This trades a brief period of running two versions for the guarantee that no run ever replays across an
incompatible code change. For compatible changes (only additive or unexecuted-step changes), a plain
rolling deploy under one app is enough.
## Related
Connect, pin vs anycast, and no-runner behavior.
Step ids, stability, and reusing an id in a loop.
Why replay makes a run survive a restart or deploy.
Start a run atomically with an app-side database write.