Production and operations

Duraton holds the durable state; your runners are stateless processes that execute steps. That split is what makes production operations simple - a runner can crash, restart, or be redeployed and the run resumes from its last checkpoint - but it leaves a few decisions to you: how many runners to run, how to route to them, and how to roll out new code without stranding runs that are mid-flight.

How many runners, and how to route

An app can register many runners; each is one process (one replica). A run is owned by an app and executed by one of that app's live runners. The routing handle is the runner id on the event:

Because runners are stateless and Duraton resends the full step memo on every invoke, different passes of the same anycast run may safely land on different replicas. Scale a stateless fleet horizontally and route anycast; reach for a pin only when a run genuinely needs one specific instance.

Deploying new code while runs are in flight

A rolling deploy replaces runners one at a time. Because Duraton replays the full memo to whichever runner answers, a single run can take early passes on the old code and later passes on the new code. That is fine - and the reason durable execution survives deploys - provided the step ids the run has already completed still resolve to the same saved results on the new code.

Safe to change in a rolling deploy:

• Adding new steps after the current point. A run that hasn't reached them yet just discovers them on a later pass.
• Changing the body of a step that hasn't run yet. Only unexecuted steps pick up new logic.

Unsafe in a rolling deploy (see the footgun below):

• Renaming, removing, or reordering steps that a run has already executed or is parked on.
• Changing a workflow's shape so an already-completed step's id no longer appears.

For those, isolate the change - see blue/green.

Graceful shutdown and draining

There is no single "drain and exit" call: graceful shutdown means stop being routed new work, let in-flight work settle, then tear down. How you do it depends on the transport.

Connect runners

Call the handle's close() on your shutdown signal. It stops the socket's liveness pings and closes the connection.

const runner = connect({ app: "shop", runner: "agent-node-1", workflows });

process.on("SIGTERM", () => {
  runner.close(); // stops liveness, closes the socket
});

runner, err := duraton.Connect(duraton.ConnectOptions{App: "shop", Runner: "agent-node-1", Workflows: workflows})
if err != nil {
	log.Fatal(err)
}
defer runner.Close()
runner.Wait()

A clean close() deregisters the endpoint immediately, so an orderly shutdown has no staleness window to wait out. Any invoke that was in flight over that socket fails with a retriable transport error, so Duraton re-dispatches it: if another replica is live it takes over at once; if this was the last runner the run parks and retries until a runner reappears.

That immediate eviction is the graceful path only. A connection lost abruptly instead of closed - a hard kill, a network partition - leaves the endpoint listed until it ages out of the staleness window (90 seconds), showing as Stale once its last-seen time falls behind. An invoke that lands on it in the meantime fails as a retriable transport error and is re-dispatched, so this costs a retry, not a run.

Serve runners

Pass an AbortSignal and abort it on shutdown. That stops the re-registration heartbeat; the runner then ages out of routing after the staleness window (90 seconds), after which no new anycast invoke is routed to it.

const controller = new AbortController();
const invoke = serve({ workflows, app: "shop", runnerUrl, signal: controller.signal });

process.on("SIGTERM", async () => {
  controller.abort();            // stop the heartbeat; the endpoint ages out after the TTL
  await sleep(RUNNER_STALE_TTL); // keep serving in-flight invokes during the drain
  server.close();                // then stop the HTTP listener
});

Aborting the signal does not actively deregister the endpoint - it relies on the TTL - so for up to that window Duraton may still route an anycast invoke to a shutting-down serve runner. Keep the HTTP server answering through the drain window (or let your platform's load balancer stop routing to the instance), then close the listener. Anything that does slip through and fails becomes a retriable transport error and is re-dispatched to a live runner.

Step ids across deploys

This is the classic footgun. A step id is hashed to the key Duraton stores its result under, and the handler re-runs top to bottom on every pass, matching each step to its saved result by that key. If a step's id changes between the pass that saved its result and a later pass, the later pass finds no saved result and treats the step as new. The docs' "keep ids stable" rule is really about this: what actually happens when you break it is silent, not a loud error.

Concretely, for a run that is in flight when you deploy renamed step ids:

• A completed step.run you renamed is a cache miss on the next pass - so its function runs again, repeating its side effect (a second charge, a second email). The old saved result is orphaned.
• A parked sleep / waitForEvent / approval you renamed is worse: the run is parked under the old id, but the new code emits a step under the new id, so Duraton parks again on the new step and the old park is orphaned. A sleep restarts its full duration; a waitForEvent waits again (and an event that had already woken the old wait is discarded); an approval creates a fresh one. The run doesn't crash - it silently re-does the wait, which looks like a hang until a fresh timeout fires.
• Reordering or changing the count of same-id steps (a loop) shifts the positional suffixes (x, x:1, x:2, ...), so every later occurrence misses its saved result and re-runs.

None of these fail loudly; they duplicate side effects or resurrect waits. The rules that avoid them:

• Keep step ids stable across a deploy - never rename, remove, or reorder a step that in-flight runs may have already executed or parked on.
• Derive loop step ids and iteration order from already-durable data so the positional suffixes stay stable across passes.
• When a change to step ids or workflow shape is unavoidable, don't roll it out under the same app while runs are in flight - isolate it.

Blue/green for incompatible changes

For a change that would break in-flight runs - renamed or reordered steps, an incompatible workflow shape - don't let old runs hop onto the new code. Isolate the two versions and let the old ones drain:

1. Deploy the new version as a distinct app (or, if you pin, a distinct runner id) so Duraton treats it as a separate routing target.
2. Cut new events over to the new app/runner.
3. Leave the old fleet running until its in-flight runs reach a terminal state - each one finishes on the same code it started on.
4. Retire the old fleet once it has drained.

This trades a brief period of running two versions for the guarantee that no run ever replays across an incompatible code change. For compatible changes (only additive or unexecuted-step changes), a plain rolling deploy under one app is enough.

Related