How many runners, and how to route
An app can register many runners; each is one process (one replica). A run is owned by an app and executed by one of that app’s live runners. The routing handle is therunner id on the event:
Because runners are stateless and Duraton resends the full step memo on every invoke, different
passes of the same anycast run may safely land on different replicas. Scale a stateless fleet
horizontally and route anycast; reach for a pin only when a run genuinely needs one specific instance.
A pinned run only makes progress while its instance is live. If that instance is down, the run
parks and retries up to the bounded no-runner
wait (default 5 minutes), then fails. Anycast has no such single point of failure as long as one
replica is live.
Deploying new code while runs are in flight
A rolling deploy replaces runners one at a time. Because Duraton replays the full memo to whichever runner answers, a single run can take early passes on the old code and later passes on the new code. That is fine - and the reason durable execution survives deploys - provided the step ids the run has already completed still resolve to the same saved results on the new code. Safe to change in a rolling deploy:- Adding new steps after the current point. A run that hasn’t reached them yet just discovers them on a later pass.
- Changing the body of a step that hasn’t run yet. Only unexecuted steps pick up new logic.
- Renaming, removing, or reordering steps that a run has already executed or is parked on.
- Changing a workflow’s shape so an already-completed step’s id no longer appears.
Graceful shutdown and draining
There is no single “drain and exit” call: graceful shutdown means stop being routed new work, let in-flight work settle, then tear down. How you do it depends on the transport.Connect runners
Call the handle’sclose() on your shutdown signal. It stops the socket’s liveness pings and closes
the connection.
close() deregisters the endpoint immediately, so an orderly shutdown has no staleness
window to wait out. Any invoke that was in flight over that socket fails with a retriable transport
error, so Duraton re-dispatches it: if another replica is live it takes over at once; if this was the
last runner the run parks and retries until a
runner reappears.
That immediate eviction is the graceful path only. A connection lost abruptly instead of closed - a
hard kill, a network partition - leaves the endpoint listed until it ages out of the staleness
window (90 seconds), showing as Stale once its last-seen time
falls behind. An invoke that lands on it in the meantime fails as a retriable transport error and is
re-dispatched, so this costs a retry, not a run.
Step ids across deploys
This is the classic footgun. A step id is hashed to the key Duraton stores its result under, and the handler re-runs top to bottom on every pass, matching each step to its saved result by that key. If a step’s id changes between the pass that saved its result and a later pass, the later pass finds no saved result and treats the step as new. The docs’ “keep ids stable” rule is really about this: what actually happens when you break it is silent, not a loud error. Concretely, for a run that is in flight when you deploy renamed step ids:- A completed
step.runyou renamed is a cache miss on the next pass - so its function runs again, repeating its side effect (a second charge, a second email). The old saved result is orphaned. - A parked
sleep/waitForEvent/approvalyou renamed is worse: the run is parked under the old id, but the new code emits a step under the new id, so Duraton parks again on the new step and the old park is orphaned. Asleeprestarts its full duration; awaitForEventwaits again (and an event that had already woken the old wait is discarded); an approval creates a fresh one. The run doesn’t crash - it silently re-does the wait, which looks like a hang until a fresh timeout fires. - Reordering or changing the count of same-id steps (a loop) shifts the positional
suffixes (
x,x:1,x:2, …), so every later occurrence misses its saved result and re-runs.
- Keep step ids stable across a deploy - never rename, remove, or reorder a step that in-flight runs may have already executed or parked on.
- Derive loop step ids and iteration order from already-durable data so the positional suffixes stay stable across passes.
- When a change to step ids or workflow shape is unavoidable, don’t roll it out under the same app while runs are in flight - isolate it.
Blue/green for incompatible changes
For a change that would break in-flight runs - renamed or reordered steps, an incompatible workflow shape - don’t let old runs hop onto the new code. Isolate the two versions and let the old ones drain:- Deploy the new version as a distinct app (or, if you pin, a distinct
runnerid) so Duraton treats it as a separate routing target. - Cut new events over to the new app/runner.
- Leave the old fleet running until its in-flight runs reach a terminal state - each one finishes on the same code it started on.
- Retire the old fleet once it has drained.
Related
Runners
Connect, pin vs anycast, and no-runner behavior.
Steps
Step ids, stability, and reusing an id in a loop.
Durable execution
Why replay makes a run survive a restart or deploy.
Transactional run start
Start a run atomically with an app-side database write.