salmon

What is left of `specs/per-node-state-machines.md`

2026-09-25T14:15:36Z

Generated from specs/per-node-state-machines-remaining.md — the repository is the canonical source, and may be ahead of this page.

What is left of `specs/per-node-state-machines.md`

Status: living plan, update as work continues. As of this writing every (R) item (R1–R9) and I1, I3, I5, I6 are done — e.g. supReapply, supDemoteEvery (Op/Supervision.hs), ConcurrencyLimit behind run serve --max-concurrency (Op/Concurrency.hs), Serve.Stale with filecontents’ contentFingerprint, Query.resolveRewrittenSelectors — and only I2 and I4 (both matters of taste) remain. The commit hashes in “Where this stands” predate the merge into master. Companion to specs/per-node-state-machines.md (the design, whose milestone list is the source of truth for 1–9) — this file is what remains, why each remaining piece is worth doing, and what order I would do it in. Read the design first if you need the “why” of the model; read this if you want to know where things stand or what to pick up next.

Where this stands

Everything below was built on branch serve-supervision (on top of 0bc3ed4) and has since merged to master. The commit hashes are that branch’s, and some did not survive into master’s history as-is: find the change by its milestone in git log rather than by hash.

Shipped

#	milestone	commit
1	`check :: IO CheckResult` — `prelim` absorbed, `Check.hs`/`Notify.hs` deleted	`52ab4f8`
2	`Salmon.Op.Dag` — the `Cofree` collapse, pure, both directions, conflicts reported	`042297e`
3	`Salmon.Op.Ledger` — per-declaration contributions, nodes and edges, retiring	`323f626`
4	both synchronous drivers over the magma and ledger; the cycle hole closed	`c12c32b`
5	`Salmon.Op.Rewrite` — cross-declaration knowledge as a registered post-fold phase	`39d773d`
6	`Op/Status` + `Op/Mailbox` + `Actions/Concurrent` — one thread per node	`14259cc`
7	`Actions/Upkeep` + `Op/Supervision` — nodes are tended, not applied once	`41f181f`
8	`Extension.managed` + `Nodes/Daemon` — a node can own its process	`ff3faaf`
9	`supStrategy` — a node's going away sends its dependants back to `WaitUp`	`f03afc2`

And three things that are not milestones:

what	commit
(R1), first of three nodes: `Systemd.systemdService` has a `check`	`f7aec15`
(I1) fixed: a bounce is believed over a stale check	`f7aec15`
`salmon-ops-serve-fixture --daemon`, so 8 and 9 can be seen by hand	`12fb625`
(R1), the other half: `CheckResult.Immaterial`, and a node that answers it parks	`1a53d95`
(R1), second of three nodes: `Filesystem.filecontents` has a `check`	`126e0d4`
(R9): `supReapply`, and `Filesystem.dir` sets it — settles (R1)'s third node too	`72e1d55`
(R7): dropped `postOrderM` (dead since milestone 4), deleted unused `historyLines`	`69692a9`/`8a3dc9e`
(R3): `stopTending` snapshots every machine's `Status` onto its node; `status`\/`query` show it	`8a3dc9e`
(R2): `force`\/`recheck`\/`pause`\/`resume [--select P]...` reach the mailbox	`484c738`
(R4), half of it: `run tree`\/`run dag` print the computed `Dag`, not the declared graph	`d57f2a5`

159 tests pass, Layer 3 included. cabal test salmon-ops-recipes --test-option=-j1. Each milestone is marked landed in the design, with its deviations recorded in place there; this table is the index, not the record.

The honest summary of where this leaves things: the execution model is finished, and the nodes have started catching up with it, but only three have. Nine milestones built a per-node state machine, a ledger, a rewrite phase, two concurrent drivers and a supervisor that can own a process and bounce what stands on it — all of which ask each node one question, “is your effect still in place, or is it cheap enough to just make sure?”, that only systemdService, filecontents and dir answer today, out of roughly ninety builtins. That is (R1) properly closed rather than (R1) proven worthwhile: the mechanism now visibly works on a real graph — the fixture self-heals a removed directory with nobody typing anything — but most of this repository’s nodes still have no opinion about their own effect going away, and giving them one remains exactly the per-node work it always was.

Left

Nothing is blocking anything else. (R1) is done — every builtin that most recipes actually declare (systemdService, filecontents, dir) now has an opinion about its own effect going away, one way or another; (R3) is done — a node’s last word about itself, and a failing one’s last output, are now visible in status/query rather than write-only; (R2) is done — force/recheck/pause/resume reach a node from the serve input language; and (I6) is done — a re-declaration that changes a node’s content is now noticed by the convergence pass itself, not only, eventually, by the tending loop. What remains is I2 and I4, both “taste” — not urgent, not bugs, better decided after real use than speculatively now.

id	what	size	note
~~R1~~	~~all three nodes done~~: `systemdService`, `filecontents` have a `check`; `dir` has `supReapply` instead	—	done; §R1
~~R9~~	~~a `Supervision` opt-in for "re-apply me on the loop, it is cheaper than asking"~~	—	done; §R9
~~R3~~	~~`statusOutput` has no reader~~ — snapshotted onto `NodeState`, shown in `status`\/`query`	—	done; §R3
~~R7~~	~~two dead bindings~~ (`postOrderM`, `historyLines`) — dropped and deleted	—	done; §R7
~~R2~~	~~no operator command addresses a node~~ — `force`/`recheck`/`pause`/`resume` do now	—	done; §R2
~~I6~~	~~a re-declaration that changes a node's content does not re-apply it~~ — the pass itself now notices, via a new `Stale` state and `filecontents`' content fingerprint	—	done; §I6
~~R4~~	~~`query` prints the declared graph, not the rewritten one~~ (`tree`/`dag` done; `query`'s selection is now rewrite-aware via a `#ref` fallback)	—	done; §R4; = `specs/advance-querying.md`
~~R5~~	~~supervisor-level restart is half wired~~ — a crashing machine now restarts in place	—	done; §R5
~~R6~~	~~no concurrency-bounding primitive~~ — a global, optional cap now exists	—	done; §R6
~~R8~~	~~`Restart` means two different things~~ (`Systemd` vs `Supervision`)	—	done; §R8
I2	`supStrategy` is authored on the dependency, not the dependant	—	taste; §I2
~~I3~~	~~`supStableAfter` carries two unrelated meanings~~ — split into `supDemoteEvery`	—	done; §I3
I4	the `RestForOne` cascade needs opting in at every hop	—	taste; §I4
~~I5~~	~~adoption refreshes a machine's supervisor but not its policy~~ — a changed policy is now a differing representative, so it is not adopted at all	—	done; §I5

The tradeoffs, in one place

Every milestone departed from the design somewhere; those are recorded in the design’s own milestone list, in place, so they are read next to what they changed. The ones that are still live decisions — where a different answer is defensible and reversing it is a real option — are the (I) items above and in §“Landed, but wanting another iteration”. The three worth knowing without reading further:

run up no longer restarts a healthy systemd unit (f7aec15). That is what giving a node a check costs, and it is the intended improvement, but it is a behaviour change on existing infra.
Supervision only runs while serve is idle (milestone 7). A piped script is therefore never supervised, which keeps serve < script deterministic and makes the feature invisible to any scripted test.
Unknown never restarts anything (milestone 7). Right, given nearly no node has a check — and the reason (R1) is worth more than any remaining milestone was.
A node with no check is no longer watched at all, it is parked (Immaterial). Strictly speaking this removes something: before, such a node was woken once a minute. What it was woken to do was call pure Unknown and go back to sleep, so nothing is lost except the illusion that it was being looked after — which is the point. It makes the (R1) gap legible instead of hiding it behind a busy-looking loop.
One builtin, Filesystem.dir, now re-applies on a timer under supervision rather than sitting parked (R9, supReapply). It is opt-in, narrow (an author-declared claim that up is cheap and idempotent), and read nowhere but Actions/Upkeep, so nothing about run up/run down changed to land it — but it is a real behaviour change under run serve: a dir node that used to sit silent between commands now calls createDirectoryIfMissing again every time its delay elapses, for as long as the effect is stable that is at most once a minute, never zero.

Landed, but wanting another iteration

These are not open work items in the sense (R1)–(R8) are: each one is implemented and shipped, and — (I1) excepted — the code does something coherent today. They are the places where the shape was decided under a single milestone’s pressure and a different answer was defensible — so they want a second pass with the whole thing built, rather than a bug report. (I1) turned out to be a bug and is fixed; (I6), the one that was not a matter of taste, is also fixed. What’s left here (I2, I4) genuinely is taste.

(I1) is fixed — it lost a running process outright, which was not a matter of taste. (I6) said something uncomfortable about what a convergence pass currently does, and is fixed too — see below.

I1. A demoted node consults its own `check` — fixed

This was a bug rather than a fork, and it is fixed; what follows is the record. salmon-ops-serve-fixture --daemon --stale-check gives a daemon node a plausible health check (“my log file exists”); before the fix, driving the loop and changing the config it stood on produced:

Signalling "web" 15
Reaped "web"
serve: daemon sent back to wait: ... stopped being up

…and nothing after it. The process was torn down and never restarted, and the node settled into Up claiming its effect was in place.

demoting sends a node to waitUp, which comes back through attempt ... Consult — and Consult asks the node’s own check first. A node whose check says Success therefore reports Skip and settles straight back into Up without re-running anything (Actions/Upkeep.hs, attempt). For a one-shot node that is merely a missed bounce; the effect really is still there, and the check is right. For a managed node it is a lie the supervisor tells about itself: the machine cancelled the action on its way out of watch, so the process is certainly gone, and any check that says otherwise is stale by construction.

So RestForOne fires only for dependants that cannot tell whether they are up. Today that is nearly every node, which is why the milestone’s tests pass and why it looks like it works. It is exactly backwards from where the tree is going: the whole of (R1) is teaching nodes to answer that question, and every node that learns to stops being bounceable. The flagship case is the casualty — a service with a working health probe reads “still up” and ignores the configuration that changed underneath it, which is the one thing this milestone was for.

The fork. Either a demotion means re-apply — attempt entered with Regardless rather than Consult, the way a restart from look already is — or it means re-evaluate, which is what it means now. The case for the current behaviour is that a spurious demotion then costs one check rather than one up, and that a node’s check is meant to be the authority on whether work is needed. The case against is that it makes the feature self-cancelling: the better a node’s check, the less RestForOne can do to it.

The middle answer was the one taken, and writing it sharpened the rule one step further: the discriminator is not “this node has a managed action” but “this machine was holding the effect when it was sent back”. A demotion out of watch re-applies (Regardless); a demotion out of resting consults. The difference matters for exactly one shape — a managed node whose action forked and exited, which afterExit then watches from resting as an unowned effect. Re-applying that would start a second copy of something already running, which is the case milestone 8 went out of its way to avoid.

So the genuinely contestable half — whether a one-shot node’s demotion means re-apply or re-evaluate — is untouched and still open, decided on its own merits rather than under the pressure of a bug. Test/UpkeepSpec.hs pins both halves, and the first of the pair fails by timing out if the Regardless is reverted.

Milestone 8’s own ordering rule is the precedent and points the same way: it consults the check before the policy because “a process that exits 0 because it daemonised is still up, and the check is the only thing that can say so”. The demotion case is the exact opposite — we stopped the process, so the check is the only thing that cannot say anything useful.

I2. The strategy is authored on the dependency, not on the dependant

§9.2 says the strategy is a per-node knob without saying which end of the edge it hangs off, and Erlang — where it is a property of the supervisor — does not settle it either, because there is no supervisor here to put it on. It went on the node that goes away: the config file declares that its going away matters, and the services reading it say nothing.

The argument for that is real and is in the design doc: the file’s author knows the content is load-bearing, while six services would each have to know separately that it might change. The argument against is equally real and is not written down anywhere — a node’s own restarts are its own business, and this is the one policy in Supervision that lets one node’s author decide something about another node’s behaviour. A RestForOne on a widely shared node is a lever with a very long arm, and nothing warns the nodes on the other end of it.

The fork. Keep it on the dependency; move it to the dependant (“bounce me when anything I stand on moves”); or have both and require them to agree, which is the conservative option and the expensive one.

A reframing that dissolves most of the “long lever” worry. The argument against assumed a bounce is a cost the dependant’s author didn’t sign up for. But the tree already asks every node’s up to be idempotent — safe to run twice — as a base convention (see “Conventions for node authors” in CLAUDE.md). Read RestForOne as “go recheck yourself” rather than “you are being torn down and rebuilt whether you like it or not”, and an unwanted bounce on a well-written node is just a wasted no-op check, not a disruption. That reframes the shared-resource case in the worked example above (§I2’s tlsCert/A/B/C): service C being bounced unnecessarily is a cheap re-verification, not an incident, provided C’s own up/check are actually idempotent — which is already the convention every node is supposed to follow regardless of RestForOne.

What this does not cover: a node whose reapplication is genuinely expensive or unsafe to repeat — slow warmup, an expensive connection pool rebuild, a migration that isn’t safely re-runnable. Such a node has no way today to resist a demotion sent by an upstream RestForOne; supStrategy only speaks from the dependency’s side, there is no dependant-side veto.

Conclusion for now: no implementation change to I2. The current dependency-side authoring is fine as long as the idempotency convention holds. The real gap is a future, separate piece of flexibility: a dependant-side mechanism for a node to declare “don’t force-reapply me from a RestForOne demotion” (or otherwise resist/absorb it), for the nodes that are the exception rather than the rule. That belongs in Extension alongside up/check/down — a per-node capability the node author supplies, the same way check itself is — rather than as another top-level field bolted onto Supervision. Not scoped further than that; noted here so it isn’t lost, not because it’s next.

I3. `supStableAfter` carried two unrelated meanings — done

It was “having been up this long forgets the earlier failures”, read by countFailure. Milestone 9 also made it “do not demote this node twice inside this interval”, read by tooSoon. One field, two jobs, and no particular reason an author would want the same number for both: how long a service has to run before a crash counts as a new crash rather than a continuing one is a different question from how often its dependants may be rebuilt behind it.

This happened because §9.3 named supStableAfter as the mitigation and adding a field looked like exceeding the brief. It was the wrong instinct: the record is meant to grow, defaultSupervision makes growing it free for every node that does not care, and this is precisely the situation the “amend defaultSupervision” convention exists to make cheap.

Landed as the fork’s first option: Salmon.Op.Supervision.supDemoteEvery :: Micros is a new field read only by tooSoon; countFailure keeps reading supStableAfter. defaultSupervision sets both to seconds 10, so nothing changes for a node that has not thought about it — every existing defaultSupervision{...} record update is untouched, since the constructor call sites that needed updating were only defaultSupervision itself (positional Supervision construction has no other caller in the tree).

I4. The cascade needs opting in at every hop

A demoted node unsettles, so a dependant of it that also declared RestForOne sees the same thing and goes back too. That is the whole mechanism, and it means the cascade stops at the first node in the chain that did not opt in: a RestForOne config, a plain service, and something downstream of the service leaves the downstream node alone.

Erlang’s rest_for_one restarts everything started after the failed child regardless of what those children think. Ours is strictly more conservative, which is the right default for a mechanism landing late — but it means the name promises more than the behaviour, and an author reading “and everything after it” will be surprised.

The fork. Leave it (and rename, or at least document the difference loudly); or make the cascade transitive from the declaring node, which means a demotion carries an “originating ref” out to the whole transitive cone rather than only to the immediate dependants that opted in.

I5. Adoption refreshes a machine’s supervisor, but not its policy — done

startUpkeep writes a new Under into every machine it adopts, so an adopted machine follows its current supervisor’s statuses, failure set, neighbour lists and halt flag. It does not rewrite ctxPolicy (Actions/Upkeep.hs), so a managed node whose Supervision changes keeps the old one for as long as it stays adopted — which under serve is indefinitely.

Nor was the change detectable: Dag.sameRepresentative compared the rendering of dynamics, and a Dynamic renders as its type alone by default, so a node whose policy changed and whose ref did not was “unchanged” and got adopted rather than replaced. This predates milestone 9 — but milestone 9 put a second thing in Supervision that matters to other nodes, so a stale policy has reach beyond its own node.

The fork was resolved with the second option: sameRepresentative now sees a changed policy. Dag.showDynamic special-cases Salmon.Op.Supervision.Supervision to render by value (via its own Show instance) rather than by the Dynamic default of its type name alone; every other Dynamic payload (Package and the rest) is untouched. A node re-declared with a changed Supervision is therefore a genuine Representative change: startUpkeep’s adoption test (Dag.sameRepresentative (machineAct m) act) fails for it, so the old machine is Released (its action cancelled through its bracket, tearing down whatever it held) rather than adopted, and a fresh machine starts under the new policy — no separate “refresh the policy in place” step was needed, because a fresh machine already starts with the right one. A node re-declared with an unchanged policy still compares equal and is adopted exactly as before, which is the overwhelmingly common case this must not regress.

This was the “more honest fix” the fork called out: the same comparison feeds dagConflicts/UpDown.Conflicting, so a policy-only disagreement between two live declarations sharing a Ref is now reported there too, not just silently resolved by adoption’s own logic.

Pinned two ways. Test.DagSpec (changedSupervisionIsAConflict / sameSupervisionIsNotAConflict) checks the pure comparison directly: two declarations of one Ref differing only in supStrategy are a dagConflicts entry; two declarations with identical Supervision are not. Test.UpkeepSpec (changedPolicyIsNotAdopted / unchangedPolicyIsAdopted, group “adoption sees a changed Supervision policy (I5)”) checks it end to end through the real mechanism: a managed node’s action never returns on its own, so startUpkeep called a second time either releases it and starts a fresh one — observed as a second spawn of the action — or adopts it and does not, depending only on whether the policy differs between the two calls. Verified the first of those two actually depends on the fix by reverting Dag.showDynamic and confirming changedPolicyIsNotAdopted times out (the stale machine is adopted and never re-runs).

I6. A re-declaration that changes what a node is does not re-apply it — done

Not a milestone-9 decision at all — it predates it, and milestone 9 is only how it came to light. Serve records convergence per Ref, and a re-declaration that changes a node’s content leaves its Ref alone: the magma’s last-writer-wins swaps in the new representative, but the node stays Converged and the gate skips it. The pass says converging (0 down, 0 up) and does nothing at all.

Watchable in the fixture: only --name web --daemon --greeting goodbye after an up ... --greeting hello produces an empty pass, and the new content only lands afterwards, when the config node’s own machine looks and finds the file saying something other than what it should. A node with no check — which is very nearly all of them — keeps the old content indefinitely. So today filecontents with changed content is a no-op on re-declaration, and the operator has no way to tell.

This is (R1) wearing a different hat, and it is the strongest argument for (R1) so far: the checks are not only how drift is noticed, they are currently the only way a deliberate change is applied at all.

Mostly closed since, by (R1)’s second node. filecontents now compares its bytes, so a re-declaration that changes a config file’s content is picked up — by the tending machine rather than by the pass, which is the second of the two paths described below and the one this document said to lean on. Two things that leaves. The pass still reports converging (0 down, 0 up), so an operator watching the pass still cannot tell that anything changed; the change lands quietly, a moment later, when the machine looks. And it still only works for a node with a check — dir is the remaining one that has none, though for dir there is nothing content-bearing to re-declare, so the residual case is narrow.

The fork was resolved with the first option: reset a node’s convergence when its representative changes. Serve.Convergence gained a Stale constructor — read exactly like Pending by gateFor (anything but Converged gets the pass’s attention; the node’s own check decides Skip vs Eval+up from there, same as ever), but kept distinct so status can tell “never touched” from “was up, now re-verifying”. Serve.record compares each incoming declaration’s Ref against whatever worldMagma already had for it via Dag.sameRepresentative — the same comparison foldDag uses to decide a conflict — and demotes a currently-Converged node to Stale when they differ.

That much closes every case sameRepresentative can already see (a changed help/notes/non-Supervision dynamics on a plain node — a Supervision change on a managed node is (I5)‘s territory instead, since gateFor skips every managed node regardless of convergence state). It does not close the flagship filecontents case on its own: sameRepresentative deliberately excludes up/check/down (functions, incomparable), and that is exactly where filecontents’ content lives — its help is the static "writes <path> with some contents" regardless of what bytes it writes, so two declarations differing only in content compared equal.

So the second half was needed too: EncodeFileContents gained contentFingerprint :: a -> Maybe Text, a pure, stable hash of the content an instance would write (Nothing by default; the EncodeFileContents (IO a) instance keeps the default deliberately, since its whole point — already documented as a hazard on checkFileContents — is that the content isn’t known until the encoder actually runs, so nothing pure is available). Text/ByteString/String/Aeson.Value all provide one (SHA256 of the encoded bytes, truncated the same way Query.shortRef truncates a Ref’s hash). filecontents appends "content-hash: " <> h to its notes when its instance has one — which is what makes a content-only re-declaration a genuine Representative change, closing the flagship case: daemon.conf going from "greeting = hello" to "greeting = goodbye" is now Stale the moment it is re-declared, not just eventually noticed by the tending machine.

Verified two ways. Test.ServeSpec’s reDeclareWithChangedContentIsAppliedByThePass is deliberately run through a piped script — under Salmon.Actions.Serve’s own idle-only tending model that is the case with zero chance for a tending machine to ever run, so it is the sharpest possible demonstration: before this landed, the exact same test left the file saying its first content forever, proven by reverting the Serve.hs half of the change and watching the assertion fail with expected: "goodbye" but got: "hello". Test.DagSpec/Test.UpkeepSpec’s (I5) cases already pin sameRepresentative’s general behaviour; no separate pure test was needed for the Stale-vs-Pending gating itself, since gateFor’s /= Converged check already treats every non-Converged constructor alike.

What’s left, narrow and pre-existing rather than new: a node whose content lives purely inside up’s closure and whose EncodeFileContents instance (if it even goes through filecontents at all) has no contentFingerprint — the IO a case, and any hand-rolled node that writes content without going through filecontents/checkFileContents at all (the fixture’s own configOp, kept hand-rolled because it also needs RestForOne, sidesteps this by putting the greeting directly into its own help text instead). Such a node still relies entirely on its own check, running later on the tending loop, exactly as before this landed — which is Serve.record’s own documented limit: it can only demote what it can see, and a node author who wants a re-declaration’s content change to register through the pass itself either needs a content-derived Representative field (as filecontents now has) or a check, per the convention (R1) already established.

Not in any milestone

The design’s milestone list is about the execution model. These are things the model now wants from the rest of the tree, plus the loose ends seven milestones left behind. (R1) is the one that matters.

R1. Nodes have no `check`, so almost nothing is actually supervised

The single highest-value item in this document, milestones 8 and 9 included. One of the three candidates below is now done, and the framing has changed underneath the other two — see “What Immaterial settled” below.

check is the only thing in the model that can notice an effect going away. Counting assignments across salmon-ops/src/Salmon/Builtin/Nodes/ and salmon-ops-recipes/src/: 21 sites in 12 files, against ~91 op nodes in the builtins alone. And the misses are the common nodes — Filesystem.filecontents and Filesystem.dir had none — the module’s two checks were replaceDirectory’s inner move and destroyDirectory, both skipIfDirectoryIsMissing — nor does Bash.run, nor did Systemd.systemdService. Two of those are now done; dir and Bash.run are not. A node with no check answers Immaterial (it answered Unknown until this change), which the upkeep FSM parks, so it is brought up once and thereafter watched by nothing. The engine is real and tested; on a real graph today it does nearly nothing.

This was already logged as an ordering question in §“Open questions” (“wants exercising on two or three real long-running nodes before milestone 7 hardens it. Tracked in todo”). Milestone 7 landing sharpens it: the question is no longer “does this shape work” but “which nodes get a check”.

Three candidates, in the order I would do them:

Systemd.systemdService — done. Systemd.checkService shells out once to systemctl show --property=ActiveState --property=UnitFileState --property=NeedDaemonReload, and Systemd.interpretShow (pure, tested in Test/SystemdSpec.hs) draws the verdict. Three departures from the one-line sketch above, each found by writing it:
- is-active alone is not enough, because this node’s own dependency rewrites the unit file before the check ever runs. Comparing the bytes on disk against what we would write can therefore only ever say “they match”, and a changed unit would be rewritten and never restarted. NeedDaemonReload is systemd’s own record of “the file changed since I loaded it” and is the only thing that still remembers.
- A transitional state is Unknown, not Failure. activating, deactivating and reloading mean the service has not gone away, and treating them as gone is how a slow starter becomes a restart loop. This is the first place in the tree where Unknown is the right answer rather than the absence of one.
- UnitFileState earns its place on its own: a unit somebody systemctl disabled is still running, so ActiveState says everything is fine right up until the next reboot.
The behaviour change is the expected one and is documented on the function: a unit that is installed, enabled, loaded and running is now skipped by run up rather than reloaded-enabled-restarted every time.

(R8) did not come due here, contrary to the prediction below: the collision only bites a module that needs both Restarts in scope, and this one never imports Salmon.Op.Supervision — the supervision policy for a systemd unit belongs on the caller’s nodes, not on this one. The interaction still to get right when someone does write one: a unit with its own Restart= is already supervised by systemd, so salmon’s Supervision for it should be OnFailure or Never and never Always — two supervisors fighting over one service is worse than one.
Filesystem.filecontents — done. Filesystem.checkFileContents compares the bytes on disk with the bytes the node would write. Correct rather than approximate (skipIfFileExists would say Success for a file with the wrong bytes), and cheap in the only sense that matters here: the node’s content is already in hand, since up is about to encode it anyway. Comparing bytes rather than decoded text also sidesteps the invalid-UTF-8 question the sketch worried about, and covers the ByteString/Aeson.Value instances for free. Four things found in the writing:
- The size is compared first, and a mismatch answers without reading. One stat, and it bounds what a node holding a few hundred bytes reads if something else has clobbered its path with something enormous.
- The reason must not quote the contents. Failure text goes into reports, and this node writes pgbouncer userlists and postgrest configurations with signing keys in them. Test/FilesystemSpec.hs pins that.
- This is what made (R1)’s first node actually work. systemdService writes its unit file through filecontents and then asks systemd whether the unit needs reloading — and systemd answers that from the file’s mtime. Rewriting byte-identical contents on every pass therefore set NeedDaemonReload=yes on every pass, so checkService said Failure on every pass and reloaded-and-restarted a healthy service. The claim in f7aec15 that a healthy unit is now skipped was true of checkService in isolation and false of the graph it sits in, until this landed. Verified against a real systemctl --user unit: rewriting identical bytes flips NeedDaemonReload to yes.
- The EncodeFileContents (IO a) instance is a hazard, and the only one. The check runs the encoder, so a side-effecting generator runs once more per look and a non-deterministic one (a timestamp) makes the node rewrite its file on every pass. Safe direction, but documented on the function; such a node wants a stable encoder or a check of its own. Nothing in the tree uses that instance today.
Filesystem.dir — settled, and not with a check. (R9) landed and dir is what it was written for: it declares supReapply rather than comparing doesDirectoryExist, so under run serve it re-runs createDirectoryIfMissing on the tending loop instead of asking a question that would have cost the same stat for no extra information — doesDirectoryExist and createDirectoryIfMissing are within noise of each other, so there was nothing to buy by asking first. Nothing changes under a one-shot run up/run down: the field is read only by Actions/Upkeep, and dir’s check still answers Immaterial either way. See (R9) for the mechanism and Test/UpkeepSpec.hs’s dirSelfHeals for the end-to-end case — a real dir node, a real rmdir behind salmon’s back, put back with nobody re-declaring anything.

Milestone 8 narrows this in one respect and widens it in another. A node that owns its process needs no check at all to be supervised — the action’s exit is the authority, which is most of what ownership was for — so Nodes/Daemon.hs works today with nothing added. It made the gap sharper for everything salmon does not own, which is every service already under systemd, and that is the gap the first candidate above has now closed: a unit that stops behind salmon’s back is noticed and restarted, and one whose file changed is reloaded.

Doing it also produced the first evidence that this list is in the right order. Giving a real node a real check is what turned (I1) from a fork into a demonstrated bug — a service node has a check, so it walked straight into being torn down and left down — and the two landed together for that reason.

What `Immaterial` settled

The complaint above bundled two things that turn out to be separable, and separating them is most of what this item needed.

The first is coverage: a node that can stop being true on its own, with nothing in the model able to notice. That is unchanged, and it is what the two remaining candidates fix.

The second was a category error in the default. “This node has no check” and “this node’s check ran and could not tell” were the same answer, Unknown, and the FSM had to treat them the same way — which meant polling 78 of 100 builtins once a minute to call pure Unknown. CheckResult now has a sixth constructor for the first case. Immaterial means there is nothing here worth asking about: applying the effect costs about what finding out would, which is exactly the property that makes those nodes idempotent in the first place (mkdir -p, ip route replace, ALTER SYSTEM SET, an append-if-missing). It is the default, so no node author writes it; the one-shot drivers map it to Required and cannot tell it from Unknown, so run up is byte-for-byte unchanged; and under Actions/Upkeep a node that answers it parks — blocked on its mailbox, its demoting dependencies and its own action, with no delay ladder — instead of polling.

Three things that buys, none of them the obvious one:

Unknown now means only what it says. It was carrying two meanings, and every rule about it had to be written for the weaker one. The systemd check’s transitional states are the case that wants the real Unknown, and they now have it to themselves.
The gap is legible. A parked node is visibly not being watched. Before, a node nobody could supervise looked identical, from the outside, to one being supervised successfully — same NextLook line every 60s. Making the engine stop pretending is what turns (R1) from a note in a document into something an operator can see in status.
The cost of the default is now proportional to what it claims. A node that says “don’t ask, just apply me when something would have applied me” is asked exactly once, learns that, and stops. It costs one check per supervisor rather than one per minute.

What it does not do is make those nodes self-healing: a parked dir whose directory somebody removed stays parked. Doing something about that is (R9), and it was deliberately not part of this change — it is a second, independent decision about whether re-applying on a loop is acceptable, and it belongs to the node author rather than to CheckResult. (R9) has since landed and Filesystem.dir now makes that decision; see §R9.

The behaviour change the remaining two carry

Each of these changes what run up does for every existing caller: a node whose check says Success stops being re-applied. That is an improvement (it is what the check convention is for, and CLAUDE.md’s idempotency section already asks for it) but it is a behaviour change on the author’s own infra, which is why milestone 7 deliberately did not smuggle any of them in. Land them one at a time, each with its own commit and its own Layer-1 test, so a regression is attributable.

R2. An operator cannot address a node, so the mailbox is unreachable — done

Salmon.Op.Mailbox was built, Upkeep.instruct was built and tested, and Force/Satisfy/Recheck/Pause/Resume all meant something to the FSM — but nothing in the serve input language could name a node, so none of it was reachable except from Haskell. Four commands close that:

force   [--select P]... [--exclude P]...
recheck [--select P]... [--exclude P]...
pause   [--select P]... [--exclude P]...
resume  [--select P]... [--exclude P]...

(Satisfy gets no command, matching this section’s own “the four instruction commands” — it is Query.forceSkip’s territory, decided at declare time, not an operator’s run-time say.) They reuse parseSelection / resolveWorldSelectors exactly as sketched — the same Set Ref status/ query/converge --select already compute — and an empty selection means every node, same as those three.

The caveat this section flagged landed as the smaller option it named: the instruction is queued, not posted. Tending gained tendingPending :: IORef (Map Ref [Instruction]); the command handler resolves the selection and queues onto it (oldest first per node, so a pause then a resume is delivered in that order) and immediately reports how many nodes matched (Instructed). Nothing is posted into a mailbox at that moment — loop already runs stopTending before every command, status included, so there is never a live one to post into regardless of whether the target is a one-shot or a holding machine. startTending drains the whole queue into Upkeep.instruct the moment the next supervisor’s machine table exists — after adoption, so both a freshly-started machine and an adopted one see it — and clears it. This is exactly “force this node next time you look at it”, and needed no change to Upkeep.startUpkeep’s signature: the queue is delivered from the caller’s side, after the call returns, not threaded through it.

A selected node that no live machine ever answers to (excluded from every active epoch, retired, or simply never reached by tending) silently drops the instruction at delivery time, same as Upkeep.instruct already does for any unknown Ref — there was nothing to queue it for once its target never showed up. Instructed’s count is therefore a statement about the selection, not a delivery receipt; the two can differ and that is not a bug.

Milestone 8’s promise is now real: pause on a node that owns a process stops tending it without touching the running service, and force on one is how to restart something that is healthy and currently has no other way to be told to. See Test.ServeSpec’s forceOverridesASatisfiedCheck (a second up with the check never once unsatisfied — the only thing that can explain it is force itself) and pauseThenResume (an effect allowed to vanish while paused, and confirmed not put back until resume).

R3. `statusOutput` has no reader — done

The bounded per-node ring is written (the machine narrates its transitions, and milestone 8 gives it real process output) and nothing read it. status couldn’t: the supervisor is stopped while any command is handled, so the TVars were gone by the time it ran.

Landed as the first of the two options sketched here: stopTending snapshots every machine’s Status — read from the Upkeep.Supervisor via Upkeep.supervisorStatuses, before Upkeep.stopUpkeep partitions it into stopped and kept — onto a new nodeStatus :: Maybe Status field on NodeState. status/query render it: a [CheckResult] on every node’s summary line, and — the thing milestone 8 made worth more than it was, since the ring now carries a managed process’s actual stdout/stderr — the tail of a failing node’s output ring underneath, capped at ten lines so one wedged node cannot bury the rest of the listing.

Freshness needed no new mechanism: every command already runs stopTending before it is handled (see loop), so a snapshot is never more than one command old, and a holding machine is re-adopted (and so re-snapshotted) into the next supervisor the next time tending starts — which happens before every command too. The second option sketched here (read a holding machine’s TVar live rather than snapshotting it) turned out to buy nothing extra given that rhythm, so it was not built.

Pinned by Test.ServeSpec.statusShowsAFailingNodesOutput: a node whose up never stops throwing is declared, fails synchronously once, and is then picked up by the idle tending loop (it is Unsettled, not yet Converged) — which is what actually produces the Failure this test reads back through status, not the declaring pass. Verified by hand too: chmod a directory read-only, declare a dir under it, and status shows [Failure "...: permission denied"] with the repeated up / error lines underneath.

R4. `query`/`run tree`/`run dag` still print the declared graph — done

Known and recorded at milestone 5. Registered Rewrites apply to run up/run down/run serve but not to the three commands that describe a graph, so query shows twenty deb nodes where run up will run one apt-get. The obstacle is structural rather than an oversight: a rewritten Dag has Refs and edges and no paths, and --select matches path globs (Query.resolveSelectors walks a Cofree). Printing the computed graph needs either a renderer that does not exist or ref-addressed patterns.

Landed for the half of this that has no --select to reconcile with paths in the first place: run tree/run dag take no selection at all, so nothing about them needed the fork above resolved before writing the renderer. CommandLine.hs’s Run RunTree/Run RunDAG now fold and rewrite the graph the same way runUp/runDown do (computedTreeDag, sharing Rewrite.wholeGraph’s “everything desired, nothing ignored” Phase) and print the resulting Dag through two new renderers: Help.printDagTree and Dot.printDagCograph. Both are one line per Ref rather than one per path — a node reached from several declarations is printed once, the way it is walked once — with dependencies listed underneath (printDagTree) or as plain edges (printDagCograph, which necessarily drops the red/orange/gray Connect/Overlay distinction Dot.printCograph draws from Shape, since a Dag has already collapsed both into “depends on”). With no rewrites registered this is the same nodes and edges as before, just collapsed to one line/node instead of one per tree position — verified by hand against the salmon-ops-serve-fixture binary (no rewrites registered): run tree/run dag on a plain bundle directive show the same four nodes either way, just without the duplicate positions a shared node used to get.

query was the holdout, and the part that actually needed the fork this section opened with resolved — it is the one of the three whose whole job is resolving a --select/--exclude pattern, which only paths could do.

Resolved with option (2): resolve a pattern against the declared graph as before, and translate the result through membersOf. Landed as Query.resolveRewrittenSelectors (Salmon.Actions.Query), a drop-in for resolveSelectors that CommandLine.hs’s QueryShow/QueryPlan handlers now call, passing the same whole-graph Rewritten run tree/run dag already compute (computedTreeDag factored into computedRewritten, so the Rewritten — not just its computedDag — is available to query too). Ordinary path-glob patterns are untouched — resolveRewrittenSelectors degrades exactly to resolveSelectors’s behaviour when no #-pattern is given, checked by test.

The fallback lookup, added alongside rather than instead of (2): a declared path genuinely cannot address a rewrite-introduced node (a package-install batch, say) at all — such a node has no position in the declared tree, since it was never declared, only ever produced after the fold. So a pattern beginning with # matches by Ref instead of by path: a prefix of either shortRef or the full ref text, checked against every declared node and every computed (rewrite-introduced) node, with a computed match expanded through membersOf back to the declared nodes it stands in for. This deliberately mirrors renderAnnotated’s own " #" <> shortRef ref disambiguation suffix (already printed today next to a colliding path, and — since run tree/run dag moved to Dag-based renderers — the same short-ref text that would identify a batch node there) so that text a render prints can be pasted straight back in as a selector, symmetric with how git short hashes work.

Both kinds of pattern union rather than override each other within one --select/--exclude, and an empty --select list still means “everything” when checked against the combined pattern list (not just its path half) — the bug the first draft had, caught by rewrittenEmptySelectStillMeansEverything before it shipped: an exclude-only --exclude '#...' with no --select at all must still select everything else, not silently narrow to nothing.

Every result is still a set of declared refs, deliberately: query plan’s phaseIgnored and a rewrite’s own collectDynamic are both keyed on declared refs (runUp’s Phase is built and consumed before any rewrite’s batching decision), so this needed no change to what run up consumes — addressing a batch by its ref and excluding it is exactly equivalent to excluding every declared package that went into it, which is the only coherent meaning available to a plan computed before any rewrite runs. query show’s rendering is unchanged too (still the annotated declared tree via printAnnotated) — only what a --select/--exclude pattern can match changed; nothing needed the tree it annotates to become the computed one, since the declared identities are still the more useful thing to show next to [selected]/[excluded].

See Test/QuerySpec.hs’s resolveRewrittenSelectors-prefixed cases, which cover: plain path patterns unchanged; a #ref pattern addressing a plain declared node directly; one addressing a batch and expanding to its declared members; a path and a #ref pattern combining within one selection; and the empty-select-still-means-everything edge case above.

R5. Supervisor-level restart — done

§“The supervision tree” wants two levels: the upkeep FSM handles the managed effect stopped, and a supervisor handles the machine managing it died. Milestone 7 had the monitoring (stopUpkeep does waitCatch on every machine and reports Escaped) and not the restart — a machine that threw was reported and gone until the next idle period rebuilt every machine anyway. That was a tolerable accident of “supervisors are rebuilt per idle period” and stopped being tolerable the moment one outlives a command (see R2).

Landed as milestone 9 made it smaller: restarting a machine in place needs nothing beyond handing the replacement its supervisor’s current state, and Upkeep.Under — a TVar a machine re-reads on every wait rather than closing over — already is that, since it exists precisely so an adopted machine sees a live supervisor rather than a dead one’s maps. startUpkeep now runs every machine through a new wrapper, restarting, instead of machine directly: a crash is reported (Escaped, on every attempt — “the honest fix is to keep it loud” turned out to mean report each restart loudly, not decline to restart) and the machine restarts in place, re-entering as Unsettled rather than wherever the dead one’s closure remembered. Unsettled is what makes the very next step a fresh Consult rather than a blind up — nothing survived the crash, not even the assumption that the effect is still there. A fixed delayFloor pause (not the adaptive ladder, which is a policy about the node, not about this module’s own bugs) separates one restart attempt from the next, so a bug that fires on every entry cannot spin a core.

One hazard found in the writing, not anticipated by the sketch above: the restart wrapper’s try @SomeException must not catch an asynchronous exception. releaseKept tears a holding machine down by cancelling its thread — throwing AsyncCancelled into it — specifically because such a machine ignores the halt flag and has no other way to be stopped; a wrapper that treated that as a crash and restarted the machine would defeat the teardown releaseKept’s caller is waiting on. SomeAsyncException is matched via fromException and re-thrown untouched instead.

Named restarting rather than supervised, which was the obvious name and already taken — Salmon.Op.Supervision.supervised :: Supervision -> Dynamic is the smart constructor that attaches a policy to a node’s dynamics, an unrelated thing. Recorded so a future reader does not reach for the same name a second time; see (R8) for the collision this project already has of this shape.

R6. Bounding concurrency — done

§“Bounding concurrency” decided in two parts and shipped the first (collections, milestone 5). The second — a bounding primitive for the case a collection cannot express, e.g. two batches fighting over the dpkg lock across different rewrites — was explicitly deferred and still is: no per-resource primitive is added here, deliberately. Two nodes contending for one specific thing is still an edge or a collection’s job, and nothing in this item changes that.

What landed is the smaller, orthogonal thing this section used to only note: a single global knob capping how many nodes are inside their own check/up/down at once across one pass, for machines where unbounded width itself is the problem (CPU/IO contention, an outbound connection limit, file descriptors) rather than any particular pair of nodes fighting over a particular resource. Salmon.Op.Concurrency.ConcurrencyLimit is a thin wrapper over a QSem; newConcurrencyLimit builds one from a positive Int (errors on <= 0, since a limit of zero would deadlock every gated action rather than mean “run nothing” — that is what excluding every node from the pass already says) and withConcurrencyLimit holds one slot for the duration of an IO action, a no-op for Nothing.

Salmon.Actions.Concurrent.upDagConcurrent/downDagConcurrent (and the walkConcurrent both share) take a Maybe ConcurrencyLimit; Nothing reproduces every caller’s behaviour from before this landed. The slot is held only around walkConcurrent‘s apply call — a node’s own check/up/down — never around the STM wait on its neighbours’ waitStability, which is what makes this safe to reason about without a deadlock analysis: a node cannot even attempt to acquire a slot until every node it depends on has settled and released its own, so two nodes never hold a slot each while blocked on one another through this mechanism — the only thing a wait through it can ever be for is a free slot, never another node’s turn.

Salmon.Actions.Serve.serveWith takes the same Maybe ConcurrencyLimit and passes it to both halves of a convergence pass — the teardown walk and the bring-up walk share one limit rather than getting one each, which is correct because converge already awaits the first before starting the second, so the two never contend for it at the same time. serve (no rewrites, for callers that don’t need them) passes Nothing, matching its signature before this landed for anyone not opting in.

run serve --max-concurrency N is the CLI surface: RunServe gained a Maybe Int (parsed with optional (option auto (long "max-concurrency" ...)), so omitting the flag is Nothing), and execCommandOrSeedWithRewrites builds the ConcurrencyLimit from it right before calling serveWith. run up/run down are untouched — they run through the sequential drivers (UpDown.upDag/downDag), which are already bounded to one node at a time by construction, so there was nothing for this knob to do there.

One thing worth being explicit about: this bounds a pass, not the tending loop Actions/Upkeep.hs runs between commands. A wide serve declaration can still start as many supervised machines as it has nodes; each one is normally idle (parked, or waiting out its own delay ladder) rather than doing work, so the unbounded-width problem this item was written for is specific to a convergence pass actually doing many things at once, which is exactly what --max-concurrency now caps. Bounding the tending loop itself was not asked for and is a different, larger question — the loop’s own steady-state cost is designed to be near zero per idle node (see Actions/Upkeep.hs’s summary), so there is little evidence yet that it needs one.

R8. `Restart` means two different things — done

Salmon.Builtin.Nodes.Systemd.Restart (rendered into a unit file’s Restart= directive; one constructor, OnFailure) and Salmon.Op.Supervision.Restart (Always/OnFailure/Never) shared both a name and a constructor. Nothing imported both, so nothing was broken — but this was the fourth collision of this kind in this work (CheckResult’s Success/Failure vs optparse’s ParserResult, Mailbox.Skip vs Report.Skip, and two Directions, the last resolved by merging them, which was not available here).

The two are genuinely different things: one is a string salmon writes into a file for systemd to read, the other is a decision salmon makes itself. Renamed Systemd.Restart to Systemd.RestartDirective (constructor OnFailure untouched, only the type name moved) rather than waiting for a caller that needs both in scope, since the section’s own prediction — a systemd node with a check (R1) is exactly the node that would want a Supervision too — is exactly the situation (R1) already created for systemdService. Nothing outside Systemd.hs named the old type (checked across both cabal.project and cabal.perso.project’s package sets), so this was a same-module rename with no call-site fallout.

R7. Two dead bindings — done

Salmon.Op.GraphFold.postOrderM (salmon-core) had no in-repo caller since milestone 4 moved both drivers onto Dag, and its own module doc still named upTree as the caller it was written for — stale as well as dead. Dropped rather than kept speculative (it was untested, and this document’s own “used or dropped” framing asked for a decision); recover it from history if a caller needs it again. foldWithContext, the module’s other export, is not dead — Actions/Dot.hs uses it — and is untouched.
Salmon.Actions.Serve.historyLines was unused and not even in the module’s export list (historyLinesMatching (const True); history goes through the Matching version directly). Deleted.

R9. “Re-apply me on the loop; it is cheaper than asking” — done

The second half of the Immaterial design, deliberately not landed with the first. Immaterial says don’t poll me; it said nothing about what a supervisor should do for a node whose effect is cheap to re-apply and can still go away — Filesystem.dir was the whole argument. A dir that somebody rmdirs was not noticed before Immaterial, was not noticed after it either, and would not have been noticed by (R1) candidate 3 as a check would have — at the cost of a doesDirectoryExist per node per minute, which is the trade Immaterial exists to avoid making silently.

The shape landed as sketched: Salmon.Op.Supervision.supReapply, a Bool field on Supervision, opt-in on the node, meaning “on the tending loop, just run up again rather than asking”. The node keeps the delay ladder it would otherwise have parked out of, and the ladder’s meaning inverts — it is now a rate limit on re-application rather than on looking. Filesystem.dir sets it; nothing else in the tree does.

The three things flagged as needing to be got right, and how each landed:

It re-runs up on a schedule, forever. Still true, still not the default, still narrow: supReapply defaults to False, defaultSupervision sets it False, and it is documented on the field as sound only for an up that is genuinely cheap and genuinely idempotent. Bash.run, cabal-build, git-repo’s clone >> pull and rsync:send-dir remain Immaterial and none of them set it.
It interacts with RestForOne. Landed by not going through unsettle/Upping at all: a successful reapply calls neither, so statusEpoch never moves and a RestForOne watcher sees nothing — which is correct, since the node never stopped being up from a dependant’s point of view. A reapply that fails still reaches a watching dependant, but through the existing markFailed/failed-set path crossing already reads, not through the epoch — so no new mechanism was needed for the one case that does need to be seen. Pinned by reapplyDoesNotDemoteDependants (success, several times, nobody sent back) and failingReapplyGivesUp (failure, folded into the ordinary supGiveUpAfter/backoff machinery via the same failed function a one-shot up failure uses).
The reporting has to distinguish it from a restart. A new Reapplying report takes NextLook’s place for such a node — filtered from serve’s output the same way Parked is, visible in status. Distinguishing it from a real restart turned out to need no extra signal beyond that: a restart passes back through Upkeep act Upping, and a reapply never does — pinned by reapplyStaysInUp, which asserts Upping is reported exactly once (the original arrival) across several successful reapplies.

One thing not anticipated when this was written: a node holding a running action (managed) had to be excluded explicitly. Such a node’s up throws by convention (see Nodes/Daemon.hs), so supReapply is read only by resting (the non-holding loop); watch (the holding one) treats anything other than Poll as Park, regardless of the field. Pinned by managedIgnoresSupReapply.

filecontents was the evidence for the other half of the original argument: it wanted a real check, because comparing bytes is better than re-applying — it is exact, it is one stat in the common case, and re-applying would have churned the mtime that systemdService reads. dir has none of those properties: there is nothing to compare beyond existence, and createDirectoryIfMissing costs about what doesDirectoryExist costs. So dir got this field rather than a check, and nothing else in the tree has both properties at once — the two R1 nodes and this one between them cover the shapes that exist today; a fourth node wanting either treatment should re-read this section’s argument rather than copy whichever one is closer.

The order I would do it in

(R1), (R9) and (R3) are done. What is left is reach and a handful of independent small items, not coverage or visibility.

~~I1~~, ~~R1’s first node~~ and ~~R1’s default~~ — done. The first two together, because the node with a real check is what proved the bounce-over-stale-check question was a bug rather than a preference. Then CheckResult.Immaterial: the half of (R1) that is one decision rather than per-node work, and that makes the rest of it visible — a node nobody can supervise now says so instead of emitting a NextLook a minute.
~~filecontents~~ — done, and it turned out to be the node that made (R1)’s first one work: systemdService’s unit file goes through it, and rewriting identical bytes was setting NeedDaemonReload on every pass. It let the fixture’s config node drop its hand-rolled check for checkFileContents, and later (see (I6) below) grew a contentFingerprint that closes the rest of what filecontents alone left open.
~~R9, and with it dir~~ — done. supReapply landed as sketched: a Bool on Supervision, read only by the non-holding loop, deliberately outside unsettle/Upping so a successful reapply cannot fire RestForOne, folded back into the ordinary failure machinery when it throws. Filesystem.dir sets it and is now the third (R1) node — closed without a check, which is the answer §R1 candidate 3 was undecided about.
~~R3~~ — done. stopTending snapshots every machine’s Status onto its NodeState before the Upkeep.Supervisor holding the live TVar is dropped; a holding machine gets a fresh snapshot too, since it is re-adopted before every command. status/query show a [CheckResult] per node and a failing one’s last output lines underneath — milestone 8’s ring finally has a reader.
~~R7~~ — done in passing: postOrderM dropped (dead since milestone 4), historyLines deleted (dead since it was written). Both trivial, both cost nothing to do the moment they were noticed rather than later.
~~R2~~ — done. force/recheck/pause/resume [--select P]... [--exclude P]... parse the same way status/query/converge --select already do, and land in a Tending-owned queue rather than a mailbox: the supervisor is always stopped by the time a command is handled, so there is never one to post into at parse time. startTending drains the queue into the next supervisor’s machines — freshly started or adopted — the moment they exist.
~~R5~~ — done. startUpkeep now runs every machine through restarting rather than machine directly: milestone 9’s Under refresh turned out to be most of what a supervisor-level restart needed to hand a replacement machine, so the remaining work was the restart loop itself (re-entering Unsettled, reporting Escaped on every attempt) and making sure it lets an asynchronous exception — releaseKept’s cancel above all — through untouched rather than treating it as a crash.
~~R4~~ — done. run tree/run dag print the computed Dag; query’s selection is now rewrite-aware too, via Query.resolveRewrittenSelectors (declared-path patterns, translated through membersOf, plus a #ref fallback for addressing a rewrite-introduced node directly). ~~R6~~ is done — a global, optional ConcurrencyLimit bounds a convergence pass’s width, reachable as run serve --max-concurrency N; the per-resource primitive it was explicitly not about is still nothing more than an edge or a collection. ~~R8~~ is done — the rename cost nothing to do ahead of a caller needing both, once checked.
I2 and I4 whenever there is an opinion to apply. Neither is urgent and neither is a bug — both are questions about what the feature means that are better answered after somebody has used it on a real graph. ~~(I3)~~, ~~(I5)~~ and ~~(I6)~~ are done.

~~(I6) has no place in that order~~ — it is done now: Serve.Convergence gained Stale, Serve.record demotes a Ref off Converged when its representative changes, and filecontents gives sameRepresentative a content-derived field to see in the flagship case that comparison alone could not. See §I6 for the two-part fix and how each half was verified.

How milestones 8 and 9 actually went

Kept because both departed from the design in ways worth knowing before touching that code, and because §9.1 and §9.3 below are the reasoning the milestone-9 departures are corrections to.

Milestone 8: `Managed` nodes

Landed as planned, with six departures recorded in place in specs/per-node-state-machines.md. Three are worth knowing here because they change what the remaining work looks like:

Lifecycle is a field, not a sum (managed beside up), as this plan recommended. If a third lifecycle ever appears, that is when to pay for the sum.
A machine holding a process is Kept across commands. This was not in the plan and is the largest thing milestone 8 added: serve stands its machines down before every command, so a supervisor that wound its processes down with it would restart every service on every status. Holding machines survive and the next supervisor adopts them, on exactly the condition §“Ref is location-addressed” named — still wanted up, and its representative unchanged.
The restart policy got supStableAfter/supGiveUpAfter, folded in rather than deferred, as recommended. Which also means milestone 9’s flapping hazard (§9.3) already has its mitigation available.

The Restart name collision the plan flagged is still latent — see (R8).

Milestone 9: `rest_for_one`

Landed with five departures, recorded in place in specs/per-node-state-machines.md. Two of them are corrections to the sketch below rather than choices, and both are worth knowing before touching this code:

§9.1’s “watch the dependencies’ statuses” cannot work as written. A level read of Stability misses every departure it is for — a dependency that fell over and recovered between two of a dependant’s waits looks identical to one that never moved, and a rewritten config file is exactly that shape. Status.statusEpoch (monotonic, bumped when a settled node unsettles) is what the dependant compares against instead.
§9.3’s use of supStableAfter as a settling delay would swallow the case the feature is for, for the same reason: the config is back within milliseconds. It is a rate limit on repeat demotions instead — an isolated departure is always honoured, a second one inside the interval is dropped.

Two things the sketch got right and one it did not anticipate: the strategy is per-node and authored on the node that goes away (§9.2), the cascade needed no code, and the thundering herd (§9.1) does not arise at all, because a machine with no opted-in dependency subscribes to nothing. What it did not anticipate is that an adopted machine had to be handed its new supervisor’s state — see the design’s fourth departure, and (R5) below, which this made smaller.

The original analysis follows, for the reasoning behind the shape.

9.1 The exact gap, as the code stands

Salmon.Actions.Upkeep.look does half of this already. When a check says the effect is gone it marks the node failed in the supervisor’s TVar (Set Ref) — so a dependant still in WaitUp holds off, which is the one-shot drivers’ Blocked containment expressed as a wait. What it does not do is touch a dependant that has already reached Up: that node is napping in resting and never looks at its dependencies again.

So the missing piece is small and precisely locatable: when a node leaves Up, its dependants’ machines have to go back to WaitUp. Salmon.Op.Status.unsettle is already the function for that. What is missing is a machine that observes it — resting‘s STM choice would gain a branch watching its own dependencies’ statuses, which is waitStability inverted (“wake me when one of these stops being Stable/TurnUp”).

That branch is also why this is last. Every node in a supervised serve would then hold a live STM subscription to its dependencies for as long as it is up, and a flapping leaf wakes its whole transitive cone. On a wide graph that is the one part of this design with a plausible thundering-herd behaviour, and it should land where it can be measured rather than early where it cannot.

9.2 It has to be a per-node choice

Erlang’s one_for_one is “restart just this node”; rest_for_one is “this node and everything after it”. §“The supervision tree” is right that the strategy is a natural per-node knob — a config-file node probably wants rest_for_one (a service reading a config that changed should be bounced), a log shipper probably wants one_for_one (nothing downstream cares).

That is a third field on Supervision, which is already the per-node policy channel and already optional:

data Strategy = OneForOne | RestForOne
supStrategy :: !Strategy   -- default OneForOne

Default OneForOne, which is today’s behaviour exactly — so this milestone changes nothing until a node opts in, which is the property that makes it safe to land at all. Note this is the opposite default from supRestart’s (OnFailure, the active choice), and deliberately: restarting a node that fell over is a statement about that node, while bouncing its dependants is a statement about other people’s nodes.

9.3 The hazard to design against

A node that flaps — check fails, check succeeds, check fails — with RestForOne dependants demotes and re-runs its whole cone on every flap. supStableAfter is the mitigation and it exists now (milestone 8): demote dependants only once the node has been down long enough to count against the tally, not on the first failed check. Milestone 9’s job is to use it, not to add it.

9.4 What to test

a dependant already Up is demoted when its dependency leaves Up, and comes back after it does;
with OneForOne (the default) it is not demoted at all;
a demoted dependant does not run up until the dependency is Stable again — i.e. the demotion goes through WaitUp and not straight to Upping;
a node with no dependants demotes nothing and costs nothing.

Relationship to the other specs

specs/salmon-as-init.md is no longer gated on this work. Its PID-2 supervisor is the upkeep FSM and its restart policy is Supervision, both landed; a node that owns a process is Nodes/Daemon.hs, and “restart this service and everything after it” is supStrategy. What it still needs from here is (R1), for anything it does not own. The Rust PID 1 remains unaffected: that boundary is about waitpid(-1), and everything here waits on specific children — deliberately, which is why the polling getProcessExitCode reaper the removed Supervised module used was not recovered along with its teardown.

specs/advance-querying.md is R4. specs/multi-user-privilege-separation.md still composes unchanged — an Invoker decorates the CreateProcess a node spawns, which is as true of a Managed node’s process as of a OneShot’s.

The salmon model

2026-09-25T14:15:36Z

Generated from resources/salmon-core.md — the repository is the canonical source, and may be ahead of this page.

The salmon model

This document describes the general, domain-independent model that lives in salmon-core — the algebraic graph, the OpGraph/Track abstractions, and the traversal semantics. Everything infra-specific (files, systemd, postgres, podman, …) is built on top of this model in salmon-ops and above; none of it is baked into salmon-core itself. If you want copy-pasteable recipes for writing infra ops, see howto-ops.md instead — this document is about the shape underneath, and where else that shape could be pointed.

The problem this is solving

There’s a recurring tension in provisioning/CI-CD/ops tooling:

One-off tools that do one narrow thing (apply these DB migrations, sync this one directory) are highly tunable but don’t compose with each other.
Do-everything tools (turn up my whole dev environment, push to prod) are powerful but their behavior becomes inseparable from their configuration — changing “generate a secret then ssh in” to “ssh in then generate a secret” means editing the tool, not just its config.

Salmon’s bet is that both needs are really the same problem at different scales, if you have one representation that’s equally comfortable expressing “create a file” and “turn a server up” — because a “turn a server up” op is, underneath, nothing but a big DAG of smaller “create a file”/“start a service”/… ops. Get that representation right, and a one-off tool and a do-everything tool are the same kind of object, just different-sized graphs.

The four core pieces

`Graph` — the algebraic graph

data Graph a = Vertices [a] | Connect (Graph a) (Graph a) | Overlay (Graph a) (Graph a)

An algebraic graph in the style of the Alga paper, simplified to a single Vertices [a] constructor instead of separate empty/vertex/ overlay. Connect g1 g2 means “g1’s nodes each precede g2’s nodes” (an ordering relationship); Overlay g1 g2 means “these two subgraphs co-occur, with no ordering implied between them.” This is the substrate; everything else in salmon-core is built on top of it.

`OpGraph` — a self-centered node with an effectful neighbor-finder

data OpGraph m node = OpGraph
    { predecessors :: m (Graph (OpGraph m node))
    , node :: node
    }

This is the key trick that makes “create a file” and “turn a server up” the same type despite wildly different dependency depth: an OpGraph doesn’t carry its whole dependency tree up front — it carries a recipe (predecessors, itself effectful in m) for producing its immediate predecessors, on demand. “Create a file” and “turn a server up” both typecheck as OpGraph m node; they just differ in how deep and effectful their predecessors recipe turns out to be when it’s actually run.

Two ways to compose two OpGraphs (from Salmon.Op.OpGraph):

x `inject` y — add y as a Connect-ed predecessor of x (“y must happen before x”).
x `overlaid` y — add y as an Overlay-ed predecessor of x (“co-occurring with x’s existing predecessors, no ordering implied”).

`Track` — “given an `a`, I know how to build the graph for it”

newtype Track m n a = Track { run :: a -> OpGraph m n }

A Track is a contravariant, divisible functor — a promise, parametrized by input type, to produce an OpGraph. Concretely: “if you hand me a database spec, I have a way to turn it into the ops that provision that database.” This is the composition mechanism for builders: it lets a piece of code that needs something (a database, a signed certificate, a running binary) depend on an abstract “here’s how to get one” rather than a concrete provisioning strategy — the caller supplies the Track, so the same consuming code can be pointed at “provision it locally” vs. “assume it’s already there” (ignoreTrack) vs. “fetch it from elsewhere” without changing.

Tracked pairs a Track with an already-realized value, so dependent values can be built compositionally: mapTracked/apTracked/bindTracked give it quasi-Functor/Applicative/Monad shapes (each one recording the accumulated dependency graph as it goes), and using/using2/using3 let you “open” one or more Tracked values to build a downstream OpGraph while automatically wiring in everything they depended on.

`Eval.expand` — materializing the graph

expand :: (Monad m) => OpGraph m node -> m (Cofree Graph (OpGraph m node))

Walks the effectful predecessors recipes all the way down and produces a fully materialized Cofree Graph — the actual, concrete dependency tree, ready to be folded over (by a traversal like upTree/downTree, or by anything else you want to write against a Cofree Graph). This is the one place where “recipe for finding neighbors” becomes “the neighbors.”

Why a graph, not a plain sequence of steps

A flat ordered script conflates two different things: “must happen before” and “happens to be listed first.” A DAG keeps them separate — two independent branches of provisioning can be expressed as unordered (Overlay) and a traversal is free to run them concurrently (run serve does, one thread per node — Salmon.Actions.Concurrent), retry just one, or report exactly which one failed, without the script author having had to think about interleaving at authoring time. The DAG shape is also what makes dedup possible: because nodes carry an explicit identity (Ref, at the salmon-ops layer — see howto-ops.md §2.1), the same logical resource reached via two different paths through the graph is recognized as one node, not provisioned twice.

`Actions`/`Act`/`Actionless` — the generic “bag of named effects”

salmon-core’s Salmon.Op.Actions module defines the generic wrapper that turns a plain node payload into something a traversal can act on uniformly:

data Act ext = Act { shorthand :: ShortHand, extension :: ext }
data Actions ext = Actionless | Actions (Act ext)

Actionless is the monoidal identity — a node that exists purely for its position in the graph (grouping/ordering) with genuinely nothing to run. salmon-core itself is agnostic about what ext actually is — it’s a type parameter. salmon-ops’s Extension (see howto-ops.md) is one particular choice of ext, tailored to shell-command-driven infrastructure provisioning (up/down, both IO ()-shaped, plus a check :: IO CheckResult). Nothing about OpGraph/Track/Eval requires that choice.

Generalizing beyond infrastructure

Everything above is agnostic to what an “op” actually does. The concrete salmon-ops layer picks IO ()-shaped shell-command effects because that’s what provisioning needs, but the same Graph/OpGraph/Track/Eval core would support a differently-shaped ext, as long as:

individual units of work can be given a stable identity (a Ref-equivalent, for dedup),
“did this succeed” can be observed (an exception, a return value, whatever fits the domain), and
“what must happen before this” can be expressed as a graph rather than a flat list.

Concretely, this points at use cases beyond “provision a server,” building on the same up/down/check shape:

CI/CD pipelines — the existing use case in this repo (see SreBox.CabalBuilding, SreBox.GeneratedSite): a build/test/publish pipeline is itself a DAG of idempotent steps, and “did this artifact already get published” is exactly the kind of check-skippable question the model was built around.
Multi-machine/fleet orchestration — SreBox.PostgresMigrations’s “upload self and re-invoke as a nested upTree on the remote machine” pattern (see howto-ops.md §5) is already a working instance of “a node’s up is itself a whole graph traversal happening somewhere else”; the same shape generalizes to fleet-wide convergence (each machine’s target state expressed as a graph, driven from a control machine).
Data pipelines — a pipeline stage (“did this table/partition get computed for this date”) is structurally the same triad as “did this file get written”: a Ref keyed on (dataset, partition), an up that computes and writes it, a check that skips recomputation if the output already exists and its inputs haven’t changed (the same shape as skipIfFileExists/skipIfNftRuleExists — see howto-ops.md §4), and a dependency graph that’s naturally a DAG already (this aggregate depends on those upstream tables). Where salmon would differ from a typical DAG scheduler (Airflow-style) is that here the dependency graph can be computed effectfully at expansion time (predecessors :: m (Graph ...)) rather than only declared statically ahead of a run — useful when which upstream partitions a stage depends on isn’t known until you’ve inspected what’s actually available.
Business process modeling / process mining — a business process (an order-fulfilment workflow, an approval chain, an onboarding sequence) is itself a graph of steps with real “must happen before” edges, “already done, skip it” idempotency (has this approval already been recorded?), and a genuine need to know when a step failed vs. was blocked by an upstream failure — exactly upTree’s Failed/Blocked distinction. Modeling a process this way rather than as an implicit control-flow graph buried in code has a second benefit specific to process mining: because the graph is a first-class, inspectable value (the same Tree/Dot rendering used for infra graphs — see howto-ops.md §9) rather than something that only exists as the emergent behavior of a program, the reported trace of a traversal (which nodes were Evaled, Skipped, Failed, Blocked, and in what order — see Salmon.Actions.UpDown.Report) is already the kind of event log process-mining tooling wants to reconstruct a real process model from, without needing separate instrumentation bolted on after the fact.
Database/schema migrations as a DAG rather than a linear chain — most migration tools force a strict total order; expressing migrations as an OpGraph would allow independent migrations (different schemas/tables) to be legitimately unordered relative to each other while still enforcing real dependencies (this migration needs that table to exist first) explicitly, and would get idempotent-rerun and partial-failure reporting for free from upTree.
Declarative, convergent configuration management generally (the Puppet/Ansible/Terraform space) — the up-is-idempotent, check-skips-if-already-satisfied, down-tears-back-down shape is exactly a “declare desired state, converge to it, converge back” model; what salmon adds relative to most tools in that space is that the DAG is a first-class, inspectable value (Tree/Dot output — see howto-ops.md §9) rather than a topological order baked into engine internals, and that a node’s dependencies can be computed (the predecessors recipe is effectful) as well as declared statically.
Anything with a “does this already exist” / “make it exist” / “tear it back down” triad and a real dependency structure between the things being managed — the infra domain is simply the first and most fully-built-out instance of that triad in this codebase, not an assumption baked into salmon-core.

None of the above beyond CI/CD and fleet orchestration are implemented in this repository today — they’re recorded here as directions the core model supports, not as existing code to go read.

Vocabulary (from the original salmon-core README)

These terms recur throughout the codebase and its documentation:

builtins — mostly atomic nodes, DAGs with small diameter (salmon-ops’s Nodes/ modules: Filesystem, Systemd, Postgres, Podman, etc.).
recipes/apps — combinations of builtins; many equivalent graphs may be valid for the same end state (salmon-ops-recipes’s SreBox/ modules).
configs — user-provided choices, evaluated on the commanding machine (the seed/Configure side of the CLI protocol — see howto-ops.md §9).
setup — machine-rationalized state, evaluated on the local/target machine (the Spec/Op side of the same protocol).
prefs — conventions parametrized by domain or service (e.g. “migrations always ship and run locally” vs. “via a remote connstring” — a convention salmon-ops-recipes deliberately enforces, per its own package description).

Where to go next

howto-ops.md — concrete, copy-pasteable patterns for writing and testing salmon-ops nodes on top of this model.
salmon-core/src/Salmon/Op/{Graph,OpGraph,Track,Eval}.hs — the actual source, all four modules short enough to read end to end in one sitting. Salmon.Op.GraphFold and Salmon.FoldBranch are the two generic folds over an expanded Cofree Graph (context carried from ancestor to descendant; root-to-node paths, which run tree and query print).
salmon-ops/src/Salmon/Op/Dag.hs — how an expanded graph is collapsed into a DAG with one node per Ref, and edges both ways; every driver walks that rather than the tree.
salmon-ops/src/Salmon/Actions/UpDown.hs — the reference traversal (upTree/downTree) built on Eval.expand and Dag; the closest thing to a “how do I fold over one of these graphs for real” example. Salmon.Actions.Concurrent and Salmon.Actions.Upkeep are the concurrent and continuously-supervising versions run serve uses — see serve-supervision.md.

Terraform integration: consuming (and optionally driving) existing Terraform-managed infra

2026-09-25T14:15:36Z

Generated from specs/terraform-integration.md — the repository is the canonical source, and may be ahead of this page.

Terraform integration: consuming (and optionally driving) existing Terraform-managed infra

Status: draft / not implemented. This is a design sketch to react to, not a committed plan.

Problem

Salmon’s own job stops at “given a machine that already exists, converge it to a declared state” — nothing in salmon-core/salmon-ops creates compute/network/DNS-zone resources at a cloud provider, and nothing in the repo talks to Terraform today (confirmed: no terraform/.tf/HCL references anywhere in salmon-*). Meanwhile there may already be Terraform state describing the machines/networks/DNS zones this project needs to target — the [[pg-ha control plane]] spec’s “Machines” leaf in particular (“control-plane seeds must be unfoldable to Machines…”) has to come from somewhere, and if that somewhere is already Terraform, salmon shouldn’t duplicate or fight it.

What “integration” could mean — two different things

It’s worth naming these as separate models up front, because they have very different blast radii and this spec recommends starting with only one of them.

Model A — read-only consumption. Terraform (run by hand, by CI, by whatever already runs it) is the source of truth for resource existence; salmon only ever reads its outputs (machine IPs, DNS zone ids, generated credentials Terraform provisioned) to build seeds/directives. Salmon never calls terraform apply/destroy.

Model B — salmon-driven apply. Salmon shells out to terraform itself, treating a Terraform root module as one more Op in the graph (up runs apply, down runs destroy), so a single salmon-x run up can bring up both the Terraform-managed layer and everything salmon layers on top in one traversal.

Recommendation: build Model A first, treat Model B as optional/future. Model A composes with “we may have already-existing terraform usage” directly — it’s non-invasive by construction, since salmon never mutates anything Terraform owns. Model B requires deciding who owns lifecycle (salmon down calling terraform destroy against existing hand-managed infra is a real footgun — see §3’s caveats) and isn’t needed to unblock the pg-ha control-plane work, which only needs machine inventory as an input.

Design goals / non-goals

Goals:

No new core (salmon-core) mechanism — reuse the same insight as the previous-seed-state design in specs/pg-ha-control-plane.md: Configure’s gen :: seed -> m a is already impure, so “read Terraform outputs” is just another impure read at seed-to-directive time, exactly like reading a previous-state JSON file.
Salmon never becomes a Terraform state owner. It reads terraform output -json (or, for Model B, drives apply/destroy against a root module it’s pointed at) — it does not generate, template, or hand-edit .tf files. Whoever wrote the existing Terraform config keeps owning it.
Salmon never manages cloud credentials itself. Both models assume terraform/its providers are already configured to run ambiently (env vars, an assumed role, a configured backend) exactly as if a human ran terraform apply at that path — salmon just shells out to the terraform binary the same way Binary.withBinary already wraps every other external tool in this codebase.

Non-goals (v1):

Generating/templating Terraform HCL from salmon seeds (the inverse direction — “salmon describes infra, emits .tf”). A materially different, much larger feature; not needed to consume existing usage.
Remote state backend management (S3 bucket + DynamoDB lock table, TFC workspace creation, etc.) — assumed to already exist if it exists at all.
Import of unmanaged resources into Terraform state — out of scope; if something needs importing, that’s a one-time terraform import a human runs, upstream of anything salmon touches.

Proposed design

1. Reading outputs into a seed (Model A)

A small new module, e.g. SreBox.TerraformState (recipes-level, not core — this is derivation logic, not an IO primitive with its own up/ down):

-- shells out to `terraform output -json`, or reads a cached copy of that
-- JSON from a file (useful for offline/repeatable seed generation, and
-- avoids requiring the `terraform` binary + provider credentials just to
-- build a directive)
readOutputs :: FilePath -> IO (Either String Aeson.Value)  -- workdir, or a captured -json file

A seed carries either a working directory (to shell out live) or a path to a previously-captured terraform output -json file — same “carry a Maybe FilePath” shape the previous-seed-state design already established:

data TerraformSource
    = LiveWorkdir FilePath (Maybe Text)   -- terraform root dir, optional workspace name
    | CapturedOutputs FilePath             -- a `terraform output -json > file` snapshot

gen then parses the specific outputs a given recipe needs (e.g. machine_ab_0_ip, machine_ab_1_ip, dns_zone_id) out of the JSON into strongly-typed seed fields, the same way any other Configure step turns loosely-typed input into a typed directive — failure to find an expected output key is a gen-time error (surfaced before any Op runs), not something recipes downstream ever need to handle.

This directly answers the pg-ha control-plane spec’s open question of “where do machine addresses come from”: ControlPlaneSeed’s machine fields become optional-override-else-read-from-Terraform, i.e. the seed carries a TerraformSource and the two pair_machine_a/pair_machine_b addresses are populated from it during gen, rather than being typed in by hand every time.

2. Recommended workflow shape

# Terraform, run however it already is (CI, human, whatever) — salmon doesn't touch this
terraform -chdir=infra/prod apply

# capture outputs once (or every time before a converge, cheap either way)
terraform -chdir=infra/prod output -json > tf-outputs.json

# salmon reads the capture, same two-phase protocol as everything else
salmon-x config --terraform-outputs tf-outputs.json ... | salmon-x run up

This keeps the existing hermetic seed→directive→ops boundary completely intact (per CLAUDE.md’s description of why that split exists) — Terraform becomes just one more impure input gen reads, alongside the previous- state file from the pg-ha spec. Nothing about Actions/Serve.hs’s World/ Epoch model needs to change either: a serve loop re-reads whatever tf-outputs.json currently says on every seed declaration, same as it would re-read any other input file.

3. Optional: `Salmon.Builtin.Nodes.Terraform` as an `Op` (Model B)

If/when salmon-driven apply is actually wanted, it fits the existing node shape cleanly — unlike nft/ip link, terraform apply is naturally idempotent (a no-op plan applies as a no-op), so this is a “prefer replace”-bucket node per CLAUDE.md’s conventions, not a prelim-skip one:

data TerraformRoot
    = TerraformRoot
    { tf_workdir :: FilePath
    , tf_workspace :: Maybe Text
    , tf_var_file :: Maybe FilePath
    }

apply :: Reporter Report -> Track' (Binary "terraform") -> TerraformRoot -> Op

up: terraform -chdir=<workdir> [workspace select <ws>] apply -auto-approve [-var-file=<f>], via the ordinary withBinary/untrackedExec path (which already throws on non-zero exit, satisfying CLAUDE.md’s “failure must not be swallowed” convention for free — no special handling needed here).

down: terraform -chdir=<workdir> destroy -auto-approve — deliberately not wired to run automatically from a generic downTree walk in v1. destroy against a root module that predates salmon’s involvement, or that other tooling/humans also touch, is exactly the kind of hard-to-reverse, shared-state action this project’s own operating conventions (see the “Executing actions with care” guidance salmon is developed under) say needs an explicit, deliberate trigger — not something that happens as a side effect of some unrelated node’s teardown pulling in a shared predecessor. Concretely: expose apply as an Op usable in up/tree/dag, but ship its down as Actionless/no-op by default, with a separate, explicitly- named destroy value (not wired into the same Op) that a caller has to opt into deliberately if they really want salmon-x run down to reach it.

4. Reading Terraform-generated secrets

Terraform providers commonly generate credentials (a random DB password, a provider-issued API key) as sensitive outputs. Treat these exactly like every other secret in this codebase per the existing convention ([[recipe key exchange agnostic]]): readOutputs surfaces them as plain values at gen time (already local, already trusted — no new transport invented), and downstream recipes take them as ordinary pre-provisioned secret values, same as they’d take a secret read from any other file today. No new secret- handling machinery needed.

Open questions

Does “already-existing Terraform usage” mean one root module or several (e.g. separate network/compute/DNS roots, possibly separate workspaces per tier/environment)? Determines whether TerraformSource needs to be a list (read/merge outputs from multiple roots) rather than one workdir — leaning towards supporting a list from the start since “one big root module” vs “several small ones” is a common enough split not to special-case away.
Output naming contract: does salmon assume specific output names (machine_ab_0_ip, etc.) that the existing .tf files would need to expose (possibly requiring someone to add outputs to already-existing config), or does gen need a mapping/config layer between “whatever outputs already exist” and “what the seed needs”? Depends entirely on what the existing Terraform code currently outputs — worth looking at before finalizing the parsing shape in §1.
Model B’s destroy exposure: even as an explicit opt-in value (not wired to down), should it require something stronger than “a Haskell value the caller chooses to reference” — e.g. a separate CLI subcommand gated behind its own confirmation prompt/flag — given the blast radius? Leaning yes, but the concrete UX depends on how destroy would actually be invoked in practice (interactively vs. from CI).
Version/binary pinning: does the environment already pin a Terraform version (a .terraform-version/required_version in existing config), and should the new Track' (Binary "terraform") check/require it, or just shell out to whatever terraform is on PATH like every other Binary track in this codebase does today (recommend the latter — consistent with existing conventions, and version mismatches surface as ordinary terraform errors rather than needing salmon-side detection)?

Phased plan

SreBox.TerraformState.readOutputs (§1) against a captured tf-outputs.json file only — no live terraform output shell-out yet, no Op involved at all, just JSON parsing into typed values. Cheapest possible slice, immediately unblocks the pg-ha control-plane spec’s “where do machine addresses come from” question.
Wire a TerraformSource into ControlPlaneSeed (from the pg-ha spec) as an alternative to hand-typed machine addresses.
Live shell-out variant (terraform output -json, not just a captured file), once the captured-file path has been exercised for real.
Model B (Terraform.apply, §3), only if/when there’s a concrete need for salmon to drive apply itself rather than assuming it already ran.

Future work

HCL generation/templating from salmon seeds (the inverse direction), if the “already-existing usage” turns out to be small enough that salmon owning it outright becomes more attractive than treating it as a fixed external input.
A destroy-with-plan confirmation flow analogous to advance-querying.md’s Plan/digest mechanism, if Model B’s opt-in destroy needs more ceremony than a bare CLI flag.

Salmon as PID 1: an init system whose unit graph is a real DAG

2026-09-25T14:15:36Z

Generated from specs/salmon-as-init.md — the repository is the canonical source, and may be ahead of this page.

Salmon as PID 1: an init system whose unit graph is a real DAG

Status: the init system itself is not implemented — milestones 1 to 6 (the Rust salmon-init, the spawn protocol, a SalmonInit.service node, ordered shutdown) have no code. All four prerequisites it consumes have shipped: 0a supervision (Nodes/Daemon.hs, Op/Supervision.hs, Actions/Upkeep.hs, Actions/Serve.hs’s tending loop), 0b history retention, 0c the serve-language socket (run serve --listen, Salmon.Actions.Serve.Socket; also --http), and 0d pull mode with a cached last document (run serve --follow ... --follow-cache, Salmon.Actions.Follow). Sections below were re-checked against the tree as of 2026-09-23; where the original text had been overtaken it was rewritten rather than annotated. A design sketch to react to, not a committed plan.

What has shipped since this was first written

The original draft’s “three things that genuinely are not here” is now one thing. The other two moved into run serve, exactly as the draft asked, and this spec now consumes them:

Draft asked for	What the tree has now
a vocabulary for "keep this running"	`Extension.managed :: Maybe (Output -> IO ExitCode)` and `Nodes/Daemon.hs` — a node whose effect is a running process, torn down by cancelling its `withAsync` (group signal, `stop_grace`, `SIGKILL`), output drained into a ring
restart policy, backoff, give-up latch	`Op/Supervision.hs` (`Restart` `Always`/`OnFailure`/`Never`, `supStableAfter`, `supGiveUpAfter`, `RestForOne`, a watchdog) read by `Actions/Upkeep.hs`'s per-node machine, with its `Tally` of consecutive failures and adaptive delay
exit-event-driven convergence	`Upkeep`'s `Up` state races the managed action; the `ExitCode` it yields is what the policy reads, after the node's own `check` (so a daemonising service that exits 0 is still up)
a "supervisor table keyed by `Ref`"	per-machine state in `Upkeep`, not in `World`; `Serve.NodeState.nodeStatus` snapshots it
`prelim`-alive / `Skippable`	`check :: IO CheckResult`; `Unknown` means "keep looking", `Immaterial` parks
a supervisor that survives its own commands	`Upkeep.Kept`: machines holding a `managed` effect outlive the supervisor that started them and are adopted by the next (`Upkeep.Under`)
`worldHistory` retention	done (`885d9f0`)
a magma keyed by `Ref`	`Op/Dag.hs`, `Op/Ledger.hs`, `Op/Rewrite.hs`
a control socket speaking the `serve` language	shipped: `run serve --listen PATH` (`Actions/Serve/Socket.hs`, `specs/generic-server.md` milestone 2)
reconfiguration by re-reading a file	shipped as pull mode: `run serve --follow` and `--follow-cache` (`specs/pull-mode.md`)

What is still genuinely missing is PID 1 itself — reaping, signals, stage-0 mounts, reboot(2) — and the process-topology decision that follows from it (next section). Everything else in this document is now a question of wiring, not of building.

Problem / goal

An init system that boots a Linux VM, converges an Op graph, and then stays up forever supervising whatever that graph declared — reading a small seed from /etc/salmon-init.json for the handful of per-machine parameters.

Scoping this to VMs rather than physical machines is what makes it tractable, and the repo is already set up for it: Salmon.Builtin.Nodes.Qemu direct-boots a kernel with -kernel/-initrd, root=vroot rootfstype=9p, net.ifnames=0 (Qemu.kernelCmdline), against a Debian.Debootstrap.rootTree chroot. Hardware is then known: virtio devices, one serial console, no firmware quirks, no disk enumeration race, a stock Debian initrd whose only job is to mount the 9p root (the toy and the qemu tier already boot this way), no udev rule engine, no network-interface naming ambiguity. And VmConfig.vm_extra_kernel_args already exists as the place init=/sbin/salmon-init would go, which means the qemu test tier from specs/qemu-test-vms.md is also the test bed for this. Physical hardware is explicitly out of scope.

The build model: a cabal-built init, not a generic one

This is the framing decision that simplifies everything downstream. The supervisor is not a general-purpose init that interprets an arbitrary machine description at runtime. It is a binary you cabal build for a particular machine role — a seed type, a directive type, and a Track' Spec composing builtins and recipes — exactly like salmon-migrator in salmon-apps. The boot graph is Haskell code, compiled in and static.

/etc/salmon-init.json then carries only what genuinely varies between two machines of the same role: hostname, addresses, paths to key material, a service count. Small, and typed by that binary’s own seed type.

Consequences worth stating up front, because each removes a whole subsystem from the design:

No generic seed language, no plugin/unit-file loading, no runtime recipe discovery. Changing what a machine runs means rebuilding and redeploying the binary, which is a deployment story this repo already has (Self.uploadSelf, SreBox.CabalBuilding).
The graph is inspectable at build time, from a laptop, with the very same binary: run tree / run dag / query plan against a candidate seed. This is the payoff of Configure being a separate hermetic step, and it is what lets the “boot order is what I reviewed” test in the Testing section exist at all.
Reconfiguration-without-reboot is a parameter change, not a shape change. A machine’s graph changes by getting a new binary; its parameters change by a new document arriving — and specs/pull-mode.md is now the mechanism for that (below), rather than a SIGHUP re-read of a local file.

Being “generally static afterwards” also means the supervisor can be built with as few runtime moving parts as possible — ideally statically linked, so that PID 1 and the supervisor are the entire userspace trusted base at boot.

Why this is genuinely attractive (the pitch)

Every init system has a dependency graph. systemd’s is After=/Before=/ Wants=/Requires= scattered across unit files, and there is no good way to see it, test it, or reason about it before boot. Salmon’s Op graph is that graph, as a first-class value, and salmon already has:

upTree — topological ordering, dedup by Ref, and failure containment (a failed node’s dependents are Blocked, not run against an unmet precondition). That is precisely correct boot semantics, already implemented.
downTree — teardown in reverse dependency order, where a node becomes free only once its last dependent is gone. That is precisely correct shutdown semantics, already implemented, including the shared-predecessor case that a naive walk gets wrong.
run tree / run dag — you can print and review the boot order, as a tree or as Graphviz, from your laptop, before booting anything.
query plan --select/--exclude — boot a subset. “Boot everything except the database” becomes a plan file, not a rescue-shell adventure.
Serve’s World — a per-node Direction + Convergence state machine across a set of active seeds, with retry of Errored/Blocked nodes on the next pass. That is a supervisor’s bookkeeping, already written.
Upkeep — the tending loop: per-node machines that keep asking whether an effect is still there, restart a managed process by its declared policy, back off, give up, bounce dependants on RestForOne, and are Kept across the supervisor’s own restarts. That is the supervisor, already written and tested (Test/UpkeepSpec.hs, Test/DaemonSpec.hs).

So a surprising amount of this spec is not “build an init system” but “notice that the init system is mostly already here, and identify the handful of things that genuinely are not.”

What genuinely is not here

1. (Shipped.) “Keep this running” is `managed`, and the init node is a second `Daemon`

The draft’s table of supervision concerns is now a table of shipped code:

Supervision concern	Where it lives now
"is this service running?"	the node's `check :: IO CheckResult`
"start it"	its `managed` action, run under `withAsync` by `Upkeep`'s `Up` state
"stop it"	cancelling that async; `Daemon.runDaemon`'s bracket escalates group-`SIGTERM` → `stop_grace` → `SIGKILL`
"it died, restart it"	the action yields an `ExitCode`; `supRestart` decides
"don't restart in a tight loop"	`Upkeep`'s delay ladder + `Tally`, `supStableAfter`, `supGiveUpAfter`
"start things in the right order"	`waitStability` over dependencies
"a dependency failed"	waited out rather than `Blocked`: the dependency's own machine keeps retrying
"a dependency went away, take me with it"	`supStrategy = RestForOne` on the dependency

So the init-system node is not a new mechanism; it is a sibling of Nodes/Daemon.hs whose managed action talks to PID 1 instead of calling createProcess itself:

SalmonInit.service :: Reporter -> Service -> Op
  check   = Query slot            -> Success | Failure | Unknown (deferred until T)
  managed = Spawn slot; block on the Exited event for it; return its ExitCode
            (cancellation => Signal slot, then wait for Exited)
  up      = throwIO NeedsSupervisor      -- exactly as Daemon.daemon does
  down    = pure ()                       -- exactly as Daemon.daemon does

managed :: Output -> IO ExitCode is precisely the shape of “ask PID 1 to run this and tell me when it stopped”, which is a strong sign the field was cut in the right place. Systemd.Service stays the vocabulary (service_user/service_group/service_umask/KillMode), rendered into a spawn request rather than a unit file. A recipe ports from systemd to salmon-init by swapping one node, and the policy it carries in Supervision needs no translation at all.

Two things the draft got right that are worth keeping explicit. First, PID 1 answering Deferred { until } maps onto check returning Unknown, not Skipped or Failure: Unknown is the one verdict Upkeep acts on by continuing to look, which is what a deferral wants. Second, the process handle lives on the node’s own machine, so the Haskell side has no pid table — but see the next section, because whose child the process is is now a real decision rather than a free one.

1b. The one new tension: `Daemon`’s bracket versus “services are children of PID 1”

Daemon.runDaemon owns its process through a bracket: the process is a child of the supervisor, and cancelling the machine kills it. That is the whole teardown story under run serve and it needs no pid table. The draft’s architecture wants the opposite — services as children of PID 1, so the supervisor can crash, restart or be replaced without any service noticing. Both cannot be true of one node.

The reconciliation is the split above: SalmonInit.service’s managed action holds no process, it holds a subscription to PID 1’s exit event for a slot. Cancellation signals through PID 1; a supervisor restart re-attaches (its check asks PID 1 and answers Success, so the node starts Settled/Standing, and its managed action subscribes to a slot that is already running rather than spawning it). The protocol therefore needs an attach verb beside spawn — “give me the exit event for the pid you are already holding in this slot” — which the draft’s Query did not cover.

What this costs: Upkeep.Kept — machines that survive the supervisor stopping — does nothing useful across a process restart of the supervisor; it was built for the in-process case (status typed at a serve prompt must not restart every service). Under salmon-init the equivalent guarantee is provided by PID 1 holding the children, and the re-attach path above is what replaces adoption. Under plain run serve on a systemd box, Daemon.daemon stays what it is. Neither node should try to be both.

2. PID 1 has duties that are not convergence at all

None of these are expressible as Ops, and all are non-negotiable:

Reaping. PID 1 inherits every orphan on the machine and must waitpid(-1) them or the process table fills with zombies.
Signals. PID 1 gets no default signal dispositions — the kernel discards any signal for which PID 1 has not explicitly installed a handler. So it cannot be accidentally killed, but every signal it wants must be handled explicitly: SIGTERM/SIGUSR1/SIGUSR2 for the shutdown/reboot conventions, SIGINT for ctrl-alt-del (after reboot(RB_DISABLE_CAD)), SIGCHLD as the supervision event source, SIGHUP for “re-read the seed”.
Never exiting. If PID 1 returns, the kernel panics (Attempted to kill init!). For a program on a managed runtime this is a severe constraint: an uncaught exception, or heap exhaustion, is a kernel panic. This is the constraint that ends up deciding PID 1’s implementation language.
Early boot. Mounting /proc, /sys, /dev (devtmpfs), /dev/pts, /run; hostname; loopback up; entropy seed.
Shutdown. SIGTERM to everything, grace period, SIGKILL, sync, unmount, reboot(2) with the right command.

3. The bootstrap paradox

A convergence engine cannot converge the preconditions of its own execution. Salmon’s engine needs /dev/null (every CreateProcess redirect), /proc/self/exe (Self.readSelfPath_linux), and a readable, writable root before it can run Configure on /etc/salmon-init.json at all. So there is a stage 0 that is hardcoded imperative code, deliberately not part of the graph, and it must be kept as small as possible because nothing in it is inspectable with run tree. Drawing that line precisely is a design decision, not an implementation detail; the proposal is to keep stage 0 to exactly: mounts, RB_DISABLE_CAD, signal handlers, the reaper, and the console.

The hard engineering problem: `waitpid(-1)` versus the GHC runtime

This deserves its own section because it is the thing most likely to sink a naive implementation, and it is not obvious.

Binary.untrackedExec — which essentially every builtin’s up goes through — uses readCreateProcessWithExitCode, which ends in waitForProcess, which calls waitpid on one specific pid. Meanwhile PID 1 must run a reaper that calls waitpid(-1, …) to collect orphans. If the reaper wins the race and reaps a child that waitForProcess is waiting for, the specific-pid waitpid returns ECHILD and untrackedExec throws — which upTree faithfully reports as a failed node. The result is a provisioning engine that fails randomly under load, with an error that points at the wrong thing entirely.

The alternative — one unified reaper, where every child (services and untrackedExec subprocesses) registers in a Map ProcessID (MVar ExitCode) and a single thread owns waitpid(-1, WNOHANG) — would require Binary.untrackedExec to stop using System.Process’s own waiting, i.e. an “exec backend” seam in the most load-bearing module in salmon-ops, existing solely to serve PID 1. Rejected.

Architecture: a Rust PID 1, a Haskell PID 2

PID 1  salmon-init        Rust. Tiny, boring, cannot panic:
       (Rust, static)     stage-0 mounts, signal handlers, THE reaper,
                          spawns/kills/setuids service processes,
                          control socket, reboot(2)
                            |
                            | spawn requests / exit events over a socketpair
                            v
PID 2  <role>-supervisor   Haskell, cabal-built per machine role: reads the
       (Haskell, static)   seed, runs Configure, expands the graph, runs the
                           Serve-style World + supervision state, uses
                           System.Process normally

Why the split. It is not primarily about language:

The waitpid conflict disappears. PID 1’s children are the supervisor, the getty, and the services. The supervisor’s own untrackedExec subprocesses are its children, reaped by System.Process as usual. Two reapers, disjoint sets, no race.
A crash in the complicated half is no longer a kernel panic. The convergence engine is where all the intricate code lives (JSON, graph expansion, subprocess management, arbitrary recipe up actions) and therefore where the bugs live. If it dies, PID 1 restarts it.
The supervisor is restartable and upgradable in place. Because services are children of PID 1, not of the supervisor, the supervisor can crash, be restarted, or be swapped for a new binary without any running service being orphaned or killed. This is what makes redeploying a rebuilt supervisor a non-event, which matters a lot given the build model above. It is also the reason spawning must live in PID 1 rather than in the supervisor, even though that is the less obvious place to put it.

Why Rust for PID 1. The split makes PID 1 so small and so tightly constrained — must never exit, must never block indefinitely, must not depend on a heap it can exhaust — that a managed runtime buys nothing there and costs the two failure modes that matter most: an uncaught exception and heap exhaustion are both kernel panics in PID 1. Rust removes both by construction: no GC, no runtime to fail to initialize before main, #![deny(panic)]-style discipline enforceable in review, and a genuinely static binary (x86_64-unknown-linux-musl) with no loader dependency at a point in boot where the dynamic loader’s own assumptions are shakiest. C would also work; Rust is preferred, and the safety argument is mostly about the reaper and signal-handling code — the exact code where C’s classic PID-1 bugs (EINTR handling, signal-unsafe calls in handlers, races on the pid table) live.

The Rust half is deliberately not extensible: it knows how to mount, reap, spawn with a uid/gid/pgid, signal, rate-limit respawns, and reboot. Every decision about what to run and in what order is on the Haskell side. If a change requires touching the Rust binary, that is a signal the boundary is in the wrong place.

The one deliberate exception is restart backoff, which lives on both sides — see the next section, since it is the one place the boundary is crossed on purpose and therefore the one place it can go wrong.

The cost is a protocol between the two: Spawn/Signal/Query requests and Exited pid status events. It is small, it must be stable (a supervisor restart must not require a PID 1 restart), and it is worth noting it is the same shape as the SalmonInit.service node’s check/managed — so the node can talk to the control socket directly and the protocol only has to be designed once. Length-prefixed JSON over a SOCK_SEQPACKET socketpair is almost certainly enough; the temptation to make it clever should be resisted, because every feature in this protocol is a feature the Rust half has to grow.

Restart backoff: PID 1 rate-limits, the supervisor decides

Backoff lives on both sides, with different jobs. Getting this division wrong is how you end up with multiplicative delays and a machine that takes twenty minutes to bring a service back, so it is worth being precise:

PID 1’s backoff is a safety property, not a policy. It is a per-slot minimum interval between respawns of the same thing, applied to everything PID 1 spawns. Its purpose is to keep the machine alive and loggable when whatever is above it is broken or absent — including the case that has no other answer at all: a crash-looping supervisor, where there is no Haskell side left to consult. It never decides whether something should run.
The supervisor’s backoff is policy. Per-service, dependency-aware, with windows and a give-up latch, and expressible in the graph.

How they compose instead of fighting

The rule is that PID 1 defers, it never refuses, and the supervisor asks rather than duplicating the timer.

When a spawn request arrives sooner than the slot’s minimum interval allows, PID 1 schedules it rather than rejecting it, and answers the request with Deferred { until }. A Query on that slot then answers “not running, spawn deferred until T” — which maps onto Unknown in the check vocabulary (1 above): Upkeep keeps looking on its ladder rather than restarting or giving up, and picks the node up once the deferral expires. The two backoffs compose into max(pid1_floor, supervisor_policy) rather than summing; Upkeep’s own ladder (double on a failing up, halve on a vanished effect) is the policy half and already exists.

Deferring rather than refusing matters: a refusal that a buggy supervisor drops on the floor means the service never comes back, whereas a deferral is self-healing. The cost is that PID 1 holds a small timer queue — bounded by the number of slots, which is bounded by the config file, so it is not an unbounded allocation.

The supervisor slot is special

If the supervisor itself crash-loops, PID 1 must cap the backoff, never give up. A give-up latch is right for a service and catastrophic for the supervisor — a machine whose supervisor has permanently stopped being restarted is a machine with no way back. So: exponential growth up to a ceiling, then a steady retry at the ceiling forever, and after N failures in a window PID 1 writes prominently to the console (the getty is already up from stage 0, so there is somewhere to write to). Visible and still trying, never silent and stopped.

The config file

PID 1 reads /etc/salmon-init.conf in stage 0. This is a different file from /etc/salmon-init.json — the latter is the supervisor’s seed, typed by that role’s Haskell seed type; this one is PID 1’s own knobs and is read by the Rust half only. Keeping them separate keeps the Rust half from ever needing to understand a role’s schema.

Non-negotiable properties, all of which follow from “PID 1 must always boot”:

Missing or unparseable is not an error. Compiled-in defaults apply; a bad line is reported to the console and skipped; a bad file is ignored wholesale. PID 1 must never fail to boot because of its own config.
The format is boring on purpose. Flat key = value lines with # comments and a [slot.<name>] grouping for per-slot overrides. Not TOML, not JSON, not YAML: every parser is panic surface inside the trusted base, and this file has maybe a dozen keys. A hand-written line parser is a feature.
Kernel-cmdline override. PID 1 reads /proc/cmdline and honours e.g. salmon_init.backoff=off, so a config that makes the machine effectively unbootable (an enormous ceiling, say) can be escaped from the bootloader or the qemu -append without editing a filesystem you may not be able to reach. Cheap to implement, and the kind of escape hatch you only regret not having once.
Re-read on SIGHUP, keeping the previous values if the new file does not parse.

The knob set should stay small — every knob is Rust surface:

# defaults for every slot
initial_delay   = 100ms
multiplier      = 2.0
max_delay       = 30s
stable_after    = 10s     # ran this long => reset backoff to initial_delay
report_after    = 5       # failures in a window before shouting to the console

[slot.supervisor]
max_delay       = 5s      # come back fast; never give up
give_up         = never

give_up = never being expressible — and being the supervisor’s default — is the config-level statement of the previous section’s rule.

Clock

Backoff must use CLOCK_MONOTONIC. At early boot the wall clock is whatever the RTC said, and it will jump when time sync happens; a backoff computed against wall time can silently become a multi-hour deferral the first time NTP corrects a skewed guest clock. This is a small detail with a very confusing failure mode, which is why it belongs in the spec rather than in the code review.

Repo consequences

This puts a non-Haskell component in a cabal multi-package project. The salmon-init Rust crate does not belong in the cabal.project package set; the plausible shape is a sibling directory with its own Cargo.toml, built independently, with the Haskell side depending on it only at image assembly time (the Debootstrap chroot gets a /sbin/salmon-init binary). That keeps cabal build all unaffected — which matters, given the project already splits packages specifically to keep build times down.

The calling convention

The kernel execs init with argv derived from the kernel cmdline; unrecognized cmdline words are passed through as argv/env. That is the reason none of this can be a fourth Command constructor: execCommandOrSeed’s entire contract is argv parsing, and argv is exactly what is unavailable here. These binaries are separate-enough things and should not pretend otherwise.

PID 1 (Rust) takes no arguments and parses none. It refuses to run unless getpid() == 1 (behind an explicit --pretend for development), because doing this by accident on a workstation would be memorable.
The supervisor (Haskell) has its own entry point, not execCommandOrSeed. It is exec’d by PID 1 with a fixed argv, and reads its seed from a path, not from flags.
The same supervisor binary is also the client when run from a shell: <role>ctl status, converge, reboot, up <seed args> — talking to the control socket, exactly as systemctl relates to systemd. This is worth keeping in one binary because the client is the thing that needs to know the graph in order to resolve --select patterns and print sensible node names, and that knowledge is compiled in.
Two sockets, not one. PID 1’s socket speaks only the small mechanical protocol (spawn/signal/query/reboot). The serve-language socket is the supervisor’s own. Keeping them separate is what allows reboot and poweroff to work even when the supervisor is wedged or restarting — which is precisely when you need them.

Usefully, the same cabal-built binary still supports the ordinary config/run tree/run dag/query plan surface when invoked normally on a developer machine — that is how the boot graph gets reviewed before it is ever booted. So the binary has three personalities (init supervisor, control client, ordinary salmon CLI) but only the first is entered without argv.

The control socket is `specs/generic-server.md`’s socket

Serve.parseServeCommand already defines a line-oriented language — up/only/down/up-directive/clear/converge/status/history/ query/load/force/recheck/pause/resume/supervise/autoconverge/ help/quit — with --select/--exclude on the node-addressing ones. That is a remarkably good fit for an init control interface. The draft proposed reusing it over a socket of this spec’s own; that socket is now specs/generic-server.md’s first milestone (a unix socket carrying the line protocol, as a second producer into the loop’s inbox), and this spec should consume it unchanged — the <role>ctl client is that spec’s terminal client, and a /dag view of a booting VM is that spec’s web UI pointed at the guest. Additions this spec still needs on top: reboot, poweroff (which go to PID 1’s socket, not this one — see “Two sockets”), and quit rejected, since quitting is a supervisor restart at best.

Boot sequence

kernel → exec /sbin/salmon-init (pid 1, Rust)
  stage 0 (hardcoded, small, cannot panic)
    reboot(RB_DISABLE_CAD); install signal handlers; start reaper
    mount /proc /sys /dev(devtmpfs) /dev/pts /run
    read /etc/salmon-init.conf  ──failure──> compiled-in defaults, carry on
    read /proc/cmdline for salmon_init.* overrides
    open /dev/console; lo up
    spawn a getty on ttyS0            <- unconditional, before anything can fail
    open the PID-1 control socket
    fork/exec the supervisor          <- restarted by pid 1, with backoff,
                                         capped, never giving up
  supervisor (pid 2, Haskell)
    read /etc/salmon-init.json  ──failure──> report + leave the getty running
    Configure IO seed directive ──failure──> report + leave the getty running
    expand to Op graph, seed the World with it, converge
    open the serve-language socket
    then: block on { exit events from pid 1, control socket, timers }
          → converge again

Note sethostname is not in stage 0: it is a per-machine parameter, so it comes from the seed and belongs in the graph. The rule for what stays in stage 0 is “things the engine needs in order to run at all”, not “things that happen early”.

Two properties worth calling out:

The getty comes up before anything that can fail. If the seed is missing, malformed, or configures to a graph that fails to converge, the machine must still be a machine you can log into. systemd’s emergency.target exists for this reason and is worth copying. “Rescue” here is not a mode — it is just “the console got spawned in stage 0 and convergence didn’t work”, which requires no extra machinery.
Reconfiguration without reboot is pull mode, not SIGHUP. The draft had the supervisor re-read /etc/salmon-init.json on SIGHUP and run Serve’s only path. specs/pull-mode.md is the same idea with the problems solved: the supervisor is started with --follow <registry> --label <role> and its parameters arrive as a JSON document (the format that spec fixes), diffed against the ledger and converged as one pass, with change detection so an unchanged poll never stands the machines down, a scheduler with backoff so a hundred booting VMs do not hammer the registry, and the fetch recorded in history as its own actor. The local file becomes the cached last document that spec asks for anyway — the thing a VM boots from when the registry is unreachable — and SIGHUP, if kept at all, is fetch (force a round now). Given the build model this only ever changes parameters; a new graph shape is a new supervisor binary, and PID 1 restarting the supervisor is the same code path as PID 1 restarting a crashed one — which, with 1b’s re-attach, no service notices.

Why the VM assumption buys so much

Spelled out, because each of these is a subsystem that does not have to be written:

Physical-machine problem	Why it disappears in the target VM
initramfs, early module loading	`-kernel`/`-initrd` direct boot with virtio built in; `Qemu.kernelCmdline` already does `root=vroot rootfstype=9p rootflags=trans=virtio rw`
udev, device naming rules	devtmpfs gives the kernel-created nodes; the device set is fixed and known
network interface naming	`net.ifnames=0 biosdevname=0`, already in `kernelCmdline`
disk enumeration, fsck, LVM, LUKS, mount ordering	9p root, no block devices to speak of
firmware/ACPI quirks, suspend/resume	not modelled at all
console detection	always `console=ttyS0`, already in `kernelCmdline`
verifying a reboot actually happened	`VmConfig.vm_monitor_socket` gives an out-of-band channel to observe and to force-reset a wedged guest

The last row is the one that makes this testable rather than merely buildable, and it is why the qemu tier is the natural home for the tests.

State that `World` doesn’t have — now mostly `Upkeep`’s

The draft asked for a supervisor table keyed by Ref with the live pid, restart counts, next-eligible time and a give-up latch, kept apart from NodeState. That is what shipped, and it shipped apart from World for the draft’s own reason: Upkeep’s per-machine Tally (consecutive failures, when the node last reached Up) and delay ladder are the counters; supGiveUpAfter is the latch (a node that gave up is parked, and Force/ Recheck starts it over — the draft’s “reported instead of consuming the machine”); and Serve.NodeState.nodeStatus is the snapshot status reads. The live pid is the one item that is deliberately not stored anywhere on the Haskell side: under run serve it lives inside the machine’s async; under salmon-init it lives in PID 1’s slot table and the machine holds a subscription (1b).

One thing survives as a genuine gap. World is an IORef: none of this persists across a supervisor process restart. The draft’s cure was “PID 1 holds the services, so a supervisor restart re-adopts”; that still holds for the processes, but the ledger, the convergence states and the tallies start from zero. Under pull mode the cached last document replays the declarations; the rest is Standing guesses from each node’s check. That is acceptable for v1 of an init (a rebooted machine starts from zero too) and is the same journal item both companion specs already rank ahead of their own work.

worldHistory’s unbounded growth, the draft’s other worry, was fixed on its own merits (885d9f0): worldEpochs keeps only graphs a pass could still walk and worldLog keeps capped per-declaration lines; and with the magma keyed by Ref (Op/Dag.hs) there is no per-declaration graph to retain at all.

Process lifecycle details worth deciding early

Process groups. Each service gets its own session (setsid) so it can be killed as a group. This is Systemd.KillMode’s Process vs a control-group equivalent; without cgroups, the pgid is the available approximation and is good enough for the VM case. cgroup v2 (for real containment and for resource limits) is a plausible v2, not a v1.
Dropping privilege at spawn. Systemd.Service already carries service_user/service_group/service_umask. Under salmon-init these stop being rendered into a unit file and travel over the spawn protocol, becoming actual setgroups/setgid/setuid calls made by PID 1 in the forked child before exec — in that order, since dropping the group after the user is a classic privilege-escalation bug, as is forgetting supplementary groups. This is the same operation specs/multi-user-privilege-separation.md proposes as applyRunAs, applied at spawn rather than by wrapping a command. The two specs share the RunAs vocabulary on the Haskell side, but not the implementation: here the fork-and-setuid that spec cautions against is the correct approach, because it is a fresh fork in a non-GC’d runtime whose only job is to exec — none of the hazards that make it a bad idea inside the Haskell engine apply.
Stdout/stderr. Services need somewhere to write. v1: a per-service file under /run/salmon-init/log/<name>, opened by PID 1 before exec. A journal is not v1.
Readiness. Type=simple only (as Systemd.ServiceType already is): “spawned” means “started”. Notify/readiness protocols are a v2 concern, and the honest v1 story for ordering against readiness is “the dependent node waits on waitStability and its own check until the dependency answers” — which Upkeep already does (failure is waited out, not contained), and which is arguably more robust than a readiness protocol anyway. A RestForOne on the dependency is the opt-in for “and bounce me if it goes away”.
Shutdown ordering. downTree gives correct reverse-dependency teardown, but a real shutdown also needs a global deadline: converge-down with a timeout, then SIGTERM everything remaining, then SIGKILL, then sync and reboot(2). The deadline cannot live in the graph.

Testing

This is unusually testable for something this invasive, because the harness already exists:

Debootstrap.rootTree + ensureVm9pBoot produce the chroot; add the Rust salmon-init at /sbin/salmon-init, the cabal-built role supervisor, and a /etc/salmon-init.json. Image assembly is the one place the two build systems meet, so it is worth making it an Op like everything else rather than a shell script beside the tests.
Boot it with vm_extra_kernel_args = ["init=/sbin/salmon-init"] — note Systemd.render_service’s quoteArg already handles the -append quoting trap that bit Qemu once (documented in specs/qemu-test-vms-progress.md).
Assert over the serial console and over SSH: services running, boot order as run tree predicted, salmon-init status agrees, a killed service comes back, a crash-looping service gives up rather than spinning, SIGHUP with a new seed converges the difference, poweroff actually powers off (observable on the monitor socket).

The “boot order matches what run tree printed on the developer’s laptop” test is the one that justifies the whole design, and it is only possible because config generation and execution are already separate hermetic steps.

Non-goals (v1)

Physical hardware; initramfs generation; udev; socket/dbus activation; cgroup resource control; timers (CronTask exists but needs cron, and a timer subsystem is its own spec); a journal; user sessions/logind; SELinux/AppArmor; /dev/initctl compatibility; being a drop-in systemd replacement for an existing distro’s unit corpus; and — per the build model — any form of runtime extensibility: no unit-file directory, no plugin path, no way to add a service without rebuilding the supervisor. The target is a machine whose entire configuration is one cabal-built binary plus a small parameter file, not a machine that also has to run somebody else’s units.

Open questions

Is the supervisor’s own restart policy declarable? Services are declared by the graph but supervised by PID 1, which also supervises the supervisor. So the one process whose restart behaviour cannot be expressed in the graph is the one that owns the graph. Probably fine and unavoidable; worth being deliberate rather than discovering it.
How small can stage 0 actually get? Every line in it is a line that run tree cannot show you. Is there a defensible way to express the mounts as ops that run under a degraded engine, or is hardcoding them honest?
Seed vs directive on disk is answered by pull mode’s document format: a document entry is either {"seed": [words]} or {"directive": {...}}, mirroring up vs up-directive. A role can publish directives (hermetic, cannot fail Configure at boot) or seeds (flexible); the cached last document is what the VM boots from. What remains open is only whether a role should refuse seeds and insist on directives for the boot path.
Does the fleet story change the build model? With labels addressing documents in a registry, “one cabal-built binary per role” is still right for the graph, but a role’s parameters now come from the registry rather than an image-baked file, which means the image is the same for every machine of a role and the label is the only per-machine input. That is a simplification; it should be stated as the target.
How static is “static”? A statically-linked GHC binary is achievable but not free, and it interacts with what the recipes actually shell out to: a supervisor with no dynamic loader still needs psql, ip, nft and friends present in the image. Worth deciding whether the goal is a static supervisor or a minimal, known image — they are different targets, and the second is probably the real one.
Interaction with specs/multi-user-privilege-separation.md. That spec’s L1 (RunAs/Invoker) and this one’s service-spawn privilege drop want to be the same vocabulary — except the drop happens in Rust here, so what is shared is the type on the Haskell side and its serialization into the spawn protocol, not the implementation. Doing that spec first makes this one smaller.

Suggested milestones

Each is independently useful and independently testable, which matters a lot for something whose failure mode is “the VM does not boot”.

In Serve, ahead of any of this:

0a. Supervision in run serve — done: managed/Daemon, Supervision, Upkeep, tending between commands (80fb410, e9fd9d9 and the per-node-state-machines series). Milestone 4 below is now wiring.

0b. worldHistory retention — done (885d9f0).

0c. The serve-language socket — specs/generic-server.md milestone 2. This spec consumes it.

0d. Pull mode with a cached last document — specs/pull-mode.md milestones 2–4. This spec consumes it for reconfiguration.

Then the init system itself:

salmon-init (Rust) as a dumb-init. Stage 0 + reaper + signals + getty
- reboot/poweroff, no convergence at all, boots to a shell in a qemu VM. Proves the PID-1 mechanics and the test harness end to end, and establishes the Rust build/packaging path into the Debootstrap image.
The spawn protocol, /etc/salmon-init.conf, and PID-1 backoff, with a hardcoded service list and the supervisor slot. Proves the process topology — “kill the supervisor, services survive, supervisor comes back and re-adopts”, which everything else leans on — and, separately, that a binary that exits immediately gets rate-limited to the configured ceiling instead of spinning, that a missing/corrupt conf still boots on defaults, and that salmon_init.backoff=off on the kernel cmdline overrides the file. That last set is cheap to test and is exactly the behaviour nobody exercises until the day it matters.
SalmonInit.service node as a sibling of Daemon.daemon (1 above): check = Query, managed = Spawn/Attach + block on Exited, cancellation = Signal. Plus seed/directive reading from the cached document and one converge pass. First real boot of a cabal-built role supervisor. Test at Layer 1 against a fake PID 1 speaking the protocol over a socketpair, before any VM.
Supervision proper: nothing to build — Upkeep tends the node from milestone 3 with its declared Supervision. The test is that killing a service under the VM brings it back by policy, and that a crash-looping one parks after supGiveUpAfter rather than spinning.
Reconfiguration: --follow a registry (0d) from inside the VM, and supervisor binary replacement without dropping services — which is 1b’s re-attach, tested by killing the supervisor and checking status from the new one shows every slot Standing.
Ordered shutdown: downTree with a global deadline, then reboot(2).

Salmon ops patterns

2026-09-25T14:15:36Z

Generated from resources/salmon-ops-patterns.md — the repository is the canonical source, and may be ahead of this page.

Salmon ops patterns

This is a companion to howto-ops.md: where that doc is a per-primitive cookbook (how to write one node, one Op, one CLI binary), this one collects recurring shapes worth reusing across recipes — patterns that combine several of those primitives to solve a problem that comes up more than once. Read howto-ops.md first if a term here (Op, Track, seed/Spec, check) is unfamiliar.

Pattern: one-time privileged bootstrap, then unprivileged forever after

Problem. A recipe needs some action that only root can perform — granting a Linux capability (setcap), installing an OS package, chown-ing a path to a different user — but you don’t want every routine invocation of the binary to require root just because one op deep in its graph does. Requiring sudo for everything is both a security smell (broader blast radius than needed) and an ergonomics problem (can’t run unattended as a normal user, breaks non-interactive/CI invocations).

Shape. Split the privileged, one-time setup from the routine, repeated work as two different seeds of the same binary, using the existing seed → spec → ops CLI protocol (howto-ops.md §9):

sudo my-salmon config bootstrap | sudo my-salmon run up   # once per machine
my-salmon config <routine-seed> | my-salmon run up        # every other time, no sudo

The bootstrap seed’s Op graph contains only the privileged, machine-wide setup: capability grants, package installs, ownership fixes. Nothing routine (booting a VM, writing a recipe’s actual state) belongs in it.
Every other seed’s graph assumes that setup already happened and never needs privilege itself.
This only works if every op in the bootstrap graph is genuinely idempotent (howto-ops.md §4) — re-running bootstrap under sudo later (e.g. after a package upgrade wipes a capability) must be a safe, cheap no-op via check, not a hazard. If you can’t make the bootstrap step idempotent, this pattern isn’t safe to recommend to users as “run it whenever” — treat it as a real migration instead.
The bootstrap seed usually needs to know which unprivileged user/group future invocations will run as (to chown/grant-to the right identity) — take that as an explicit seed argument rather than inferring it from $SUDO_USER/similar, so sudo my-salmon config bootstrap --for alice is unambiguous about who it’s provisioning for.

Worked example. Salmon.Builtin.Nodes.Capabilities.grantCapabilities (salmon-ops/src/Salmon/Builtin/Nodes/Capabilities.hs) is exactly this kind of bootstrap-only op: it needs CAP_SETFCAP (in practice, root) to run, but its check (getcap-based) makes every subsequent run a no-op — see its haddock. The qemu test tier (specs/qemu-test-vms-progress.md §0.2) combines it with Salmon.Builtin.Nodes.User.chown into one bootstrap graph, currently exposed as a standalone fixture binary (salmon-ops/fixtures/QemuHostSetupFixture.hs) rather than a real bootstrap seed on a unified CLI binary — folding it into the latter shape (a proper Seed = Bootstrap User | BootVm VmSpec | ...) is the natural next step if/when this tier grows a real production binary instead of remaining test-only support code. salmon-toy-qemu-pg-ha (salmon-apps/src/QemuPgHaToy.hs) already has the two-seed shape — a prereqs seed run once under sudo (root filesystems, and /etc/ssh handed to the unprivileged user), then up/client without it — though its two capability grants are still a documented manual setcap, not a node in prereqs.

Related, but not this pattern: if the privileged step isn’t safely re-runnable (e.g. a real schema migration, a one-shot data backfill), don’t reach for “bootstrap seed” — that’s ordinary migration territory (Salmon.Builtin.Migrations), which has its own once-ever semantics instead of check-based idempotency.

GCP Support Plan for Salmon

2026-09-25T14:15:36Z

Generated from specs/gcloud-support.md — the repository is the canonical source, and may be ahead of this page.

GCP Support Plan for Salmon

Status: phase 1 (gcloud-first) is implemented through all ten steps of §15: Salmon.Builtin.Nodes.Gcp.{Core,Storage,Compute,SshAccess,Iam,ArtifactRegistry,CloudRun,LoadBalancing} (instance-group and serverless-NEG backends), plus modules the plan did not list (ResourceManager, Billing, ServiceUsage, SecretManager), the recipes SreBox.Gcp.{VmProvision,CloudRunDeploy,PostgrestCloudRun,PreviewEnvironment}, salmon-gcp-toy and Test.GcpSpec. Not done: phase 2 (direct REST calls), the SreBox.Gcp.WebStack recipe of §2, CloudRun job executions, and a configurable gcloud path (both deferred by §17). Kept as the design record.

This document proposes adding Google Cloud Platform (GCP) resource support to Salmon. The goal is to enable Salmon DAGs that turn up VMs, load balancers, Artifact Registry repositories, Cloud Storage buckets, and CloudRun services, while provisioning the VMs over SSH using an SSH-CA trust model.

The design stays within Salmon’s existing patterns: resources are modelled as Op nodes, GCP tools are wrapped via Salmon.Builtin.Nodes.Binary, and VM provisioning reuses the existing Self, Ssh, Keys, and Rsync machinery.

1. Strategy: gcloud-first, API-second

For the same reason Salmon wraps systemctl, psql, podman, and nft rather than linking their native APIs, the fastest path is to model GCP resources as nodes that shell out to gcloud.

Phase	Approach
Phase 1	Wrap `gcloud` CLI commands via `Binary` nodes.
Phase 2	Replace hot or latency-sensitive paths (VM status polling, operation waiting) with direct REST calls via `http-client` if needed.
Auth	Use Application Default Credentials (ADC). A single `Gcp.Core` validation node runs before any resource node.

This gives Salmon’s existing execution model — sequential, concurrent, and supervised (run serve) drivers — for free.

2. Package Layout

New modules under `salmon-ops/src/Salmon/Builtin/Nodes/Gcp/`

salmon-ops/src/Salmon/Builtin/Nodes/Gcp/
  Core.hs              -- project, zone, region, auth, common CLI wrappers
  Compute.hs           -- GCE instances, instance groups, templates
  LoadBalancing.hs     -- backend services, URL maps, forwarding rules, health checks, NEGs
  ArtifactRegistry.hs  -- repositories, IAM, docker/podman auth
  Storage.hs           -- GCS buckets, IAM, lifecycle
  Iam.hs               -- service accounts, role bindings
  CloudRun.hs          -- CloudRun services and revisions
  SshAccess.hs         -- OS Login / metadata keys / SSH-CA glue

New recipes under `salmon-ops-recipes/src/SreBox/Gcp/`

  WebStack.hs          -- VM + LB + bucket + CloudRun service
  VmProvision.hs       -- network + vm + ssh access + self-provisioning
  WebStack.hs          -- VM + LB + bucket + CloudRun job

3. Shared Core: `Gcp.Core`

module Salmon.Builtin.Nodes.Gcp.Core where

data Project = Project { projectId :: Text }
  deriving (Eq, Ord, Show)

data Zone = Zone { zoneName :: Text }
  deriving (Eq, Ord, Show)

data Region = Region { regionName :: Text }
  deriving (Eq, Ord, Show)

data GcpError = GcpCliError Text Int Text
  deriving (Exception, Show)

-- | Track for the gcloud binary.
gcloud :: Track' (Binary "gcloud")

-- | Validates Application Default Credentials.
-- Almost every other GCP op depends on this.
applicationDefaultCredentials :: Op

The ADC node should run gcloud auth application-default print-access-token and succeed only if a token is returned. This catches misconfigured environments before any resource operation.

4. Compute Engine VMs: `Gcp.Compute`

Types

data MachineType
  = E2Medium
  | E2Standard2
  | N2Standard4
  | Custom Text
  deriving (Eq, Show)

data BootDisk = BootDisk
  { bootDiskSizeGb :: Int
  , bootDiskImage  :: Text
  }
  deriving (Eq, Show)

data Instance = Instance
  { instanceName         :: Text
  , instanceProject      :: Project
  , instanceZone         :: Zone
  , instanceMachineType  :: MachineType
  , instanceBootDisk     :: BootDisk
  , instanceNetwork      :: Text
  , instanceSubnet       :: Text
  , instanceServiceAccount :: Maybe Text
  , instanceMetadata     :: Map Text Text
  , instanceTags         :: [Text]
  }
  deriving (Eq, Show)

Op

instance :: Reporter Report -> Track' (Binary "gcloud") -> Instance -> Op

up: gcloud compute instances create ...
down: gcloud compute instances delete --quiet ...
check: gcloud compute instances describe --format='value(status)'
- RUNNING → Success
- TERMINATED or absent → Failure
- PROVISIONING, STAGING, STOPPING → Unknown
- SSH readiness is checked separately by Gcp.SshAccess.sshAvailable, not by the VM node itself.

External networking

Keep VMs private by default. If outbound internet is required, depend on an existing Cloud NAT or add a Gcp.Compute.CloudNat node. For provisioning access, use IAP tunneling or the SSH access model described below.

SSH availability check

Because the VM node only reports GCE-level RUNNING, add a separate support node for verifying that SSH is actually reachable. This is especially useful before Self.uploadAndCallSelf runs.

data SshEndpoint = SshEndpoint
  { sshHost     :: Text
  , sshPort     :: Int
  , sshIdentity :: FilePath
  }

sshAvailable :: Reporter Report -> SshEndpoint -> Op

up: no-op (or a connect probe)
check: ssh -o ConnectTimeout=5 -o BatchMode=yes -i <identity> <host> true exits 0 → Success; else Failure
down: no-op

Recipes that provision a VM should depend on sshAvailable before invoking Self.uploadAndCallSelf.

5. SSH Access Model

The objective is to create a VM and then SSH into it to run a Salmon binary for local provisioning. We prefer SSH-CA over long-lived per-instance keys.

Enable OS Login at project or organization level.
Upload the SSH CA public key to Google Cloud Identity via the OS Login API.
Salmon signs short-lived user certificates using existing Keys primitives.
Instances get metadata enable-oslogin=TRUE.
Users connect with signed certificates; Google validates the CA.

data OsLoginConfig = OsLoginConfig
  { osLoginProject     :: Project
  , osLoginCaPublicKey :: FilePath
  }

enableOsLogin      :: OsLoginConfig -> Op
uploadOsLoginCaKey :: Text -> FilePath -> Op

Caveat: uploading a CA to Cloud Identity requires domain-wide delegation or admin credentials. This is cleanest if you control the Google Workspace / Cloud Identity domain.

Option B: Project metadata SSH keys + Salmon CA (recommended starting point)

Salmon generates an SSH CA key pair (Keys.SSHKeyPair).
Salmon injects the CA public key into project or instance metadata.
Instances trust that CA.
Salmon signs user/host certificates as needed.

data MetadataSshCa = MetadataSshCa
  { sshCaProject   :: Project
  , sshCaPublicKey :: FilePath
  }

installMetadataCaKey :: MetadataSshCa -> Op

This avoids Cloud Identity Admin SDK complexity and is self-contained within a project.

Use gcloud compute ssh. This is the simplest but loses the SSH-CA model.

Recommendation

Start with Option B (metadata SSH-CA) for a self-contained Salmon setup. Migrate to Option A once organization-wide trust and Cloud Identity admin automation are in place.

6. VM Self-Provisioning

Once SSH access works, reuse Salmon’s existing remote-provisioning machinery:

Salmon.Builtin.Nodes.Self.uploadAndCallSelf
Salmon.Builtin.Nodes.Self.uploadAndCallSelfAsSudo
Salmon.Builtin.Nodes.Ssh.preExistingRemoteMachine
Salmon.Builtin.Nodes.Rsync.sendFile

A recipe composes the pieces:

module SreBox.Gcp.VmProvision where

provisionedVm ::
  Reporter Report ->
  Instance ->
  Self.SelfPath ->
  Ssh.Remote ->
  Track' directive ->
  directive ->
  Op
provisionedVm r inst selfpath remote runRemote spec =
  let vm        = Gcp.Compute.instance r gcloud inst
      sshAccess = Gcp.SshAccess.metadataCaAccess r gcloud inst
      provision = Self.uploadAndCallSelfAsSudo
                    ... selfpath remote Ssh.preExistingRemoteMachine runRemote CLI.Up spec
  in
    provision
    `inject` sshAccess
    `inject` vm

This mirrors the existing SreBox.PostgresMigrations.remoteMigrateOpaqueSetup pattern.

7. Load Balancers: `Gcp.LoadBalancing`

GCP L7 load balancers consist of many small resources. Expose a single high-level recipe rather than forcing users to wire every component manually.

data Backend
  = InstanceGroupBackend InstanceGroup [Int]        -- ports for named ports
  | CloudRunBackend Text                            -- CloudRun service name
  deriving (Eq, Show)

data ApplicationLoadBalancer = ApplicationLoadBalancer
  { albName        :: Text
  , albProject     :: Project
  , albRegion      :: Region
  , albNetwork     :: Maybe Text                    -- required for instance groups; optional for serverless
  , albBackends    :: [Backend]
  , albHealthCheck :: Maybe HealthCheck             -- required for instance groups; omitted for CloudRun
  }

applicationLoadBalancer :: Reporter Report -> ApplicationLoadBalancer -> Op

Internally creates:

Health check (for instance-group backends only)
Instance group + named ports, or serverless NEG for each CloudRun backend
Backend service (one per backend, or a single service with multiple backends)
URL map
HTTP(S) target proxy
SSL certificate (managed or self-provided)
Forwarding rule + external/global IP

Check: verify every sub-resource exists and the backend service reports healthy backends (for instance groups) or that the NEG points to a deployed CloudRun service.

The recipe must reject mixing instance-group and CloudRun backends in a single backend service if GCP does not allow it; otherwise create separate backend services and route by URL map path rules.

8. Artifact Registry: `Gcp.ArtifactRegistry`

data RepoFormat = Docker | Maven | Npm | Python | Apt | Yum

data ArtifactRepo = ArtifactRepo
  { repoName     :: Text
  , repoProject  :: Project
  , repoLocation :: Region
  , repoFormat   :: RepoFormat
  }

artifactRepository :: ArtifactRepo -> Op

up: gcloud artifacts repositories create ...
check: gcloud artifacts repositories describe ...
IAM nodes grant roles/artifactregistry.reader / writer.

Then reuse existing Salmon.Builtin.Nodes.Podman with image names like:

{region}-docker.pkg.dev/{project}/{repo}/{image}:{tag}

Add a helper to configure local auth:

configureDockerAuth :: Project -> Region -> Op
-- runs: gcloud auth configure-docker {region}-docker.pkg.dev

9. Cloud Storage: `Gcp.Storage`

data Bucket = Bucket
  { bucketName   :: Text
  , bucketProject :: Project
  , bucketLocation :: Region
  , bucketUniformBucketLevelAccess :: Bool
  }

bucket :: Bucket -> Op

up: gcloud storage buckets create gs://... --location=...
check: gcloud storage buckets describe gs://...
IAM nodes for access grants.
Optional lifecycle and CORS nodes if needed.

Use a pre-check before create to make up idempotent.

10. IAM: `Gcp.Iam`

A generic IAM node is useful for creating and attaching principals to resources. It can create service accounts and grant roles on projects or individual resources.

data Principal
  = ServiceAccount Text              -- account ID
  | User Text
  | Group Text

data IamBinding = IamBinding
  { iamPrincipal :: Principal
  , iamRole      :: Text              -- e.g. "roles/storage.objectViewer"
  , iamResource  :: Text              -- project ID or resource URI
  }

serviceAccount :: Project -> Text -> Op       -- create service account
iamBinding     :: IamBinding -> Op            -- grant role on resource

serviceAccount up: gcloud iam service-accounts create <id> --project=<project>
serviceAccount check: gcloud iam service-accounts describe <id>@<project>.iam.gserviceaccount.com
iamBinding up: gcloud <resource-type> add-iam-policy-binding <resource> --member=<member> --role=<role>
iamBinding check: gcloud <resource-type> get-iam-policy <resource> and verify the binding exists
down: remove the binding or delete the service account

Use this for granting CloudRun service accounts access to buckets, Artifact Registry, and secrets.

11. CloudRun Services: `Gcp.CloudRun`

For now we model CloudRun services and revisions, not job executions. The image comes from the Artifact Registry repository provisioned earlier, so a CloudRun service node depends on both the repository and a pushed image.

data CloudRunService = CloudRunService
  { crsName           :: Text
  , crsProject        :: Project
  , crsRegion         :: Region
  , crsImage          :: Text           -- full Artifact Registry URL
  , crsEnv            :: Map Text Text
  , crsServiceAccount :: Text
  , crsIngress        :: IngressSetting
  , crsMaxInstances   :: Maybe Int
  }

data IngressSetting = All | Internal | InternalAndLoadBalancing

cloudRunService :: CloudRunService -> Op

up: gcloud run deploy <name> --image=<image> --region=<region> ...
check: gcloud run services describe <name> --region=<region> and verify the active revision points at crsImage.
down: gcloud run services delete <name> --region=<region> --quiet

Image lifecycle: a CloudRun service node does not build or push images. It depends on an upstream node that pushes a Podman-built image to Artifact Registry. The image URL is part of the service spec, so changing the image triggers a new revision on up.

Job executions: deferred to a later phase. When needed, add cloudRunJob and cloudRunJobExec nodes separately.

12. Example DAG: Tenant Stack

tenantStack :: TenantId -> Op
tenantStack tenant =
  let bkt    = Gcp.Storage.bucket    (tenantBucket tenant)
      repo   = Gcp.ArtifactRegistry.artifactRepository (tenantRepo tenant)
      vm     = Gcp.Compute.instance    ... (tenantVm tenant)
      sshCa  = Gcp.SshAccess.metadataCaAccess ... tenant
      remote = Ssh.Remote "salmon"   (vmIp tenant)
      provision = provisionedVm ... vm sshCa remote tenantSpec
  in
    provision
    `inject` repo
    `inject` bkt

Dependencies ensure the bucket and repository exist before the VM starts, and SSH access is in place before the remote Salmon binary runs.

13. Idempotency and Check Mapping

Resource	Check command	Result mapping
ADC	`gcloud auth application-default print-access-token`	exit 0 → `Success`; else `Failure`
VM	`gcloud compute instances describe --format=value(status)`	`RUNNING` → `Success`; `TERMINATED`/absent → `Failure`; transitional → `Unknown`
SSH available	`ssh -o ConnectTimeout=5 -o BatchMode=yes ... true`	exit 0 → `Success`; else `Failure`
OS Login CA	list user's SSH keys and compare CA fingerprint	match → `Success`; else `Failure`
Metadata CA	`gcloud compute project-info describe` metadata	key present → `Success`; else `Failure`
Load balancer	describe all sub-resources	all present and healthy → `Success`; else `Failure` or `Unknown`
Artifact Registry	`gcloud artifacts repositories describe`	exists → `Success`; else `Failure`
GCS bucket	`gcloud storage buckets describe gs://...`	exists → `Success`; else `Failure`
IAM binding	`gcloud <resource> get-iam-policy <name>`	binding present → `Success`; else `Failure`
CloudRun service	`gcloud run services describe`	exists and active revision matches image → `Success`; else `Failure`

For resources with an update operation, prefer describe → update-or-create so that up remains idempotent even when parameters change.

14. Secrets Handling

Use existing Salmon.Builtin.Nodes.Secrets for:

SSH CA private keys.
Service account JSON files, if ADC cannot be used.

Never commit service account keys into directives. Pass them as file paths generated by Secrets.sharedSecretFile.

15. Implementation Order

Gcp.Core + ADC validation node.
Gcp.Storage.bucket — simplest resource; good proving ground.
Gcp.Compute.instance + status-based check.
Gcp.SshAccess — choose metadata CA or OS Login and implement key injection.
Gcp.SshAccess.sshAvailable — verify SSH is reachable before remote provisioning.
SreBox.Gcp.VmProvision — VM + SSH + Self.uploadAndCallSelf.
Gcp.Iam — service accounts and role bindings.
Gcp.ArtifactRegistry repository + docker auth.
Gcp.CloudRun services and revisions from pushed images.
Gcp.LoadBalancing recipe last, because it has the most moving parts and depends on VM/instance-group or CloudRun nodes.

16. Risks and Mitigations

Risk	Mitigation
Async operations	Poll inside `up` for `RUNNING` / ready state, or model intermediate states as explicit nodes.
API rate limits	Salmon's concurrent driver is unbounded; add dependency edges between GCP ops or introduce a rewrite that batches `gcloud` calls.
Output fragility	Prefer `--format=value(...)` or JSON output parsed with Aeson over human-readable text.
IAM propagation delay	Retry on auth failures; do not trust `check` alone in the first seconds after a grant.
gcloud not installed	Add a `Binary "gcloud"` provider node that fails early with a clear message.
Network access to private VMs	Use IAP tunneling, a bastion, or OS Login with Identity-Aware Proxy.

17. Decisions

VM check: reports only GCE-level RUNNING status. SSH readiness is verified by a separate Gcp.SshAccess.sshAvailable support node.
Load balancer backends: support both GCE instance groups and serverless NEGs for CloudRun.
CloudRun: model services and revisions first, deployed from Podman images pushed to Artifact Registry. Job executions are deferred to a later phase.
IAM: add a generic Gcp.Iam module for creating service accounts and granting roles on projects or resources.
gcloud binary: support gcloud on PATH only for now. Configurable binary paths are deferred.

Advanced querying: targeting `run` at a subset of nodes

2026-09-25T14:15:36Z

Generated from specs/advance-querying.md — the repository is the canonical source, and may be ahead of this page.

Advanced querying: targeting `run` at a subset of nodes

Status: implemented. query show/query plan/query extract-directive, --select/--exclude path globs (plus #ref selectors for rewrite-introduced nodes), the Plan artifact with its directive digest, run up --plan (--force-stale-plan as the dumb escape hatch), and forceSkip all shipped in Salmon.Actions.Query and Salmon.Builtin.CommandLine; serve’s status/query/converge --select reuse the same resolver. Still open as written below: re-validating a plan’s refs at run time, tag-based addressing, selection-only execution, and a plan for downTree. Where the code differs from the design, “Deviations from this design” at the end says how and why. Open question 2 is settled (see there). Kept as the design record.

Problem

Today execCommandOrSeed (Salmon.Builtin.CommandLine) only offers whole-graph operations: run Up executes every node, run Tree/run DAG describe every node. There is no way to say “run everything except this node” or “show me just this subtree” without hand-editing the recipe. As graphs grow (a single initialize already nests user/group/chown/ssh-key subtrees), operators want to:

inspect a subtree or subgraph in isolation (debugging, code review of a recipe change, onboarding),
run Up while skipping a known-bad or intentionally-deferred node, without losing the rest of the DAG’s ordering/dedup/failure-propagation behavior.

Design goals / non-goals

Goals:

A query surface to talk about “this subtree”, “this subgraph”, “all of X except Y” against an already-expanded graph.
A run-compatible execution mode that skips excluded nodes but otherwise preserves exact DAG semantics (ordering, Ref dedup, Failed/Blocked propagation).
Safe by construction against the seed/config/query/run split already being four independent process invocations: nothing should let an exclusion list computed against one graph get silently applied to a different one.

Non-goals (v1):

Running only a subtree in isolation (as opposed to “everything, minus some exclusions”) — interesting, but a distinct feature; see “Future work”.
A general boolean query language (intersection, negation of a whole expression, etc.) — v1 covers union-of-selections minus union-of-exclusions, which is what the motivating use case needs.
Down/teardown-with-plan — downTree has no prelim-equivalent today (see CLAUDE.md), so “skip on down” needs its own design; out of scope here.

Why the addressing/digest scheme has to start from the directive, not the seed

Recall the existing two-phase protocol (Salmon.Builtin.CommandLine):

my-salmon config <seed-args...>   # seed -> directive JSON  (Configure IO seed directive: impure)
my-salmon run Up|Tree|DAG         # directive JSON -> Op -> execute            (Track' directive: pure)

Configure’s own haddock is explicit that this split exists so the “turn a seed into a directive” step can be impure (reads files, env vars, whatever) while “turn a directive into ops and run them” is meant to be hermetic. In every existing recipe, Track' directive builds an Op = OpGraph Identity Actions' — the Identity means expand is pure. Given the same directive JSON, the DAG shape (nodes, Refs, edges) is fully deterministic. All the non-determinism in the whole pipeline lives in the seed → directive step.

That has one immediate consequence for this design: a query command must consume the directive (the same JSON run already reads from stdin), not the seed. If query instead re-ran Configure from a seed, two separate invocations of an impure Configure (one for query, one later for run) could silently produce two different directives — and therefore two different graphs — while looking like “the same seed” to a human. Consuming the directive sidesteps that: it’s already the hermetic boundary the project chose on purpose.

This also gives us the digest mechanism for free: hashing the directive’s canonical JSON bytes pins the exact DAG shape a plan was computed against, and run can cheaply re-check that hash against whatever directive it’s handed before trusting a plan.

Node addressing

Reuse the path format run tree (Salmon.Actions.Help) already prints — /initialize/chown/deb etc. — as the human-facing selector syntax, since operators already read that format when debugging today.

/initialize/chown/deb            -- exact path
/initialize/chown/*               -- one segment wildcard (direct children)
/initialize/chown/**               -- subtree wildcard (any depth)

Important subtlety: a path is a position in the expanded tree, not a node identity — the same Ref can appear at multiple paths (that’s exactly the repeated-subtree phenomenon from the passwordless/chown example earlier in this project). So resolving a selector is a two-step process:

Match the pattern against every path in the expanded Cofree Graph (same traversal Help.printHelpCograph and Dot.printCograph already do), collecting the Ref at each matching path.
Selections and exclusions are ultimately sets of Refs — this is also what upTree’s dedup-by-Ref already keys on, so “exclude this Ref” composes cleanly with “the second occurrence of this node is Redundant anyway”: whichever occurrence is walked first is the one whose prelim gets forced.

Query language (v1)

Repeated flags, applied as one union-then-difference, no operator precedence to think about:

--select PATTERN     -- may repeat; union. Omitted entirely = "everything".
--exclude PATTERN     -- may repeat; union, then subtracted from the selection.

resolvedRefs = union(matches(select_i)) \ union(matches(exclude_j))

This is enough to express both motivating cases:

“show me this subtree”: --select '/initialize/chown/**'
“run everything except this node”: --exclude '/initialize/passwordless'

New CLI surface

Add a query subcommand alongside config/run, and let run up accept an optional plan.

data Command seed
    = Config seed
    | Query QueryCommand
    | Run BaseCommand

data QueryCommand
    = QueryCommand
    { queryMode :: QueryMode
    , querySelect :: [Text]   -- PATTERN, repeatable
    , queryExclude :: [Text]  -- PATTERN, repeatable
    }

data QueryMode
    = ShowMatches   -- human-readable, like `run tree` but annotated
    | EmitPlan       -- JSON `Plan` to stdout

data BaseCommand
    = Up
    | UpWithPlan FilePath   -- new: run Up, but honor an emitted Plan
    | Tree
    | DAG

Both Query and UpWithPlan read the directive from stdin exactly like Run does today — no new input channel, same hermetic boundary.

# inspect a subtree
my-salmon config <seed-args> | my-salmon query show --select '/initialize/chown/**'

# build an exclusion plan
my-salmon config <seed-args> | my-salmon query plan --exclude '/initialize/passwordless' > plan.json

# apply it
my-salmon config <seed-args> | my-salmon run up --plan plan.json

The `Plan` artifact

data Plan = Plan
    { planDirectiveDigest :: Text        -- sha256 of the canonical directive JSON
    , planExcludedRefs :: [Ref]          -- resolved at `query plan` time
    , planExcludedPatterns :: [Text]     -- kept for humans re-reading the file
    }

query plan emits this after resolving patterns against the directive it was handed. run up --plan plan.json:

Reads the directive from stdin (unchanged).
Recomputes sha256 of the same canonical encoding and compares it to planDirectiveDigest. Mismatch → refuse to run (non-zero exit, loud error naming both digests), unless --force-stale-plan is passed. This is the guard against the two-invocation race: if anything upstream (a re-run of config with different seed args, a change to the recipe binary between builds, …) produced a different directive than the one query plan saw, the plan is refused rather than silently mis-applied.
Expands the directive into the Op graph as usual.
Applies forceSkip planExcludedRefs (see below).
Runs upTree unchanged.

`forceSkip`

forceSkip :: Set Ref -> Op -> Op

Walks the graph and, for every node whose ref is in the given set, replaces prelim with pure Skippable, leaving up, down, ref, dynamics, and the graph topology completely untouched. OpGraph’s derived Functor/Traversable (salmon-core/src/Salmon/Op/OpGraph.hs) already maps a function over every node in the structure including through the effectful predecessors, which is the natural place to hang this rewrite.

Net effect on upTree (see Salmon.Actions.UpDown): a forced-skip node is walked exactly like today, dedup-by-Ref still applies, but it reports Skip unconditionally instead of running its own prelim/up. Everything that depends on it runs exactly as if it had genuinely been up-to-date — this is the “keep the original DAG ordering” property asked for: nothing is cut out of the graph, so a dependent that (incorrectly) assumed the excluded node’s effect doesn’t silently get reordered or vanish, it just sees a no-op predecessor.

Caveat worth documenting prominently: this can absolutely produce a broken system if a dependent’s correctness genuinely required the excluded node’s up to have run (e.g. excluding user creation but not the things that log in as that user). The tool enforces graph-shape consistency (the digest check); it cannot and should not try to enforce semantic safety of an operator’s chosen exclusion set.

`query show` output sketch

Mirrors run tree’s indentation, annotating matched lines:

/initialize
/initialize/file-contents writes /etc/sudoers.d/salmon with some contents
/initialize/file-contents/directory ensures /etc/sudoers.d exists, including subdirs
/initialize/passwordless removes a system user's password          [excluded]
/initialize/passwordless/user creates a system user
...
/initialize/chown sets ownership of /home/salmon to salmon:salmon   [selected]

Open questions

Should query plan’s resolved Refs be re-validated at run time against the freshly-expanded graph (i.e. warn/fail if an excluded Ref from the plan no longer appears anywhere), separately from the digest check? Given the digest already pins the whole directive, this may be redundant, but a friendlier error message (“this plan excludes a node that doesn’t exist in this graph”) might be worth the extra check.
--force-stale-plan: allow proceeding on a digest mismatch by re-resolving planExcludedPatterns (not planExcludedRefs) against the new directive? That would make the plan resilient to some classes of directive drift (e.g. an unrelated field changed) at the cost of re-introducing the exact risk the digest was meant to close. Leaning towards: no, keep the escape hatch dumb (skip the check entirely, operator’s responsibility) rather than clever (silently re-resolve and hope the patterns still mean the same thing). Settled: the escape hatch stayed dumb. --force-stale-plan prints a warning naming both digests and applies planExcludedRefs as they are, without re-resolving the patterns.
Tag/dynamics-based addressing (nodes opting into stable labels via the existing dynamics field, independent of tree position) would be more refactor-resistant than path globs, but requires node authors to annotate their ops. Worth a v2 if path-glob addressing turns out too fragile once recipes change shape.

Future work

Selection-only execution (“run just this subtree, treating its own external dependencies as already-satisfied”) rather than everything-minus-exclusions. Same forceSkip machinery, inverted: force-skip everything not reachable from the selection’s roots, then run the selection’s own roots normally. Deferred because it changes the “predecessor’s a real dependency” invariant in a way that needs its own correctness argument (a truly-required predecessor outside the selection would need to be forced skippable, which is a much easier way to shoot yourself in the foot than the exclusion case).
The same Plan/digest idea applied to downTree, once teardown gets a prelim-equivalent.

Deviations from this design

The digest is of the raw stdin bytes, not of canonical JSON. Query.digestBytes hashes exactly the bytes run/query read off stdin. aeson gives no guarantee that decoding and re-encoding reproduces the same bytes across invocations, and a digest that could change between query plan and run up would refuse good plans. As a result, a directive that was reformatted, even with the same meaning, no longer matches its plan. That errs on the safe side, and --force-stale-plan is the way past it.
Plan gained planDirective, empty unless query plan --embed-directive fills it with the directive’s bytes. That makes the plan a self-contained, replayable artifact, for example for an audit trail. query extract-directive PLAN prints the embedded directive, so callers never read the field directly.
There is no prelim any more. Extension’s prelim merged into check :: IO CheckResult (see CLAUDE.md’s “check, not prelim”). forceSkip replaces check with pure Skipped, and Skipped exists for this purpose alone: it records a decision about the node, not a fact about its effect. run up --plan does not go through forceSkip either. It passes the excluded refs to upDag as an UpDown.Gate, so the exclusion composes with collection rewrites: a batch runs if any of its members is wanted. The report is the same Skip either way.
#ref selectors were added beside the path globs. A rewrite can introduce a node, such as a batch, that has no declared tree position for a path to match. A pattern starting with # matches by Ref instead: a prefix of shortRef or of the full ref text. It expands through Rewrite.membersOf to the declared nodes it stands for (Query.resolveRewrittenSelectors). A plan’s exclusion set still holds declared refs only.

Multi-user / privilege separation: running parts of a graph as a lesser identity

2026-09-25T14:15:36Z

Generated from specs/multi-user-privilege-separation.md — the repository is the canonical source, and may be ahead of this page.

Multi-user / privilege separation: running parts of a graph as a lesser identity

Status: draft / not implemented. This is a design sketch to react to, not a committed plan.

Problem

One upTree/downTree is one Haskell process, and a process has exactly one uid. Today that uid has to be the maximum of every privilege any node in the graph needs — in practice root, because somewhere in the graph there is an apt-get install, an nft add rule, or a systemctl daemon-reload. Every other node in the graph then also runs as root, whether or not it needs to.

That produces three distinct problems, and it is worth separating them because they have different fixes:

No way to drop privilege for a node that doesn’t need it. A Cabal.build, a Git.repo clone, an Npm/Spago build, a Web.download — none of these need root, and running them as root means an arbitrary build.sh/postinstall script in a third-party dependency runs as root. There is no vocabulary in the codebase to say “this node runs as builder”.
Identity-switching is hardcoded, per node, in the wrong place. Salmon.Builtin.Nodes.Postgres.psqlAdminRun_Sudo (salmon-ops/src/Salmon/Builtin/Nodes/Postgres.hs:398) bakes sudo -u postgres into the CreateProcess of all twelve of its admin commands. That module already carries the todo this spec is answering (Postgres.hs:394):
todo: workaround chmod and sudo hack with some calling preference
- we’ll need to request more than a Track' (Binary "psql") but some more complex logic with sudo, the user, and the right binary
The consequences of hardcoding are concrete: the choice of mechanism (sudo vs runuser vs setpriv) is not a caller’s to make, sudo becomes an undeclared dependency (nothing in the graph installs it or configures sudoers), and a caller that already runs as postgres still pays for a sudo hop.
Files created by the root process are unreadable by the lesser identity, and the workaround is a security hole. Filesystem.filecontents (Filesystem.hs:51) is ByteString.writeFile under the process umask: root-owned, 0644. So Postgres.adminScript (Postgres.hs:342) copies the migration script into /opt/salmon/postgres/migrations/admin/ and then runs chmod a+r on it (ChmodAdminScript, Postgres.hs:404) purely so that the sudo -u postgres psql -f ... on the next line can read it. That makes every admin migration script world-readable on the box. The same shape would be far worse applied to Secrets.sharedSecretFile or Keys.

The ask is: what vocabulary should salmon grow — ops, decorators, or something else — to express this?

The structural finding: the decoration cannot live on `Op`

The instinctive answer is an Op -> Op decorator:

runAs :: User -> Op -> Op          -- does not work

It cannot work, and understanding why determines the whole design. Extension.up has type IO (). By the time you hold an Op, the command has already been rendered: withBinary (Binary.hs:80) called prepare to build a CreateProcess, closed over it in untrackedExec, and handed the node author an opaque Reporter Report -> IO (). An Op -> Op decorator receives that closure and has no way to look inside it, let alone rewrite its cmdspec. up’s type erases everything.

So an Op-level decorator has exactly one implementation available to it: wrap the whole IO () in a forkProcess+setuid (Layer 4 below), which brings in the threaded-RTS hazards and semantic cliffs discussed there.

The privilege decision has to be made upstream of Extension.up — at the Command/withBinary boundary, where the CreateProcess still exists as a value. That is the good news: it is exactly one chokepoint, and ~100 of the call sites in the repo go through it.

What already exists to build on

The repo is not starting from nothing here; several pieces are already the right shape.

Existing	What it gives us
`Binary.Command { prepare :: arg -> CreateProcess }` (`Binary.hs:72`)	A pure `arg -> CreateProcess`, i.e. a value that can be rewritten by a combinator before `withBinary` closes over it.
`Track'`/`tracking`/`inject`	The mechanism to make "and `sudo` must be installed, and the user must exist" real graph predecessors rather than ambient assumptions.
`User.User`/`Group`/`Owner`/`chown` (`User.hs`)	The identity vocabulary already exists as typed values, plus a `Track' Group`/`Track' User` convention for provisioning them.
`Capabilities.grantCapabilities` (`Capabilities.hs`)	Precedent for "grant a binary less than root instead of running the caller as root" — the fine-grained end of this same spectrum.
`Systemd.Service.service_user`/`service_group`	Identity separation already solved for the runtime of a service; this spec is the same idea for the provisioning of it.
`Self.callSelfAsSudo` (`Self.hs:99`)	Precedent for re-invoking the salmon binary itself under a different identity with a JSON directive on stdin — currently only over SSH, but the local case is the same trick.
`Query.Plan` + `run up --plan` + `Query.forceSkip`	Machinery to execute a subset of a graph, digest-pinned to a directive. This is what makes Layer 3's partitioning possible with almost no new code.
`Extension.dynamics :: [Dynamic]` + `collectDynamics`	The established way to attach out-of-band typed metadata to a node and recover it by a whole-graph analysis — the natural carrier for a privilege-domain tag.
`UpDown.Gate`	A caller-supplied per-node "does this traversal want this node" — already the hook a domain-filtered pass would use.

Proposed layering

Four layers, deliberately independent. Each is useful alone; none requires the next.

Layer	What it segments	New code	Recommendation
L0	Nothing — partition by hand with existing `query plan` + `sudo -u self run up --plan`	none	Document it now; it is available today
L1	Individual binary calls (`RunAs` + `Invoker`)	small, in `Binary.hs`	Build this first. Answers the `Postgres.hs:394` todo directly
L2	File ownership/mode at creation	small, in `Filesystem.hs`	Build second. Without it L1 is half a solution
L3	Whole subgraphs, by declared domain, via re-exec of self	medium	Design now, build if L1+L2 prove insufficient
L4	Individual ops, via `forkProcess`+`setuid`	small but hazardous	Document why not; keep as escape hatch

The user’s framing — “if it’s too complicated to segment ACLs/user around an Op, then at least segment the binary calls” — maps onto L4-vs-L1. The recommendation is that L1 is not the consolation prize: it is the better answer, because it puts the decision where the information still exists, whereas L4 puts it where the type has already thrown the information away.

L0: partition by hand, today

Worth writing down because it needs no code and it validates the L3 model before anyone builds L3. The querying work already landed, so:

my-salmon config ... > directive.json

# everything the unprivileged builder owns
my-salmon query plan --select '/**/cabal-build/**' --select '/**/git-repo/**' \
  < directive.json > builder.plan
# everything else
my-salmon query plan --exclude '/**/cabal-build/**' --exclude '/**/git-repo/**' \
  < directive.json > root.plan

sudo -u builder my-salmon run up --plan builder.plan < directive.json
sudo             my-salmon run up --plan root.plan    < directive.json

The Plan’s directive digest is what keeps this honest: both passes are pinned to the same directive.json, so a plan computed against one graph cannot be silently applied to another.

Two real limitations, which are also the argument for L1/L3:

Ordering across the partition boundary is lost. Each invocation walks its own subset with correct internal ordering, but if a root node must run between two builder nodes, no ordering of the two passes expresses that. The partition has to be a topological cut, and nothing checks that it is.
Addressing is by path glob, so a recipe refactor that renames a shorthand silently changes which nodes land in which privilege domain. Silently mis-partitioning privilege is a bad failure mode. L3 fixes this by moving the tag into the recipe.

L1: `RunAs` + `Invoker` — segmenting the binary calls

The types

-- in Salmon.Builtin.Nodes.Binary (or a new Salmon.Builtin.Nodes.Privilege)

-- | Whose identity a command runs under.
data RunAs
  = -- | Inherit the salmon process's own identity. The default; today's
    -- behaviour for every node that doesn't hardcode sudo.
    RunAsSelf
  | -- | Drop (or raise) to another account, by a chosen mechanism.
    RunAsUser !Mechanism !User.User
  deriving (Show, Eq, Ord, Generic)

instance Hashable RunAs

-- | How the switch is performed. Pluggable on purpose: which one is correct
-- depends on what the salmon process already is, and on what the target box
-- has configured.
data Mechanism
  = -- | @sudo -u \<user\> -- \<cmd\> \<args\>@. Works from a non-root caller,
    -- but needs a sudoers rule (and NOPASSWD, since there is no tty).
    ViaSudo
  | -- | @runuser -u \<user\> -- \<cmd\> \<args\>@. Needs the caller to already
    -- be root, but needs no sudoers configuration and cannot prompt.
    ViaRunuser
  | -- | @setpriv --reuid \<uid\> --regid \<gid\> --init-groups -- \<cmd\>@.
    -- Lowest-level; no PAM session at all.
    ViaSetpriv
  deriving (Show, Eq, Ord, Generic)

Recommended default: ViaRunuser, not ViaSudo. Salmon’s whole premise is that the process starts privileged. From root, runuser is strictly simpler than sudo: no sudoers file to provision (which is itself an unmodelled dependency today), no PAM password path that could block a headless run forever, no env_reset surprises. sudo stays available for the case where salmon is not root and needs to go up. setpriv for when even a PAM session is unwanted.

Note the -- in every rendering: it terminates option parsing, so a command whose first argument happens to start with - cannot be reinterpreted as a flag of the wrapper.

The rewrite

-- | Rewrites a rendered command to run under another identity.
applyRunAs :: RunAs -> CreateProcess -> CreateProcess

It rewrites cmdspec only. RawCommand path args becomes RawCommand "runuser" (["-u", user, "--", path] <> args); a ShellCommand becomes the wrapper invoking a shell with -c (and is worth discouraging — see “Gotchas”).

The carrier: `Invoker`

A RunAs alone is not enough, because switching identity has dependencies: the wrapper binary must be installed, and the target account must exist. Those are graph edges, and salmon already has the vocabulary for graph edges. So the thing a node asks for is not a bare Track' (Binary x) but:

-- | Everything needed to invoke a binary: where the binary comes from, whose
-- identity it runs under, and how that identity is provisioned.
data Invoker (x :: Symbol)
  = Invoker
  { invoker_binary :: Track' (Binary x)
  , invoker_runAs :: RunAs
  , invoker_identity :: Track' RunAs
    -- ^ provisions the wrapper binary + the target account; 'ignoreTrack'
    -- when both are pre-existing.
  }

-- | Lift today's signature unchanged: same binary, no identity switch.
asSelf :: Track' (Binary x) -> Invoker x
asSelf t = Invoker t RunAsSelf ignoreTrack

-- | The parallel of 'withBinary', threading the identity through.
withInvoker ::
  Invoker x -> Command x arg -> arg -> ((Reporter Report -> IO ()) -> Op) -> Op

withInvoker is withBinaryStdin with two changes: it applies applyRunAs to the CreateProcess before closing over it, and it injects run invoker_identity invoker_runAs alongside the existing binary track. That second half is the part that earns its keep — it makes “this op needs sudo installed and the postgres account to exist” visible in run tree/run dag and orderable by upTree, instead of being an assumption that fails at runtime.

invoker_identity being a Track' (rather than a hardcoded Debian.sudo) is the same convention as the existing rule that recipes must not bake in a secret-transport mechanism: take the provisioning as a parameter, let the caller pass ignoreTrack when the account and wrapper are already provisioned out of band. A recipe that hardcodes Debian.sudo would be making the same mistake psqlAdminRun_Sudo makes today, just one level up.

Two different consumers, one type

It is worth being explicit that L1 serves two cases that look similar and are not:

Node-author-fixed identity. “psql admin commands run as the postgres account” is intrinsic to the node; no caller should override it. Here the node keeps constructing its own RunAs internally, and the only thing that changes versus today is that the mechanism and the dependencies become values rather than a string literal in prepare.
Caller-chosen identity. “build this cabal target as builder” is a deployment choice. Here the smart constructor’s signature has to change from Track' (Binary "cabal") to Invoker "cabal".

Only the second forces signature churn, and only on the nodes that want it.

Migration

There are ~100 withBinary/withBinaryStdin call sites and ~89 Track' (Binary …) parameters in salmon-ops. A big-bang migration is neither necessary nor desirable:

Add RunAs/Mechanism/applyRunAs/Invoker/asSelf/withInvoker. Nothing changes for anyone; withBinary keeps working unchanged and is redefined as withBinary t = withInvoker (asSelf t).
Fix Postgres: delete psqlAdminRun_Sudo, keep a plain psqlAdminRun that renders bare psql, and have the admin nodes build Invoker psql (RunAsUser ViaRunuser "postgres") …. This resolves Postgres.hs:394 and is a good single-module proof of the design.
Convert nodes to Invoker only when a caller actually wants to choose — Cabal, Git, Npm, Spago, Web, Tar are the plausible first set (the “build stuff” cluster, which is where running as root is least defensible). Call sites become asSelf debianCabal mechanically.

`Ref` identity — a correctness requirement, not a nicety

If a node’s identity is caller-chosen, its mkRef key must include the RunAs, otherwise upTree’s dedup collapses “clone this repo as builder” and “clone this repo as ci” into one node and silently drops the second. This is easy to get wrong because it only bites in graphs that actually use two identities for the same command. Rule to state in CLAUDE.md’s “Conventions for node authors”: any node taking an Invoker includes its RunAs in the Ref key. Cost is zero when the identity is fixed, so there is no reason to make it conditional. This is why RunAs needs Hashable.

`CommandIO` too

withBinaryIO/CommandIO (Binary.hs:163, used by WireGuard.privateKey/ publicKey where stdin/stdout need redirecting) builds its CreateProcess in IO. applyRunAs applies just as well there — it is still a CreateProcess -> CreateProcess at the end. Worth doing at the same time so the two paths don’t diverge, even though no current CommandIO user needs an identity switch.

L2: ownership and mode at file-creation time

L1 lets psql run as postgres; it does not let postgres read the script. Without L2, every L1 identity switch grows its own chmod a+r, and the repo ends up with more instances of the exact hack this spec is trying to remove.

Today Filesystem.dir/filecontents create paths under the process umask, and User.chown (User.hs:185) is a separate op whose own haddock says it “does not itself create the path”. Composing them means create-then-chown, which has a window where the file exists with the wrong owner and mode. Irrelevant for a migration script; not irrelevant for Secrets.sharedSecretFile or Keys.

Proposal: an optional ownership/mode decoration applied at creation.

data Ownership
  = Ownership
  { ownership_owner :: Maybe User.Owner   -- ^ user:group; needs root
  , ownership_mode  :: Maybe FileMode     -- ^ e.g. 0o640
  }

dirWith          :: Ownership -> Directory -> Op
filecontentsWith :: (EncodeFileContents a) => Ownership -> FileContents a -> Op

Implemented in-process with System.Posix.Files (setFileMode/setOwnerAndGroup, or openFd with the mode up front) rather than by shelling out — no subprocess, no chmod/chown binary dependency, and the mode can be correct from the moment the inode exists. dir/filecontents stay as they are, defined as the Ownership Nothing Nothing case.

The Postgres.adminScript fix then reads: write the script 0640 root:postgres into the admin dir, drop ChmodAdminScript from PsqlAdmin entirely, and the world-readable window disappears along with it.

Open question worth deciding early: should Ownership be a field on FileContents/Directory rather than a separate …With constructor? A field is tidier and forces every call site to consider it; a separate constructor keeps the ~40 existing filecontents call sites untouched. Leaning towards the separate constructor for the same “additive, no churn” reason as asSelf.

L3: declared privilege domains + partitioned execution

L1 handles “this command runs as X”. It does not handle “this whole subtree — including its in-process file writes, its nested upTree, its non-subprocess ups — runs as X”. If that turns out to be needed, the answer is not to make up polymorphic in identity; it is to run a second salmon process under that identity, and salmon already knows how to do that.

The tag

Reuse Extension.dynamics, which exists for exactly this kind of out-of-band annotation:

newtype PrivilegeDomain = PrivilegeDomain Text   -- e.g. "builder", "root"

inDomain :: PrivilegeDomain -> Op -> Op   -- attaches via dynamics

Unlike L0’s path globs, this survives recipe refactors: the tag travels with the node, and collectDynamics recovers the partition from an expanded graph.

The execution

A --domain selector on query plan resolves the tag to a Ref set, giving exactly the Plan files L0 built by hand — but derived from the recipe rather than from glob spelling. Then either the operator runs the passes (as in L0), or a driver command does it:

my-salmon run up --partitioned < directive.json

which, for each domain in topological order, re-execs /proc/self/exe run up --plan <domain>.plan under that domain’s RunAs, with the directive on stdin. This is Self.callSelfAsSudo (Self.hs:99) with the SSH hop removed — the same “serialize a directive, re-invoke myself under another identity, check the child’s exit code” pattern that is already load-bearing for remote provisioning.

The hard part is not the mechanism, it is the cut: the partition must be topologically consistent, i.e. there must exist an ordering of domains such that no node in an earlier domain depends on a node in a later one. This is checkable on the expanded graph and must be checked — an unchecked partition fails as “provisioning silently ran in the wrong order”, which is much worse than a refusal. run up --partitioned should refuse to run an inconsistent partition and print the offending edge.

Note this is strictly more expressive than L1 for the sandboxing question but strictly less fine-grained in ordering: within-domain ordering is exact, cross-domain ordering is coarse. L1 preserves the exact DAG. That is the real tradeoff between the two, and it is why L1 should come first.

L4: `forkProcess` + `setuid` per op

The literal Op -> Op decorator, for completeness:

runOpAs :: RunAs -> Op -> Op   -- wraps `up`/`down` in fork + setgid/setuid

The child calls setGroups/setGroupID/setUserID (in that order — dropping the group after the user is a classic bug, as is forgetting supplementary groups), runs the original up, and _exits; the parent getProcessStatuses and rethrows a non-zero status as an exception so upTree sees a Failed.

Why it should not be the default:

forkProcess under the threaded RTS is hazardous. Only the calling thread survives into the child; any lock held by another capability at fork time is held forever in the child. Salmon’s own Reporter machinery and any node doing concurrent IO are exactly the things that make this bite. It is not “usually fine” so much as “usually fine until a report is being flushed”.
In-memory effects are lost. ups that mutate an IORef, populate a Tracked value, or run a nested upTree (e.g. PostgresMigrations.remoteMigrateOpaqueSetup’s continuation) do that work in a process that then exits. The parent sees only an exit code.
It is one-way. The parent must be root; a node cannot regain privilege. Fine as a constraint, but it means the ambient uid still has to be the maximum over the graph, which is the thing we set out to avoid.

Keep it documented as an escape hatch for a node whose up is genuinely in-process and genuinely needs another uid. Do not build recipes on it.

Cross-cutting concerns

Teardown. At L1 this is free: down closes over the same rewritten CreateProcess, so a node torn down runs under the identity that brought it up. At L0/L3 it is a real gap — run down --plan does not exist, and specs/advance-querying.md explicitly listed down-with-plan as a non-goal because downTree has no prelim-equivalent. Partitioned teardown therefore needs its own design; it is not just “same thing, reversed”, because the domain ordering also reverses and the “a failed down blocks its predecessors” containment has to survive being split across processes. Another reason to prefer L1.

serve. L1 composes with Serve without any thought: the loop stays one root process and the identity switch happens per subprocess. L3 does not — serve holds the World in memory across declarations, and a design that re-execs a fresh process per domain has nowhere to put that state. If partitioned execution and serve are both wanted, that is a genuine open design question, not an implementation detail.

Reporting. Binary.Report’s CommandStart/CommandStopped carry the CreateProcess, so the rewritten (wrapper-prefixed) command shows up in reports automatically — which is what we want for auditability: the report should say what actually ran. The debuggability cost is that a CommandFailed now carries the wrapper’s stderr, so “user not in sudoers” and “psql syntax error” arrive through the same channel and look similar. Worth a distinct report constructor, or at least making applyRunAs record the original cmdspec alongside the rewritten one.

Environment and working directory. applyRunAs rewrites cmdspec and leaves CreateProcess’s env/cwd alone, but the semantics of both change under a wrapper. sudo resets the environment by default (env_reset), so an env set by prepare may not survive; runuser - versus runuser differ on whether a login shell is set up. Worse, cwd is applied by the parent before exec, so a cwd the root process can enter but the target user cannot (the Cabal.build case — Cabal.hs sets cwd to the project dir) fails in a confusing way. Rule for node authors: a node that sets cwd and takes an Invoker must ensure the directory is reachable by the target identity — which is a direct dependency on L2 landing.

Testing. Test.PostgresInitSpec shims ["apt-get", "sudo", "bash", "chmod"] inside its container. Recommending runuser as the default mechanism means adding a runuser shim; dropping ChmodAdminScript at L2 means the chmod shim can eventually go. Both are cheap, but the shim list is a reminder that mechanism choice is observable by the test harness and should be settled before the Postgres migration, not after. Beyond that, L1 is unusually testable without containers: applyRunAs is a pure CreateProcess -> CreateProcess, so its renderings (including the -- boundary) belong in the cheap in-process test layer.

Security. Three things deserve stating rather than assuming:

The -- argument boundary is not optional; without it a command argument beginning with - is parsed by the wrapper.
ShellCommand composed with an identity switch is a quoting hazard — applyRunAs should either refuse ShellCommand or be very explicit about what it does with it. Postgres.CreateDB’s existing bash -c string is the one place this already matters.
Dropping privilege is only meaningful if the lesser user cannot trivially regain it. A builder account that owns a file the root pass later executes, or that can write into a directory root reads from, has not actually been contained. L2 is what makes L1’s containment real, which is the main argument for treating them as one piece of work rather than two.

Open questions

Naming: Invoker/RunAs, or something closer to the todo’s own vocabulary (“calling preference”)? RunAs reads well at call sites (RunAsUser ViaRunuser "postgres"); Invoker is the weaker of the two names.
Should Invoker also carry the cwd/env policy (i.e. “this identity always runs with this env”), or is that overloading it?
Is Mechanism per-Invoker, or a single deployment-wide default with per-node override? A box configures sudoers once; carrying the choice on every invoker may be more knob than anyone wants.
Does the PrivilegeDomain tag (L3) subsume RunAs (L1), i.e. should an Invoker name a domain and let a top-level table map domains to identities? That would make L1→L3 a smooth progression rather than two mechanisms, at the cost of indirection in the common single-identity case.

Suggested order of work

RunAs/Mechanism/applyRunAs + pure tests on the rendering. No call site changes.
Invoker/asSelf/withInvoker; redefine withBinary in terms of it.
L2 Ownership on Filesystem.
Migrate Postgres: plain psqlAdminRun + an Invoker, delete ChmodAdminScript, closing Postgres.hs:394. This is the end-to-end proof that L1+L2 solve a real case that exists today.
Add the Ref-includes-RunAs rule to CLAUDE.md’s node-author conventions.
Convert the build cluster (Cabal, Git, Npm, Spago) to Invoker, and see whether “builds run as builder” is now expressible without L3.
Only then decide on L3.

`run serve`: convergence and supervision

2026-09-25T14:15:36Z

Generated from resources/serve-supervision.md — the repository is the canonical source, and may be ahead of this page.

`run serve`: convergence and supervision

This is a companion to howto-ops.md: where that doc covers how to write one Op, this one covers what happens to a whole graph of them once you run them through run serve instead of a one-shot run up. Read howto-ops.md first if Op, check, Track, or the seed → spec → ops CLI protocol are unfamiliar — this doc assumes all of them.

It’s organized around the question a new user actually has: what do I get for nothing, how do I see it happening, and what do I have to write myself to get more?

1. The mental model in three sentences

run up/run down are one-shot: they walk a graph once and stop. run serve is long-running: it reads seed declarations from stdin (one per line — up <seed args>, down <seed args>, only <seed args>, and a handful of operator commands), keeps a World of everything it’s been told to want, and converges that World after every command. Between commands, while nothing is waiting on stdin, it also tends every node it’s converged — a second, independent mechanism that notices drift and fixes it (or doesn’t, depending on what the node tells it) without anybody typing anything.

Those are the two halves of “supervision”: a convergence pass (driven by you, typing or piping commands) and a tending loop (driven by idle time). Almost everything below is about what each one does for a node that says nothing about itself, and what a node can say to get more out of either.

2. What you get for free

Take any existing salmon binary — built the ordinary way, via Salmon.Builtin.CommandLine.execCommandOrSeed, with plain Ops that have no check, no Supervision dynamic, no managed action — and point it at run serve instead of run up. You get, with zero changes to your nodes:

Convergence bookkeeping across re-declarations. Nodes unify by Ref across every seed that mentions them. Re-declaring an unchanged seed is a no-op (nothing pending, nothing re-run). Retiring a seed tears down exactly the nodes no other live seed still wants — a directory two files share survives until the last file using it is gone, never torn down out from under the other.
status/history/query introspection. status lists every node this world still cares about, its wanted direction (up/down), and its convergence (Pending/Stale/Converged/Errored/Blocked) — plus, if the node has ever been tended, its own last check verdict and (for a failing node) the tail of its output, and (underneath) every path a currently-active seed’s graph reaches it at — the exact text a --select/--exclude pattern matches, pasteable straight back in. Without this a pattern could only be guessed; status is where it comes from. history lists what was declared, when, and by whom — a typed line, a loaded file, or (§12) a fetched document. query annotates nodes [selected]/[excluded] against a --select/--exclude pattern without acting on anything — useful for checking a pattern before you force/ pause with it for real.
Re-declaring with different content is noticed by the pass itself, not just eventually by a background loop — as long as something about the declaration that changed is visible to Salmon.Op.Dag.sameRepresentative (shorthand, help, notes, most dynamics). A node goes Stale rather than staying silently Converged, and the very next convergence pass re-checks it. (§8 below is about making your own content-bearing nodes participate in this — the stock Filesystem.filecontents already does.)
force/recheck/pause/resume, addressable by --select/--exclude path globs, reach any node regardless of whether it has a check:
- force re-applies a node even if its own check currently calls it satisfied — the only way to tell a healthy-looking node “do it anyway”.
- recheck makes a tended node look now instead of waiting out its current delay.
- pause/resume stop/restart tending a node without touching whatever effect it currently holds.
A managed node (one that owns a running process, see §7) is supervised with a sane default the moment it exists — restarted OnFailure, OneForOne (demotes nobody else), kept running across every other serve command, even with a completely default Supervision. You don’t have to opt in to get some policy; you only have to opt in to get a different one.
Idle cost is proportional to what a node claims. A node with no check answers Immaterial (“asking would cost what applying costs”) and is parked — looked at once, then left alone on its mailbox until something (a dependency moving, an operator’s force/recheck) wakes it. It is not polled every minute doing nothing useful. This means an undecorated graph costs almost nothing to tend, but it also means it is not self-healing — see the gap this leaves in §4.
A piped script stays deterministic. run serve < script.txt queues every line before the first pass can even finish, so there is never an idle moment for the tending loop to run in — what you get is exactly the sequence of convergence passes the script describes, nothing more. This is what makes run serve usable for CI/scripted setups, not only interactive sessions. Reach for supervise off if you want the same determinism in an interactive session, and supervise on (the default) to get it back.
Declaring and converging can be split apart. By default every up/ only/down/clear converges immediately, as if converge had been typed right after it. autoconverge off turns that off: declarations still record and are visible to status/query right away, but nothing is applied until you type an explicit converge (optionally restricted with --select/--exclude). Useful for stacking up several declarations — up a, down b, up c — and reviewing the combined result with query/status before anything actually runs. run serve --no-autoconverge starts a session already in that state, for a script or session that always wants to review before acting.

None of this requires writing a single check or Supervision dynamic. What it does not give you for free: a node whose effect can be perturbed from outside salmon (a config file hand-edited, a directory rmdir’d, a service that crashes) is not put back unless that node has a check that says so, or is a managed process. That’s the next section.

3. Try it yourself, five minutes

The quickest way to see all of this without writing any code is the fixture shipped with the repo, salmon-ops-serve-fixture (salmon-ops/fixtures/ServeFixture.hs — read its own module haddock for the full tour). But the same shape works on any salmon binary you already have; substitute your own seed args for <seed> below.

$ my-salmon run serve
up <seed>
status

You should see serve: mode: interactive, then your nodes listed, up/Converged. Now perturb something underneath it from another terminal — if any of your nodes is a Filesystem.filecontents or a Filesystem.dir, edit or delete the file/ directory by hand. Then:

status

If the perturbed node has a check (every filecontents does, and every dir re-applies on a timer instead — see §5 and §6), it’s back, without you typing anything else in between — the tending loop already noticed and fixed it while you were looking the other way. If it’s a plain node with no check, it stays broken until you force it:

force --select '/**'

Now try a re-declaration with different content (if your seed’s args feed into something content-bearing):

only <same seed, different content argument>
status

Watch for a line showing a node Stale rather than jumping straight back to Converged with nothing having happened — that’s the pass itself noticing the change (§8), not the tending loop catching it later.

Finally:

pause --select '/**'
# perturb something again — it should stay broken
resume --select '/**'
# and now it's fixed again
quit

help at any point prints the full command reference; help TOPIC (e.g. help select, help force) prints more about one command. Two commands this doc does not otherwise use: load FILE runs a file of these lines, in order, as if typed, and up-directive/only-directive/down-directive FILE declare straight from a directive JSON file instead of seed args.

4. What’s free vs. what needs decoration

capability	free, zero decoration	needs
convergence bookkeeping, re-declare/retire semantics	✅	—
`status`/`history`/`query`	✅	—
`force`/`recheck`/`pause`/`resume` reachability	✅	—
discovering a node's `--select`/`--exclude` path	✅ (`status`/`query` print it)	—
disambiguating two nodes that share one path (identical shorthand)	✅	a `#ref` selector (§9)
`Stale` on a re-declaration that changes `help`/`notes`/most `dynamics`	✅	—
`Stale` on a re-declaration that only changes content baked into `up`	❌	a `check`, or a content-derived `notes`/`dynamics` field (§8)
self-healing when an effect is perturbed from outside	❌	a `check` (§5)
a process salmon owns, restarted when it exits	❌ (node must declare it)	`managed` (§7)
choosing how a managed node is restarted, or whether its going away bounces dependants	uses a sane default	a `Supervision` dynamic (§6)
a non-`managed` node re-applying itself on a timer instead of parking	❌	`supReapply` (§6), narrowly
addressing a batch/rewrite-introduced node that has no declared path	❌	a `#ref` selector (§9)
bounding how many nodes converge at once	❌ (unbounded by default)	`--max-concurrency N` (§10)
reports a script can parse	❌ (text by default)	`--json` (§11)
fetching declarations from a registry instead of typing them	❌ (stdin only)	`--follow DIR --label L` (§12)
a second operator or a tool attached to a running `serve`	❌ (stdin only by default)	`--listen PATH` (§13)
reads, commands and a live event stream over HTTP, a terminal client, a web page	❌	`--http PATH` (§14), `--http-tcp HOST:PORT` with TLS and a token for a network
a status document a fleet reader can fold	❌	`--status-sink PATH` (§12)

5. Decorating nodes: `check`

This is the single highest-leverage thing you can add to a node, and it’s covered in full in howto-ops.md §4 for the one-shot angle (idempotency). Here’s what each CheckResult means specifically to the tending loop, which is a second, independent reader of the same function:

Immaterial (the default, for a node with no check at all) parks the node. It is looked at once on the way up, then left alone — not polled. This is right for a node whose up is already idempotent and cheap to ask about (mkdir -p, ip route replace) — there is nothing cheaper to put on a timer than what applying costs. It is not right for a node whose effect can drift without your say-so, because nothing will ever notice.
Success/Failure text are what let a node be supervised at all. A check that can tell the effect is gone (Failure) is the only thing that can trigger a restart under Supervision’s Restart policy, or notice a re-declared node’s new content (§8), or fire a RestForOne demotion (§6). Without a real check, a node is brought up once and then genuinely nobody is watching.
Unknown (a check that ran and genuinely couldn’t tell — e.g. a systemd unit mid-restart) never triggers anything under the tending loop; treating “I couldn’t look” as evidence of anything would spin a node at its delay floor forever. It’s different from Immaterial: Unknown means a check exists and sometimes can’t say; Immaterial means there’s no check worth having.
Completed means the effect ran to completion and stopped on purpose — a job, not a service. Converged, not running. Only a Restart = Always policy (§6) re-applies on this.
Skipped is not something your check should ever return itself — it’s what Query.forceSkip/a run up --plan exclusion rewrites a check into, a statement that someone decided this node is satisfied, not a statement about the effect.

Worked template, following Filesystem.checkFileContents’s shape (compare what’s there against what you’d write, cheaply and without quoting secrets into the failure text):

myCheck :: MyConfig -> IO CheckResult
myCheck cfg = do
    there <- probeTheEffect cfg
    pure $ case there of
        Nothing -> Failure "missing: <identifying text, no secrets>"
        Just actual
            | actual == expected cfg -> Success
            | otherwise -> Failure "drifted: <identifying text>"

6. Decorating nodes: `Supervision`

Salmon.Op.Supervision.Supervision is a handful of optional fields, carried on a node’s dynamics (the same channel Package uses — see howto-ops.md §7 on dynamics if this is unfamiliar):

import Salmon.Op.Supervision (defaultSupervision, supervised, Strategy (..), Restart (..), seconds)

op "my-node" nodeps $ \actions -> actions
    { dynamics = [supervised defaultSupervision
        { supRestart = OnFailure       -- Always | OnFailure | Never (default OnFailure)
        , supStrategy = OneForOne      -- OneForOne | RestForOne      (default OneForOne)
        , supReapply = False           -- re-run `up` on a timer instead of parking (default False)
        , supWatchdog = Nothing        -- Maybe Micros: report if silent this long (default Nothing)
        , supStableAfter = seconds 10  -- how long up resets the failure tally (default 10s)
        , supDemoteEvery = seconds 10  -- RestForOne rate limit, see below     (default 10s)
        , supGiveUpAfter = Nothing     -- Maybe Int: stop retrying after N consecutive failures
        }]
    , ...
    }

Prefer amending defaultSupervision field-by-field, as above, rather than writing out the constructor positionally — the record has grown several times already.

A node that says nothing here behaves exactly as if none of this existed — every field’s default is chosen so an undecorated graph is unaffected. What each buys you, beyond the default:

supRestart = Never — for a node whose up is destructive to repeat, or whose failure means something worse happened upstream that an operator should look at rather than salmon silently retrying.
supRestart = Always — the “restart a service that exits cleanly on reload” case. Don’t set this on a job that’s meant to run once.
supStrategy = RestForOne — this node’s going away should bounce whatever depends on it. The case this exists for is a config file: a service reading it should be re-verified (not necessarily restarted — see below) whenever the file changes underneath it. Authored on the node that goes away, not on its dependants, because only the config file’s author knows its content is load-bearing.
- The safety condition: this reframes a RestForOne bounce as “go re-verify yourself,” which is cheap only if the dependant’s up/ check are genuinely idempotent — the same baseline convention every node in this tree is already supposed to follow. A dependant whose reapplication is genuinely expensive (a slow warmup, an unsafe-to-repeat migration) has no way today to resist a RestForOne demotion sent from upstream — don’t put RestForOne on a node whose dependants might not be able to afford being asked to re-verify on every change.
- supDemoteEvery is the rate limit: a node is demoted by a dependency at most once per this interval, so a flapping dependency can’t rebuild the whole cone behind it on every flap. An isolated departure is always honoured whenever it comes, however soon.
supGiveUpAfter = Just n: stop restarting after n consecutive failures (a service that crashes once a week never latches off, because supStableAfter forgets the streak once it’s run that long between crashes). A node that’s given up says so in status/a force starts it over; nothing else touches it until you do.
supWatchdog = Just (seconds n): report a node as possibly-wedged if it’s gone this long without doing anything observable. Reports only — nothing kills a wedged up, since not every node has a bracket to kill it through.
supReapply = True: for a node with no check at all (answers Immaterial), re-run up on the tending loop’s delay ladder instead of parking. This is narrow — sound only for an up that is genuinely cheap and idempotent (Filesystem.dir is the one builtin that sets it: createDirectoryIfMissing costs about what asking first would). Don’t reach for this as a substitute for writing a real check on anything whose up is a build, a clone, or otherwise not free to repeat.

7. Decorating nodes: owning a process (`managed`)

A node whose effect is a running process — not “create a file,” but “keep this running” — fills in managed instead of relying on up alone. Salmon.Builtin.Nodes.Daemon.daemon is the builtin for this:

import qualified Salmon.Builtin.Nodes.Daemon as Daemon
import System.Process (proc)

myService :: Op
myService = Daemon.daemon reporter $
    Daemon.defaultDaemon "my-service" (proc "/usr/bin/my-service" ["--config", path])

Three things that follow from managed existing at all, distinct from everything above:

A one-shot run up cannot bring this node up at all — its up deliberately throws (NeedsSupervisor) rather than silently no-op’ing. Only run serve can hold a running action.
run serve treats it specially. It’s invisible to the convergence pass in both directions (there’s nothing for a one-shot up/down to do with it); instead its machine races the action itself, reads its ExitCode against supRestart, and is kept running across every other serve command rather than stood down and restarted on every status.
Decorate it with Supervision exactly as any other node (§6) — the defaults already give you OnFailure restart and OneForOne (no bouncing of dependants). RestForOne on the config file this process reads, not on the process itself, is the combination the fixture’s own --daemon walkthrough demonstrates end to end (ServeFixture.hs’s module haddock).

8. Making your own content-bearing nodes `Stale`-aware

§2 mentioned that a re-declaration with different content is noticed by the pass, if something about it is visible to Dag.sameRepresentative (shorthand/help/notes/most dynamics — deliberately not up/check, which are functions and not comparable). Filesystem.filecontents already does this for you: its EncodeFileContents instances carry a pure contentFingerprint, and filecontents puts it into notes when one is available. If you’re writing your own content-bearing node — a hand-rolled one, or a new EncodeFileContents instance — the same trick is available to you:

myConfigNode :: MyConfig -> Op
myConfigNode cfg =
    op "my-config" (deps [enclosingDir]) $ \actions -> actions
        { help = "writes " <> path
        , notes = ["content-hash: " <> hashOf cfg]   -- <-- this line is the whole trick
        , ref = mkRef "my-config" path
        , check = myCheck cfg                         -- still worth having regardless, per §5
        , up = writeTheFile path cfg
        , down = removeFile path
        }
  where
    hashOf = Text.take 12 . Text.decodeUtf8 . Base64.URL.encode . SHA256.hash . encode

Without this, a re-declaration that only changes cfg is indistinguishable from one that changes nothing at all — the node stays Converged, and you’re relying entirely on check (if you wrote one) running on its own timer to notice, which won’t happen at all under a piped script (§2’s last bullet) and may take a while even interactively.

9. Addressing nodes precisely: `--select`/`--exclude`

Every command that takes --select PATTERN/--exclude PATTERN (status, history, query, converge, force/recheck/pause/resume) resolves it as a /-separated glob against each node’s declared tree position — * matches exactly one segment, ** matches any depth including zero. Patterns may repeat (union within each of --select/--exclude); omitting --select entirely means everything. A few examples:

--select '/root/web/**'      everything under the web subtree
--select '/**' --exclude '/root/db/**'   everything except the db subtree

A path is built from op kinds (shorthand, e.g. directory, file-contents), not from any identifying value a recipe passed in — two nodes at the same tree position with the same shorthand (say, two files a recipe declares in a loop) get the exact same path, and a path pattern necessarily selects both together. status’s path line is where you’d notice this: two nodes printing the identical path is the tell. When that happens, address one of them directly instead with a #-prefixed pattern, matching by Ref rather than by path — the same (short or full) hash text status/run tree/run dag/query show already print next to it:

--select '#AbCd1234'    matches by a Ref fragment (short or full)

If your binary registers a Rewrite (e.g. Debian.Package.batchPackages, which collapses every declared deb node into one apt-get batch — see howto-ops.md/CLAUDE.md’s Op/Rewrite.hs section if this is new to you), the rewrite-introduced node (the batch itself) was never declared and so has no tree position — and no status line — of its own either; a #-match against it expands to every declared node it stands in for, but only for run up/run down/query show/query plan, which see the computed, post-rewrite graph. serve’s own status/query/converge --select (this section, otherwise) resolve against the declared graph and so cannot name a batch this way at all — restrict by the declared nodes that feed it instead.

10. Bounding concurrency

run serve --max-concurrency N caps how many nodes are inside their own check/up/down at once, across one convergence pass. Useful on a machine where unbounded width itself is the problem (CPU/IO contention, an outbound connection limit) rather than two specific nodes fighting over one resource — that case is still an edge (a dependency) or a collection’s job, not this flag’s. Omit it for the old, unbounded behavior (the default).

11. Machine-readable reports: `--json`

Everything above prints text: serve:-prefixed lines for the loop itself, Shown UpDown.Reports for what each node did. run serve --json (and run up --json/run down --json, the same flag) replaces all of that with one JSON object per line on stdout, flushed as each report happens, so my-salmon run up --json | jq streams and a script can watch a serve for converge-stop without parsing prose. The text output is unchanged when the flag is absent.

Every object has a kind (the report’s constructor, kebab-cased: declared, converge-start, done, failed, wedged, …), a stream (serve for the loop’s own reports, updown for what a node did, follow for the fetcher’s under --follow (§12); the tending loop’s reports arrive nested inside serve’s tended), a ref whenever the report is about one node ({"short": ..., "full": ...}, the same short tag status/query show print after #, so it pastes back in as a selector), and the node’s shorthand/help/notes under node. Report text is public: notes, failure messages and a status’s output ring go out verbatim, so keep secrets out of them (see the filecontents failure text for the convention). Salmon.Reporter.Tagged is the encoding, and Test/ReportJsonSpec.hs holds a golden object per constructor. Sequence numbers exist only on --http’s /events (§14), where the same objects go out with a seq added.

Two things the flag does not cover. A node’s own subprocess output — the Binary.Reports a node’s builder was handed a reportPrint for — is not one of the four streams and still prints as text, so a binary whose nodes were built with reportPrint (all of salmon-apps today) interleaves those lines with the JSON ones; a consumer should skip lines that are not JSON. And run tree/run dag/query are renderings of their own, not reports, and are untouched.

12. Pull mode: `--follow`

run serve --follow REGISTRY --label L [--label L]... [--follow-base S] ... makes the loop fetch its own declarations instead of only waiting to be typed at. A registry is anything that answers “the latest document for this label”; the simplest is a directory with one JSON document per label at DIR/<label>.json (the others are below), each the desired set of seeds for that label — not a log of commands:

{
  "salmon": 1,
  "id": "web-api@2026-09-23T10:41:07Z",
  "seeds": [
    { "seed": ["--dir", "/tmp/play", "--name", "web", "--file", "index.html"] },
    { "directive": { "...": "a directive JSON, as `up-directive` takes" } }
  ]
}

salmon is the format version (only 1), id is whatever the publisher calls this revision, and anything else at the top level is ignored. A host following several labels wants the union of their documents.

What happens on a change: the fetcher diffs the document against the one it last applied for that label and injects one batch — up for each seed newly present, down for each seed no longer present and not carried by any other followed label either — which the loop runs with autoconverge held off, restores, and converges once. Seeds you typed interactively are never in that diff (unless you typed the exact seed a document then drops: the ledger identifies a seed by its directive, not by who declared it).

What happens when nothing changed: nothing. The registry’s mtime and size say whether to read the file at all, the sha256 of the bytes says whether anything changed, and an unchanged round is invisible to the loop. That rule is load-bearing: every line reaching the loop stands the tending machines down (§1), so a fetcher that injected on every poll would keep the supervisor from ever reaching a steady state. Poll as often as you like.

history tells the fetcher’s declarations from yours:

serve: seeds:
  #0 up       [active] --dir /tmp/play --name web --file index.html [fetched /srv/reg label=web-api id=web-api@2026-09-23T10:41:07Z sha256=32ea59311d97]
  #1 up       [active] --dir /tmp/play --name api --file openapi.json [fetched /srv/reg label=web-api id=web-api@2026-09-23T10:41:07Z sha256=32ea59311d97]
  #2 up       [active] --dir /tmp/play --name scratch --file notes

(#2 was typed; a line run from load <file> says [loaded <file>].)

The first fetch runs before standard input is read, so the first convergence is deterministic — what the registry said at startup — and later changes arrive live, handled like any typed command. A document that fails to parse, a seed the binary cannot parse, or a seed whose config step throws, is reported and skipped; the loop keeps serving and the last good document stays in force.

When rounds run, and when a change is applied

Two schedules, pointing in opposite directions, both on the --follow-* flags (seconds unless said otherwise):

flag	default	what it is
`--follow-base`	30	seconds between rounds while they succeed (`--follow-interval` is the older name for the same thing)
`--follow-factor`	2	how much slower each consecutive failed round makes the next one
`--follow-cap`	600	the longest a failing registry is left alone
`--follow-jitter`	0.2	every delay is scaled by a draw from `[1-j, 1+j]`, so a fleet does not poll in step
`--follow-debounce`	5	how long the registry must be quiet after a change before the change is applied; `0` applies at the round that saw it
`--follow-max-wait`	60	the longest a change waits while the registry keeps changing
`--follow-cache`	none	a directory to keep each label's last applied document in, replayed at startup if the registry cannot be reached (below)
`--follow-refuse-older`	off	refuse a document whose `published` is older than the one already applied for its label (below)
`--follow-timeout`	30	the longest one HTTP fetch may take (the `http(s)://`, `dns:` and bucket registries)
`--follow-workdir`	see below	where a `git+` registry is checked out
`--follow-bucket-endpoint`	none	an S3-compatible endpoint for an `s3://` registry, path-style

Toward the registry: a round that succeeds — changed or not — schedules the next one one base away; a round that fails (the registry threw, or the bytes do not parse; a label with no document is not a failure, the registry answered) climbs a ladder, min(cap, base · factor^(n-1)) after n failures in a row, and the first success steps off it. follow: 3 failed round(s) in a row; next in 120s is what that looks like.

Toward the loop: a changed document is not applied at once. It is set aside (follow: web id=web@2 ...: changed, waiting for the registry to go quiet) and applied once no round has seen a further change for debounce, or max_wait after the first pending one, whichever comes first — and what is applied is the diff from the document the loop last heard about to the latest one, so a publisher writing three times in a row is one batch and one pass, and a half-published state is never applied. Several labels changing inside one window are one batch too. The startup round is the exception and applies at once: nothing to coalesce yet.

fetch cuts both short: a round now, the ladder forgotten, and whatever is pending afterwards applied without waiting out the window — for the operator who just published and does not want to wait. Without --follow it only says nothing is being followed.

Across a restart: `--follow-cache`

The world is in memory. Without more, a host restarted while its registry is unreachable comes up empty, tears nothing down, and looks converged — worse than no puller at all. --follow-cache DIR closes that: after every batch the fetcher writes each label’s just-applied document to DIR/<label>.applied.json (bytes, sha256 and id; written to a temp file and renamed, so a crash mid-write leaves the previous entry), and at startup a label whose fetch fails — the registry directory is missing, or the file does not parse — is replayed from there:

follow: fetching web failed: user error (registry directory does not exist: /srv/reg)
follow: web: registry unreachable; replaying the cached document id=web@1 sha256=8a8e1c180390
follow: web id=web@1 sha256=8a8e1c180390: 1 seed(s) up, 0 down
serve: epoch #0 up (4 nodes, 1 active seed(s))

A replayed document is treated exactly as a fetched one from then on — same diff, same batch, same [fetched ...] in history — and its digest is what the registry’s answer is later compared against, so a registry that comes back with the same bytes injects nothing (the starvation rule holds across restarts) and one that comes back with a different document is diffed against the replayed one, not applied from scratch. A label the registry answers “no document” for is not replayed: the registry answered. A cache entry that cannot be read is reported once and ignored, one that cannot be written is reported and the batch goes in regardless; the cache never takes the loop down. Without the flag nothing is cached, and a restart against an unreachable registry declares nothing, as before.

Which mode is this?

status now starts with which guarantees apply to the world:

serve: mode: replay
serve: nodes:
  ...

interactive — nothing is followed; every declaration was typed, loaded or sent by a client.
following — a fetcher is running and the world is what the registry last said.
replay — the registry could not be reached at startup and at least one label’s world is its cached document: the last thing this host knew, not necessarily what the registry says now.

replay turns into following at the first round in which every label answers, changed or not. It is only ever entered at startup: after a successful round the world already is the registry’s last word, a round failing later changes nothing about it (the last good document stays in force), and follow: N failed round(s) in a row is what says the registry is gone. Under --json the status object carries "mode"; the HTTP surface’s /status is the same object, and /dag’s envelope carries the same field.

Refusing to move backwards: `--follow-refuse-older`

A document may carry a published timestamp (RFC 3339) at its top level. Nothing reads it unless --follow-refuse-older is given, under which a fetched document published before the one already applied (or pending) for its label is reported and left alone:

follow: web id=web@0: published before the document already applied; refused (--follow-refuse-older)

That is what a registry serving from a lagging replica would otherwise do to a host. Off by default; a document without published, on either side, is never refused. A published that does not parse is a malformed document, not an ignored annotation.

The registries: what `--follow` can name

The backend is chosen by the shape of the address, and each one owns the rule that turns a label into an address and the cheap “has it moved?” test that keeps an unchanged round from reading anything:

`--follow`	the document for `<label>`	unchanged when	notes
`/srv/reg`	`/srv/reg/<label>.json`	mtime and size match	the directory missing is a failed round, not "no document"
`git+URL[#BRANCH[:SUBDIR]]`	`SUBDIR/<label>.json` at `origin/BRANCH` (the remote's default branch without `BRANCH`)	the branch points at the same commit	cloned once into `--follow-workdir` (default `checkout` under `--follow-cache`, else a temp directory named by the repository), then `git fetch` + `git reset --hard` every round; the subdirectory comes after the branch because URLs have colons of their own (`git+ssh://h:22/r#main:hosts`, `git+https://h/r#:hosts`); credential prompts are off, so a private repository fails rather than hangs
`http://…` / `https://…`	`<base>/<label>.json`, or the URL with `{label}` replaced (`https://h/seed/latest/{label}`)	`304` to `If-None-Match` (`ETag`) or `If-Modified-Since` (`Last-Modified`)	`404` is "no document"; `5xx`, `403`, a refused connection or `--follow-timeout` running out is a failed round
`dns:ZONE`	a `TXT` record at `<label>.ZONE` reading `v=salmon1 url=<https url> sha256=<hex>`, then that URL	the record's `sha256` is the one last seen — one lookup, no HTTP at all	a body that does not hash to what the record announces is refused with that reason (a failed round, never applied); no record is "no document"; the lookup is `dig +short`, so `dig` must be installed
`s3://BUCKET/PREFIX` / `gs://BUCKET/PREFIX`	`https://BUCKET.s3.amazonaws.com/PREFIX/<label>.json`, `https://storage.googleapis.com/BUCKET/PREFIX/<label>.json`, or `ENDPOINT/BUCKET/PREFIX/<label>.json` under `--follow-bucket-endpoint`	as HTTP	the HTTP backend under a template: public or presigned objects only — no SDK, no credentials, and a private bucket's `403` is a failed round that says so

Three things hold for every backend. The stamp above decides whether to read, the sha256 of the bytes decides whether anything changed, and only a changed document reaches the loop — so a git commit --allow-empty, a re-uploaded identical object or a rewritten identical file injects nothing. history names the registry as you gave it ([fetched git+https://h/r#main:hosts label=web ...]). And a fetch that throws leaves the last good document in force and climbs the ladder, whatever threw.

The DNS shape is the cheap one for a fleet: a host’s round is one UDP lookup answered from the resolver’s cache until the record’s TTL runs out, and the controller publishes by writing a record — which salmon can already do as a node (SreBox.MicroDNS, SreBox.DNSRegistration). Publish the document first and the record second, since a record announcing a digest the store does not yet serve is refused until it does.

Before anything is applied: the verifier

Every document — from any registry, and a cached one on replay, since a cache file is as writable as a registry file — goes through Follow.followVerify on its raw bytes before it is parsed. A refusal is reported and is a failed round:

follow: refusing the document for web (sha256=1f0d2c9a7b3e):
unsigned document: a signing key is configured (--follow-key) and this document carries no signed envelope

The bytes are neither injected nor cached; the last good document stays in force. Without --follow-key the verifier accepts everything (Follow.noVerifier): unsigned mode is the default, and a document is taken as the registry serves it.

Signed documents: `--follow-key`

With --follow-key FILE (repeatable) the host requires every document — fetched from any registry, and a cached one on replay — to be a signed envelope carrying a signature by one of those keys; any one suffices. The round trip needs no tool but salmon-fleet:

$ salmon-fleet keygen --out fleet.key
salmon-fleet: wrote fleet.key (private, 0600) and fleet.key.pub (public); key id 51d3c2152fb2…
$ cat fleet.key.pub
{"crv":"Ed25519","kty":"OKP","x":"Esc7UxOvyQCXne0_TqOseUq2e5CHmdFhjsr4wglADHk"}
$ salmon-fleet sign --key fleet.key < web.json > /srv/reg/web.json
$ cat /srv/reg/web.json
{"document":{"id":"web@1","salmon":1,"seeds":[{"seed":["--dir","/tmp/play","--name","web","--file","index.html"]}]},
 "salmon-signed":1,
 "signatures":[{"alg":"EdDSA","key":"51d3c2152fb2…","sig":"+rS1PN29nf1o…"}]}
$ my-salmon run serve --follow /srv/reg --label web --follow-key fleet.key.pub
follow: /srv/reg for web every 30s (...)
follow: web id=web@1 sha256=b2014a4513e7: 1 seed(s) up, 0 down

The document rides inside the envelope as you wrote it (annotations and all); the signature is over its canonical bytes — aeson’s own encoding of the parsed value, keys sorted — so a registry or a proxy that re-serialises the envelope (other key order, other whitespace) leaves the signature valid, and only a change of content breaks it. What the loop parses is the document inside; the sha256 in reports, history and the cache is that of the bytes as fetched, the envelope’s. Keys are JWK files (the format Keys.jwkKey already writes), Ed25519, and a key’s id is its RFC 7638 thumbprint.

Hand-edit the file inside its envelope, and the host says why:

follow: refusing the document for web (sha256=6fd186d7f55f):
no signature verifies against any of the 1 configured key(s): signature by 51d3c2152fb2 does not verify: the document was altered after signing, or signed by another key

Serve a plain document to a host started with a key:

follow: refusing the document for web (sha256=4ed68ba04fd8):
unsigned document: a signing key is configured (--follow-key) and this document carries no signed envelope

An envelope that does not parse, one with no signatures, or one signed by a key the host does not hold are refused the same way, each naming its cause; and a --follow-key file that does not load is an exit 1 with the path before any loop starts — a host that then refused everything, or accepted everything, would be worse than none. To rotate a key, run hosts with both the old and the new --follow-key while documents are re-signed, then drop the old one; nothing more than that exists (no revocation, no key in the document).

Not there yet (specs/pull-mode.md): other sinks (a bucket object, an HTTP POST), and authenticated bucket access.

Status flows back: `--status-sink`

A host in pull mode converges with nobody watching. --status-sink PATH makes it write down what came of it — a JSON document, to a temp file renamed over PATH so a reader never sees half of one — after every convergence pass, after every follow injection, and every --status-sink-interval seconds (default 10) otherwise:

{
  "salmon-status": 1,
  "host": "web-3",
  "written": "2026-09-24T10:41:07.12Z",
  "mode": "following",
  "labels": [{"label": "web", "id": "web@42", "sha256": "…", "applied": "2026-09-24T10:40:58.51Z"}],
  "status": { "kind": "status", "mode": "following", "nodes": [ ... ] },
  "last": {
    "converge": { "stream": "serve", "kind": "converge-stop", "ok": true, "remaining": 0 },
    "follow":   { "stream": "follow", "kind": "injected", "label": "web", "document": "web@42", ... }
  }
}

host is the machine’s node name (uname -n) unless --status-sink-host NAME says otherwise. Give one when two loops on one machine each write a document (the fold shows two rows naming one host otherwise, and does not pick), or when the node name means nothing to whoever reads the directory (a container’s generated hostname).

status is the very object status --json prints (and /status answers); labels is the document each followed label last applied; last holds the most recent converge-stop and the most recent follow report, as --json prints them. The fetcher’s own reports (injected, backoff, replayed, …) are a --json stream in their own right now — "stream": "follow" — which is what lets the sink carry them.

The sink never touches the loop: it is a reporter watching the loop’s stream for its triggers and a reader of the world through the same accessor /status uses, so a write stands no tending machine down. A path that cannot be written is reported once (serve: status sink PATH could not be written:), and again only after a write has succeeded in between; the loop keeps serving. A host gone quiet therefore shows as a document whose written is old — not as one that says all is well.

Fleet status is a fold over a directory of these, computed by whoever reads it. salmon-fleet status DIR is that reader, one line per host:

$ salmon-fleet status /srv/status
host      mode       labels                      converged  errored  age   flags
web-1     following  web=web@42@32ea59311d97     4/4        0        3s
web-2     following  web=web@42@32ea59311d97     3/4        1        5s
db-1      replay     db=db@7@d00ef24caea4        3/3        0        94s   stale

--label L keeps only hosts whose applied documents include L, --stale SECONDS (default 60) sets when a host is flagged, --json emits the rows as one array. It only reads; a stale host is a visible fact, not a decision, and nothing here decides a host is dead. Two documents naming one host (two loops on one machine, as in the tests) are two rows.

13. A second way in: `--listen`

run serve --listen PATH binds a unix socket at PATH and accepts the same line protocol on it — up, status, force --select ..., quit, every command §3 typed on stdin — from any number of clients at once, while stdin keeps working alongside. Milestone 2 of specs/generic-server.md; Salmon.Actions.Serve.Socket is the implementation.

my-salmon run serve --listen /run/my-salmon.sock < /dev/null &
printf 'status\n' | socat - UNIX-CONNECT:/run/my-salmon.sock
ssh -L /tmp/remote.sock:/run/my-salmon.sock host   # then the same, locally

Four things to know:

Each client reads exactly the reports for its own lines, as JSON lines in the §11 encoding, whatever the loop’s own stdout is set to (text by default, JSON under --json; it sees everything either way). What another client typed, and what the tending loop says between commands, never reaches a client — the loop stamps every report with who typed the command it belongs to, and the socket only echoes the ones stamped for it. There is no per-client text mode.
A client hanging up is not quit. It is reported on the loop’s stdout (serve: PATH#N hung up) once every line that client typed has been handled, and the connection is closed then — so printf 'status\n' | socat ... gets its answer even though it half-closes immediately. quit from a client ends the loop exactly as it does from stdin.
Under --listen, stdin’s end of input does not end the loop either. With a socket to talk to, the process is expected to outlive whatever started it (< /dev/null &, a unit file), so stdin is one more source whose hang-up is reported and read past; only quit — typed anywhere — or a signal ends it. The same holds under --http/--http-tcp (§14). Without any of them, stdin closing ends the loop as it always has.
The socket is owner-only (mode 0600) and the path is checked before it is taken. A stale socket file (its serve died without removing it) is replaced; one something still answers on is refused (AlreadyListening), as is a path holding something that is not a socket. Permissions are the whole access story: there is no authentication, and no TCP — see the spec’s security section for why a salmon server must never listen on a network without both (--http-tcp, §14, is the one listener that does, and it has both).

The commands are still one inbox: a line from a client stands the tending machines down before it runs, same as a line from stdin, and two clients’ lines interleave at line granularity in arrival order.

14. HTTP on a socket: `--http`

run serve --http PATH binds a second unix socket and serves HTTP on it — reads of the live world as JSON, and the same command language as POST. Milestone 3 of specs/generic-server.md; Salmon.Actions.Serve.Http is the implementation. It is its own path rather than HTTP detected on --listen’s socket, so use both flags if you want both; the socket file has the same owner-only mode and the same live/stale checks as §13’s.

my-salmon run serve --http /run/my-salmon.http < /dev/null &
C='curl -s --unix-socket /run/my-salmon.http'

$C http://x/dag | jq '.nodes[] | {shorthand, ref: .ref.short, direction, convergence,
                                  deps: [.dependencies[].short]}'
$C http://x/status | jq .        # the object `status` prints under --json
$C http://x/history | jq .       # likewise `history`, plus an `elided` count
$C http://x/help/seed | jq -r .seed   # this binary's own `config --help`

$C -X POST -d 'up --name web --file index.html' http://x/command      # sync
$C -X POST -d 'up --name api' 'http://x/command?async'                # {"seq": n}
$C -X POST -H 'content-type: application/json' -d '{"line": "status"}' http://x/command

curl -sN --unix-socket /run/my-salmon.http http://x/events            # live, forever
curl -sN --unix-socket /run/my-salmon.http 'http://x/events?since=42' # replay after 42, then live
curl -sN --unix-socket /run/my-salmon.http 'http://x/events?stream=upkeep,updown&origin=stdin'

What the -N client sees while another posts an up (the fixture binary, salmon-ops-serve-fixture run serve --json --http /tmp/x.http --events-ring 64, abridged):

id: 3
data: {"kind":"enqueued","line":"up --dir /tmp/play --name web --file index.html","origin":{"kind":"other","name":"/tmp/x.http#0"},"seq":3,"stream":"server"}

id: 4
data: {"active_seeds":1,"direction":"up","epoch":0,"kind":"declared","nodes":3,"origin":{...},"seq":4,"stream":"serve"}

id: 6
data: {"kind":"eval","node":{"shorthand":"directory",...},"origin":{...},"ref":{...},"seq":6,"stream":"updown"}
...
id: 12
data: {"kind":"converge-stop","ok":true,"origin":{...},"remaining":0,"seq":12,"stream":"serve"}

id: 13
data: {"from":"/tmp/x.http#0","kind":"hung-up","seq":13,"stream":"serve"}

id: 14
data: {"down":0,"kind":"supervising","seq":14,"stream":"upkeep","up":3}

id: 16
data: {"delay_us":2000000,"kind":"reapplying","node":{"shorthand":"directory",...},"ref":{...},"seq":16,"stream":"upkeep"}

The ?async answer was {"seq":3}: everything numbered above 3 with that origin is that command; from 14 on, with no origin, it is the machines tending between commands — which a sync POST never shows, since tending happens exactly when no command is being handled.

What to know:

Reads never touch the inbox. /dag, /status, /history and /help/seed read the loop’s own World directly — they do not stand the tending machines down, do not wait behind a command, and answer while a node’s up is still running. The price is that a read is at most one command old: each node’s status is the snapshot the last command took (§11’s status field, null for a node never tended). Motion between commands is on /events, below, and /status and /dag carry a seq — the last event number at the moment of the read — so that /events?since=<that seq> starts exactly where the snapshot left off.
/dag is the graph a pass walks, not the declared tree: one object per Ref, with dependencies and dependants as ref lists both ways, the node’s shorthand/help/notes/dynamics (the fields §8’s Stale detection compares), and its direction/convergence/status/ paths as status lists them. It is populated the moment something is declared — under autoconverge off every node reads pending with its edges already in place — and a retired seed’s nodes stay in it with direction: "down" until their teardown is done. A node whose current representative won a collision — two seeds describing one Ref differently, or one graph reaching it from two differently-described nodes — carries a conflict with the kept and replaced representatives (the same four fields) for as long as some live declaration still wants the losing version, so a client that did not catch the pass’s conflicting event can still show the pair. A batch a Rewrite would introduce is not shown; the nodes it would stand in for are. The envelope’s top-level mode is §12’s (interactive, following, replay), read at the moment of the request — the same value /status opens with, so a client showing nodes as tended knows whether they are.
POST /command is one line of §3’s language, text/plain or {"line": "..."}, and it is handled like any other line: it stands the machines down first and takes its turn in the inbox. Synchronous by default, the response is a JSON array of exactly the reports that line produced (§11’s objects), returned when the loop has finished with it — what a script or a CI step wants. ?async returns 202 {"seq": n} the moment the line is queued; n is the number of the enqueued event on /events, and that command’s reports are the events above n carrying its origin. quit works from here too and answers [].
/events is one stream, numbered, replayable. text/event-stream: each event is id: <seq> and one data: line holding the §11 object with seq added, plus origin (the object history entries use) when the report was produced for a command. Three streams and the server’s own: serve, updown, upkeep (the tending machines’ reports, which reach a client here and nowhere else) and server (enqueued, and gap). One counter numbers everything — enqueues and reports, from the loop and from machine threads — so one cursor is enough. ?since=N replays what the ring still holds after N, then continues live; the ring keeps the last --events-ring N events (default 2048), and a client further behind than that is sent {"kind":"gap","from":<oldest>,"stream":"server"} first (no id), never a silent skip. ?stream=a,b and ?origin=NAME filter on the server. An idle stream carries a comment line every 15 seconds so proxies and read timeouts keep it open; hanging up is all a client has to do to unsubscribe. curl -N or any EventSource reads it.
salmon-tui PATH is a terminal over all of the above (milestone 6; salmon-tui https://HOST:PORT --token-file FILE [--cacert FILE] for a --http-tcp listener, see “Reaching it over the network”; salmon-apps, over Salmon.Client.Http and the pure Salmon.Client.Model). It reads /dag once, follows /events from that snapshot’s seq, and draws a header (socket, mode, seq, converged/errored/total, the current pass, supervising/not supervising, stream=live|reconnecting), the node table in /dag’s order — ref, shorthand, direction, state, last check, last event — and a footer. j/k (or the arrows) move, g/G jump to the first/last row, enter expands the selected node (help, notes, paths, edges, check reason, error, the output ring of the last snapshot), r re-reads /dag, q quits leaving the server as it was, and : opens a command line: the line is sent as POST /command?async and the footer echoes the seq it was queued at. That line is the only thing on the screen that stands the tending machines down — every read bypasses the loop, so the TUI can stay open on a box without perturbing it, and the footer says so. It holds no state the server does not: a declared or a gap makes it re-read /dag (rebased onto what it was showing), and a lost stream is retried with ?since= the last number it saw. Works unchanged over ssh -L /tmp/remote.http:/run/my-salmon.http host — it is a client of the socket, not a mode of serve. What the fixture looks like right after :up --dir /tmp/play --name web --file index.html --file style.css:
```
/tmp/x.http mode=interactive seq=17 converged=4 errored=0 total=4 converged  stream=live
  ref        shorthand              dir  state     check        last event
> MTE5MDg2   file-contents          up   converged -            done &#35;8
  ODQwMTE0   directory              up   converged -            reapplying &#35;16
  bjU3MjUz   serve-fixture-bundle   up   converged -            parked &#35;17
  bjczNjY4   file-contents          up   converged -            done &#35;10
&#35;17 upkeep parked bjU3MjUz serve-fixture-bundle
j/k move  enter expand  : command (async; stands the machines down)  r re-read /dag  q quit
```
Permissions are the whole access story on the socket. No token is asked for on it; notes, help and report text are as public as the logs they already go to. Do not put this socket where an untrusted user can open it. Reaching the same server over a network is the next paragraph, and it is TLS with a token or nothing.

Reaching it over the network: `--http-tcp`

Milestone 8 of specs/generic-server.md. The same HTTP — every route above, /events included — can also listen on a TCP address, and the only way to spell that is with all three of a certificate, its key and a token file:

my-salmon run serve --http /run/my-salmon.http \
    --http-tcp 0.0.0.0:8443 --tls-cert /etc/my-salmon/server.pem \
    --tls-key /etc/my-salmon/server.key --token-file /etc/my-salmon/token < /dev/null &
# stderr, once: serve: exposing HTTP on 0.0.0.0:8443 with TLS, token from /etc/my-salmon/token

T="Authorization: Bearer $(cat /etc/my-salmon/token)"
curl -s --cacert ca.pem -H "$T" https://host:8443/status | jq .
curl -s --cacert ca.pem -H "$T" -X POST -d 'up --name web --file index.html' https://host:8443/command
curl -sN --cacert ca.pem -H "$T" 'https://host:8443/events?since=0'

What the fixture binary does with each way of getting it wrong (the refusals are exit 1 before anything is bound or read; the option check itself is a pure function, CommandLine.validateTcpOptions):

$ salmon-ops-serve-fixture run serve --http-tcp 127.0.0.1:8443
--http-tcp needs --tls-cert, --tls-key, --token-file: a salmon server never listens on a network without TLS and a token
$ salmon-ops-serve-fixture run serve --http-tcp 127.0.0.1:8443 --tls-cert tls/server.pem
--http-tcp needs --tls-key, --token-file: a salmon server never listens on a network without TLS and a token
$ salmon-ops-serve-fixture run serve --token-file token
--token-file need --http-tcp HOST:PORT to apply to; there is no network listener without it
$ ls -l token
-rw-r--r-- token
$ salmon-ops-serve-fixture run serve --http-tcp 127.0.0.1:8443 --tls-cert tls/server.pem --tls-key tls/server.key --token-file token
--token-file token is readable by others; a token anyone on the box can read is not one (chmod 600 it)
$ salmon-ops-serve-fixture run serve --http-tcp :8443 --tls-cert tls/server.pem --tls-key tls/server.key --token-file token
--http-tcp: no host in ":8443"; spell the address, 0.0.0.0 included

and, once it is up (chmod 600 token first), what a client sees:

$ curl -s --cacert tls/server.pem https://localhost:8443/status
{"error":"a bearer token is required"}                       # 401, WWW-Authenticate: Bearer
$ curl -s --cacert tls/server.pem -H "Authorization: Bearer $(cat token)" https://localhost:8443/status
{"kind":"status","mode":"interactive","nodes":[],"seq":2,"stream":"serve"}
$ curl -s --cacert tls/server.pem -H "Authorization: Bearer wrong" https://localhost:8443/dag
{"error":"a bearer token is required"}
$ curl -si http://localhost:8443/status                      # plain HTTP on the TLS port
HTTP/1.1 426 Upgrade Required
$ curl -s --unix-socket /run/my-salmon.http http://x/history | jq -c '.seeds[] | .origin'
{"kind":"other","name":"127.0.0.1:51510#0"}                  # the unix socket: no token, and who typed the line

Eight things to know:

There is no plaintext option, behind any flag. Http.Bind has a unix constructor and a TLS constructor and nothing else; --http-tcp without all three files is refused with every missing one named, and the three files without --http-tcp are refused too, since silently unused is how a listener ends up open by accident. HOST is spelled, always: :8443 is refused, 0.0.0.0:8443 is how listening on every address is written, [::1]:8443 for IPv6. A salmon server is root on the box, one up away — the spec’s security section is binding on this.
The token is on every route of the TCP listener, Authorization: Bearer <token>, compared in constant time against the file’s content with surrounding whitespace removed (so echo secret > token is fine). Reads and /events are not exempt: a node’s output ring is as sensitive as a command. 401 with {"error": ...} otherwise, and a refused command is never queued. The token file must not be readable by others (chmod 600), and must not be empty. Checking it queues nothing — a read is still a read.
A browser signs in at /auth, because nothing lets a page put a header on a navigation or an EventSource. GET / with no credential is a 303 to /auth, a form asking for the token; posting the right one answers 303 back to / with a __Host-salmon-session cookie (HttpOnly, Secure, SameSite=Strict), which the TCP listener then accepts wherever it accepts the bearer header. The cookie is not the token — 32 random bytes minted per sign-in and known only to the running process — so the token is never stored in a browser, and restarting the server signs every browser out. A wrong token is a 401 and the form again. On the unix socket /auth has nothing to do and redirects to /.
Signing out is the page’s sign out button, a plain form posting to /auth/logout: the session is revoked, the cookie expired, and the browser sent back to /auth. Every tab of that browser shared the session, so each is signed out with it — an /events stream the session opened is cut at once, and a page that then gets a 401 goes to /auth on its own. A GET of /auth/logout is refused, so a link or a prefetch cannot sign anybody out. The button shows only when GET /auth/session says the page holds a session, so never on the unix socket.
A session also ends on its own. --session-lifetime S (default 43200, 12h) counts from sign-in and ends a session however busy it is, cutting its open event stream on time; --session-idle S (default 3600, 1h) ends one nothing has used, where an open event stream is use, so a page left open to watch does not idle out. 0 turns either off. The cookie carries the lifetime as Max-Age, a page whose session ended lands on /auth?ended (“Your session ended; sign in again”), and ended sessions are dropped at every sign-in — so the server holds at most the sign-ins of one lifetime, however long it runs.
The unix socket is unchanged, token-free, and the same server: one event ring, one seq counter, one inbox, whichever listener a request came in on. What differs is the origin a command is typed under: PATH#n on the socket, the client’s own ADDR:PORT#n over TCP, so history says who typed a line from the network.
Plain HTTP on the TLS port is refused by warp-tls with 426 Upgrade Required before any route is reached; a client with a wrong CA sees a failed handshake. Both are the listener working, and neither is traced on stderr — the startup line is deliberately the only thing written there.
Mint the certificate however you like; the tree can do it. Certificates.certificateAuthority writes a v3 self-signed certificate a client can pin with --cacert; a CA-issued one works the same. Note that Certificates.selfSign and caSign write X.509 v1 certificates (openssl x509 -req without extensions), which OpenSSL-based clients accept and crypton-based Haskell clients reject (LeafNotV3).

salmon-tui reaches it too, given the URL, the token file and — for a self-signed certificate — the certificate to pin:

salmon-tui https://host:8443 --token-file token --cacert server.pem

Without --cacert the system’s trust store decides, and a self-signed certificate fails the handshake before the token is sent; there is no flag that turns verification off. An http:// address is refused outright, and so is a token file others can read, as the server refuses its own. The web UI reaches it through /auth above. Mutual TLS and a read-only token are the spec’s own v2.

The web UI: `GET /`

The same socket serves a page at / (its script and stylesheet under /ui/, compiled into the binary, so there is nothing to install beside it) that draws the world as the graph it is and drives it. Milestone 7 of specs/generic-server.md: the static picture, the live one, the actions and the seed form. It is a client of the routes above and nothing more — it fetches /dag, lays the nodes out in layers with dependencies above dependants and an edge per dependencies entry, one box per node (short ref, shorthand, direction · convergence, the last event and check verdict), coloured by convergence and dashed for a node wanted down; then it subscribes to /events?since=<the snapshot's seq> and applies what arrives: eval/ done/failed pulse the box and move its colour, next-look updates the check verdict, converge-start/converge-stop and the counts go in the header. It keeps no state the server does not: a declared, a cleared, a converge-stop, a gap or a dropped stream all mean “fetch /dag again and resubscribe from its seq”, and the reload button is that by hand. Clicking a node opens a panel with its help, notes, dynamics, paths, dependencies and dependants (each a link), the last check and its reason, and the output ring. Below 700px wide the graph gives way to a list.

Everything the page does is one POST /command?async and then the event stream: the answer is the seq the line was queued at and the origin it was queued under (a toast shows both), and the events above that seq carrying that origin are what the command did — they outline the nodes it touched (the amber “touched” outline in the legend, until the loop’s hung-up for that origin says the line has been handled), and they go into the log under the command line. The page never uses the synchronous form: a sync up holds the request for the whole pass, and the page is the thing that would be waiting. Four places send a line:

The node panel has force, recheck, pause and resume for the selected node, sent as <verb> --select #<short ref> — the # selector from §9, so what the box prints is what the command names. A node does not know which seed declared it and /history does not say which nodes an epoch declared, so retiring “the seed behind this node” is the operator’s choice: the panel lists every live declaration under retire a seed, each with its down.
The header has the world’s commands: converge, supervise on|off, autoconverge on|off, fetch (which the loop answers “nothing is being followed” without --follow) and clear, which asks first since it retires every seed. quit is deliberately not there: the page is served by the process it would be stopping, and leaving the loop is the one thing that should take a terminal.
The seed form (the seeds button) shows /help/seed — this binary’s own config --help, and the loop’s command reference under it — a text field for the seed words, and up/only/down, sending <verb> <words> as typed. /history is listed under it, one row per declaration with its epoch, verb, words, origin and whether it is still active, a down on each active row, and the words clickable to put them back in the field. The list is fetched again on every declared and cleared.
The command line at the bottom (: focuses it, as in vi and less; Esc leaves it) sends any line of §3’s language as typed — status and help included, whose reports land in the log rather than on the page.

The page itself never handles a token. A browser cannot open a unix socket, so open the page on the TCP listener and sign in once at /auth (see “Reaching it over the network”); the session cookie the browser keeps then carries every fetch and the EventSource until sign out:

my-salmon run serve --http /run/my-salmon.http \
    --http-tcp 127.0.0.1:8443 --tls-cert server.pem --tls-key server.key \
    --token-file token < /dev/null &
xdg-open https://localhost:8443/        # redirected to /auth; paste the token

A self-signed certificate is a browser warning to click through (or add it to the browser’s trust store). Without a TCP listener, a forward of the unix socket still works — ssh -L 8080:/run/my-salmon.http host from another machine, or socat TCP-LISTEN:8080,bind=127.0.0.1,reuseaddr,fork UNIX-CONNECT:/run/my-salmon.http locally — but bind it to 127.0.0.1: the port inherits none of the socket’s file permissions, and whoever reaches it has the socket. The layout is a small longest-path layering with barycentre ordering written in salmon-ops/ui/ui.js itself — no bundler, no framework, no vendored library — so the three files are readable as they are served.

15. Gotchas

A piped script is never supervised. If you’re testing self-healing and piping a script in, you won’t see it — there’s no idle moment for the tending loop to occupy. Type interactively, or drive the loop from a fifo, to actually observe §3’s perturb-and-watch-it-heal behaviour.
Immaterial means parked, not polled. A node with no check looks exactly as “up” in status whether or not anything is actually watching it — the absence of supervision is deliberate and silent by design (the alternative, polling to ask a question with no useful answer, would cost something for nothing). If you want self-healing, write a check.
sameRepresentative’s blind spot is up/check/down. Two declarations that differ only inside those functions compare equal for Stale-detection purposes (§8) — put something comparable in notes if you need a content-only change to register through the pass itself rather than the tending loop.
There are two different querys. my-salmon query show|plan|... is a top-level, one-shot CLI command that inspects a directive on stdin before any run up/run serve even starts. serve’s own in-loop query [--select]... command inspects the live, already-converging World. They share selector syntax but operate on different things.
A node’s check is the only thing that can notice it going away, and therefore the only thing that can fire a RestForOne demotion. A config node with no check never notices its own file changing, and nothing standing on it is ever bounced, however that node is decorated otherwise.

16. Where to read more

resources/howto-ops.md — writing and testing the Ops this doc assumes.
CLAUDE.md’s “salmon-ops layer” section — the implementation-level summary of every module named above (Actions/Upkeep.hs, Actions/Serve.hs, Op/Supervision.hs, Op/Dag.hs, Op/Rewrite.hs), written for whoever is modifying salmon itself rather than using it.
specs/per-node-state-machines.md and specs/per-node-state-machines-remaining.md — the original design and its running status; read these for the why behind a given tradeoff, or to see what’s still explicitly left as “taste, not yet decided” (supStrategy living on the dependency rather than the dependant; the RestForOne cascade requiring opt-in at every hop).
salmon-ops/fixtures/ServeFixture.hs — a runnable, hands-on tour of everything in §3, §6 and §7, including the --daemon/--stale-check flags that demonstrate RestForOne and what a health check is and isn’t allowed to say about a process salmon just tore down.

A generic salmon server: `serve` behind an API, with clients that show the DAG

2026-09-25T14:15:36Z

Generated from specs/generic-server.md — the repository is the canonical source, and may be ahead of this page.

A generic salmon server: `serve` behind an API, with clients that show the DAG

Status: milestones 1 to 8 below are implemented (--json via Salmon.Reporter.Tagged; run serve --listen via Salmon.Actions.Serve.Socket; run serve --http via Salmon.Actions.Serve.Http with /dag, /status, /history, /help/seed, POST /command[?async]; GET /events via Salmon.Actions.Serve.Events and --events-ring; mode on status//dag; salmon-tui over Salmon.Client.Http/Salmon.Client.Model; the web UI under salmon-ops/ui/ served at GET /; and --http-tcp with --tls-cert, --tls-key, --token-file), each with its deviations recorded in place in the milestone list. Milestone 8’s closing “Not done: the clients” has since been done: salmon-tui https://HOST:PORT --token-file FILE [--cacert FILE], and a browser signs in at /auth for a session cookie (--session-lifetime, --session-idle, sign-out at /auth/logout). Still open: the web UI’s “Not yet” items under milestone 7 (a Conflicting pair side by side, batch and RemoteOp expansion, history as a timeline), client certificates and a read-only token. Kept as the design record. Companion to specs/pull-mode.md (which is about a host fetching its declarations); this one is about talking to a running serve — a web UI, a terminal UI, tooling — and about finally showing the DAG as what it is while it is being converged and tended.

Problem

run serve’s entire surface is one Handle in and a text Reporter out (salmon-ops/src/Salmon/Actions/Serve.hs, serveWith; the reader thread hGetLines into a TChan, loop reads it). Everything else follows from that:

One operator, one terminal. No second client can attach, no tool can drive a running supervisor, and there is no auth because there is nothing to authenticate to.
A remote serve cannot be addressed. Self.callSelf only ever runs a one-shot run up over ssh. ssh host bin run serve works, but the only thing that can then speak to it is that ssh session’s stdin.
The DAG is invisible while it runs. run tree/run dag print a static picture before anything happens (Help.printDagTree, Dot.printDagCograph); status prints a flat list after. The one thing salmon is best at — a graph with per-node state, failure containment and Conflicting/Blocked distinctions — is never rendered as a graph while a pass or the tending loop is acting on it.

The author of the line protocol is the first to lament it. This sketch is the replacement.

What already exists as a value

None of this needs a new engine; it needs a transport and a read model over values the loop already maintains.

World (Serve.hs): the ledger, the magma (one representative per Ref), and a NodeState per node — nodeShorthand, nodeHelp, nodeDirection, nodeConvergence (Pending/Stale/Converged/Errored/Blocked) and, since (R3), nodeStatus: the node’s own last CheckResult, when it last did anything observable, and its output ring.
Dag (Salmon.Op.Dag): dagNodes, dagDependencies, dagDependants, dagOrder, dagConflicts — already the collapsed, rewrite-applied structure run tree/run dag print, with membersOf for a rewrite-introduced node standing in for declared ones.
Three report streams, all plain sum types: Serve.Report (Started, BadCommand, Loading, … the loop’s own events), UpDown.Report (Skip/Eval/Done/Failed/Blocked/Conflicting/ Instructed/…) and Upkeep.Report (Acted, Upkeep/Downkeep state changes, NextLook, Wedged/Unwedged, demotions). Reporter is contravariant and already has encodeJSON; what is missing is ToJSON instances on these types (most carry an Act ext, which needs a serializable projection — shorthand, ref, help, notes — not the closures).
Selectors: parseSelection/resolveWorldSelectors resolve --select/--exclude path globs and #ref prefixes against the live world; status prints the paths a pattern can match.
RemoteOp (CommandLine.hs): the dynamic that lets run dag draw a remote call’s subgraph locally (injectRemoteSubgraphs). A live view can use the same thing to expand a remote node.
Mailbox instructions (Salmon.Op.Mailbox.Instruction: Force/Satisfy/Recheck/Pause/Resume) and the force/recheck/ pause/resume commands that queue them.

Design

The server is generic because the protocol never interprets a seed

Every salmon binary is a different program (ParseRecord seed, its own Configure, its own Track' directive), so a server bolted onto serve has to be binary-agnostic to be worth writing once. It is, as long as the protocol carries seeds as opaque word lists — exactly what the line parser already takes after up/only/down — and speaks about nodes only in terms of Ref, path, shorthand, help, notes and state. A client that speaks that drives any salmon binary; it never knows what pgpair primary=db1 means.

The one place this leaks: composing a seed. The server exposes each binary’s own --help text for config (it is optparse-generic’s, and already exists), and optionally a JSON schema derived from the ParseRecord instance, so a UI can offer a form instead of a free-text line. That is the only non-generic surface and it is bounded.

Three surfaces, one inbox

The loop stays as it is: a single consumer reading lines off one TChan, calling stopTending before each. The server is another producer into that channel (the same generalization specs/pull-mode.md milestone 1 asks for), plus a reader of World and the report streams. Concretely:

Commands — POST /command with {"line": "up pgpair primary=db1"} or a structured form {"verb":"up","seed":[...]}; the server writes the line to the inbox. Both a sync and an async mode: sync (the default) blocks and returns the reports that command produced, since the loop already knows when a command’s reports end (step returns) — what curl and a CI step want; ?async returns immediately with the sequence number at which the command was enqueued, and the client reads its reports off /events from there — what a UI wants. The existing grammar is the API, unchanged. stdin keeps working alongside.
Reads — GET /dag returns the computed Dag as JSON: one object per Ref with ref, short, shorthand, help, notes, direction, convergence, check (last CheckResult), output (tail), paths (the declared positions a selector can match), members (for a rewrite-introduced node), remote (a nested Dag, from RemoteOp), and dependencies/dependants as ref lists. It is built by reading the magma — worldDag from worldMagma plus the ledger’s precedence, exactly as a convergence pass builds its own walkable structure — so it exists the moment anything has been declared, whether or not a pass has run yet (autoconverge off with declarations recorded is the ordinary case: every node Pending, edges in place). Plus GET /status, GET /history, GET /help/seed. Reads never touch the inbox and never stop tending: they read the IORef World and the tending snapshot.
Events — GET /events, server-sent events (one connection, text, proxies and curl understand it; WebSocket adds nothing here): one stream, every Serve.Report, UpDown.Report and Upkeep.Report as JSON, each tagged with its kind and the Ref it concerns and a monotonic sequence number so a client that reconnects can ask ?since=N. The server is one more Reporter — the codebase’s contravariant reporter type (Salmon.Reporter; the tracer word is taken by the ops themselves) — composed with reportBoth beside the text reporter the binary already has, contramapped from each report type into one tagged sum. Nothing in the loop learns that a server exists; a client filters. This is the stream a live DAG view animates from.

Transport, in order: a unix socket carrying the line protocol first — serveWith already takes any Handle, so this is nearly free, it makes a remote serve addressable through ssh -L, and it lets a second client attach today; then HTTP on that socket or a TCP port with the three surfaces above; TLS and auth when it listens on anything but localhost or a unix socket (see below).

Reads and the tending loop

stopTending before every command exists because a command is about to act. A read is not, so reads bypass the inbox and never stand the machines down — which is exactly the property the loop’s (R3) snapshot design was built for: nodeStatus is snapshotted by stopTending and re-snapshotted on the next, so a read sees a snapshot that is at most one command old. For a live view that is not enough; the event stream is what carries the between-commands changes (Upkeep.Reports are emitted by the running machines, not by the loop). So: GET /dag for the picture, /events for the motion, and a client rebuilds the current state as dag ⊕ events since the dag's sequence number. The server records the sequence number at which each snapshot was taken and returns it with the snapshot, so the client knows where to resume.

Serialization: what an `Act ext` becomes

Act ext holds closures (up/check/down/managed) and Dynamics. The wire projection is exactly the fields Dag.sameRepresentative compares — shorthand, help, notes, the rendering of dynamics (with Supervision by value, as Dag.showDynamic already does) — plus the Ref. That is not a coincidence: it is the set of things that are comparable, and a UI that shows exactly those is showing what serve itself can see. RemoteOp is the one dynamic that gets special treatment (expanded to a nested Dag), same as run dag does today.

Modes, so a client knows which guarantees apply

status and /dag report the loop’s mode: replay (a piped script or startup replay — every line queued, never supervised, deterministic), interactive (tending between commands), and, once pull mode exists, following (documents arriving on the scheduler). A web client showing a node as “supervised” while the loop is replaying a script would be lying.

Clients

Web UI. Renders /dag as a graph (a layered DAG layout — dagre or ELK — not force-directed; dependency direction is the information), colours nodes by convergence, overlays check, animates Eval/Done/Failed/Blocked as they arrive on /events, shows a Conflicting pair side by side, collapses a rewrite-introduced batch to its members on click, expands a RemoteOp node into its subgraph, and turns a click on a node into force/recheck/pause/resume with the #ref selector the server already accepts. history is a timeline. A config form built from /help/seed composes an up. Dot output stays as the export.
Terminal UI. Same API, same read model, for the box with no browser: a tree view (the run tree shape) with live state, a report pane tailing /events, and a command line that is the existing grammar. This should be a client of the socket, not a mode of serve, so it works against a remote host over ssh -L unchanged.
Tooling. curl, jq, a CI step asserting every node is Converged before proceeding; a salmon-fleet status that folds /dag from N hosts (or from the pull-mode status sink, which should emit the same JSON).

The web UI is a separate package (salmon-web, PureScript is already in the tree via Spago/purescript-bridge; the bridge can generate the client types from the Haskell ones, which is exactly what it is for). The server itself lives in salmon-ops next to Serve.hs if it can stay light (a small WAI app), else in its own package so salmon-ops does not grow a warp dependency for every binary that never listens.

Security

A unix socket inherits filesystem permissions and needs nothing else. For TCP: TLS (the Certificates nodes can mint the cert; this is what they are for), and a bearer token or client certificate — the tree already has JWT signing (SreBox.JWTSigning). Commands and reads are the same privilege in v1 (an up is as sensitive as reading the output ring of a node holding a pgbouncer userlist); a read-only token is an obvious v2. Default to localhost or a unix socket; never listen on 0.0.0.0 without TLS and auth configured — a salmon server is root on the box, one up away.

notes, help, output rings and report text are public — the convention already in force, since all of them end up in logs (this is why Filesystem.checkFileContents names files and never quotes them, and why filecontents puts a fingerprint in notes, never content). The server inherits that convention rather than adding a redaction layer: it ships exactly what the text reporter would print. A real story for sensitive data (what a node may put where, and what a reporter may emit) is needed and is another spec, not this one; until it exists, node authors should assume anything they write into a report or an Extension text field is readable by anyone who can read the logs.

Interaction with pull mode

The two sketches meet at the inbox and at the status JSON. Pull mode adds a producer (the scheduler) and a consumer of /dag-shaped JSON (the status sink). A fleet view is then either a salmon-fleet that folds sink objects (no server needed on any host) or a web UI that connects to N hosts’ /events (a server on every host). Both should work; the former is the cheap one and lands first.

Non-goals (v1)

Multi-user access control, audit trails, read-only roles.
A server that manages other hosts (that is pull mode plus a fleet fold; this server speaks for one World).
Editing seeds or directives in the UI beyond composing an up line.
WebSocket, gRPC, or any second wire format: SSE + JSON.
Rendering the declared graph (the query tree with Connect/Overlay colouring); the live view is the computed Dag, the same as run tree/ run dag since (R4). query’s tree stays a CLI concern.

Decisions taken

POST /command has both modes: sync by default, returning the command’s reports; ?async returning the enqueue sequence number for a client that reads /events.
/dag reads the magma. It is worldDag over worldMagma and the ledger, the same structure a pass walks, and exists from the first declaration on — no pass required.
notes and every other report text are public. No redaction in this server; a sensitive-data story is a separate spec.
One tagged event stream, produced by composing a server Reporter beside the existing text one (Salmon.Reporter’s contravariant combinators), contramapped into one sum. Clients filter.

Open questions

Sequence numbers: one counter for the whole loop, or per report kind? One (a client resuming wants a single cursor), but Upkeep reports are emitted from machine threads while Serve/UpDown reports come from the loop, so the counter has to be taken under the same MVar the concurrent driver already serialises runReporter through. Answered in milestone 4: one counter; the critical section is the numbering reporter’s own STM transaction, which the drivers’ (several, local) MVars compose over.
Whether /dag should include nodes only retiring declarations still describe (wanted TurnDown, not yet down). Yes, with direction: down — a teardown in progress is the most useful thing to watch — but the UI needs to draw them differently from live ones.

Suggested milestones

ToJSON for the three report streams and a --json reporter flag on every shipped binary. No server yet; run up | jq works; every later client reuses the encoding. Test: golden JSON for each constructor.
Unix socket carrying the line protocol (run serve --listen <path>), as a second producer into the inbox, stdin unchanged. Test: two clients, interleaved commands, reports go to the client that typed them. Shipped (Salmon.Actions.Serve.Socket, Test/ServeSocketSpec.hs). Two deviations: “stdin unchanged” holds for what stdin accepts, not for what its end of input does — under --listen stdin EOF is a hang-up and only quit ends the loop, since a server started < /dev/null & must not exit at once; and a HungUp report was added to Serve.Report, because “every line this client typed has been handled” is a fact only the loop knows and the socket needs it to close the connection at the right moment.
/dag, /status, /history, /help/seed as JSON over HTTP on the socket. The Act projection lands here. Test: /dag equals what Help.printDagTree would print, structurally. Shipped (Salmon.Actions.Serve.Http, run serve --http PATH, Test/ServeHttpSpec.hs), with POST /command in both modes. Two deviations: it is a second unix socket beside --listen’s rather than HTTP detected on the same one (the line protocol reads through a Handle that cannot give peeked bytes back, so sharing meant rewriting both over raw sockets plus a warp Internal shim, for the price of one flag); and /dag is worldDag unrewritten — no members, no remote, no output/paths beyond what status already carries — since the loop’s registered Rewrites run per pass and a rewrite-introduced node has no NodeState to project. /history folds history-elided’s count in as an elided field rather than answering with two objects.
/events (SSE) with sequence numbers and ?since=. Test: a client that reconnects mid-pass misses nothing. Shipped (Salmon.Actions.Serve.Events, GET /events in Salmon.Actions.Serve.Http, --events-ring N, Test/ServeEventsSpec.hs). The open question on sequence numbers is answered: one counter, and the critical section is an STM transaction owned by the numbering reporter (counter, ring and broadcast written together), not the concurrent driver’s MVar — there is no single such MVar to take (one per walk, one per supervisor), but each is held while runReporter runs, so the transaction composes under all of them. Three deviations: every POST /command publishes an enqueued event numbered from the same counter (so the numbering is dense and the ?async number is an event a client can see); a Tended report is delivered as its inner Upkeep.Report under stream: "upkeep" rather than as {"kind":"tended"}; and ?stream=/?origin= filter server-side after all, since a terminal client over a slow link wants less on the wire. /dag and /status carry seq, read before the snapshot so a race replays rather than skips.
mode in status//dag. Shipped: status and /status with specs/pull-mode.md milestone 4 (Serve.Mode on StatusReport), /dag as a top-level mode on the envelope, read from the server’s accessor at the moment of the request; one ToJSON Serve.Mode in Salmon.Reporter.Tagged serves both. /help/seed does not carry it.
Terminal client against the socket. Shipped (salmon-tui PATH in salmon-apps, over Salmon.Client.Http and the pure Salmon.Client.Model in salmon-ops, Test/ClientModelSpec.hs): /dag once, /events from its seq, a node table with direction, state, last check and last event, enter for a node’s help/notes/output, : for a command sent ?async with its seq echoed, r to re-read, reconnect with ?since=, re-read on gap. Three deviations from the “Clients” section. It is a table in /dag’s order (the Dag‘s first-seen order, what run tree prints), not a tree view: with the edges in both directions on every node and a node appearing once whatever number of paths reach it, a tree would repeat nodes and a table with an expand does not, so the tree waits for the web UI’s layered layout. There is no separate report pane tailing /events: the last event is one footer line and each node’s row carries the last event about it, which is what a pane tailing the stream would mostly be showing; a scrollback of events is a place the client would hold state the server does not. And the model drops a replayed event per stamp (a node’s own seq, the loop’s own seq) rather than by one cursor, because a /dag snapshot carries the nodes’ state and not the pass’s, so a re-read after declared would otherwise swallow the converge-stop of a pass the client had already shown starting. Two things the spec did not say that the client needed: a snapshot must be rebased onto a folding model (Model.rebase), and declared must trigger a re-read, since an event names nodes by ref and no event describes a node the client has never seen.
Web UI: static graph from /dag, then live from /events, then actions, then the seed form. Shipped, all four steps (GET / and /ui/* in Salmon.Actions.Serve.Http, the files under salmon-ops/ui/ embedded at build time with file-embed; resources/serve-supervision.md §14 “The web UI”). Three deviations from the sketch above. It is not a separate salmon-web package and not PureScript: three static files — one page, one ES module, one stylesheet, no bundler — served by the binary itself, since a UI that ships inside the thing it watches needs no deploy step and the read model is small enough that generated client types would cost more than they save; the bridge stays an option for the seed form. The layered layout is neither dagre nor ELK but a longest-path layering with barycentre ordering written in ui.js, because vendoring a bundle for a graph of tens of nodes is the wrong trade and the layout is a hundred lines. And a browser cannot open a unix socket, so the page is reached through a TCP forward (socat, ssh -L) until milestone 8, which added --http-tcp and the /auth sign-in, so a browser now reaches it directly. The actions and the seed form add three more deviations, all on the write side. Every write is POST /command?async and never the synchronous form — the page reads the outcome off /events by the request’s origin, which is what marks the nodes a command touched and fills the log under its command line; the sketch’s “what a UI wants” turned out to be the whole of it. The seed form is /help/seed’s text in a <pre> and a free-text field for the words, not a form derived from the ParseRecord — no schema is served, and the bridge stays unused. And “retire the seed behind this node” is not a per-node action: a node does not know its declaring seed and /history does not list an epoch’s nodes, so the panel offers every live declaration’s down instead and the operator picks. Also deliberate: no quit on the page (the page is served by the process it would stop), and no bearer token sent: over --http-tcp the browser signs in at /auth and carries a session cookie instead (milestone 8). Not yet: the Conflicting pair side by side (as of 2026-09-25 /dag carries it — conflict: {kept, replaced} on the node, held while a live declaration still wants the losing version — so only the page’s half remains), collapsing a batch to its members and expanding a RemoteOp (neither is on /dag, see milestone 3’s deviations), and history as a timeline — it is a table under the seed form.
TCP + TLS + token, opt-in, with the loud default described above. Shipped (run serve --http-tcp HOST:PORT --tls-cert FILE --tls-key FILE --token-file FILE; Http.withHttpServerOn/Http.Bind/ Http.requireToken, CommandLine.validateTcpOptions, Test/ServeTlsSpec.hs, resources/serve-supervision.md §14). The same application on a warp-tls listener beside the unix socket, one Server for both; a bearer token on every route of the TCP listener, compared in constant time; the unix socket unchanged and token-free. The loud default is stricter than the text above: not “never on 0.0.0.0 without TLS and auth” but never on any address without both — there is no plaintext TCP constructor or flag, --http-tcp without all three files exits 1 naming the missing ones, and the host is always spelled (:8443 is refused, 0.0.0.0:8443 is how listening everywhere is written), so “default to localhost” is not a default but a choice the operator types. Four deviations. A bearer token from a file, not a JWT: SreBox.JWTSigning signs for other services and a verifier here would need a key store, an audience and a clock for what a chmod 600 file already gives; and no client certificate, which is the v2 the text names. The token file must not be readable by others and must not be empty — two refusals the text did not ask for. Commands typed over TCP are attributed to the client’s ADDR:PORT#n, not to the listener, so history says who. And the Certificates nodes can mint the certificate, as the text says, with one caveat found on the way: selfSign/caSign (openssl x509 -req) write X.509 v1 certificates, which crypton’s validation rejects (LeafNotV3) while OpenSSL-based clients accept; certificateAuthority (req -x509) writes v3, and is what the test pins. Not done: the clients — salmon-tui and the web UI need a --token and a TCP address to use this.

Builtins, recipes and binaries

2026-09-25T14:15:36Z

Generated from README.md — the repository is the canonical source, and may be ahead of this page.

Builtins, recipes and binaries

Builtin nodes

salmon-ops/src/Salmon/Builtin/Nodes/ — mostly atomic, small-diameter ops, one module per concern:

Module	Covers
`Bash`	ad-hoc shell script ops
`Binary`	the shared subprocess-running plumbing (`untrackedExec`, exit-code-checked exec) every other builtin is built on
`Cabal`	building Haskell projects with cabal
`Capabilities`	Linux file capabilities (`setcap`/`getcap`), so a binary can do one privileged thing unprivileged
`Certificates`	TLS certificate generation/signing
`Continuation`	chaining/sequencing op continuations
`CronTask`	cron job management
`Daemon`	a process salmon owns and keeps running — the one builtin with a `managed` action, for where there is no systemd (see `resources/serve-supervision.md`)
`Demo`	a toy graph (`collatz`) for trying the drivers on
`Debian.Debootstrap`, `Debian.Package`, `Debian.OS`	Debian package installs and base-system setup
`Filesystem`	directories, file contents, copy/move/replace-directory (the canonical small example — see `resources/howto-ops.md`)
`Gcp.*`	Google Cloud: projects and billing (`ResourceManager`, `Billing`, `ServiceUsage`), `Iam`, `Storage`, `ArtifactRegistry`, `CloudRun`, `Compute`, `LoadBalancing`, `SecretManager`, `SshAccess`, `Monitoring` (notification channels and Cloud Run alert policies) — driven through `gcloud`; see `resources/gcp-toy-validation.md`
`Git`	git repository operations
`Keys`	key material management
`LinuxBridge`	Linux bridge and tap devices, a real L2 network for qemu VMs to sit on
`Netfilter`	`nft` firewall rules (with `check`-based idempotency, since `nft add rule` itself isn't idempotent)
`Nginx`	nginx site/config management
`Npm`	npm package operations
`PgBouncer`	PgBouncer connection-pooler configuration
`Podman`	container image/volume/network/env lifecycle
`Postgres`	cluster creation, users/groups/grants, WAL streaming replication, `pg_hba.conf` management
`Qemu`	a qemu VM as a systemd unit, booted from a debootstrap chroot over 9p (`specs/qemu-test-vms.md`)
`Routes`	IP routing table entries
`Rsync`	file/secret transport over rsync
`Secrets`	secret material placement
`Self`	uploading and re-invoking this binary on a remote machine
`Spago`	PureScript/Spago builds
`Ssh`	SSH remote-machine plumbing
`Sysctl`	kernel parameter tuning
`Systemd`	systemd unit management
`Tar`	archive creation/extraction
`Upx`	binary compression
`User`	OS user/group accounts
`Web`	HTTP-facing ops
`WireGuard`	WireGuard VPN interfaces, keys, and routing

Recipes

salmon-ops-recipes/src/SreBox/ — opinionated compositions of the builtins above, where this project’s own conventions get enforced:

Module	Covers
`CabalBuilding`	building and publishing cabal-based binaries
`DNSRegistration`	DNS record registration
`Environment`	environment/machine bootstrapping
`Initialize`	initial setup sequencing
`JWTSigning`	JWT signing key management
`MicroDNS`	a minimal DNS server setup
`PostgresInit`	database/user/group/grant setup for a Postgres cluster (locally or driven onto a remote machine via `Self`)
`PostgresMigrations`	shipping and running migrations, local or remote-connstring
`PostgresBackup`	periodic `pg_dump` backups: the script, the schedule, and a node that notices when no recent dump exists
`PostgresPair`	two machines, one Postgres cluster, and a declared primary: switchover, operator-decided failover, `pg_rewind` rejoins, and pgbouncer routing that follows — see [`resources/postgres-pair.md`](/docs-postgres-pair.html)
`PostgresTemplate`	template databases: build once, lock, hand out clones (`salmon-migrator config template`/`clone`)
`PostgresTls`	Postgres authenticating clients by certificate, and the material that makes it possible
`Postgrest`	PostgREST service configuration
`WireGuardVpn`	a full static-server/dynamic-client WireGuard VPN, transport-agnostic on key exchange
`Gcp.CloudRunDeploy`	build a podman image, push it to Artifact Registry, deploy it to Cloud Run
`Gcp.CloudRunAlerts`	the standard Cloud Monitoring alerts for a Cloud Run service (5xx ratio, p99 latency, memory, instances at max) to one email
`Gcp.PostgrestCloudRun`	PostgREST on Cloud Run talking to a Postgres elsewhere over a client certificate
`Gcp.PreviewEnvironment`	several Cloud Run deploys composed into one named node: a preview environment
`Gcp.VmProvision`	turn up a GCE instance, then run a salmon binary on it over SSH via `Self`

salmon-ops-recipes-experimental (not part of the default package set — see Build) holds SreBox.CertSigning (certificate signing workflows), SreBox.KitchenSinkBlog, SreBox.KitchenSinkMultiSites, SreBox.GeneratedSite, and the Salmon.Builtin.Nodes.Acme builtin (ACME/Let’s Encrypt certificate issuance).

Binaries

salmon-apps/ — each one is a small Main over a recipe:

Binary	Does
`salmon-migrator`	Postgres migrations, template databases and clones (`config template`/`clone`)
`salmon-pgpair`	a Postgres primary/standby pair whose primary is a declaration — see [`resources/postgres-pair.md`](/docs-postgres-pair.html)
`salmon-pg-backup`	take a Postgres dump now, or install the cron job that keeps taking one, here or on another machine
`salmon-init-locally`	the local-machine salmon setup (sudoers, the salmon user and group)
`salmon-gcp-toy`	a tiered, throwaway exercise of the GCP builtins against a real project — see [`resources/gcp-toy-validation.md`](/docs-gcp-toy-validation.html)
`salmon-toy-qemu-pg-ha`	the `salmon-pgpair` demo on three qemu guests, with a client that keeps writing while the primary moves
`salmon-fleet`	the controller's side of pull mode: `status DIR` folds the hosts' status documents into one line per host; `keygen`/`sign` make signed documents
`salmon-tui`	a terminal client for `run serve --http` (or `--http-tcp`)

A Postgres pair whose primary is declared, not discovered

2026-09-25T14:15:36Z

Generated from resources/postgres-pair.md — the repository is the canonical source, and may be ahead of this page.

A Postgres pair whose primary is declared, not discovered

Two machines, one Postgres cluster, and a sentence that says which machine is the primary today. Changing that sentence moves the primary; nothing else does. This is SreBox.PostgresPair, the salmon-pgpair binary, and salmon-toy-qemu-pg-ha, a demo that builds its own machines to show it.

If you want the design argument rather than the instructions, read specs/pg-switchover.md; this file is about using what it describes.

What it is for, and what it is not

It is for a planned switchover (maintenance, a machine you want to drain) and an operator-decided failover, on two machines, where a few minutes of somebody’s attention is an acceptable price for the machine you did not buy. It is also, deliberately, a thing to break on purpose: every failure mode below is something the test suite causes and then asserts about.

It is not automatic failover. Deciding that a machine is dead needs consensus, consensus needs three voters, and salmon has neither — so this recipe never promotes on a hunch. When it cannot prove a promotion is safe it refuses, and the operator’s answer to a refusal is the one field that accepts a loss by name:

--may-discard A     "I accept losing the writes on A that B does not have"

For the unattended-at-3am version, specs/pg-patroni.md sketches the Patroni-backed shape, where salmon provisions and supervises and Patroni decides.

The shape

Three kinds of node, and only one of them mentions a role:

Node	Says
`member` (one per machine)	how to be either half of the pair: replication settings, `pg_hba` lines for the peer, the replication and rewind roles. Nothing about which half it is.
`bouncerSetup` (one per bouncer)	pgbouncer's own configuration, and a routing file it includes but does not own
`pairRole`	"this pair's primary is on B" — the only declaration an operator edits

That split is the whole ergonomic claim: a switchover changes one node’s declaration, so under run serve the machines underneath are not marked stale by it, and an operator reading a diff sees one word.

pairRole’s check asks both machines and every bouncer what they are; its up takes one step at a time until the declaration holds. The state is re-derived every pass — observe, decide, act, observe again — so an up killed half-way through a switchover is finished by the next one. There is no progress file that could disagree with the machines.

Using it

salmon-pgpair config --primary A --a 10.0.0.2 --b 10.0.0.3 --bouncer 10.0.0.4 --seed B \
  | salmon-pgpair run up

salmon-pgpair config --primary B --a 10.0.0.2 --b 10.0.0.3 --bouncer 10.0.0.4 \
  | salmon-pgpair run up          # the switchover

run tree prints what the first of those declares before anything runs:

pg-pair-role (n2504805732311083189) primary of app is on A
  <- pg-pair-bouncer
  <- pg-pair-seed
  <- pg-pair-member
  <- pg-pair-member
pg-pair-bouncer (n671955809932212377) pgbouncer 10.0.0.4 in front of app
pg-pair-seed (7089073737010868211) seeds 10.0.0.3 from 10.0.0.2
  <- pg-pair-member
  <- pg-pair-member
pg-pair-member (n6118828710077022853) member of app on 10.0.0.3
pg-pair-member (2659614637461449593) member of app on 10.0.0.2

What it assumes was done before it ever ran, because a recipe that ships secrets has chosen a transport for everyone who uses it: both machines have a Postgres cluster and the two .pgpass files the pair names, and the bouncer has pgbouncer, a userlist.txt and the .pgpass for its admin console. The recipe is given paths.

--seed B is the first clone of a pair’s life, and the only way back from a standby that has fallen too far behind (see the slot budget below). It is safe to leave declared — the clone does nothing once the two sides share a system identifier, and refuses a machine holding a cluster it does not recognise.

What a switchover actually does

PauseBouncers -> StopMember A -> Promote B -> RepointBouncers B -> Rejoin A -> Done

PauseBouncers holds the clients rather than dropping them. With pool_mode = transaction, PAUSE waits for transactions in flight and queues what comes after, so a client sees latency where it would otherwise see an error.
StopMember is a clean stop, which hands the standby everything it has not got, including the shutdown checkpoint the promotion waits for. It also pins wal_keep_size first, because a clean shutdown ends in a checkpoint and a checkpoint recycles the WAL a later rewind reads.
Promote waits for the server to say it promoted.
RepointBouncers rewrites the routing file, RELOADs, and RESUMEs. Never a restart: a restart drops every client the bouncer is there to hold.
Rejoin brings the old primary back as a standby through pg_rewind, onto the new primary’s history — same machine, same data, only the records that diverged replaced. Not a re-clone.

Traffic moves through pgbouncer’s admin console, and the routing lives in its own file pulled in with %include. That is a seam between two writers: bouncerSetup owns the ini, so a change there is applied by a restart; the role node owns the routing file (bouncerSetup writes it only when it is missing), and applies a change gently. One file with two writers is how a switchover becomes an outage.

When it refuses, and why

A refusal is the recipe saying it cannot prove the next step is safe:

It says	Because
the two machines hold different clusters	their system identifiers differ, so every step below would be applied to somebody else's data. Checked before anything else, and `--may-discard` does not override it
both machines are primaries	resolving a split brain means rewinding one onto the other, which is a loss. Name the side
the peer did not stop cleanly	a crashed cluster's `pg_controldata` records its last checkpoint, not the end of its WAL: the standby may be missing anything written after it
cannot confirm the peer has stopped	it is unreachable, and promoting without fencing is how two primaries happen
the peer standby is ahead	promoting the declared side would lose the difference

And two verdicts that are not failures, and act on nothing:

AwaitStreaming — the peer is pointed here and not connected: a partition, or a standby that came back a second ago. Waiting is right; rewinding at it would stop it and then fail, because whatever keeps it from streaming keeps pg_rewind from reading too.
Degraded — the declaration holds but the pair is one machine short. Reported as Unknown, the one verdict run serve acts on by continuing to look.

The slot budget

Each member streams with a replication slot the pair names, created by the member that rejoins. A slot is a promise to keep WAL until the standby has it, and an unbounded promise is how a machine that is merely down takes the machine that is up with it — so max_slot_wal_keep_size caps it. Past the cap the slot is invalidated, the WAL is recycled, and the standby can never catch up.

That is a good trade and a terrible surprise, so the pair says it out loud: the check reports the lost slot by name and does nothing. pg_rewind would succeed and change nothing; the only way back is a re-seed, and wiping a machine is an operator’s decision, declared with --seed.

Watching it happen

salmon-toy-qemu-pg-ha builds three qemu guests, puts the pair on two of them and pgbouncer on the third, and writes through the bouncer while the primary moves:

t=$(cabal list-bin salmon-toy-qemu-pg-ha)

sudo $t config prereqs             | sudo $t run up   # once: three rootfses
$t config up --primary A --seed B  | $t run up        # once: the pair
$t config client --seconds 120     | $t run up &      # a client, writing
$t config up --primary B           | $t run up        # the demo

The client prints three numbers when it stops. On the machine this was written on:

  inserts acknowledged through the bouncer: 460
  inserts that came back an error:          0
  acknowledged rows missing afterwards:     0

prereqs is the only part that needs root — debootstrapping a root filesystem, regenerating an initrd that can mount a 9p root, and handing /etc/ssh to whoever runs the rest. Everything after it is an unprivileged user with two capabilities granted once (capsh needs cap_net_admin, qemu needs cap_dac_override,cap_chown,cap_fowner; see specs/qemu-test-vms-progress.md).

After the demo B is the primary: pause machine B’s guest, declare up --primary A, and it refuses; add --may-discard B and the refusal turns into a failover.

What is tested, and what that is worth

Test.PostgresPairSpec covers the decision table at Layer 0 — every state two machines can be found in, including every refusal — without a database.

Test.PostgresSwitchoverSpec and Test.PgPairDemoSpec cause the failures on real machines:


S1	switch A→B→A with a client attached: every acknowledged insert present, zero client errors
S2	the controller killed after each step in turn: a plain rerun finishes it
S3	the primary crashed with un-replicated writes: refused without the flag, rewound with it
S4	the two machines partitioned: the check is `Unknown`, and the standby is not touched
S5	a failover across a partition that hides the primary from the controller too: two primaries, then the loser rewound
S6	a standby that falls off the slot budget: said, not silently re-seeded
S7	both machines stopped, in either order, and the stale one declared primary
S8	a stranger's cluster where a member should be: refused, nothing deleted

Between them these cost the recipe twelve defects, and not one was on the happy path: S1 and S8 found nothing, and every other defect needed a machine that stopped, or was cut off, without being asked to. That is the argument for writing the list as causes rather than as features.

Property-based testing for `run serve`

2026-09-25T14:15:36Z

Generated from specs/serve-property-testing.md — the repository is the canonical source, and may be ahead of this page.

Property-based testing for `run serve`

Status: v1 implemented, in salmon-ops-recipes/test/Test/ServeModelSpec.hs. It generates up/down/only/clear/converge sequences over a fixed three-seed universe, folds them through an independent shadow model, and checks that the real loop in piped-script mode agrees on its final World and on the per-node up/down counts: invariants 1, 2 and 3 (prop_convergesLikeModel) and 6 (prop_clearSettlesToEmpty, plus a bookkeeping check inside the first). Invariant 7 is covered implicitly, as suggested below: the suite finds each node in worldNodes by a Ref computed from its name alone (nodeRef), so a Ref that depended on history would make the lookup miss and the property fail. A third property, prop_producersTakingTurnsAgree, came later with Serve.serveProducers and is not one of the invariants below. Invariants 4 and 5 stay example-based in Test.ServeSpec, per the non-goals. Rewrite-registered batching (v2) is not written yet. The three decisions at the end are resolved; the rest of this document is kept as the design record.

Problem

Test.ServeSpec (see CLAUDE.md’s note on Salmon.Actions.Serve) is entirely example-based: each test picks one specific command sequence by hand and asserts on the outcome. That style is good at pinning a known regression down precisely (see the autoconverge off tests added on serve-supervision), but it only ever checks the sequences somebody thought to type. The bug this spec is a reaction to — autoconverge off failing to also stop the idle tending loop, and later force having nowhere to deliver its instruction once tending was correctly stopped — was found by hand, once, in an interactive session. A property test stating “no IO happens on any node while autoconverge is off, whatever the preceding history” would have caught both for free, and would keep catching the next variant of the same mistake without anyone having to think of the exact scenario again.

Serve.hs’s own model is unusually well-suited to this: a World is a pure fold over a sequence of commands (record/retract/converge/etc.), and the whole point of the module (per its own haddock) is that “everything else is derived” from that fold. That’s exactly the shape property-based testing wants: generate a sequence, fold it, check an invariant of the result — not “guess an interesting sequence and hand-write it.”

Design goals / non-goals

Goals:

A handful of invariants, checked against randomly generated command sequences, over the exact Serve.serveWith entry point the example tests already drive (no separate model of the implementation to keep in sync — the shadow model is of the domain, e.g. “which seeds are live”, not of Serve.hs’s internals).
Shrinking that produces a short, readable failing command sequence, since that is most of the value of property testing over examples — a hand-picked regression test is only as good as the report that led to it.
Reuse of the existing test fixtures/plumbing (runServeWith, the Spec seed type, withSession) rather than a parallel test harness.

Non-goals (v1):

Properties that need real idle time (the tending loop, withSession’s forked-thread sessions). Real threads and real delays make shrinking fight the scheduler instead of the command sequence; the hand-written withSession examples already cover that territory and should stay example-based. v1 is scoped to piped-script mode (runServeWith), which is a pure function of the command list — no idle moment ever occurs, so there is nothing nondeterministic to shrink around (see CLAUDE.md’s note that a piped script is never supervised).
Rewrite-registered batching. Interesting, but it’s a second axis of complexity on top of plain declare/converge; a v2 concern once the plain case has a harness worth extending.
Testing the tending FSM itself (Test.UpkeepSpec already does that, at the right level — a single machine’s state transitions, not a whole serve session).

Invariants

Numbered for reference, not priority — see “Suggested starting set” below.

Convergence does what it says. After any converge (or an autoconverging declaration), every node whose final resolved direction is TurnUp and was not already Converged had up called on it at least once since the previous convergence; dually, TurnDown nodes had down called. This generalizes the motivating example (“if the latest event is up node, we observe up called by the end”) to the state at the end of an arbitrary history rather than just after one command.
No spurious re-application. A node that was already Converged in some direction, whose direction and content did not change, gets zero additional up/down calls from a later converge. The idempotence half of (1) — easy to eyeball as “it converged”, easy to miss “it converged again for no reason” in a hand-read transcript.
Shared-node teardown safety. If two live declarations both want a node (the shared-directory case retireMultiFileBundle/onlySupersedes already cover by hand), retiring one must never call down on it while the other is still live. Generalizes those two fixed examples across arbitrary interleavings of up/down/only/clear.
autoconverge off is a strict no-op on IO. From autoconverge off until either converge or autoconverge on, no up/down fires for any node, regardless of how many up/down/clear/only commands happen in between. This is the bug this spec exists because of; see “Non-goals” above for why it stays example-based (withSession) rather than becoming a v1 property despite being the original motivation — the interesting failure mode (idle tending applying a deferred node) is precisely the one piped-script mode cannot exercise at all.
force/recheck are scoped. Forcing node A never causes IO on node B, no matter what else is pending. Same real-idle-time caveat as (4) — the instruction only does anything once a machine exists to receive it, which piped-script mode never starts. Stays example-based in v1 for the same reason as (4).
Settling is total. clear followed by enough converges drives worldNodes/worldEpochs/worldLedger/worldMagma all empty, whatever the preceding history was — generalizes assertWorldSettled off its one fixed sequence.
Ref stability under reordering. Declaring the same seed args always resolves to the same Ref, independent of what else was declared before/after/around it — mkRef’s content-addressing should not depend on history shape. Cheap to check as a side-assertion inside (1)/(3)/(6) rather than its own property: whenever the model says “seed X is up”, the real node’s Ref should be the one first seen for seed X, ever.

Suggested starting set for v1

(1), (2), (3), (6) — all piped-script-only, all checkable against the exact harness sketched below with no new infrastructure beyond a generator and a shadow model. (4) and (5) are the ones the bug this spec reacts to actually lived in, but per “Non-goals” they need real idle time to be meaningful, so they stay as the withSession examples already on serve-supervision (autoConvergeOffAlsoStopsIdleTending, autoConvergeOffForceStillReachesANamedNode, etc.) rather than becoming property tests in this first pass. (7) is cheap enough to fold into whichever of (1)/(3)/(6) lands first rather than write standalone.

Harness sketch

Universe. A small fixed set of seed ids and node names — 2–3 seeds sharing 1–2 node names on purpose (mirroring Test.ServeSpec’s existing program/Spec fixture: a seed is a list of file names under a shared directory). A bigger universe dilutes exactly the shared-node interactions these invariants are about; a bigger history length is where the interesting coverage should come from, not a bigger alphabet.

Command generator. Gen [ServeCommand]-shaped, restricted to Up/Down/Only/Clear/Converge over that fixed universe (no AutoConverge/Force/Instruct in v1 per “Non-goals”). Ordinary list shrinking (shrink the list, then shrink individual seed choices toward the first seed id) should already produce short, readable counterexamples, since QuickCheck/Hedgehog both shrink lists well out of the box.

Spy. A stub Track' Spec (reusing the existing Spec/program approach) whose up/down each bump a per-node counter in one IORef (Map Ref Int) (or Map NodeName Int if working in terms of the model’s own naming rather than the real content-addressed Ref) rather than touching the filesystem — matching neverRuns/neverRunsNamed’s existing pattern in Test.ServeSpec, extended to record counts per node rather than one global counter or two hand-picked ones.

Shadow model. Something close to what Ledger/worldNodes already compute, but written independently and at the seed level rather than mirrored from the implementation:

data Model = Model
  { modelLive :: Set SeedId          -- declared and not yet retired
  , modelUpCount :: Map NodeName Int -- expected cumulative `up` calls
  , modelDownCount :: Map NodeName Int
  }

Folding a command into a Model should be short enough to visibly not share logic with Serve.hs — the whole point is an independent restatement of “what should be true”, not a shrunk copy of Ledger.hs.

Running it. Feed the generated [ServeCommand] as lines to runServeWith (piped-script mode, deterministic, one convergence pass per autoconverging command — see CLAUDE.md). Fold the same command list through the shadow model. Compare: final World’s per-node Direction/Convergence against the model’s modelLive-derived expectation (invariant set (1)/(2)/(3)/(6)), and the spy’s counters against the model’s expected call counts.

Decisions needed before writing code

Library. Neither salmon-ops-recipes.cabal’s test-suite nor any other package in this tree currently depends on QuickCheck/tasty-quickcheck (or hedgehog/tasty-hedgehog). This is a new, if standard and light, test-only dependency to add consciously rather than as a side effect of the first property test — tasty-quickcheck is the smaller addition given tasty/tasty-hunit are already in use, but worth a deliberate choice over hedgehog’s (arguably nicer) generator/shrinker story.
Where the shadow model lives. As its own small internal module (Test.ServeModelSpec or similar) versus inline in Test.ServeSpec — the existing file is already large; a model-based property suite is a distinct enough concern (and reusable enough, if Rewrite-aware properties get added later) to justify its own module from the start.
Number of commands per generated sequence. Long enough to hit multi-seed overlap reliably, short enough that a first failing run is already close to minimal before shrinking does its work — needs a bit of experimentation once the harness exists rather than a guess up front.

How they were decided

Library: QuickCheck, through tasty-quickcheck, as the smaller addition beside tasty/tasty-hunit. Both are test-only dependencies of salmon-ops-recipes, and nothing else in the tree depends on them.
Where the shadow model lives: its own module, Test.ServeModelSpec, separate from Test.ServeSpec.
Commands per sequence: between 1 and min 30 (size + 5) (genCommands). With three seeds that overlap pairwise, that is enough to reach the shared-node cases, and shrinking brings a failure down to a few lines. The model’s two subtleties (a node with no check is applied whenever it is freshly tracked, and a node that settles TurnDown leaves worldNodes) were found this way. Its haddock records them.

Implementation progress: `specs/qemu-test-vms.md`

2026-09-25T14:15:36Z

Generated from specs/qemu-test-vms-progress.md — the repository is the canonical source, and may be ahead of this page.

Implementation progress: `specs/qemu-test-vms.md`

Status: living doc, update as work continues. Companion to specs/qemu-test-vms.md (the design) — this file tracks what’s actually built, what’s still untested, and exactly what to do next. Read the spec first if you need the “why.”

Work on this branch is being committed incrementally as it lands (see git log); this doc may still lag the latest working-tree state by a change or two at any given moment.

0.3 Headline: fixed the flake — two real teardown bugs, both pre-existing (2026-09-08)

Following a flake reported in §0.2 (Postgres replication VM test timing out only inside the full 20-test suite, not standalone): the actual cause was found by inspecting the live machine rather than guessing from logs — ps showed 6 orphaned qemu-system-x86_64 processes left running from earlier interrupted/crashed test runs, eating ~3GB RAM and real CPU, which starved the timing-sensitive replication test under full-suite load. Cleaning them up and rerunning made the flake disappear, but the real fix was finding why teardown wasn’t cleaning them up — two separate, pre-existing bugs, neither introduced by §0.2’s privilege work:

LinuxBridge.tap declared the shared bridge as its own graph dependency (deps [bridge ...]). downTree’s “release a predecessor once its last dependent is torn down” rule is correct in general (the directory/two-files case documented in CLAUDE.md) but wrong here: the bridge is explicitly meant to persist across many taps/VMs coming and going, and each tap’s downTree call has no visibility into other taps still relying on the same bridge (different process, different traversal). Result: tearing down any single VM deleted the shared bridge out from under every other still-running VM, leaving their taps NO-CARRIER — directly observed on the live machine (ip link show salmontest0 → “Device does not exist” while several taps sat orphaned). Fixed by dropping the graph dependency entirely and instead running bridge’s own Op via a nested upTree inside tap’s up (same accepted “nested traversal, check the Bool, throwIO if False” pattern as PostgresMigrations.remoteMigrateOpaqueSetup — howto-ops.md §5) — brings the bridge up as a precondition without ever making it this tap’s own teardown-reachable predecessor.
Systemd.systemdService never actually set a down action at all — it defaulted to Extension’s no-op, silently contradicting Qemu.hs’s own module haddock, which already claimed “down goes through plain systemctl stop”. This is why the orphaned VMs existed in the first place: every single withVmAt teardown, even a clean, successful one, left the qemu process running forever — down deleted the tap and the unit file (via configContents’s own real down) but never stopped the service itself. Fixed by adding a Stop SystemCtlCall and wiring down = stop — confirmed via ps/systemctl --user list-units immediately after a test run: no leftover process, no leftover unit file, where before there reliably was one of each per withVmAt call.

Confirmed via ps/ip link/systemctl --user list-units before and after: the full 20-test suite (cabal test salmon-ops-recipes) passed clean twice in a row post-fix (168s, 186s), leaving zero qemu processes and zero unit files behind either time — no flake recurrence.

0.2 Headline: the whole tier runs unprivileged now, no `sudo` at all (2026-09-01/08)

Following up on §4’s privilege open question: dropped the requirement that the whole test binary run as root. Test.Harness.hasVmPrivileges now accepts either real root or a one-time capability grant; withVmAt runs qemu as the invoking user (via LinuxBridge.Tap’s tapOwner and Qemu.VmConfig’s vm_user/vm_group) instead of root:root. New Salmon.Builtin.Nodes.Capabilities (setcap/getcap, idempotent) plus a salmon-qemu-host-setup-fixture executable do the one-time host grant as a real Op graph instead of a shell snippet — see its own haddock for the exact commands. Confirmed passing fully unprivileged: QemuSmokeSpec (20s) and PostgresReplicationSpec (69s, two VMs) both green with no sudo anywhere in the test invocation.

Two real, boot-validated bugs found getting there, neither guessable from code review:

Granting cap_net_admin to ip itself does not work. strace on a failing unprivileged ip link add ... type bridge showed ip unconditionally calling capset({...}, {effective=0, permitted=0, inheritable=0}) at startup — iproute2 drops its entire capability set on exec and only trusts the ambient set afterwards, which a plain file-capability grant can never populate (the kernel zeroes ambient for any exec of a “privileged” file, by design). Confirmed via a clean control test first: ping (file-cap cap_net_raw) works fine unprivileged, proving the capability mechanism itself was never the problem. Fix: grant the capability to capsh instead, and have ip invocations go through capsh --inh=cap_net_admin --addamb=cap_net_admin -- -c "ip ...args..." (Salmon.Builtin.Nodes.LinuxBridge.ipLinkCommand) — ambient capabilities do propagate across exec and are what iproute2 actually honors. Raising ambient itself needs the capability in both the process’s permitted and inheritable sets (--inh= first) since exec does not carry a file’s inheritable bit into the new process’s own inheritable set. Works identically for real root (whose permitted set is already full) and for an unprivileged user with the grant on capsh, so the wrapping is unconditional now, not privilege-mode -specific.
A qemu VM’s systemd unit can’t live under /etc/systemd/system unprivileged (plain permission denied writing there). Fix: added Systemd.Scope (System/User) to Salmon.Builtin.Nodes.Systemd; Qemu.VmConfig gained vm_systemd_scope/vm_unit_dir, and Test.Harness.withVmAt now uses Systemd.User against a resolved ~/.config/systemd/user, with systemctl --user and default.target swapped in for multi-user.target/network-online.target (which don’t exist in the user manager). Systemd also rejects User=/Group= in a user-manager unit, so render_service omits them for User scope. PgBouncer/Postgrest/MicroDNS’s existing Systemd.Config call sites were updated to explicit System//etc/systemd/system — no behavior change for them.

Not yet chased down: the Postgres replication VM test flaked once when run as part of the full 20-test suite (cabal test salmon-ops-recipes) — walsender process due to replication timeout inside the guest, a stale-pidfile postgres restart loop — while passing cleanly twice standalone. Smells like timing/resource contention from running back-to-back with everything else rather than a regression from the privilege changes above (this project already has one documented unrelated Layer 2 podman flake under load), but not confirmed either way.

0.1 Headline: Phase 5’s real recipe test now passes for real (2026-08-21)

Test.PostgresReplicationSpec (see §1) now passes under sudo against real pg-primary/pg-standby rootfses — closes §3 item 4. Getting from “builds clean, never run” to a real pass took four more bugs, none guessable without actually booting the VMs:

resolveFixtureBinary took the first line of cabal list-bin’s stdout, not the last: under sudo (root, no prior cabal config), cabal prints a one-line notice (“Config file path source is default config file.”) to stdout before the actual bin path, so the scp target became that notice string instead of a path. Fixed to take the last non-blank line.
postgresql-client isn’t pulled in by the postgresql meta-package the way postgresql-client-17 is — the fixture’s Debian.psql/ Debian.pg_ctl map to the generic postgresql-client package name, which wasn’t in the rootfs’s --include list, so the guest (no network route past boot, by design) failed trying to fetch it. Fixed by chroot-installing it into the master rootfs (host has network) before copying out — see §2.
The cluster ended up on port 5433, not 5432: pg_createcluster (during the postgresql package’s postinst, run inside a chroot on the host) auto-picks the next free port by probing the host’s own running processes — since it shares the host’s network stack, it saw something already on 5432 and picked 5433. The fixture hardcodes primaryPort = 5432. Fixed by forcing port = 5432 in the rootfs’s postgresql.conf directly.
Test.Harness.sshToVm silently mis-delivers any argument containing embedded whitespace. ssh joins every argument after the destination with a single space and ships the result as one string for the remote shell to tokenize — exactly like typing the words by hand at a terminal. So an args element that’s a whole SQL statement or a cmd 2>&1 redirection doesn’t arrive as one remote token: the remote shell re-splits it on spaces along with everything else. E.g. ["psql", "-tAc", "SELECT state FROM pg_stat_replication;"] arrived remotely as psql -tAc SELECT state FROM pg_stat_replication; — -tAc only captured SELECT, and waitForStreaming’s query silently malformed on every single poll, meaning it could never have detected real streaming state regardless of whether replication actually worked. Same class of bug as the Systemd.render_start ExecStart= quoting fix in §0 item 4 below — just in the test harness instead of production code. Fixed by adding Test.Harness.quoteForRemoteShell (single-quotes a string so it survives ssh’s space-join as one token) and applying it at each call site in Test.PostgresReplicationSpec that needs one. Deliberately not made automatic inside sshToVm itself: Test.QemuSmokeSpec’s existing sshToVm access ["echo smoke-ok"] relies on the remote shell’s own re-splitting to turn one Haskell string into two remote words — quoting every argument unconditionally would instead hand the remote shell one literal token "echo smoke-ok" (a program name with a space in it) and break that passing test.

Confirmed passing standalone (--pattern 'Postgres replication', ~100s) and with the rest of the non-root suite around it unaffected (Podman’s Layer 2 container test failed in one non-root sanity run, but that’s an unrelated, pre-existing environmental flake — Salmon.Builtin.Nodes.Podman wasn’t touched by any of this work).

0. Headline: the Layer 3 tier now works end to end (2026-08-20)

Test.QemuSmokeSpec (new) boots a real qemu VM via Test.Harness.withVm against a hand-built debootstrap rootfs and SSHes into it for real — confirmed passing under sudo (root is required, see §2):

cabal build salmon-ops salmon-ops-recipes:test:salmon-ops-recipes-test
sudo PATH="$PATH" dist-newstyle/build/x86_64-linux/ghc-9.8.2/salmon-ops-recipes-0.1.0.0/t/salmon-ops-recipes-test/build/salmon-ops-recipes-test/salmon-ops-recipes-test --pattern Qemu
# Qemu (Layer 3, real VM boot via withVm)
#   boots the smoke rootfs and answers SSH: OK

Getting there took four real production bugs, each hand-validated by booting an actual VM and reading its serial console — none of these were guessable from code review alone:

NIC naming (predicted in §3.1 below, confirmed for real): the virtio-net device came up as ens4, not eth0, breaking the ip= kernel arg silently. Fixed by adding net.ifnames=0 biosdevname=0 to Qemu.kernelCmdline unconditionally (this whole tier already assumes a single, always-eth0 NIC).
9p root never mounts: the stock debootstrap initrd never even attempts a 9p mount of its own root (9pnet/9pnet_virtio/9p are kernel modules, not builtin, and nothing loads them) — panics with /dev/root does not exist. Fixed with a new op, Debootstrap.ensureVm9pBoot, that appends those modules to /etc/initramfs-tools/modules and regenerates the initrd via a chroot. Also required renaming the 9p mount tag from /dev/root to vroot (and root=vroot on the cmdline): initramfs-tools’s local_device_setup only skips its udev block-device wait for a ROOT that neither starts with /dev nor contains = — anything else, 9p tags included, it waits forever since 9p never produces a udev block device.
security_model=mapped breaks /sbin/init: run-init: /sbin/init: Too many symbolic links encountered — mapped doesn’t round-trip Debian’s /bin -> usr/bin-style symlinks faithfully. Since qemu (and this whole tier) already runs as root on the host, switched to security_model=passthrough (real symlinks/ownership preserved, no uid remapping) in Qemu.qemuArgs.
Systemd.render_start never quotes ExecStart= args (found only once #1–#3 above were fixed and the VM booted but SSH still never answered): it joins args with a bare Text.unwords. -append’s value is one multi-word string; unquoted in the unit file, systemd’s own ExecStart= parser splits it back into several separate qemu arguments, so the kernel cmdline silently never arrives intact. Fixed by quoting any arg containing whitespace/shell metacharacters in Systemd.hs. This is a general Systemd.hs bug, not qemu-specific — just never exercised before since nothing else here passes a multi-word single ExecStart= argument.

A fifth issue was in the test setup, not production code: the original plan (§4 step 2, now superseded) had a human manually copying their own ~/.ssh/id_ed25519.pub into the rootfs’s authorized_keys. Running the privileged tier under sudo doesn’t forward the invoking user’s ssh-agent, so pubkey auth via a personal key silently never succeeds and waitForSsh just times out. Fixed by having withVm generate its own ephemeral SSH CA + signed client key per boot (Test.Harness. ensureVmSshAccess, using the existing but previously-unused Keys.sshKey/Keys.signKey CA-signing primitives) and provision the guest’s sshd to trust it (TrustedUserCAKeys + PasswordAuthentication no drop-in) — no manual key step needed any more. This surfaced one more bug along the way: Keys.signKey never passed -n <principal> to ssh-keygen -s, and modern OpenSSH (checked against 9.6p1) hard-rejects a certificate with an empty principal list (Certificate lacks principal list) — contrary to older folklore that an empty list means “valid for any principal.” Fixed by adding a [Principal] parameter to signKey (safe: grep confirmed nothing else in the codebase called it yet).

1. Files touched so far

File	State	What
`salmon-ops/src/Salmon/Builtin/Nodes/LinuxBridge.hs`	new	`bridge`, `tap`, `bridgeAddr` ops.
`salmon-ops/src/Salmon/Builtin/Nodes/Qemu.hs`	new	`VmConfig`, `resolveKernelInitrd`, `qemuArgs`, `setup`. Boot-validated fixes: `security_model=passthrough`, `mount_tag=vroot`/`root=vroot`, `net.ifnames=0 biosdevname=0` baked into `kernelCmdline`, `-cpu host` paired with `-enable-kvm`.
`salmon-ops/src/Salmon/Builtin/Nodes/Debian/Debootstrap.hs`	modified	`vmEssentials :: Includes` (kernel + openssh-server), plus new `ensureVm9pBoot` op (9p initramfs modules + `update-initramfs` via chroot).
`salmon-ops/src/Salmon/Builtin/Nodes/Systemd.hs`	modified	`render_start` now quotes `ExecStart=` args containing whitespace/shell metacharacters — real bug fix, not qemu-specific.
`salmon-ops/src/Salmon/Builtin/Nodes/Keys.hs`	modified	`signKey` gained a required `[Principal]` parameter (`-n` to `ssh-keygen -s`) — modern OpenSSH rejects principal-less certs.
`salmon-ops/salmon-ops.cabal`	modified	registered `LinuxBridge` and `Qemu` in `exposed-modules`.
`salmon-ops-recipes/test/Test/Harness.hs`	modified	Layer 3 section: `testBridge`/`testBridgeCidr`/`testVmAddr`/`testVmAddr2`/`ensureTestBridge`/`withVm`/`withVmAt`, `VmAccess`/`sshToVm`/`scpToVm` (replaces the old bare-`Ssh.Remote` interface — see §0 item 5), `ensureVmSshAccess`. `withVm` is now `withVmAt testVmAddr`; `withVmAt` takes the guest address as a parameter so more than one VM can be up at once on the shared test bridge (nested calls, one address each) — added for §3 item 4. `quoteForRemoteShell` added 2026-08-21 (see §0.1 item 4) — single-quotes a caller's `sshToVm` argument so it survives ssh's own space-join as one remote token; not applied automatically inside `sshToVm` itself (would break `Test.QemuSmokeSpec`'s existing `["echo smoke-ok"]` call, which relies on the old join-then-resplit behavior).
`salmon-ops-recipes/test/Test/QemuSmokeSpec.hs`	new	Layer 3 smoke test: boots the smoke rootfs via `withVm`, asserts SSH answers. Skips loudly (not fail) without root, without `qemu-system-x86_64`, or without a pre-built rootfs at `/var/lib/salmon-test-vms/smoke/root`.
`salmon-ops-recipes/test/Test/DebootstrapSpec.hs`	new	Layer 3: runs `Debootstrap.rootTree` `inject` `Debootstrap.ensureVm9pBoot` as real `Op`s (not the equivalent hand-run chroot script) against `/var/lib/salmon-test-vms/debootstrap-op-smoke/root`, checks the 9p modules got written, then reruns once more to confirm idempotency. Closes §3 item 2. Gated behind root/`debootstrap`-on-PATH (skip loudly) plus, for the first (network-heavy) run only, the opt-in env var `SALMON_TEST_RUN_DEBOOTSTRAP=1` — deliberately does not wipe the rootfs between runs, so once debootstrapped once, every later run is offline/seconds-long via the two ops' own `prelim`s. Confirmed passing under `sudo` (`1562.57s` first run, network-bound).
`salmon-ops-recipes/test/Test/PostgresReplicationSpec.hs`	new	Layer 3 port of the hand-run `salmon-ops/fixtures/PostgresReplicationFixture.hs`: boots a primary VM (`testVmAddr`) and a standby VM (`testVmAddr2`) via `withVmAt`, `scpToVm`s the already-built fixture binary (resolved via `cabal list-bin`, no hardcoded path) onto each, drives it over SSH exactly like the old fixture's manual `podman exec` steps, then — unlike the old fixture, which just told a human to eyeball `psql` — polls `pg_stat_replication` for `streaming`, inserts a row on the primary, and polls the standby until that row actually shows up. Closes §3 item 4 (2026-08-21) — confirmed passing under `sudo` for real (~100s), after the four bugs in §0.1. On a failed fixture run or a `waitForStreaming` timeout, dumps `pg_lsclusters`/postgres logs/`pg_stat_replication`/`pg_stat_wal_receiver`/a standby→primary ping into the failure message (failure-path only, no extra SSH round-trips on the success path).
`salmon-ops-recipes/test/Test/QemuResolveKernelSpec.hs`	new	Layer 1, no root/VM needed: builds a scratch `boot/` dir via `temporary`'s `withSystemTempDirectory` and checks `Qemu.resolveKernelInitrd`'s three cases (one match resolves, zero throws, two — a held-over old kernel — throws `"ambiguous"` rather than silently picking one). Closes §3 item 3 (2026-08-21).
`salmon-ops-recipes/test/Main.hs`, `salmon-ops-recipes.cabal`	modified	wired `Test.QemuSmokeSpec`, `Test.DebootstrapSpec`, `Test.PostgresReplicationSpec`, `Test.QemuResolveKernelSpec` in; added `unix` to test-suite `build-depends` (for `getEffectiveUserID`).

cabal build salmon-ops salmon-ops-recipes salmon-ops-recipes:test:salmon-ops-recipes-test and cabal test salmon-ops-recipes (non-root; all Layer-3-needing tests vacuously skip, everything else including a real Layer 2 podman test passes — 17 tests, all green) both clean as of 2026-08-20. The sudo-run Qemu and Debootstrap tests are the real, non-vacuous confirmations (see §0, §3 item 2); the Postgres replication test is written and building but not yet run for real (see its row above).

2. Environment state (this machine, checked 2026-08-20)

iproute2 (ip), qemu-system-x86 (provides qemu-system-x86_64), debootstrap (1.0.134ubuntu2): all installed.
KVM available and now exercised: /dev/kvm exists, 40 vmx/svm flags in /proc/cpuinfo. Test.Harness.withVm now defaults to vm_enable_kvm = True with -cpu host (see §3.1 for the fix history); confirmed passing under sudo, boot+SSH completing in single-digit seconds in the two runs measured so far, vs. ~80–140s for the earlier TCG-only runs.
The smoke rootfs lives at /var/lib/salmon-test-vms/smoke/root, built via (see §0 item 2 for why ensureVm9pBoot’s initramfs fix also needs applying — the rootfs on disk currently has that fix hand-applied via chroot, not yet re-derived by actually running the new Debootstrap.ensureVm9pBoot op against it):
```
sudo debootstrap --include=linux-image-amd64,openssh-server stable /var/lib/salmon-test-vms/smoke/root
```
No authorized_keys provisioning needed any more (§0 item 5) — withVm handles its own access.
A second rootfs, /var/lib/salmon-test-vms/debootstrap-op-smoke/root, built by Debootstrap.rootTree/Debootstrap.ensureVm9pBoot themselves (via Test.DebootstrapSpec, see §1) rather than by hand — this is the one that actually proves those ops work, as opposed to the smoke rootfs above which still carries a hand-applied 9p fix.
/var/lib/salmon-test-vms/pg-primary/root and /var/lib/salmon-test-vms/pg-standby/root (§1) built 2026-08-21 — postgres is baked in at debootstrap time rather than apt-installed inside the guest at test time, since the test bridge has no NAT/internet route out of a guest past boot. Rather than debootstrapping each separately (they need the identical package set), built once as a pg-master rootfs and copied out twice:
```
sudo debootstrap --include=linux-image-amd64,openssh-server,postgresql,sudo stable /var/lib/salmon-test-vms/pg-master/root
# apply Debootstrap.ensureVm9pBoot's fix (see §0 item 2) to pg-master once
# chroot-install postgresql-client into pg-master (see §0.1 item 2 — not
# pulled in by the `postgresql` meta-package the way postgresql-client-17 is)
# force `port = 5432` in pg-master's postgresql.conf (see §0.1 item 3 —
# pg_createcluster auto-picked 5433 since the chroot install shares the
# host's own network stack/port-in-use probing)
sudo rsync -aHAX --numeric-ids /var/lib/salmon-test-vms/pg-master/root/ /var/lib/salmon-test-vms/pg-primary/root/
sudo rsync -aHAX --numeric-ids /var/lib/salmon-test-vms/pg-master/root/ /var/lib/salmon-test-vms/pg-standby/root/
```
Safe here specifically because Test.Harness.sshToVm uses StrictHostKeyChecking=no/UserKnownHostsFile=/dev/null (no host-key verification, so duplicate host keys across the two copies don’t matter), and postgresql’s postinst doesn’t start the service or write instance-specific state during a chroot debootstrap (services don’t autostart in a chroot) — the copied data directories are just the vanilla package-created default cluster; primary vs. standby role is entirely a runtime distinction made by the fixture, not baked into the rootfs.

3. What’s still open

~~KVM never actually exercised end to end.~~ Done (2026-08-20). Flipped Test.Harness.withVm’s Qemu.vm_enable_kvm to True and reran QemuSmokeSpec under sudo — passed. First KVM pass was noticeably slower than the proven TCG baseline; root cause: -enable- kvm was passed without -cpu host, so the guest still ran the generic emulated qemu64 CPU model — KVM only pays off once the guest actually gets a KVM-aware CPU model. Fixed in Qemu.qemuArgs: if cfg.vm_enable_kvm then ["-enable-kvm", "-cpu", "host"] else []. Rerun after the fix passed in ~5.5s (one earlier run hit ~3.75s) — both far faster than the ~80–140s TCG figure recorded in §2, though with only two data points this could partly be host-side caching rather than a clean KVM-vs-TCG comparison; not worth chasing further unless boot time becomes a problem again. vm_enable_kvm = True is now the harness default.
~~Debootstrap.ensureVm9pBoot itself is untested as a salmon Op.~~ Done (2026-08-20). Test.DebootstrapSpec runs Debootstrap.rootTree inject Debootstrap.ensureVm9pBoot for real against a fresh rootfs, checks the 9p modules land, and reruns once more to confirm both ops’ prelims report Skippable the second time — passed under sudo (1562.57s, almost entirely debootstrap’s own package downloads). Gated behind SALMON_TEST_RUN_DEBOOTSTRAP=1 for that first real run and does not wipe the rootfs between runs, so it doesn’t silently re-fetch ~100+ packages (and burn a metered connection) on every suite run — see its row in §1.
~~resolveKernelInitrd’s prefix match~~ Done (2026-08-21). Added Test.QemuResolveKernelSpec — a Layer 1 (pure filesystem, no root/VM/debootstrap) test that builds a scratch boot/ dir via withSystemTempDirectory and checks all three cases: exactly-one-match resolves, zero matches throws "no vmlinuz-*", and two matches (simulating a held-over old kernel) throws "ambiguous vmlinuz-*" rather than silently picking one. All three pass. No rootfs/VM bugs found — resolveKernelInitrd’s existing ambiguity handling was already correct, just previously unverified.
~~Phase 5 of the spec: pick a real recipe Layer 2 can’t exercise well and write its first real Layer 3 test.~~ Done (2026-08-21). Test.PostgresReplicationSpec (§1) ports salmon-ops/fixtures/PostgresReplicationFixture.hs’s hand-run, eyeballed podman flow onto two real qemu VMs with actual pass/fail assertions (pg_stat_replication reaches streaming, a row written on the primary shows up on the standby) — confirmed passing under sudo for real (~100s), after the four bugs in §0.1 (a cabal list-bin output-parsing bug, a missing postgresql-client package, a wrong postgres port, and a silent sshToVm argument-quoting bug affecting every multi-word remote command this test ran). Needed generalizing withVm into withVmAt (a caller-chosen guest address) so two VMs can be up at once on the shared test bridge, plus a new scpToVm to get the compiled fixture binary onto each guest.

4. Design decisions made while implementing (not equally emphasized in the spec)

bridgeAddr/Cidr in LinuxBridge.hs — not in the original spec write-up, added because the host side of the test bridge needs an address for SSH to route through. Same idempotency shape as bridge/ tap (ip addr show dev grep via prelim).
- Test subnet: 10.99.0.0/24, host (bridge) 10.99.0.1, guest fixed at 10.99.0.2 (Test.Harness.testBridgeCidr/testVmAddr) — single-VM- at-a-time assumption for v1, matching the spec’s “prove the tier end to end” framing before anything like an address pool.
- Bridge name: salmontest0, left standing across test runs (persistent, matching the spec’s leaning in its bridge-lifecycle open question).
Qemu.setup’s systemd unit runs qemu as root:root (Test.Harness.withVm sets vm_user/vm_group = "root") rather than threading through LinuxBridge.tap’s tapOwner for an unprivileged user — simplest given this whole tier already assumes privileged execution (documented prerequisite, per the spec’s privilege open question), avoids a second permissions mechanism to get right on the first pass. This also motivated the security_model=passthrough choice in §0 item 3.
Graceful VM shutdown via the qemu monitor socket (spec §2) is not implemented — down goes through plain systemctl stop, i.e. SIGTERM. Explicitly called out as acceptable for v1’s disposable-VM use case in Qemu.hs’s module haddock; revisit if abrupt termination ever causes a real problem (e.g. corrupting guest filesystem state between runs).
Test SSH access is now entirely withVm’s own responsibility (a fresh per-boot CA + signed key, see §0 item 5) rather than something the caller pre-provisions — a deliberate narrowing from the original “caller supplies authorized_keys” plan once the sudo/agent problem showed up in practice. Production recipes (e.g. a real CA-backed service) still follow the project’s key-exchange-agnostic convention; this only changes how the test harness itself authenticates to its own disposable, harness-owned VM.

Validating the GCP builtins against a real project

2026-09-25T14:15:36Z

Generated from resources/gcp-toy-validation.md — the repository is the canonical source, and may be ahead of this page.

Validating the GCP builtins against a real project

This is the playbook for running salmon’s Salmon.Builtin.Nodes.Gcp.* builtins against real GCP, in a sandbox you are willing to destroy. It exists because the GCP nodes’ automated tests are all Layer 0: they check the pure verdict functions (interpretInstanceStatus, interpretBillingDescribe, …) and the rendered gcloud argument lists. Nothing in cabal test ever talks to Google, so “does an up actually converge, and is it idempotent, and does down take it all away” is a question only a real project can answer.

Three pieces do that:

salmon-apps’s salmon-gcp-toy binary (salmon-apps/src/GcpToy.hs), a tiered, throwaway stack built out of the Gcp builtins, driven through the usual seed → directive → ops protocol.
salmon-apps/scripts/gcp-toy-validate.sh, which runs that binary through up → up → down and reports what each pass did.
salmon-apps/scripts/gcp-toy-serve.sh, which keeps that binary up as a run serve server instead — the HTTP surface, the web UI, salmon-tui — and changes what it wants one word at a time (tier0, tier1, tag v2, down), with tier 2’s two passes driven through pull mode by editing a registry document. Use it once tier 0 is clean under the validator.

The blast radius is bounded by the project: by default the toy creates the project it works in, which makes the project the deepest node of the graph, so run down tears every resource down individually first (that is the part being validated) and then deletes the project, which sweeps anything a buggy down left behind.

What the toy declares

Tiers are cumulative, ordered by cost. Pick one with --tier.

Tier 0 (≈ free — nothing here is billed beyond negligible storage):

Node	Resource
`Gcp.Core.applicationDefaultCredentials`	validates ADC before anything else runs
`Gcp.ResourceManager.project`	the project itself (skipped with `--existing-project`)
`Gcp.Billing.linkBillingAccount`	links the billing account
`Gcp.ServiceUsage.enableService`	`storage`, `iam`, `artifactregistry` (and `run` at tier 1, `monitoring` with `--alert-email`)
`Gcp.Storage.bucket`	`<project>-<prefix>` , uniform bucket-level access
`Gcp.Iam.serviceAccount`	`<prefix>-sa@<project>.iam.gserviceaccount.com`
`Gcp.ArtifactRegistry.artifactRepository`	`<prefix>-repo`, docker format
`Gcp.Iam.iamBinding` ×2	`storage.objectViewer` on the bucket, `artifactregistry.reader` on the repo

Tier 1 (cents) adds SreBox.Gcp.CloudRunDeploy.buildPushDeploy: podman builds an image, logs in to <region>-docker.pkg.dev with an isolated authfile, pushes, and Gcp.CloudRun.cloudRunService deploys it as <prefix>-hello running as the tier-0 service account, --max-instances 1.

The image is either FROM --base-image (Google’s cloudrun/container/hello sample by default, plus a label carrying --image-tag so each tag really is a distinct image) or your own --containerfile PATH, built with that file’s directory as the podman build context. Whatever you deploy must serve HTTP on $PORT and be linux/amd64, or the revision never becomes ready. The service is deployed without --allow-unauthenticated: the toy asserts the deploy happened and runs the expected image, it never issues an HTTP request to it.

With --alert-email ADDRESS, tier 1 also declares SreBox.Gcp.CloudRunAlerts.standardAlerts on the service: one Gcp.Monitoring.notificationChannel (<prefix> alerts, an email channel to that address) and four Gcp.Monitoring.alertPolicy nodes — <prefix>-hello: 5xx ratio, p99 latency, memory and, since the service has --max-instances 1, instances at max. Both resources are addressed by display name (Cloud Monitoring assigns the ids), and a policy carries a salmon-fingerprint user label of what it was rendered from: a second pass skips the four, editing one in the console makes the next pass update it back, and run down deletes what a lookup by name finds. Alerting is free; what this tier exercises is create, skip, update and delete against the real API, which the Layer 0 tests on the rendered gcloud argv and policy JSON cannot.

Tier 2 (an e2-micro’s hourly rate, plus a reserved IP) is specs/gcloud-support.md §6’s “objective”: a VM salmon boots, trusts and then provisions with this same binary.

Node	Resource
`Gcp.Compute.address`	a reserved regional external IP, `<prefix>-ip`
`Gcp.Compute.firewallRule`	`tcp:22` from `--ssh-source-range` to instances tagged `<prefix>-ssh`
`Filesystem.filecontents`	the startup script, passed as `--metadata-from-file`
`Keys.sshKey` ×2, `Keys.signKey`	a CA and a client key, and a certificate for `--vm-user`
`Gcp.SshAccess.installMetadataCaKey`	the CA's public key, into project metadata
`Gcp.Compute.gceInstance`	the VM, claiming the address and carrying the tag
`Gcp.SshAccess.sshAvailable`	waits for sshd to answer as that user, with that certificate
`Self.uploadAndCallSelfAsSudoWith` (via `SreBox.Gcp.VmProvision.provisionedVm`)	rsyncs this binary over and runs `run up` on it there

The startup script is what closes the gap installMetadataCaKey leaves: nothing on a GCE instance reads that metadata key by itself. It fetches the CA from the metadata server into /etc/ssh/salmon_ca.pub, points sshd’s TrustedUserCAKeys at it, creates the login user the certificate names as its principal (with no OS Login, a principal must be a local account), gives it passwordless sudo, and makes sure rsync is there for the upload.

Tier 2 runs in two passes, and the script drives both. GCP picks the address, so the first pass reserves it and stops; the driver then reads the IP (gcloud compute addresses describe, the call Compute.readAddress wraps) and re-issues the directive with --vm-ip, and the second pass declares the same graph plus the provisioning step. That is not a wart of the toy: an Op naming the host has to be built before any up runs, so something outside the graph has to carry the address across.

What proves it worked is the file the uploaded binary writes on the VM, /var/lib/salmon-toy/provisioned; the script reads it back over ssh. The binary runs there with the same directive, tagged OnVm, which is why the payload is declared in the same Track' as everything else.

Tier 3 (a forwarding rule’s hourly rate on top of tier 2) puts a regional external Application Load Balancer in front of that VM.

Node	Resource
`Gcp.Compute.subnet`	the proxy-only subnet, `<prefix>-proxy`, `REGIONAL_MANAGED_PROXY`/`ACTIVE`
`Gcp.Compute.instanceGroup`	an unmanaged, zonal group, `<prefix>-ig`
`Gcp.Compute.instanceGroupMember`	the tier-2 VM, put in it
`Gcp.Compute.firewallRule`	`tcp:<--lb-port>` from the proxy range and the health-check ranges, to instances tagged `<prefix>-lb`
`Gcp.LoadBalancing.applicationLoadBalancer`	health check, backend service, named ports, URL map, target proxy, forwarding rule
`Systemd.systemdService` (on the VM)	`salmon-toy-web.service`, a `python3 -m http.server` over a page salmon wrote

Three of those exist only because a regional external ALB is an Envoy fleet rather than a Google frontend, and that is what the tier is really testing: the proxies run inside the VPC, in a proxy-only subnet that must already exist in the region; they reach the backends from that subnet’s range, so the backend firewall has to allow it — as does the separate 35.191.0.0/16 + 130.211.0.0/22 pair the health checks come from, which is a different source entirely and the usual reason a balancer that came up cleanly still answers 502; and a VM is not a backend, an instance group is.

The web server is declared on the VM side, by the tier-2 payload. That is deliberate: a forwarding rule that merely exists proves nothing, so what the script checks is a 200 carrying the project id, and the only thing that can put that body there is salmon running on the machine. A green tier 3 is therefore a second, independent proof that the tier-2 hand-off worked — this time through the front door.

Step 1 — dry run, no GCP calls

run tree only expands the graph; it never shells out to gcloud. Do this first, with a throwaway billing id, to see what a given set of flags declares:

cd <repo>
cabal build salmon-gcp-toy
TOY=$(cabal list-bin salmon-gcp-toy)

$TOY config --project salmon-toy-$(date +%s) \
      --billing-account 000000-000000-000000 --tier 0 \
  | $TOY run tree

config also validates the flags (project id shape, prefix length, that --containerfile exists, that a created project has a billing account), so a typo fails here rather than half-way through an up.

Step 2 — find the real ids

gcloud billing accounts list   # ACCOUNT_ID -> --billing-account
gcloud organizations list      # ID         -> --organization (omit if you have none)

The billing account must be one the active account can see in that listing and OPEN: True — a closed account cannot be linked, and an id you cannot see fails with a permission error that reads as if the id might not exist. The script pre-flights this (gcloud billing accounts describe) before it declares anything, because linking is the first step that touches something the caller may not own, and by then a project has already been created.

Step 3 — point gcloud at the sandbox

The toy needs credentials twice over: the gcp-adc node checks application default credentials, while every gcloud invocation uses the active account. They are separate logins, and a stale one is the most common cause of a first-run failure.

gcloud config configurations create salmon-sandbox   # leaves other configs alone
gcloud auth login
gcloud auth application-default login
gcloud config get-value account                      # confirm the sandbox identity

The script prints the active account and the ambient project when it starts. The ambient project should not matter: every node passes --project explicitly. If a run only works when the ambient project happens to be right, that is a bug in a node, and worth reporting.

Step 4 — the real run

salmon-apps/scripts/gcp-toy-validate.sh -- \
  --project salmon-toy-$(date +%s) \
  --organization YOUR_ORG_ID \
  --billing-account YOUR_BILLING_ACCOUNT \
  --tier 0

Then, once tier 0 is clean, the same with --tier 1 (needs podman), and/or --containerfile ./myapp/Containerfile to deploy your own app. Tier 2 adds --vm-zone (default <region>-b), --vm-machine-type (e2-micro), --vm-image-family/--vm-image-project (Ubuntu 24.04 LTS), --vm-user (salmon) and --ssh-source-range (0.0.0.0/0 — narrow it to your own address if the sandbox is not disposable). Tier 3 adds --lb-proxy-range (192.168.100.0/24) and --lb-port (8080).

Script options, before the --: -y skips the confirmation prompt, --keep skips teardown (it then prints the run down command to finish up later). Everything after the -- goes verbatim to salmon-gcp-toy config. OUT=<dir> and SALMON_GCP_TOY=<binary> override the log directory and the binary.

Use a fresh project id every run. A deleted project sits in DELETE_REQUESTED for ~30 days and nobody can reuse its id in that window — $(date +%s) in the id is there for exactly this. Gcp.ResourceManager’s check reports that state with its own message rather than as a plain “not found”, because the up that follows is going to fail on the id, not on credentials.

What the passes mean

Pass	What a clean result looks like	What a dirty one tells you
`up` #1	converges first time	a failure triggers one retry; "converged ONLY ON RETRY" means something needed time — IAM propagation after creating a service account, or a just-enabled API — that no node waits out yet
`up` #2	every node with a real `check` reports `Skip`	a node listed as "RE-APPLIED DESPITE A CHECK" has a `check` that never says `Success`, i.e. it is not idempotent under `run up`
`down`	succeeds, then the project is `DELETE_REQUESTED` (or, with `--existing-project`, every resource fails to `describe`)	a failed `down` leaves that node standing and `Blocked`s everything it depends on — including, deliberately, the project delete, so the leftovers are still there to look at

Nodes that legitimately have no check today re-apply on every pass; the script lists them separately rather than counting them as findings. Tier 2 adds the expensive members of that list: rsync:sendfile re-uploads the binary and ssh:call re-runs the remote directive every pass. The remote run is itself idempotent — it streams its own report back over ssh, and on the second pass it reports Skip for its file node, which the script surfaces.

Two nodes also cannot go down on a workstation, by design rather than by accident, and the script says so instead of calling the teardown failed: deb tears down with apt-get remove, which needs root and would uninstall a system package salmon did not put there; and directory refuses a non-empty directory, which the ssh key dir always is, because Keys.sshKey deliberately “keeps keys around”. The logs (gcp-toy-runs/<timestamp>/up-1.log, up-2.log, down.log, plus tree.txt and directive.json) hold the full UpDown report stream.

The script exits non-zero on any failure, retry, unexpected re-apply, or leftover.

Things that will bite

Org policies. A freshly created organization enforces several policies by default. None of the tier 0/1 resources needs a service-account key, which is the usual casualty, but an unexpected up failure mentioning constraints/... is a policy, not a salmon bug.
gcloud prompts. The script exports CLOUDSDK_CORE_DISABLE_PROMPTS=1; a prompt in a non-interactive up would otherwise hang forever.
Bucket names are global. The toy scopes them by project id, so two sandboxes cannot collide, but a name you pick by hand can collide with the whole world.
Permissions. Creating a project and linking billing need resourcemanager.projects.create on the parent and billing.resourceAssociations.create on the billing account (i.e. roles/billing.user, which only a billing administrator can grant — being able to see an account, or to create projects, does not imply it). --existing-project avoids both if you would rather have someone else create the project and attach billing.
A failed run is resumable. Nothing is torn down when up fails twice, and re-running with the same --project reuses the project rather than burning a new id: its check reports ACTIVE and the node is skipped. Fix the cause (or the flag), re-run the same command line, or run down < directive.json to drop what was built.
--workdir is deleted by down. The Containerfile and the podman authfile live there, and the directory node that holds them removes the directory on teardown — so point it at a scratch path, not at a directory with anything else in it. (Tier 1’s first real run failed here: podman logout empties the authfile but leaves it, and the leftover file kept the directory from being removed. Podman.login’s down now deletes the file it caused to exist.)
Eventual consistency is real, and nodes now ride it out. Two cases were found by running this toy: a create issued seconds after its API was enabled is denied for up to a minute (PERMISSION_DENIED ... (or it may not exist), even for a project owner), and a binding naming a just-created service account is rejected by the service owning the resource. Gcp.Core.retryingIO is the shared remedy: bucket/repository/Cloud Run creates retry ~6×10s, Iam.iamBinding 5×3s, and Iam.serviceAccount’s up additionally waits for its own describe to answer. Retries show up in the logs as repeated CommandStart reports for one node, which is how to tell “needed the retry” from “worked first time”.
Tier 2 needs a local glibc no newer than the VM’s. The binary is rsynced and run as-is, so the image family has to be at least as new as the machine running the toy. Ubuntu 24.04 (glibc 2.39) is the default because that is what this was developed against; an older image will fail at exec time with a version error, not at build time.
Tier 2’s --ssh-source-range defaults to the whole internet. A throwaway VM reachable on 22 by anybody, trusting only a certificate, is an acceptable risk for an hour; narrow it anyway when the project is not disposable.
Tier 3’s proxy range must avoid 10.128.0.0/9. The default network is an auto mode VPC, and that whole block belongs to the subnets GCP creates per region on its own — including for regions that do not exist yet. Hence the 192.168.100.0/24 default. It also has to be /26 or larger, and only one ACTIVE proxy-only subnet may exist per network per region, so a second concurrent tier-3 run in the same project (not the same organization) will collide.
A tier-3 backend is UNHEALTHY for a minute or two after up. The balancer answers 502 until the first health checks pass, which is why the script waits up to five minutes for the page rather than fetching once. If it never arrives, the script prints the backend’s health, because the cause is nearly always a firewall rule rather than the balancer.

Gaps this does not cover

The VM’s own teardown is the project delete. down removes the instance, the address and the firewall rule as declared nodes, but nothing checks that the guest was left in any particular state.
Host identity is trust-on-first-use, per recipe. The user is authenticated by certificate, but the host is not: VmProvision keeps a known-hosts file next to the client key and accepts a new host on sight. Signing host certificates with the same CA would close that, and needs ssh-keygen -s -h support in Keys plus a way to get each VM’s host key signed at boot.
The serverless-NEG backend is still unexercised. Tier 3 drives Gcp.LoadBalancing’s InstanceGroupBackend; the CloudRunBackend branch (a serverless NEG in front of tier 1’s Cloud Run service) renders but has never been run. Mixing the two in one balancer is not an option — a backend service holds one kind of backend — so exercising it means a second balancer.
HTTPS, and anything past the default route. The URL map has one default service and the forwarding rule is plain :80; managed certificates, host and path rules, and the --network-carrying form of the forwarding rule are all rendered-but-unrun.
Two credentials, one identity assumed. gcp-adc validates ADC; Core.printAccessToken (the registry login password) uses the active account. Log both in as the same identity.
CloudRun’s check is a substring match on the image, so img:1 matches img:10, and a changed env var or service account is not noticed.

Pull mode: a `serve` that fetches its own declarations

2026-09-25T14:15:36Z

Generated from specs/pull-mode.md — the repository is the canonical source, and may be ahead of this page.

Pull mode: a `serve` that fetches its own declarations

Status: milestones 1 to 7 below are implemented (Salmon.Actions.Follow, Salmon.Actions.Follow.Scheduler, Salmon.Actions.Follow.Registry and its git/HTTP/DNS backends, run serve --follow, --follow-cache, mode in status, --status-sink, salmon-fleet status, the verify-before-inject hook, and signed documents behind --follow-key); the rest is a design sketch to react to, not a committed plan. It grew out of a fleet-management assessment; the companion idea (a generic salmon server with web/terminal clients that render the live Dag) is a separate sketch and is only referenced here where the two meet.

Problem

Today’s run serve only ever waits to be pushed at. Its whole input surface is one Handle: serveWith forks a reader thread that hGetLines into a TChan and the loop reads commands off that channel (salmon-ops/src/Salmon/Actions/Serve.hs, serveWith/readInto/loop). The only way a declaration reaches a remote machine is a controller that ssh-es in — and what it runs there is a one-shot run up with the directive on stdin (Salmon.Builtin.Nodes.Self.callSelf), never a serve.

So the shape we have is hub-and-spoke push: a controller holds every declaration, opens a connection per host per change, and learns an exit code back. That has three limits that get worse with every host added:

Every host needs an inbound path from the controller (ssh, a key, a firewall rule), and the controller must be up at the moment a change is wanted.
A serve on a remote host is unreachable once started. ssh host bin run serve works, but nothing can then talk to it except that one ssh session’s stdin.
There is no fleet-level state. Each serve is an island; nothing answers “which hosts have converged to which declaration”.

The missing mode is the inverse: a host that periodically fetches its desired state from somewhere and converges on it — the kubelet / Puppet-agent shape. It is the cheapest route to a fleet because it needs no control plane and no inbound port on any host. Nothing in it requires consensus; it requires a place to put documents.

controller ──writes──▶ dumb store (file / git / bucket / HTTP)
                            │ fetch                 ▲
              ┌─────────────┼─────────────┐         │ status
        host A: serve   host B: serve   host C: serve
          --follow        --follow        --follow

What the loop already has

Most of a puller’s semantics exist; what’s missing is the transport and one ordering rule.

load <file> runs a file’s lines through the command language, nested loads included, depth-capped (loadFile, maxLoadDepth). A puller is “load from somewhere else, on a trigger”.
only <seed> retires every other seed, up/down add/retire one, and clear retires all. Re-declaring an unchanged seed is a no-op because the ledger unifies by directive (Salmon.Op.Ledger, and Serve.record’s Stale-vs-unchanged comparison).
autoconverge off + converge separates recording declarations from acting on them (Serve.hs, AutoConverge), so a fetched document can be applied as one atomic batch of declarations followed by a single pass, rather than N passes.
up-directive <file> declares from a directive JSON rather than seed args, so a document can carry either spelling.
The reader is the only thing that assumes stdin. serveWith already takes any Handle; the loop reads Maybe String off a TChan. Generalizing the reader to a merged source of lines is a small change.

Design

The fetched thing is a declarative document, not a command log

The document is the desired set: the seeds this host should have live. It is not a sequence of up/down commands, and it is JSON, not seed lines — JSON is what up-directive already speaks, what every tool that might produce or inspect a document (CI, a web UI, jq, a controller written in anything) speaks, and what a signature can be computed over unambiguously. Seed-line spelling stays available inside it, as an array of words, so a document can carry either a seed or a fully-configured directive:

{
  "salmon": 1,
  "id": "web-api@2026-09-23T10:41:07Z",
  "seeds": [
    { "seed": ["base-packages"] },
    { "seed": ["app", "--version", "42"] },
    { "directive": { "...": "a directive JSON, as `up-directive` takes" } }
  ]
}

salmon is a format version; id is opaque, chosen by the publisher, and is what history records (below). Anything else at the top level is ignored by v1 so publishers can annotate — except published, an optional RFC 3339 timestamp that milestone 4 gave a meaning (see the open questions).

Reason for a document rather than a log: a log needs a cursor, exactly-once delivery, and a story for a host that missed the middle of it. A document is idempotent — safe to re-fetch, safe to fetch twice, safe to fetch after a month offline. The puller diffs the document against the live ledger and emits, on the loop’s inbox:

autoconverge off
up   <seed A>        # in document, not live
up   <seed B>        # in document, already live -> no-op by the ledger
down <seed C>        # live, not in document
autoconverge on      # (restore whatever it was)
converge

only is the tempting one-liner but it is the wrong primitive: it retires everything not named, which conflicts with a host following several labels (each document covers only its own label’s seeds) and with an operator who typed something interactively that the document shouldn’t know about. The diff is explicit about what it retires; only is not.

force/recheck/pause/resume are not state and do not belong in the document. They stay on a push channel (stdin today; a socket later) or in a separate, consumed-once side file if there’s a real need.

Labels are addresses into a registry

A label is not a selector inside one big fleet file; it is syntax for addressing the latest document in a larger registry. A host started with labels web-api and canary fetches two documents — “latest for web-api”, “latest for canary” — and its desired set is their union. A registry is anything that can answer “latest document for <label>”, and the label is spliced into an address by a small template the registry backend owns:

registry backend	how `<label>` becomes an address	latest / change detection
directory	`<dir>/<label>.json`	mtime + content hash, or inotify
git repo	`<repo>/<label>/latest.json` (a subdirectory per label; history is git's)	commit id
HTTP	`https://controller.example/seed/latest/<label>`	ETag / `If-None-Match`, else hash
bucket	`gs://<bucket>/<label>/latest.json` (`Gcp/Storage`)	object generation
DNS	`<label>.<zone>`, e.g. `web-api.controller.salmon.example`	see below

The DNS backend is the “hack” worth spelling out because it is cheap to poll and salmon already runs DNS (SreBox.MicroDNS, SreBox.DNSRegistration), so a controller can publish records with nodes that exist today. Two shapes:

Index only. A TXT record at <label>.<zone> carrying v=salmon1 url=<where the JSON is> sha256=<digest>. The host polls DNS (one UDP round-trip, cached by TTL, no connection to the controller) and fetches the payload only when the digest changes. DNS is the registry’s index; HTTP/git/bucket is its storage. This is the recommended shape.
Inline. For a document small enough, the TXT record is the document (base64, chunked at 255 bytes as TXT allows). Fine for a one-seed label; not the general case.

Either way the record’s TTL is a natural, controller-chosen lower bound on how fast a change propagates, and the zone’s serial is a fleet-wide “did anything change” bit for free.

Every backend above is already something salmon can do as an op. That is the point of the second observation: the agent’s inbox can be a node in its own graph — “the latest document for <label> from <registry> is at this local path” is a Filesystem/Git/Storage/Web node with the usual check/up/failure reporting, and the puller reads a local file that this node keeps fresh. That gets fetch failures into the same Report stream as everything else instead of a separate log.

Labels are given at start (--follow <registry> --label web-api --label canary); a label file the host re-reads on each round is a cheap extension so that re-labelling doesn’t need a restart, and is not v1.

Two labels whose documents disagree about one effect site are not a registry-level error: both seeds are declared, and the Dag reports the collision as Conflicting exactly as it would for two interactive ups. The registry is not where that is resolved.

A directory registry ships first (it’s also the test harness: write a file, watch the world change), git second (the natural “desired state is a repo” workflow), HTTP and DNS-index after, bucket last.

Change detection happens before injection — the one rule that’s easy to get wrong

Every line that arrives on the inbox stops tending: loop calls stopTending before handling any command, by design (a command is about to act on the nodes). A naive poller that injected on every tick — even “nothing changed” — would therefore starve the supervisor: at a 30s poll the machines would never reach their 60s check ceiling, and a managed node’s watch would be interrupted every tick.

So the fetcher hashes / ETags what it got and injects only on change. An unchanged poll must be invisible to the loop. Corollary: the poll interval and the tending loop are independent; polling can be aggressive without costing supervision anything.

A scheduler owns the rounds: backoff toward the registry, debounce toward the loop

Polling rounds must be decoupled from inbound events, because there will be both: a round is not triggered by a line arriving on stdin or a socket, and a line arriving does not reset the round clock. The scheduler is one thread per followed registry producing “fetch now” ticks, and it has two jobs that point in opposite directions.

Toward the registry: exponential backoff on failure. A round that fails (unreachable, 5xx, unparseable, signature bad) schedules the next one at min(cap, base · factor^n) with jitter; a round that succeeds resets n. Configuration is the usual four numbers (base, factor, cap, jitter) with defaults in the tens-of-seconds to minutes range. This protects the registry from a fleet hammering it during its own outage, and the jitter spreads a fleet that all rebooted at once. A successful round that observed no change keeps the base interval; there is no reason to slow down while quiet, and the base interval (plus DNS TTL, for that backend) is already the propagation bound the operator chose.

Toward the loop: debounce on change. A change observed in a round does not inject immediately. The scheduler opens a quiet window (debounce, default a few seconds, configurable up to minutes) and injects the latest document seen once no further change has been observed for that long, with a max_wait after which it injects regardless. This coalesces a publisher that writes three times in a row (a CI job pushing per-label documents one after another, an operator saving a file twice), so the controlled system sees one diff-batch and one convergence pass, not three — and so a half-published state is never applied. It also bounds how often the supervisor is stood down by document traffic, which is the starvation rule above restated as a rate.

Both knobs live on the follow, not on the seed: --follow <registry> --poll 30s --backoff 5s..10m --debounce 5s. A fetch command on the loop’s input language triggers a round out of schedule (an operator who just published and doesn’t want to wait), and is the one place inbound events touch the scheduler.

Upkeep has a ladder of the same shape (double to a cap, halve to a floor), but it is a per-node question about checks; this one is per-registry about fetches. They should not share code beyond a small Backoff value type.

Merged input, one inbox

Rather than a second loop, the reader becomes a set of producers into the existing TChan:

stdin (today’s behaviour, unchanged, and still what a piped script uses);
the fetcher, injecting a diff-batch on change;
later, a socket (the generic-server sketch).

The loop does not care which producer a line came from. This preserves the property the loop’s own comment defends: a piped script has every line queued before the first pass ends and is never supervised. A fetcher whose first fetch happens before the loop starts (i.e. on startup, synchronously) behaves the same way — deterministic first convergence — and only later changes arrive “live”, handled by the tending loop as any interactive command would be. status should say which it is (see “what this doesn’t solve”).

Groups, canaries and rollouts are registry writes

Because a label addresses a document, host groups, canaries and staged rollouts are writes to the registry, with no service tracking membership: publish app --version 42 under canary, watch the status sink, then publish it under web-api. A host in both groups gets the union, which for two versions of one app is a Conflicting in its Dag — visible, and the publisher’s mistake to fix, not the registry’s. This is also the first meaningful use of a unified Remote type (today Self, Ssh and Rsync each define their own {user, host} record): a host is a name plus its labels.

The fetcher is an actor in `history`

Every declaration the fetcher makes is recorded in history with its provenance — registry, label, document id, digest — as a distinct actor from an operator’s typed line or a loaded file. LogEntry grows an origin (Typed | Loaded path | Fetched registry label id digest), and history prints it. Cheap, and it is the only way an operator can tell “I typed this” from “the document said so” when a host does something surprising.

Status flows back the same way

After every pass the host pushes its status snapshot — the same JSON the generic-server sketch wants for its Dag endpoint — to a sink: a file, an HTTP POST, or a bucket object keyed by host. Fleet status is then a fold over those objects, computed by whoever reads the store (a script, the web UI, a salmon-fleet status subcommand), not by a running service. The store is the only shared dependency and it’s a dumb one.

Same rule as fetching: pushing status is itself an op (a filecontents, a Storage upload), so a sink that’s down shows up as a Failed node, not a silently stale dashboard.

Signed documents

Pulling inverts trust: today a host trusts whoever holds an ssh key to it; in pull mode it trusts the source. TLS to the store covers transport. For the document itself, the tree already has JWT signing (SreBox.JWTSigning), Keys, and Certificates — a document can carry a detached signature the puller verifies against a key it was started with, before any line is injected. A document that fails verification is reported and ignored; the last good one stays in force. Not in v1, but the hook (verify-before-inject) should be there from the start so it’s a function to fill in, not a restructuring.

Shipped (milestone 7 below: Salmon.Actions.Follow.Signature, --follow-key, salmon-fleet keygen/sign, Test.FollowSignatureSpec), with deviations listed there. The one worth reading here: the key is a JWK, not something Certificates produces — the tree’s JWT signing (SreBox.JWTSigning) is HMAC over a shared secret, which a host cannot be handed without also handing it the power to sign, and Keys.jwkKey is the one public-key format already written by the tree; the signature is EdDSA over Ed25519 through jose, not a JWS.

Bootstrap is the existing push pattern, once

Self.uploadSelf, then ssh host bin run serve --follow <registry> --label … under a Systemd.systemdService unit (which gives restart-on-crash and survives reboots). After that the controller only writes documents and never ssh-es again. The push pattern isn’t replaced; it’s demoted to “day zero”.

Interaction with the rest of the loop

stopTending before every command — honoured unchanged; that’s why change detection is load-bearing (above).
Persistence. A host that can’t reach the store keeps converging on its last fetched document — the right behaviour — but only if that document survives a restart. World is an IORef today; a puller that forgets on restart is worse than no puller (it comes up empty, tears nothing down, and looks converged). The fetcher should at minimum cache the last verified document on disk and replay it on start; the journal for World proper is its own item and should land first or alongside. The cache shipped in milestone 4 (--follow-cache); the World journal has not, and is still its own item.
supervise off/autoconverge off typed interactively should be respected by the fetcher — it must not silently re-enable either. The diff batch reads the current setting and restores it.
Rewrites (Salmon.Op.Rewrite) already run once per convergence pass over the whole ledger; a document-driven batch is exactly one pass, so batching works as it does today with no change.
Two operators. An interactive up for a seed the document doesn’t mention is left alone by the diff (it only retires seeds it previously declared — the fetcher owns a contribution in the ledger’s sense, and retires only its own). This is the ledger’s set-not-refcount semantics doing its job.

What this does not solve

Liveness / consensus. Nothing decides a host is dead, same as today and deliberately (see SreBox.PostgresPair’s reasoning). A host that has stopped pushing status is stale in the sink, which is a visible fact, not a decision.
The store’s availability is one shared dependency. That is a storage problem (pick a durable store), not a control-plane problem.
Mid-pass arrivals. A document change that arrives while the loop is idle is handled by the tending loop like any interactive command, not replayed deterministically. Correct, but status should report mode: following vs mode: replay so a test or an operator knows which guarantees apply. It does, as of milestone 4, with one shift in what replay means: not “the startup round was applied synchronously” (that is always true, and following covers it) but “the registry could not be reached at startup and the world is the cached document” — the case an operator actually needs to be told about. interactive is the third value, for a loop with no --follow at all.
Secrets in documents. The document names seeds; seeds that need secret material should keep using pre-provisioned files (see the recipe key-exchange-agnostic convention), not inline them.

Non-goals (v1)

A server that pushes to hosts (that’s the generic-server sketch, and a socket per host).
Rollout orchestration (wait for A before B) beyond what labels + registry writes give.
Any selector or query language: a label is an address, nothing more.
~~Signing (hook only).~~ Shipped in milestone 7; still out: rotation and revocation beyond “several --follow-key flags”, signing inside a registry.
A label file re-read at runtime (labels are start-time flags in v1).

Decisions taken

JSON documents, not seed lines, for tool support and for something a signature and a digest can be computed over; seed-line spelling survives as a word array inside them.
The fetcher is a first-class actor in history, with registry, label, document id and digest.
A scheduler decouples rounds from inbound events, with exponential backoff (base, factor, cap, jitter) toward the registry and a debounce window (plus max_wait) toward the loop — the first protects the registry, the second the controlled system.
Labels address documents in a registry; the registry backend owns the template that turns a label into an address (a directory, a git subdirectory, an HTTP path, a bucket prefix, or a DNS name under a zone).

Open questions

DNS-index record format: one TXT with url= and sha256= as sketched, or a URI record plus a TXT digest? And whether the inline-document variant is worth having at all. Settled as sketched (milestone 6): one TXT at <label>.<zone> reading v=salmon1 url=<https url> sha256=<hex>, fields in any order after the version, a record longer than one string joined the way TXT readers do. Two things decided it. A single record is one lookup and one atomic write for the publisher — a URI plus a TXT can be observed with one updated and the other not, which is exactly the “index announces what the store does not serve” state the digest check refuses, only now manufactured by the index itself. And the sha256= field is the change detection: the record’s digest is the stamp, so a round that finds it unchanged makes no HTTP request at all, which a URI record would not carry. The URI variant was not done; nothing stops a later reader from also accepting it. The inline-document variant was not done either and is not planned: base64 chunked over 255-byte strings for a document that is JSON anyway buys nothing a small HTTP store does not, and the record size would bound the document.
Does the document id need to be ordered (so a host can refuse to move backwards if a registry serves a stale copy from a lagging replica), or is “latest is whatever the registry says” enough? Leaning: an optional published timestamp, refuse-older as a flag, off by default. Settled as the leaning says (milestone 4): id stays opaque; a document may carry published (RFC 3339) at its top level, and under --follow-refuse-older a fetched document published before the one already applied or pending for its label is reported Stale and not injected. Off by default; without published on both sides the latest is whatever the registry says. Two details worth knowing: the comparison is against the pending document when there is one, not only the applied one, since “do not move backwards” has to hold inside a quiet window too; and a published that does not parse is a malformed document rather than an ignored annotation, because a flag that silently skipped a mistyped timestamp would not be doing its one job.
Whether debounce should also apply to the first fetch at startup (probably not: startup wants the deterministic synchronous fetch, and there is nothing to coalesce yet). Settled as the leaning says: the startup round is synchronous and what it finds is injected at once, before standard input is released; the window applies from the first scheduled round on. A restart therefore applies the registry’s current document immediately, even if the publisher is mid-write — which is the same exposure milestone 2 had, and the price of a deterministic first convergence.

Suggested milestones

Reader generalization. serveWith takes a list of line producers instead of one Handle; stdin is one producer. No behaviour change; Test.ServeSpec still passes untouched.
Document format + directory registry. The JSON shape, `–follow --label `, one document per label, union across labels, mtime+hash change detection, diff-batch injection, fetcher-owned contribution, fetcher origin in `history`. Layer 1 test: write the file, assert the world; rewrite identical content, assert *no* command was injected (the starvation rule, as a test). *Shipped* (`Salmon.Actions.Follow`, `Test.FollowSpec`), with two deviations: the batch is one structured `Serve.Batch` inbox entry rather than text lines (so the loop, which alone knows the `autoconverge` setting, restores it, and no other producer's line can land mid-batch), and the "fetcher-owned contribution" is computed in the fetcher as the union across its labels — the ledger keys a declaration by its directive, so it cannot retire "only its own" copy of a seed an operator also typed. Rounds ran on a fixed `--follow-interval` until milestone 3.
Scheduler. Backoff with jitter toward the registry, debounce with max_wait toward the loop, the fetch command. Test: three writes inside the window yield one pass; a failing registry is polled on the ladder, not the base. Shipped (Salmon.Actions.Follow.Scheduler, Test.FollowSchedulerSpec), with three deviations: the knobs are six flat flags (--follow-base/-factor/-cap/-jitter/-debounce/-max-wait, with --follow-interval kept as the base’s older name) rather than the --poll 30s --backoff 5s..10m spelling above; a label with no document is not a failed round (the registry answered), so only a throwing registry or unparseable bytes climb the ladder; and the ladder’s first rung is the base itself (min(cap, base · factor^(n-1))), so a single failure is retried no later than a success would have been. One consequence for the loop: Batch carries an origin per command, since a window can close over several labels at once and history must still say which document each declaration came from.
Cached last document + mode in status. Shipped (--follow-cache DIR, --follow-refuse-older, Serve.Mode, Test.FollowCacheSpec), with four deviations. The cache is written after every injection (bytes, sha256 and id per label, a temp file renamed into place), not after a verified convergence — the fetcher does not learn whether the loop’s pass succeeded, and “the last document the loop was told about” is the right thing to come back to anyway. A label is replayed when its startup fetch fails, which covers the registry throwing and its bytes not parsing (a half-written file at the moment of a restart is the realistic case), but not the registry answering “no document” — the registry answered, and the cache is not deleted either, so it will be replayed the next time the registry is unreachable. replay is entered only at startup and turns to following at the first later round in which every label answers; a failure after that is Backoff, not a mode, since the world is still the registry’s last word. And the directory registry now throws when its directory is missing instead of answering “no document”, because the cache hinges on telling those apart. The Followed record (Serve.hs) is the loop’s only view of the fetcher — the fetch hook and an IO Mode — and mode lives on StatusReport itself, so the HTTP /status gets it from the same encoder.
Status sink (file first), and a salmon-fleet status that folds a directory of them. Shipped (Salmon.Actions.Serve.StatusSink, Salmon.Actions.Fleet, salmon-fleet in salmon-apps, Test.StatusSinkSpec), with four deviations from the sketch above. The sink is not an op in the host’s graph. “Status flows back” wanted a filecontents/Storage node so a sink that is down shows as a Failed node; but a node is applied by a pass, and the document must be written after the pass it describes, which a node inside that pass cannot do. It is a reporter beside the loop’s (watching for ConvergeStop and Follow.Injected) plus a read-only accessor to the world and a timer, and a failed write is a Serve.SinkFailed report — the same information, on the same stream, once per run of failures. It writes more often than “after every pass”: also after every follow injection and every --status-sink-interval seconds (10) with nothing happening, so that a host that has gone quiet is visibly one whose written is old. The fetcher’s reports had to become a --json stream first (stream: "follow", the fourth constructor of Salmon.Reporter.Tagged); until then Injected/Backoff/Replayed printed as text under --json and no sink could carry them — which the sketch did not anticipate because it predates --json. And the host is uname -n, so two loops on one machine write two documents naming one host, and the fold shows two rows rather than picking; --status-sink-host NAME is that override, added once (2026-09-25) rather than speculatively. salmon-fleet status DIR [--label L] [--stale S] [--json] is the fold: one line per document, host order, converged/errored/total off status.nodes, stale past --stale (60s) as a flag and never a decision. Only the file sink exists; a bucket object or an HTTP POST is the same document handed to a different writer.
Git registry, then HTTP, then the DNS index over HTTP, then bucket. Verify-before-inject hook with a no-op verifier. Shipped (Salmon.Actions.Follow.Registry and Registry.Git/.Http/ .Dns, Follow.followVerify, Test.FollowRegistrySpec), with five deviations from the table under “Labels are addresses into a registry”. The git template is <subdir>/<label>.json at a branch (git+URL[#BRANCH[:SUBDIR]]), not <repo>/<label>/latest.json — one file per label beside the others is what a publisher edits, and a subdirectory per label would have made history git’s twice. The registries are not nodes in the agent’s own graph: the “inbox as a node” idea above would have put a fetch under stopTending (a pass is what runs a node’s up) and so under the very starvation rule the fetcher exists to respect; a fetch failure reaches the same Report stream as FetchFailed/Backoff instead. The bucket backends are the HTTP one under a URL template (virtual-hosted S3, GCS, or path-style under --follow-bucket-endpoint) — public or presigned objects only, no SDK, no object generation as the stamp (the ETag serves), and no authenticated access, which is its own item if wanted. The DNS resolver is dig +short through Binary behind a Resolver record (nothing in the tree resolved DNS, and one TXT lookup did not justify a resolver library’s footprint); the record format is settled under the open questions. And the hook is followVerify :: Digest -> ByteString -> IO (Either Text ()) on the raw bytes, after the digest comparison and before the parser, run on a cache replay as well as on a fetch — a refusal is Rejected, a failed round, neither injected nor cached, the last good document staying in force as the “Signed documents” section asks. Three flags joined the --follow-* family for the backends’ sake: --follow-timeout, --follow-workdir, --follow-bucket-endpoint.
Signed documents. Shipped (Salmon.Actions.Follow.Signature, --follow-key FILE repeatable, salmon-fleet keygen --out FILE and salmon-fleet sign --key FILE, Test.FollowSignatureSpec), with five deviations from the sketch and the work item’s brief. The envelope wraps the document rather than signing its bytes: {"salmon-signed": 1, "document": <the document as fetched>, "signatures": [{"key", "alg", "sig"}]}, the signature over the canonical bytes of the document member — Data.Aeson.encode of the parsed value, whose sorted keys and single spelling per scalar are what let a registry re-serialise an envelope without breaking it; both sides parse-then-encode with the same function, and no canonical-JSON library is involved. The verifier hands the loop the inner document, so Verifier became Digest -> ByteString -> IO (Either Text ByteString) (the bytes to parse) rather than Either Text (); noVerifier returns its input. The digest is the envelope’s, not the document’s, everywhere the fetcher keeps one (change detection, Rejected, history, the cache): the cache keeps the bytes as fetched so a replay is verified exactly as a fetch, its integrity check is over those bytes, and Rejected for an envelope that does not even parse has no other digest to name; the brief asked for the document’s, and that would have meant a second digest per node for the sake of one report field. The key is a JWK, not PEM: the brief said “PEM public key” and also “the tree’s existing key format”, and those disagree — Keys.jwkKey writes JWK through jose, and nothing in the tree parses PEM, so JWK it is; --follow-key reads either the public file or the private one (taking its public half). Ed25519 via jose, which already sat on crypton in the plan; bestJWSAlg means an RSA or EC JWK signs too, while none and the HMACs are refused before jose sees them (its verify of none against an empty signature answers true). Refusals say which: unsigned under a key, an envelope that does not parse, no signatures, every signature failing (naming the key and whether it was unknown, tampered, or an algorithm a public key cannot verify). A --follow-key that does not load exits 1 with the path; the flag without --follow is refused as --label is. Unsigned mode is the default and the docs say so.

Future work: what a running world knows, and who gets to see it

2026-09-25T14:15:36Z

Generated from specs/future-work.md — the repository is the canonical source, and may be ahead of this page.

Future work: what a running world knows, and who gets to see it

Status: ideas, not a plan. Four notes taken after trying run serve with the HTTP surface, the web UI and salmon-tui on salmon-gcp-toy (PR #8, #9), each elaborated to the point where the next spec could start. They share one theme: the loop now has a lot of knowledge about each node — what it found, what it last said, where its effect lives, what a remote copy of it is doing — and almost none of it is addressable from outside.

1. Facts a node discovers

The itch. salmon-gcp-toy tier 2 takes two passes because GCP picks the address: the first pass reserves it, the driver reads it back with Compute.readAddress, and feeds it in as --vm-ip for the second (resources/gcp-toy-validation.md). The same shape recurs whenever up learns something the declaration could not say: a project number, a Cloud Run service URL, a bucket’s generated name, a generated password’s fingerprint, a container’s assigned port. Today that value lives, at best, in a node’s output ring (NodeState.nodeStatus, snapshotted by stopTending) or in prose in notes, and the operator re-derives it with gcloud.

The shape. A node publishes facts: a small Map Text Value it is willing to state after up or check. Two ways to get them out, and the cheap one first:

As reports. A Fact !(Act ext) !Text !Value constructor on UpDown.Report (or its own small stream in Tagged), emitted by up and check through the reporter a node already has. It then reaches everything for free: --json, the socket clients, /events, the status sink, the UI’s per-node panel. Nothing changes in Extension; a node opts in by calling a helper.
As state. NodeState gains nodeFacts, folded from those reports by the loop (last writer wins per key), so /dag and status show a node’s current facts without replaying events, and a fleet fold can compare them across hosts (salmon-fleet status --fact project-number).

Where it bites. Feeding a fact back into a declaration — the --vm-ip case — is the actual two-pass problem, and it is not solved by either of the above. Options, roughly in order of ambition: a pull-mode document that names a fact ({"seed": [..., "--vm-ip", {"fact": "#addr/ip"}]}, resolved by the fetcher against the world before injection, so the second pass is a document the controller can write generically); a Rewrite phase that substitutes facts into dependants at expand time (closest to Rewrite.hs’s “cross-declaration knowledge lives here” rule, but a declaration whose words depend on a fact has a Ref that changes when the fact does, which Serve.record would read as a re-declaration — perhaps correctly). Start with facts-as-reports; the feedback loop is its own spec.

2. Structured, clickable notes

The itch. notes :: [Text] is prose. A GCP node’s most useful note is a console URL; a Cloud Run service’s is its endpoint; a Postgres node’s is a connection string one should not paste into a log. The web UI renders notes as text (textContent, deliberately: notes are untrusted), the TUI prints them, and neither can offer a link.

The shape. Beside notes, an Extension.links :: [Link] with Link { linkLabel :: Text, linkKind :: LinkKind, linkHref :: Text } and LinkKind = Console | Endpoint | Logs | Doc | Other Text. Structured rather than “notes that look like URLs” so the UI can render an anchor with a label, the TUI can print [console] https://..., /dag can carry them typed, and a fleet fold can list every host’s endpoints. The builtins that know a URL attach it: Gcp.* (console pages per resource kind), CloudRun (service URL), Storage (bucket URL), Systemd (journalctl -u as a Logs link the TUI can run), Qemu (the serial console socket).

Where it bites. sameRepresentative compares notes; whether links count as identity (a changed console URL is not a changed effect) needs a decision — probably not compared, like dynamics payloads other than Supervision. And links are where secrets creep in (a presigned URL, a connstring with a password): the sensitive-data story the generic-server spec defers is a prerequisite for Endpoint links at least, or links must be declared public by construction like notes are today.

3. A node’s latest messages

The itch. After an up, the UI panel and the TUI say “never tended” or show a stale check, because /dag’s per-node status is the snapshot stopTending last filed (R3 in Actions/Serve.hs), and the tending machine’s live output ring is behind a TVar nobody outside Upkeep can reach. The events ring has everything a node ever said, but only as one stream to scan.

The shape. Two reads, both cheap because the data already exists:

GET /node/<ref>: the node’s /dag projection plus its last n events from Events’s ring filtered by ref (the ring is a Data.Sequence of tagged objects; an index Map Ref (Seq seq) maintained at publish makes this O(n) in the answer, not the ring). ?since= for a client that wants only what is new. The UI panel and the TUI’s enter view get their “last events” from here instead of from what they happened to see live.
Live output ring. Upkeep.Supervisor exposes a read of each machine’s current Status (check verdict, reason, output ring, statusEpoch) and serveObserved’s accessor includes it, so /node/<ref> and /dag report the machine’s current status when one is running and the snapshot only when none is. That is the freshness gap D1 and D2 both hit (“check reads - until a next-look arrives”).

salmon-tui and the web UI are the consumers; status --select on the socket could print the same thing in text.

4. Remote ops as a DAG with live events

The itch. Self.callSelf runs ssh host bin run up with a directive on stdin and streams the remote’s text output into the local node’s output ring. run dag can draw a remote subgraph through injectRemoteSubgraphs and the RemoteOp dynamic, but /dag cannot (B3’s finding: the magma has no path to a remote subgraph), and a remote node’s eval/done/failed never become local events. So the UI shows one box, remote-call, going from pending to converged after minutes, with everything interesting hidden inside it.

The shape, in three steps that each stand alone:

Parse what comes back. callSelf invokes run up --json and decodes each line with the Tagged encoding (the wire format that item A1 made a contract), re-emitting every remote report through the local reporter with a host stamp — a Remote host inner constructor on Tagged, or a host field beside stream. Text stays available for a remote binary too old to speak JSON. The remote’s Facts (section 1) ride along unchanged.
Draw the remote subgraph in the magma. At expand time, a RemoteOp’s subgraph is folded into the Dag under its remote-call node with refs prefixed by host (so two hosts running the same recipe do not collide on one Ref, which mkRef’s location-addressing would otherwise make happen), marked remote: host in the /dag projection. The UI draws it as a nested box; the events from step 1 land on those nodes by ref. Concurrent’s failure containment is unaffected: the remote-call node is still the one that fails locally.
The fleet DAG. Once a remote host runs its own serve --follow (pull mode), the controller no longer calls it; it reads the host’s /dag and /events (E1’s TLS listener, Salmon.Client.Http from D1) and composes them into one view keyed by host — the aggregation salmon-fleet status lacks (C4’s finding: nothing summarises by label or document id). That is the “fleet control” picture from the capabilities assessment, and it needs nothing more from the loop: it is a client.

Step 1 is small and immediately visible; step 2 is the one that changes Dag.hs; step 3 is the one that makes pull mode and the server meet.

Not here

Consensus, and any decision that a host is dead; those stay out on purpose (specs/pg-switchover.md’s argument). The sensitive-data story deferred by specs/generic-server.md, which sections 2 and 4 both lean on.

Local qemu VMs + tap/bridge networking for recipe testing

2026-09-25T14:15:36Z

Generated from specs/qemu-test-vms.md — the repository is the canonical source, and may be ahead of this page.

Local qemu VMs + tap/bridge networking for recipe testing

Status: phases 1 to 5 of the phased plan are implemented: Salmon.Builtin.Nodes.LinuxBridge (bridge/tap), Debootstrap.ensureVm9pBoot (9p root, -kernel/-initrd), Salmon.Builtin.Nodes.Qemu (a VM as a systemdService), and the Layer 3 tier in Test.Harness (withVm/withVmAt, Test.QemuSmokeSpec), whose first real recipe test is Test.PostgresReplicationSpec. The first “Future work” item shipped too: Test.PostgresSwitchoverSpec and Test.PgPairDemoSpec run several VMs on one bridge. Not done: phase 6 (a raw disk image) and snapshot/clone support. How it was built, and what departed from this design, is in specs/qemu-test-vms-progress.md. Kept as the design record.

Problem

salmon-ops-recipes/test/Test/Harness.hs already tiers tests by IO cost/ blast-radius, and its Layer 2 (podmanTrack/withContainer) dogfoods Podman.pullImage/Podman.runContainer to get a real, disposable sandbox for recipes that need a real service (Postgres, in Test.PostgresInitSpec). That works well for anything that fits in a container, but several things this project increasingly needs to test do not:

systemd units as PID 1 actually manages them (Systemd.hs is used all over salmon-ops/SreBox, but a container’s systemd, if present at all, behaves differently from a real boot),
real network interfaces/routing (WireGuard.hs, Routes.hs, Netfilter.hs), where a podman container’s network namespace doesn’t exercise the same code paths as a genuine host interface,
multi-machine topologies where “machine” needs to mean something closer to a real boot (kernel, init, network stack) — exactly what the [[pg-ha control plane spec]] (specs/pg-ha-control-plane.md) needs to test the diagonal replication pair, bouncer failover, etc. against something more realistic than two podman containers.

There’s an unfinished start at this: Debian.Debootstrap.rootTree builds a debootstrapped root filesystem at a path (idempotent via skipIfFileExists on etc/issue) but nothing turns that into something bootable, and there’s no qemu or bridge/tap networking builtin at all — Netfilter.hs literally has -- TODO: Ip, Ip6, Arp, Bridge, NetDev and stops there.

Goal

A Layer 3 addition to the same test-harness tiering: real qemu VMs, on a local bridge, reachable over SSH, disposable the same way Layer 2’s podman containers are — dogfooding new salmon builtins (bridge/tap setup, qemu VM lifecycle) as the sandbox provisioner, same philosophy as Layer 2.

Design goals / non-goals

Goals:

New builtins follow every existing convention: op/Track'-shaped, idempotent up (see CLAUDE.md’s “Conventions for node authors”), down implemented (this project’s downTree machinery assumes it, and disposable-sandbox teardown is the whole point here).
Reuse Systemd.hs for the VM process lifecycle instead of inventing a new “manage a long-running process” mechanism — a qemu VM is just another systemd unit from the host’s point of view (see §2).
Reuse Ssh.hs for reaching into a running VM, exactly like Layer 2 uses podman exec.
Finish Debootstrap enough to produce something qemu can boot, without inventing a whole image-building subsystem.

Non-goals (v1):

Production VM hosting (this is a test sandbox mechanism, not a new “salmon runs your workload in a VM” feature — no live migration, no resize, no snapshots).
libvirt/virsh — plain qemu-system-x86_64 + a monitor socket is enough for scripted start/stop; libvirt’s XML/daemon layer adds nothing v1 needs.
Multi-host bridging (VXLAN, etc.) — a single-host Linux bridge is enough for “several VMs on one test machine talk to each other and to podman containers if needed.”

Proposed builtins

1. `Salmon.Builtin.Nodes.LinuxBridge` (new)

The bridge + tap primitives Netfilter.hs’s TODO never got to:

data Bridge = Bridge { bridge_name :: Text }

data Tap = Tap { tap_name :: Text, tap_bridge :: Bridge, tap_owner :: Maybe User.User }

bridge :: Reporter Report -> Track' (Binary "ip") -> Bridge -> Op
tap    :: Reporter Report -> Track' (Binary "ip") -> Tap -> Op

up: ip link add name <br> type bridge && ip link set <br> up; ip tuntap add dev <tap> mode tap [user <owner>] && ip link set <tap> master <br> && ip link set <tap> up. Neither ip link add nor ip tuntap add is idempotent (both fail with “File exists” on retry) — same shape as Netfilter.rule’s problem, so use the same fix already established as this project’s convention: prelim checks ip link show <name> and reports Skippable if it’s already there, rather than trying to force the ip invocation itself to be idempotent. down: ip link delete <name>.

2. `Salmon.Builtin.Nodes.Qemu` (new)

A VM as a systemd unit, mirroring Nginx.setup/PgBouncer.setup’s exact shape (render a start command, hand it to Systemd.systemdService):

data VmConfig
    = VmConfig
    { vm_name :: Text
    , vm_memory_mb :: Int
    , vm_smp :: Int
    , vm_disk :: FilePath          -- see §3, the boot image
    , vm_tap :: LinuxBridge.Tap    -- depends on §1
    , vm_mac :: Text               -- stable MAC so the host can predict/reserve a DHCP lease if needed
    , vm_monitor_socket :: FilePath
    , vm_extra_args :: [Text]
    }

setup :: Reporter Systemd.Report -> Track' (Binary "systemctl") -> Track' (Binary "qemu-system-x86_64") -> VmConfig -> Op

up/down are exactly “start/stop the systemd unit” (free, via Systemd.hs) — no new process-management code. Command line:

qemu-system-x86_64 -name <vm_name> -m <memory_mb> -smp <smp>
  -drive file=<disk>,if=virtio,format=raw
  -netdev tap,id=net0,ifname=<tap>,script=no,downscript=no
  -device virtio-net-pci,netdev=net0,mac=<mac>
  -monitor unix:<monitor_socket>,server,nowait
  -nographic -serial mon:stdio
  -enable-kvm   -- if /dev/kvm exists; fall back to TCG otherwise (slow but portable, worth keeping as a fallback for CI boxes without nested virt)

Graceful shutdown on down ideally goes through the monitor socket (system_powerdown) rather than systemctl stop sending SIGTERM straight to qemu — worth a small Qemu.shutdown helper that writes to the monitor socket and polls for the process to exit before falling back to a hard stop, so the guest gets a real ACPI shutdown instead of losing an in-flight up/write. Exact mechanism (raw socket write vs qemu-system-x86_64’s own -monitor command tooling, if any exists on the host) is an implementation detail to work out against a real qemu version, not a design blocker.

3. Finishing `Debootstrap`: from chroot dir to bootable disk

Two options, both worth having eventually but starting with the first:

a. 9p virtfs passthrough (recommended v1 default) — skip image-building entirely; boot the existing RootTree directory straight off the host filesystem via qemu’s virtfs:

-fsdev local,id=root,path=<rootTree.path>,security_model=mapped
-device virtio-9p-pci,fsdev=root,mount_tag=/dev/root
-kernel <rootTree.path>/boot/vmlinuz-*  -initrd <rootTree.path>/boot/initrd.img-*
-append "root=/dev/root rootfstype=9p rootflags=trans=virtio rw console=ttyS0"

No mkfs/loop-mount step, no separate image artifact to keep in sync with the chroot, fast to rebuild (debootstrap again just overwrites the dir, same idempotency the node already has). Tradeoff: 9p root is nonstandard enough that a few recipes’ assumptions (real block device semantics, fsync behavior a Postgres data directory cares about) might not transfer 1:1 to production behavior — acceptable for “does the recipe’s up/down/ check logic run correctly,” not for filesystem-performance testing.

b. Raw disk image (future work, if 9p’s divergence bites) — extend Debootstrap with a variant that targets a loop-mounted .raw/.img file instead of a plain directory (losetup, mkfs.ext4, mount, run debootstrap against the mountpoint, install a bootloader or keep using direct -kernel/-initrd boot to skip GRUB entirely), producing a real block-device-backed VM. More moving parts (loop device idempotency/cleanup needs its own care — a stale loop device from a crashed previous run is exactly the kind of thing down needs to handle), so deferred until 9p proves insufficient.

Either way, RootTree.includes needs to grow to cover what a bootable VM needs that a plain chroot doesn’t: a kernel package (linux-image-<arch>), openssh-server (so §4’s SSH-based test harness can reach in), and enough of an init to reach multi-user (Debian’s default systemd-sysv — already implied by debootstrap unless --variant=minbase was used, worth confirming RootTree isn’t passing that).

4. Test harness: Layer 3

-- Test.Harness additions, mirroring podmanTrack/withContainer/podmanExec_
qemuTrack :: Track' (Binary "qemu-system-x86_64")
withVm :: VmConfig -> (Ssh.Remote -> IO a) -> IO a   -- boots, polls SSH readiness, runs action, tears down
vmExec_ :: Ssh.Remote -> Text -> IO ()

withVm runs Qemu.setup’s Op through runUp/runDown exactly like withContainer does for podman — real IO, no mocking, dogfooding the new builtins as their own test infrastructure. “Boot readiness” is poll-SSH- until-it-answers (a VM takes real seconds to boot, unlike a podman container being “up”), with a timeout that fails loudly rather than hanging a test suite — same “skip loudly, don’t hang” spirit as requireExecutable.

requireExecutable "qemu-system-x86_64" (already-generic) gates the whole tier the same way Layer 2 gates on podman, so a machine without qemu skips these tests instead of failing.

Open questions

KVM availability in CI/dev containers: nested virtualization may not be available everywhere this test suite runs. -enable-kvm needs a /dev/kvm-exists fallback to TCG (§2 already notes this) — worth confirming up front whether TCG boot times are tolerable for a test suite before committing to “VMs boot fast enough to be a normal test tier” as an assumption.
Bridge lifecycle scope: one shared bridge reused across test runs (persistent, created once, VMs’ taps attach/detach per test), or a fresh bridge per test run (fully disposable, more sudo ip-shaped setup/teardown noise per test)? Leaning towards one persistent bridge (named distinctly, e.g. salmontest0) with per-test taps, mirroring how Layer 2 doesn’t recreate podman’s network each test either.
Privilege: ip link add/tuntap add and (for KVM) /dev/kvm access typically need root or specific capabilities/group membership (CAP_NET_ADMIN, the kvm group). Does the test harness assume the invoking user already has these (documented prerequisite, same as podman needing to be installed/usable), or does it need a sudo-wrapped path? Recommend the former (documented prerequisite) to match how Layer 2 already assumes a working, usable podman rather than trying to grant privileges itself.
Where do prebuilt kernel/initrd come from: debootstrap installs /boot/vmlinuz-*/initrd.img-* only if a kernel package is in includes (§3) — confirm the target suite’s kernel package name (linux-image-amd64 on Debian stable) and that update-initramfs runs automatically as part of package postinst inside the chroot (it should, via the chroot’s own dpkg triggers) rather than needing an explicit step.

Phased plan

LinuxBridge (§1): bridge + tap nodes, prelim-based idempotency, down. Test by hand (ip link show) before anything qemu-shaped depends on it.
Extend Debootstrap’s includes/confirm kernel+ssh presence (§3a); by hand, boot the resulting chroot directly with a one-off qemu command line (no salmon Op yet) to validate the 9p+-kernel approach works at all before wrapping it in a node.
Qemu node (§2) wrapping the now-validated command line as a Systemd.systemdService, against the §1 bridge.
Test.Harness Layer 3 (§4): withVm, SSH-readiness polling, one smoke test that boots a VM and runs a trivial command over SSH.
Pick one existing recipe that Layer 2 can’t exercise well (a Systemd- or WireGuard-dependent one) and add its first Layer 3 test, proving the tier end to end.
Raw disk image variant (§3b), only if 9p’s divergence from a real block device turns out to matter for something concrete.

Future work

Multi-VM topologies on the same bridge for testing the [[pg-ha control plane]]’s diagonal replication pair against two real VMs instead of two podman containers, once Layer 3 itself is proven out.
Snapshot/clone support (qemu -snapshot or backing-file qcow2 images) to make repeated test runs cheaper once the raw-image variant (§3b) exists.

Two-node Postgres with salmon-driven switchover and failover

2026-09-25T14:15:36Z

Generated from specs/pg-switchover.md — the repository is the canonical source, and may be ahead of this page.

Two-node Postgres with salmon-driven switchover and failover

Status: implemented — phases 1 to 6 of the phased plan and the disaster catalogue S1–S8, in SreBox.PostgresPair (member, seedMember via pair_seed, bouncerSetup, pairRole, pairOp), salmon-pgpair and salmon-toy-qemu-pg-ha, tested by Test.PostgresPairSpec, Test.PostgresSwitchoverSpec and Test.PgPairDemoSpec. The paragraphs below record how it got there; where an earlier one says something is not done (the re-seeding half of phase 6), a later one says it since was. Still open: the questions at the end (pair_synchronous is not offered; the old primaryReplicationSetup/standbyReplicationSetup still exist beside the symmetric members). Kept as the design record.

Done: the prerequisites P1-P5; the state table and the probe (SreBox.PostgresPair, phase 2), covered at Layer 0 including every refusal; the role node over ssh (phase 3) and pair_may_discard’s failover and split-brain paths (phase 5), covered by Layer 3 tests that move a real primary between two VMs and back (S1, minus the client assertions), stop a controller after each step in turn and let an ordinary pass finish it (S2), kill a primary outright and fail over onto its standby (S3), cut the two machines off from each other and change nothing (S4), and fail over across a partition that hides the old primary from the controller too, then heal it and rewind the loser (S5).

Also done: the slot budget (phase 6), in that each member streams with a slot the pair names and the rejoin creates, and a slot that falls off the budget is reported rather than rewound at (S6). The re-seeding half of phase 6 is not: there is no seeding node, so “re-seed it” is still something an operator does by hand – which is why the check can only name the problem.

The disaster catalogue S1-S8 is written, and running it is what most of the design above was decided by.

Phases 3 and 4 are now whole: member configures a machine to be either half of the pair without naming a side, bouncerSetup stands a pgbouncer up in front, pairOp is the three of them as one declaration, and salmon-pgpair is a binary an operator can type at. PauseBouncers and RepointBouncers have commands behind them, and decide asks the bouncers rather than assuming there are none.

Also done: the seeding clone (seedMember, declared through pair_seed), and S1’s client assertions – 460 inserts through pgbouncer across a switchover, no errors, nothing lost.

There are two ways to watch it. Test.PgPairDemoSpec asserts it on three VMs the harness boots. salmon-toy-qemu-pg-ha is the same thing as a binary that makes its own guests, where moving the primary is a command you type – and where the three steps that used to be a README (debootstrap, ensureVm9pBoot, handing /etc/ssh to the unprivileged user) are a prereqs seed that declares them.

Not done: nothing in this spec, though pg-patroni.md is still a sketch.

Companion: pg-patroni.md covers the other end of the range, with automatic failover and three voters. pg-ha-control-plane.md is the wider bouncer/app/LB picture that either one plugs into.

What this is for

A primary and a streaming standby on two machines. Salmon moves the primary between them when an operator changes a declaration. Nothing decides on its own that a machine is dead.

Two audiences:

Disaster tests in the harness. The declared primary is a knob a test turns. The same node that runs a switchover can also bring a scenario back to a known state, and every failure mode below is something a Layer 3 test should be able to cause on purpose and then assert about. See “Disaster scenarios”, which is as much the point of this spec as the recipe itself.
Services with a low SLA, where a few minutes of downtime and a human deciding “fail over now” are acceptable, and two machines are what the budget allows.

It is not for anything that needs to survive an unattended failure at 3am. That takes consensus, and consensus takes three voters. See pg-patroni.md.

What it promises:

Event	Data lost	Time to recover
Planned switchover	none: the old primary is stopped cleanly and the standby is checked to have its last record before promotion	seconds; with pgbouncer `PAUSE`, clients see a delay rather than errors
Unplanned failover (the primary died)	whatever had not reached the standby, i.e. the replication lag at the moment of death	however long the operator takes to decide, plus seconds
Network partition, no operator action	nothing: salmon refuses to act	none: the primary stays up and the standby falls behind

An optional pair_synchronous could take the unplanned row’s data loss to zero, at the usual two-node price: when the standby is down, every commit blocks. See “Open questions”.

Where the layout question landed

Two machines are fine for this design, and bad for automatic failover. The costs of the cross-replicated shape (each machine primary for one cluster and standby for the other):

each machine must be sized to carry both primaries, since that is what happens when one dies; spreading them buys headroom, not smaller boxes;
losing one machine takes away the redundancy of both clusters at once;
maintaining a machine means switching both clusters over first.

The primary’s location is a per-pair field in the directive, so “A is primary for everything” and the cross-replicated shape are the same recipe with different values. Start with the former. Nothing below depends on the choice.

Prerequisites: fixes to what exists, each worth doing on its own

These are defects in Salmon.Builtin.Nodes.Postgres and PgBouncer as they stand. The rest of the design assumes they are fixed.

P1. standbyReplicationSetup can wipe a promoted standby. cloneFromPrimaryScript’s guard is “does standby.signal exist”. Promotion deletes that file, so the next run up with an unchanged directive runs rm -rf "$datadir" and clones from the old primary again. If the old primary is down, which is the usual reason for promoting, the clone fails after the delete. The guard should compare system identifiers instead: the local pg_controldata’s “Database system identifier” against the primary’s SELECT system_identifier FROM pg_control_system().

Equal: this data directory already belongs to the cluster, as a standby or as a promoted one. Never touch it.
Different: it is foreign. Clone only if it holds nothing but what pg_createcluster made (no databases beyond postgres/template0/ template1), and otherwise refuse with the reason.

P2. pg_rewind is impossible on these clusters today. It needs wal_log_hints = on or data checksums, and nothing in salmon-ops/src sets either. Without it, rejoining an old primary means cloning it again from scratch. Add wal_log_hints = on to primaryReplicationSetup’s settings. It is restart-only, so it has to be in place before there is data worth protecting. Add max_slot_wal_keep_size too (see scenario S6): without a cap, a standby that is down for long enough fills the primary’s disk through its slot.

P3. primaryReplicationSetup restarts the primary on every run up, because restartCluster (via clusterCtl) has no check. The natural check for a restart node is “is any setting waiting for one”: SELECT count(*) FROM pg_settings WHERE pending_restart. That is Success at zero, which is the same move as checkService reading NeedDaemonReload. startCluster’s named-cluster path has the same non-idempotent start that detectVersionAndStartMainCluster had before the templates PR.

P4. A changed pgbouncer.ini is never applied by run up. PgBouncer.setup goes through Systemd.systemdService, whose checkService says Success for an active, enabled unit whose unit file has not changed. configFiles rewrites the ini as a dependency, but nothing reads the ini’s change, so the service is skipped. Routing a switchover through pgbouncer needs a RELOAD. Do it over the admin console rather than with a restart, which would drop every client connection. That also needs an admin user in BouncerConfig.

P5. The replication password travels inside a bash -c string. cloneFromPrimaryScript builds PGPASSWORD=... pg_basebackup in the script text. Use a passfile pre-provisioned on the machine, per the recipes’ secret-transport rule.

Shape of the recipe

A new SreBox.PostgresPair, three kinds of node.

1. Members are symmetric

Built, as member / memberScript. It runs over ssh from the controller like everything else here, which is why it renders SQL rather than reusing the nodes in Nodes/Postgres.hs: those are ops that run on the machine they configure, and nothing in this recipe does. What it does reuse is that module’s opinion about which settings replication needs (replicationSettings), so one place decides. Role passwords are read on the member out of the .pgpass files the pair already names, and fed to psql on standard input — never in the script, never on a command line, never in a report.

Both machines get the same configuration, whichever is primary today:

wal_level, max_wal_senders, max_replication_slots, hot_standby, wal_log_hints, max_slot_wal_keep_size;
pg_hba lines for replication from the peer, and for the rewind role (below) from the peer;
the replication role and the rewind role. Roles live in the catalog, so they are created on whichever member is primary and replicate to the other.

No member node mentions a role in its ref, help or notes. A switchover then changes exactly one node’s declaration, the role node’s, and under run serve (I6) the members are not marked Stale by it. This replaces the primaryReplicationSetup/standbyReplicationSetup asymmetry for this recipe. Those two stay for the existing fixture and Test.PostgresReplicationSpec.

The rewind role needs a plain login, not a replication one, with EXECUTE on pg_ls_dir(text, boolean, boolean), pg_stat_file(text, boolean), pg_read_binary_file(text) and pg_read_binary_file(text, bigint, bigint, boolean). That is the documented minimum for pg_rewind --source-server without a superuser.

2. Seeding the standby is a one-time clone

This is the pg_basebackup step, with P1’s guard. It runs once in a pair’s life, and again only if a standby is lost beyond repair (S6). It is deliberately not how an old primary rejoins after a switchover; that is pg_rewind, inside the role node.

Not built yet, and S6 is what says how it should behave when it is: a lost slot leaves the pair in a state only a re-seed fixes, and the role node deliberately does not fix it. Wiping a machine’s data directory is a decision about losing whatever is on it, which is the same class of decision as pair_may_discard and belongs to the same place — an operator saying so — rather than to a pass that runs unattended. So the seeding node should be separately declared, and the role node’s job is to name the problem precisely enough that the operator knows which machine to declare it for.

3. The role node: “this pair’s primary is on B”

data Side = A | B

data Member = Member
    { member_remote :: Ssh.Remote       -- how the controller reaches it
    , member_host :: Postgres.Host      -- how the peer and the bouncers reach it
    , member_cluster :: Postgres.ClusterName
    , member_port :: Postgres.Port
    }

data Pair = Pair
    { pair_name :: Text
    , pair_a, pair_b :: Member
    , pair_primary :: Side
    , pair_may_discard :: Maybe Side   -- see "Failover and split brain"
    , pair_repl_role, pair_rewind_role :: Postgres.RoleName
    , pair_passfile :: FilePath        -- on each member, pre-provisioned
    , pair_bouncers :: [Bouncer]       -- admin-console endpoints to PAUSE/RELOAD/RESUME
    , pair_step_timeout :: Int         -- seconds to wait for the standby to catch up
    }

ref = mkRef "pg-pair-role" pair_name: keyed on the pair, never on who is primary, so switching primaries changes that one node rather than creating another. notes carry the declared primary and pair_may_discard, so a re-declaration is a visible change (I6).

The node runs on a controlling machine and reaches both members over Ssh.callWith (or Nodes/Self.hs’s callSelf for anything bigger than a script). It cannot run on a member: the member that dies might be the one running it.

Observation, then a pure verdict

One ssh round trip per member returns a small key=value report:

cluster status from pg_lsclusters;
if it is running: pg_is_in_recovery(), the timeline, the current or replay/receive LSN, the upstream host from pg_stat_wal_receiver, the configured upstream from primary_conninfo, and system_identifier;
if it is stopped: pg_controldata’s cluster state, latest checkpoint location, minimum recovery point, timeline and system identifier;
if it is running: the replication slots it holds and each one’s wal_status, which is the only place the fate of the other member’s catching-up is written down;
from the bouncers, whether each one is paused (SHOW DATABASES has a paused column).

data Observed
    = Unreachable Text
    | Stopped { o_sysid :: Word64, o_timeline :: Int, o_checkpoint :: Lsn, o_clean :: Bool }
    | Primary { o_sysid :: Word64, o_timeline :: Int, o_lsn :: Lsn, o_slots :: [(Text, Text)] }
    | Standby { o_sysid :: Word64, o_timeline :: Int, o_upstream, o_configured :: Maybe Text, o_received, o_replayed :: Lsn }

data Step
    = Done
    | Degraded Text           -- the declaration holds, but the pair is not redundant
    | PauseBouncers
    | StopMember Side          -- a clean, fast shutdown
    | AwaitCatchUp Side Lsn    -- the standby must have received past this
    | AwaitStreaming Side      -- pointed here, not connected: wait, do not rewind
    | Promote Side
    | Rejoin Side              -- pg_rewind -R against the primary, then start
    | StartMember Side
    | RepointBouncers Side     -- rewrite ini, RELOAD, RESUME
    | Refuse Text

nextStep :: Pair -> Observed -> Observed -> [BouncerState] -> Step

parseObserved and nextStep are pure, in the Systemd.interpretShow / Postgres.interpretTemplateRow pattern. Layer 0 can then cover the whole table below, including every refusal, without a database.

A standby is asked two questions about its upstream, and the difference between them is what keeps a partition from becoming an outage. o_upstream is where it is streaming from, which is empty the instant the connection drops; o_configured is where primary_conninfo tells it to stream from, which it keeps through a partition. Reading only the first, a standby that cannot reach its primary is indistinguishable from a standby that belongs to somebody else – and the answer to the second of those is Rejoin, which stops the standby and then fails, because whatever keeps it from streaming keeps pg_rewind from reading too. A partition would take the standby down. Reading both, the answer is AwaitStreaming: wait, and if the waiting runs out say so as Unknown rather than as a failure, because “it came back a second ago” and “it has been cut off for an hour” are the same observation.

Two of those positions are not the field whose name they carry, and writing S3 is what found it. A stopped cluster’s position is max(latest checkpoint, minimum recovery point): a standby that was stopped replayed past its last checkpoint, and only the second field says so. A standby’s is max(received, replayed), because pg_last_wal_receive_lsn() is NULL — not a position — in a server that has received nothing since it started, which is the state of every standby whose primary has just died, i.e. exactly when a failover needs the number.

check is observe >>= nextStep: Done is Success, Degraded is Unknown, and anything else is a Failure naming the step.
up loops: observe, compute the step, perform it, and repeat until the step is Done or Degraded, throwing on Refuse. There is a bound on iterations so that a fault in the table cannot spin forever.

Because the state is re-derived from observation on every turn, up can resume from any point. A controller killed between “stop A” and “promote B” leaves a pair the next run recognises and finishes. There is no progress file to lose or to disagree with reality.

Degraded maps to Unknown on purpose. Under run serve, Unknown “restarts nothing” (see Actions/Upkeep.hs). “The primary is where it should be, and the peer is unreachable” is exactly a case where supervision should keep looking and must not act.

The table (declared primary: B, peer: A)

Checked in order, first match wins:

Observed A	Observed B	Step
any sysid ≠ B's sysid		`Refuse "not the same cluster"`
Standby streaming from B	Primary	bouncers point at B and are unpaused? `Done` : `RepointBouncers B`
Standby or Stopped, and B's slot for A is `lost`	Primary	`Degraded "the slot is lost"`: the WAL it needs has been recycled, so waiting produces nothing and a rewind changes nothing. Only a re-seed helps, and that is an operator's call
Standby, pointed at B, not streaming	Primary	`AwaitStreaming A`, and `Unknown` once the waiting runs out
Standby, pointed anywhere else	Primary	`Rejoin A` (the rewind is a no-op if nothing diverged)
Stopped	Primary	`Rejoin A`
Unreachable	Primary	`RepointBouncers B` if needed, else `Degraded`
Primary, not `may_discard`	Primary	`Refuse "two primaries"`
Primary, `may_discard = A`	Primary	`StopMember A`, then `Rejoin A`: A's divergent writes are lost, as declared
Primary	Standby of A, streaming	`PauseBouncers`, then `StopMember A`
Primary	Standby of A, not streaming	`AwaitStreaming B`: a clean stop hands the tail over through that connection, so stopping A without one strands whatever B has not got
Stopped, `may_discard = A`	Standby	`Promote B`: the loss is already accepted, and waiting on a machine that will send nothing more is a slower way to the same place
Stopped, crashed	Standby	`Refuse "A did not stop cleanly"`
Stopped cleanly at c	Standby, received < c, still streaming	`AwaitCatchUp B c`: the tail is in flight
Stopped cleanly at c	Standby, received < c, not streaming	`StartMember A`: nothing will arrive from a stopped machine, so start the one that holds the records and let B catch up from it
Stopped cleanly at c	Standby, received ≥ c	`Promote B`
Unreachable, not `may_discard`	Standby	`Refuse "cannot confirm A is stopped"`
Unreachable, `may_discard = A`	Standby	`PauseBouncers`, then `Promote B`
Standby	Standby	promote B only if B has received at least as much as A, else `Refuse`
Stopped	Stopped	`StartMember B`

Paused bouncers are a state of their own. If up dies with them paused, every client is stuck, so any bouncer found paused makes the check fail, and every terminal step, Refuse included, resumes the bouncers salmon paused.

Rejoin is pg_rewind -R --source-server=<B> as the rewind role, then a start — with three things around it that the bare command does not do, each of which was a failure before it was a line of script.

The recovery configuration is written here, not by -R. pg_rewind is entitled to decide no rewind was needed at all, which is the ordinary case after a clean switchover, and what it then does about -R is not worth betting a second primary on. So primary_conninfo is rewritten every time. So is primary_slot_name — deleted, and for a sharper reason: slots are not replicated, so a member that comes back naming the slot it used to stream with names it on a machine that has never heard of it, and a standby whose slot is missing does not fall back to streaming without one. It retries forever (replication slot "..." does not exist) while looking, to every other query, like a healthy standby: pg_is_in_recovery() is true, the timeline is right, only pg_stat_wal_receiver is empty. Streaming with no slot costs WAL retention, which is what phase 6’s slot budget is for. Streaming with a slot that is not there costs everything.

A crashed target is recovered before the rewind, by us. pg_rewind refuses a target that was not shut down cleanly, and since 13 it fixes that itself by running postgres --single -D <datadir> — which takes the configuration to be in the data directory. On Debian it is in /etc/postgresql, so that step fails, and it fails in exactly the case a failover is about: the machine that died. The rejoin therefore reads pg_controldata’s cluster state and, for anything but a clean stop, runs the single-user recovery itself with -c config_file=. Single-user, never a start: a server that listens is a second primary, and this one still believes it is the primary.

Neither that recovery nor the stop before it may throw away what the rewind is for. A clean shutdown ends in a checkpoint, and a checkpoint recycles the WAL before it — which is the WAL pg_rewind then reads, walking back to the last checkpoint the two machines share. So both StopMember and the recovery pin wal_keep_size to what pg_wal already holds, and the rejoin takes the pin off again once it has been used. Without it the rewind fails with could not open file .../pg_wal/... immediately after the step that was supposed to enable it. In an ordinary switchover nothing older than the shutdown checkpoint is ever wanted, so this never shows; after a split brain the histories parted long before, and stopping the loser is precisely what destroys the record of how.

And the slot it will stream with is made here too. -R does not create one, and slots are not replicated, so a rejoining member creates its own on the machine it is about to stream from — over the replication connection, which is the one path the pair already requires — and then checks that it is there, because a standby naming a slot the primary does not have retries forever while looking healthy to every query but pg_stat_wal_receiver. The name is derived from the pair and the side (salmon_pair_<name>_<side>, lower-cased, anything else an underscore, 63 characters) rather than declared, so that a member computes the same name the member it rejoins would and a slot nobody can name is not a slot nobody can drop. The same step drops the slot this member held for its peer back when it was the primary: nothing consumes it here, and a slot nobody consumes goes on pinning every segment behind it.

Failover and split brain: `pair_may_discard`

The operator’s escape hatch is one field, and its name states what it costs: “I accept losing writes on this side that the other side does not have.”

The same flag covers two cases:

Failover: A is unreachable and B is promoted without proof that A has stopped.
Split-brain resolution: both sides are primaries and A is rewound onto B’s history.

At configure time it must name the side that is not the declared primary; any other value is rejected there.

What a stopped peer proves

A peer that is not running is read off pg_controldata, and what that file is worth depends on how it stopped:

Shut down cleanly. The last record written is a shutdown checkpoint, so the checkpoint location is the end of its WAL: a standby that has reached it has everything. This is the ordinary switchover, and it needs no declaration from anybody.
Crashed. The checkpoint location is wherever the last periodic checkpoint happened to land, and any amount of acknowledged WAL may follow it, with nothing on disk to say how much. Comparing the standby against that number would read as “the standby has everything” precisely when it is least likely to be true.

So a crashed peer is a refusal until pair_may_discard names the side. It is the same rule as the unreachable peer’s, arrived at by a different road: in both, promoting means losing writes nobody can count. Database cluster state is the field that tells them apart, and anything other than shut down / shut down in recovery — including a value this parser does not recognise — counts as a crash, which is the direction that refuses.

Salmon cannot fence a machine it cannot reach. What makes promoting B safe enough for a low SLA is routing, not fencing: if pg_hba only admits application traffic from the bouncers’ hosts, then repointing the bouncers is the fence. A reachable A that comes back as a primary is stopped and rewound by the next pass. Its writes in the gap are the loss the flag already declared. The recipe should say this plainly in its haddock and in the check’s Failure text, instead of implying a fence it does not have.

Routing

Implemented; what follows is what was built and why.

The bouncers’ upstream is always the declared primary. Admin nodes that must run on the primary (database, user, templates, clones, migrations) also target the declared primary, so the Track' Postgres.Server they take is simply B’s. This is where the design is simpler than Patroni’s, where the primary has to be discovered at run time.

A switchover is PAUSE on every bouncer, the Postgres steps, a rewritten routing file, RELOAD, then RESUME. With pool_mode = transaction, PAUSE waits for transactions in flight, so clients see latency rather than errors. That makes scenario S1’s “no client errors” a real assertion.

The routing lives in its own file, not in the ini, pulled in with %include. That is the seam between two nodes that would otherwise fight over one file. PgBouncer.setup owns the ini and watches it, so a change there is noticed and applied — by a restart, which drops every client the bouncer exists to hold. The role node owns the routing file, which is not watched, and applies a change the gentle way. Each file has one writer, and the one that moves traffic never restarts anything.

bouncerSetup writes the routing file only when it is missing, for the same reason: after that it is the role node’s, and a pass that rewrote it would move clients without pausing them first.

Reading a bouncer is SHOW DATABASES on the admin console, by column name — the columns have changed between pgbouncer versions, and counting them is a way to read the wrong one. A bouncer that cannot be reached reads as sending clients nowhere, which is not Done: a pair whose clients are going somewhere unknown has not arrived.

Disaster scenarios

A catalogue of what to cause and what to assert, meant to run as Layer 3 tests. Each is a scenario description plus the assertions it must pass; if the Patroni spec reuses them against its own backend, the expected outcomes differ and the causes do not.

Machines: three VMs:

pg-primary and pg-standby (rootfses that already exist, plus pgbouncer);
a third VM for the bouncer and a client loop.

The controller is the test process, reaching the guests with the harness’s SSH CA key, which should map onto Ssh.ClientOpts’ identity and known-hosts fields.

Ways to cause trouble, all things the tree already has. Two of the three are written, in Test.PostgresVms and SreBox.PostgresPair:

a crash: crashCluster — systemctl kill -s KILL postgresql@... for the unit’s whole cgroup, then a wait for the cluster to actually be gone;
a controller killed part-way: convergeUpTo, a step budget, which from the machines’ side is indistinguishable from the controller dying after step k;
a partition: partitionFrom / partitionFromEverythingFor, nft rules built out of Netfilter’s own Table/Chain/Rule vocabulary and rendered by its own command, so the rule text is the tree’s; only the running of it differs, since the guests have no salmon on them and the argv goes over ssh. A partition that hides a machine from the controller cannot be lifted by the controller — the command would have to travel the path it cut — so that one is handed to the machine as cut, wait, heal, and left to run detached.

Alongside them, what a scenario asserts with: dataDirectoryIdentity (the inodes a rewind keeps and a re-clone cannot), assertPrimaryIs / assertInRecovery, and waitFor, since every one of these waits is on a machine doing something in its own time.

#	Scenario	Assert
S1	Switch A→B, then B→A, with a client inserting through the bouncer the whole time	every acknowledged insert is present; the client saw zero errors; a rerun of `run up` changes nothing
S2	S1, with the controller killed after each step k in turn	a plain rerun finishes; no bouncer is left paused; the result is the same as S1
S3	Crash A while it holds writes the standby never got; declare B, first without `may_discard` and then with it	the first pass refuses, and promotes nothing; the second promotes B, which serves writes; A rejoins as B's standby through `pg_rewind`, not a re-clone (its data directory is never unlinked); the replicated rows survive and the un-replicated ones are gone, which is what the flag said
S4	Partition A from B, with no change to the declaration	the check is `Unknown`; nothing is promoted, and in particular the standby is not stopped or rewound; after the partition heals, B catches up on its own and the pair is `Done`
S5	Partition A from B and from the controller, fail over to B with `may_discard = A` while A still holds writes B never got, then let it heal	without the flag the pass refuses, twice: once while A cannot be reached, and again once both machines call themselves primaries; with it, B is promoted, then A is stopped and rewound onto B's history; A's writes behind the partition are gone, as declared
S6	Stop the standby; write past `max_slot_wal_keep_size` on the primary	the primary's `pg_wal` is bounded rather than following the standby down; the slot reports `wal_status = 'lost'`; the check is `Unknown` with the slot named in it; a pass does nothing at all, and in particular the standby's data directory is not unlinked — re-seeding is an operator's decision about throwing data away, not a step
S7	Stop both, in either order, and declare the side that stopped first	the pair converges on the declaration without losing what the other machine wrote after it: B starts, and if it is behind, A is started again so B can catch up from it before the ordinary switchover runs
S8	A stranger's cluster (fresh `initdb`) where a member should be: same address, same cluster name, same port	`Refuse`, in both directions and with `may_discard` naming either side — the flag says whose writes may go, which presumes one cluster, and is not a licence to wipe a machine that was never in the pair. Neither data directory is touched, and the stranger's own databases are still there

S2 and S8 matter most. S2 proves resumability, which is the design’s main claim. S8 proves the refusal that P1 is about, one level up: P1 guards the clone against cloning over a stranger, and S8 guards every step against being applied to one.

All eight are written, in Test.PostgresSwitchoverSpec, and between them they cost the recipe twelve defects. Where those came from is the argument for writing a catalogue of causes rather than of features: not one of the twelve is on the happy path. S1, the scenario in which nothing goes wrong, found nothing, and so did S8, whose guard was written first and stayed right. Every other defect needed a machine that stopped, or was cut off, without being asked to.

Writing S3 alone turned up six defects: the two positions that were not what their names said (see “Observation, then a pure verdict”), the crashed peer treated as a clean one, and the three things the rejoin now does around pg_rewind. None of them is reachable from the happy path, because every one needs a machine that stopped without being asked to – which is the argument for the rest of this catalogue, and the reason the scenarios are written as a list of causes rather than of features.

S4 and S5 turned up three more, in the same spirit. A partitioned standby read as somebody else’s standby and would have been stopped and rewound, so a broken link between the two machines would have taken the standby down with it. Deciding that a member is unreachable took minutes, because nothing bounded ssh’s own retrying — on the one failure this recipe exists for. And the clean stop of a divergent primary recycled the WAL the rewind of it then needed.

S6 turned up the state the pair had no answer for at all: once slots are in use, a standby that falls off the budget can never catch up, and every earlier version of the table would have gone on rewinding at it. S7 turned up the two worst, both of them one declaration away – stopping a primary while the standby was not connected to receive its tail, and then waiting out a budget for records a stopped machine was never going to send. Between them they could take a healthy pair to one machine stopped and the other unpromotable.

Phased plan

P1–P5, each with its own test. P1 gets a Layer 3 case in PostgresReplicationSpec: promote the standby, rerun run up, and assert the data survives.
Observed, parseObserved, nextStep, and Layer 0 tests over the whole table.
Symmetric members, the seed clone and the role node, without bouncers; S1 (minus the client assertions), S2, S7, S8.
Bouncer routing, PAUSE/RELOAD/RESUME; the rest of S1.
pair_may_discard, failover and split brain; S3, S4, S5.
The slot budget and re-seeding; S6.

Open questions

pair_synchronous. Synchronous replication gives zero loss on failover, but with two nodes it blocks every commit while the standby is down. If it is offered, Promote must also clear synchronous_standby_names on the new primary, or the survivor blocks waiting for the dead member. Leaning towards leaving it out of v1.
Minimum Postgres version: 13. It brings max_slot_wal_keep_size, wal_status, pg_rewind -R and its automatic crash recovery, and a reloadable primary_conninfo. The templates already need 13 for DROP DATABASE ... WITH (FORCE).
Should the symmetric members replace primaryReplicationSetup / standbyReplicationSetup, once the fixture and PostgresReplicationSpec are ported? They are strictly less capable, and they are where P1 and P3 live.
Controller placement for real deployments. In the harness it is the test process. For a low-SLA service it is whatever machine runs run up or run serve for the pair. Is it worth also stating that it must not be one of the two members?

Specs

2026-09-25T14:15:36Z

Specs

The specs/ directory holds design sketches: opinionated documents to react to, some of which then got built. Each starts with a Status line saying what of it has shipped, which is reproduced here. These pages are generated from the salmon repository — the repository is the canonical source, and may be ahead of what is published here.

Advanced querying: targeting run at a subset of nodes — Status: implemented. query show/query plan/query extract-directive,
Future work: what a running world knows, and who gets to see it — Status: ideas, not a plan. Four notes taken after trying run serve with the
GCP Support Plan for Salmon — Status: phase 1 (gcloud-first) is implemented through all ten steps of §15:
A generic salmon server: serve behind an API, with clients that show the DAG — Status: milestones 1 to 8 below are implemented (--json via
Multi-user / privilege separation: running parts of a graph as a lesser identity — Status: draft / not implemented. This is a design sketch to react to, not a
Pinning an application’s traffic to a chosen interface via dedicated network namespaces — Status: draft / not implemented. This is a design sketch to react to, not a
Per-node state machines: a supervised tree over a folded graph — Status: milestones 1 to 9 below are implemented, each marked landed with
What is left of specs/per-node-state-machines.md — Status: living plan, update as work continues. As of this writing every
HA Postgres + bouncer + app-instance + LB control plane — Status: draft / not implemented as a whole. Of the phased plan, only the
Postgres with automatic failover: salmon provisions, Patroni decides — Status: draft / not implemented. A design sketch to react to, not a committed
Two-node Postgres with salmon-driven switchover and failover — Status: implemented — phases 1 to 6 of the phased plan and the disaster
Pull mode: a serve that fetches its own declarations — Status: milestones 1 to 7 below are implemented (Salmon.Actions.Follow,
Local qemu VMs + tap/bridge networking for recipe testing — Status: phases 1 to 5 of the phased plan are implemented:
Implementation progress: specs/qemu-test-vms.md — Status: living doc, update as work continues. Companion to
Salmon as PID 1: an init system whose unit graph is a real DAG — Status: the init system itself is not implemented — milestones 1 to 6
Property-based testing for run serve — Status: v1 implemented, in salmon-ops-recipes/test/Test/ServeModelSpec.hs.
Lighter-weight process isolation for services: systemd hardening, bubblewrap, firejail — Status: draft / not implemented. This is a design sketch to react to, not a
Terraform integration: consuming (and optionally driving) existing Terraform-managed infra — Status: draft / not implemented. This is a design sketch to react to, not a

Pinning an application's traffic to a chosen interface via dedicated network namespaces

2026-09-25T14:15:36Z

Generated from specs/netns-traffic-pinning.md — the repository is the canonical source, and may be ahead of this page.

Pinning an application’s traffic to a chosen interface via dedicated network namespaces

Status: draft / not implemented. This is a design sketch to react to, not a committed plan.

Problem

Sometimes one particular app/service’s traffic needs to go out a specific interface — e.g. only one app instance’s egress should go through a WireGuard tunnel while everything else on the box uses the normal uplink, or (on a multi-homed box) a specific service needs to be pinned to a secondary physical NIC. This is occasional, per-service, not a host-wide routing policy change.

This is exactly the open edge specs/service-sandboxing.md left dangling: its “network-namespace tradeoff” section flagged that Tier B’s --unshare-net gives a sandboxed process zero connectivity until something wires it back up, and deferred solving that. This spec is that “something” — and it turns out solving it well also directly answers “how do I pin an app to an interface,” because the namespace boundary and the interface-selection mechanism are the same mechanism: whatever interface(s) live inside a given network namespace are the only ones a process joined to it can use, full stop. No ip rule/fwmark policy routing needed, no risk of a routing-table rule silently not matching what you expected — the kernel simply doesn’t hand that process any other device.

Why a persistent, named namespace instead of bwrap’s `--unshare-net`

specs/service-sandboxing.md’s Tier B creates its network namespace ephemerally, as a side effect of the sandboxed process starting (bwrap --unshare-net ...). That’s fine for “isolate this process from the network,” useless for “wire this process to a specific interface”: there’s no namespace to attach a veth peer or move a device into before the process exists, and no stable name to reference it by afterwards.

Linux (via ip netns) also supports persistent, named network namespaces, independent of any process — created with ip netns add <name>, they show up as bind-mounted files under /run/netns/<name> and stick around until explicitly deleted. This is the right primitive here: the host can fully wire a namespace’s networking (create/move interfaces, assign addresses, set routes) before anything joins it, and systemd has a native way to join a unit’s process to one: NetworkNamespacePath= (systemd ≥245). This composes cleanly with everything in specs/service-sandboxing.md’s Tier A — network-namespace selection and mount/capability hardening are orthogonal Systemd.Service fields, not competing mechanisms.

Recommendation: for anything network-namespace-related, prefer a persistent named ip netns + NetworkNamespacePath= over bwrap’s --unshare-net. Tier B (bwrap) remains valuable for the other namespaces (mount, in particular) it’s already recommended for; this spec effectively narrows bwrap’s job back to “everything except networking” and gives networking its own, more host-controllable mechanism.

Proposed design

1. `Salmon.Builtin.Nodes.NetNamespace` (new)

Same shape as Salmon.Builtin.Nodes.LinuxBridge (prelim-based idempotency, since neither ip netns add nor ip link set ... netns ... is idempotent on its own):

newtype NetNs = NetNs {netnsName :: Text}
    deriving (Eq, Ord, Show)

-- | Creates a persistent, named network namespace.
netns :: Reporter Report -> Track' (Binary "ip") -> NetNs -> Op

-- | Moves an already-existing interface into a namespace. Depends on both
-- the namespace and (by 'Ref', not a hardcoded call) whatever created the
-- interface in the first place -- callers pass that Op in explicitly since
-- this module has no idea whether it's a WireGuard iface, a veth end, or
-- anything else.
moveInterface :: Reporter Report -> Track' (Binary "ip") -> NetNs -> LinuxBridge.DevName -> Op -> Op

-- | A veth pair: one end moved into the namespace, the other attached to a
-- 'LinuxBridge.Bridge' (reusing the qemu spec's bridge primitive, here for
-- a very different purpose -- wiring a process sandbox to an uplink instead
-- of a VM).
data VethPair = VethPair { veth_inside_name :: LinuxBridge.DevName, veth_outside_name :: LinuxBridge.DevName, veth_ns :: NetNs, veth_bridge :: LinuxBridge.Bridge }
vethIntoBridge :: Reporter Report -> Track' (Binary "ip") -> VethPair -> Op

prelim for netns: ip netns list (grep for the name) — same “does the effect already exist” shape as every other ip-backed idempotency check in this codebase. moveInterface’s prelim: ip netns exec <ns> ip link show <dev> succeeding means it’s already there. down for netns: ip netns delete <name> — same teardown-ordering concern Salmon.Actions.UpDown.downTree’s module-level docs already describe for any predecessor shared by several dependents (a namespace shouldn’t be deleted while a device still sits in it, or while a systemd unit is still joined to it) — nothing new to design here, downTree’s existing reverse-dependency-order teardown already handles this as long as the Op graph’s edges reflect the real dependency (interface-in-namespace depends on namespace existing, exactly as sketched above).

2. Setting up routing inside the namespace

Once an interface is moved in, configuring anything about it — including the namespace’s own default route — has to run inside that namespace, not the host’s. Routes.route/Netfilter’s existing ops build plain proc "ip" [...]/proc "nft" [...] CreateProcess values; rather than duplicating those modules with namespace-aware copies, a small generic combinator lets every existing Command-shaped builtin in this codebase be reused unmodified:

-- | Rewrites any CreateProcess to run inside a namespace via `ip netns exec`.
inNetNamespace :: NetNs -> CreateProcess -> CreateProcess
inNetNamespace ns cp = case cmdspec cp of
    RawCommand path args -> cp{cmdspec = RawCommand "ip" (["netns", "exec", Text.unpack ns.netnsName, path] <> args)}
    ShellCommand s -> cp{cmdspec = RawCommand "ip" ["netns", "exec", Text.unpack ns.netnsName, "sh", "-c", s]}

-- | Same, at the Command level -- wrap once, every Op built from the
-- wrapped Command runs inside the namespace, no changes needed to
-- Routes.hs/Netfilter.hs/WireGuard.hs/anything else.
wrapCommand :: NetNs -> Command sym arg -> Command sym arg
wrapCommand ns (Command prepare) = Command (inNetNamespace ns . prepare)

This is the one genuinely new idea worth calling out: no existing builtin needs to change. Routes.route reporter (wrapCommand ns Routes.ipcommand ... ) — wait, Routes.route’s current signature takes a Track' (Binary "ip"), not a Command directly, so the wrapping has to happen one layer up, at whichever Command value a builtin’s smart constructor closes over internally. This is a real, small prerequisite change: Routes.route/WireGuard.iface/etc. would need their internal ipcommand/wgcommand value exposed as a parameter (or a namespace-aware variant added) rather than hardcoded in the function body, mirroring how Systemd.systemdService already takes its Track' (Binary "systemctl") as a parameter instead of assuming one binary track globally. Small, mechanical, per-module change — flagged rather than hidden, same spirit as specs/service-sandboxing.md’s “not a source-breaking change hidden behind a default” note about threading Hardening through Service.

3. Joining a systemd unit to the namespace

Extend Systemd.Service (the same record specs/service-sandboxing.md is already extending with service_hardening) with one more optional field:

data Service
    = Service
    { -- ...existing + service_hardening from specs/service-sandboxing.md...
    , service_network_namespace :: Maybe FilePath   -- NetworkNamespacePath=/run/netns/<name>, or Nothing (today's behavior: host's default netns)
    }

render_service emits NetworkNamespacePath=<path> only when Just. The namespace itself, and everything inside it, is provisioned by an ordinary Op dependency (§1/§2 above) the same way PgBouncer.setup’s config files are a dependency of its systemd unit — deps [netns r ip ns, ...] at the service’s own op construction, so the namespace and its interface exist before systemctl start ever runs.

4. Worked example: pin one app instance’s egress through WireGuard

appNs :: NetNs
appNs = NetNs "ns-internaltool-a-0"

wgIfaceOp :: Op
wgIfaceOp = WireGuard.iface reporter ipTrack "wg-internaltool-a-0" (Ipv4Cidr "10.66.0.2" 32)
    -- ...peer/key setup as any other WireGuardVpn.hs-style recipe already does...

pinnedOp :: Op
pinnedOp =
    Systemd.systemdService reporter systemctl trackConfig cfg
        `inject` NetNamespace.moveInterface reporter ipTrack appNs "wg-internaltool-a-0" wgIfaceOp
        `inject` defaultRouteInsideNs
  where
    defaultRouteInsideNs =
        Routes.route reporter (wrapCommand appNs Routes.ipcommand |> asTrack) (Routes.Route Routes.Default "wg-internaltool-a-0" Nothing)
        -- ^ pseudocode: see §2's note that Routes.route needs a small parameter
        -- change before this composes as cleanly as sketched
    cfg = /* Systemd.Config with service_network_namespace = Just "/run/netns/ns-internaltool-a-0" */

The app instance’s own binary needs no awareness of any of this — it just opens sockets normally; the kernel only ever shows it wg-internaltool-a-0 (plus loopback). No policy-routing rule to get subtly wrong, no fwmark to leak across a recipe boundary.

5. Worked example: pin traffic to a specific physical uplink

Same shape, but §4’s WireGuard interface is replaced by §1’s VethPair into a LinuxBridge.Bridge that’s itself attached to (or routes toward) the secondary physical NIC — reuses the qemu spec’s bridge primitive for a third purpose now (VM networking, then test-VM networking, now process traffic pinning), which is a good sign the primitive is at the right level of abstraction rather than over-fit to one caller.

Open questions

DNS resolution inside the namespace: ip netns exec bind-mounts /etc/netns/<name>/* over /etc/* if present — so a pinned app needing working DNS resolution needs /etc/netns/ns-internaltool-a-0/resolv.conf written (a plain FS.filecontents op, nothing new needed) pointing at a resolver actually reachable from inside the namespace (not necessarily the host’s own /etc/resolv.conf contents, if that resolver is only reachable via the interface being deliberately excluded).
Is this actually static-per-restart, and is that acceptable? NetworkNamespacePath= is read at unit start; changing which namespace (or what’s inside it) takes effect on the next restart, not live. Given the “sometimes” framing in the original ask, confirm that “toggle by redeploying/restarting the service” is the right granularity, as opposed to something that needs to flip while the process keeps running (which would need an entirely different mechanism — e.g. runtime policy routing after all).
The Command-parameterization prerequisite (§2): how many existing builtins actually need this before the worked examples stop being pseudocode — Routes.hs for sure (default route inside the namespace); Netfilter.hs possibly, if per-namespace firewall rules are ever wanted inside a pinned namespace too. Worth doing as its own small prerequisite pass (mirroring specs/service-sandboxing.md’s Tier A prerequisite of threading Hardening through Service’s call sites) rather than ad hoc per-recipe.
ip link add ... netns <ns> direct-create vs. create-then-move: modern iproute2 can create a link directly inside a target namespace in one command; §1 sketches the safer, universally-supported create-in-host-namespace-then-move sequence instead (lets WireGuard.iface stay completely unmodified). Worth revisiting once this is actually run against the target iproute2 version — may simplify the sequencing, doesn’t change the design.
Relevance to the pg-ha control plane vs. WireGuardVpn: is the actual motivating case an app-instance (from specs/pg-ha-control-plane.md) needing to reach something only over a VPN, or is this more about the control-plane’s own management traffic, or something outside either spec entirely (the original ask didn’t specify)? Affects which recipe gets the first real integration in the phased plan below.

Phased plan

Salmon.Builtin.Nodes.NetNamespace: netns, moveInterface, inNetNamespace/wrapCommand. Hand-validate against a throwaway interface (a dummy/veth, not WireGuard yet) before anything real depends on it.
The Command-parameterization prerequisite (open question above) for Routes.hs specifically — smallest slice that makes §4’s worked example real instead of pseudocode.
Systemd.Service’s service_network_namespace field + NetworkNamespacePath= rendering, threaded through existing call sites with Nothing (behavior-preserving), same discipline as specs/service-sandboxing.md’s Tier A rollout.
End-to-end worked example (§4), against a real WireGuard tunnel, on whichever recipe the answer to the last open question points at.
VethPair/§5 (physical-uplink pinning), only once §4 is proven and if a concrete need for the non-VPN case shows up.

Future work

/etc/netns/<name>/ resolv.conf management as its own small helper if this pattern gets reused often enough to be worth a smart constructor rather than a one-off FS.filecontents call each time.
Revisiting whether a live-reconfigurable mechanism (policy routing after all, or some other dynamic scheme) is ever actually needed, depending on the answer to the “static-per-restart” open question above.

Lighter-weight process isolation for services: systemd hardening, bubblewrap, firejail

2026-09-25T14:15:36Z

Generated from specs/service-sandboxing.md — the repository is the canonical source, and may be ahead of this page.

Lighter-weight process isolation for services: systemd hardening, bubblewrap, firejail

Status: draft / not implemented. This is a design sketch to react to, not a committed plan.

Problem

Every service recipe in this repo today (PgBouncer.setup, Nginx.setup, SreBox.Postgrest.setupPostgrest, and the “internaltool”-style app instances sketched in specs/pg-ha-control-plane.md) runs as a plain Systemd.systemdService unit: a [Service] stanza with User/Group/ WorkingDirectory/ExecStart and nothing else — Systemd.Service (Salmon.Builtin.Nodes.Systemd) has no isolation directives at all today. The only heavier isolation option this project has is Salmon.Builtin.Nodes.Podman — a full container (own image, own network namespace by default, its own lifecycle/pull/build machinery).

That leaves a wide gap on the isolation spectrum for exactly the case the [[pg-ha control plane]] spec’s app-instances (internaltool.a.0, postgrest.b, …) sit in: several independent, mutually-untrusting-ish app instances sharing a machine (the “shared tier” from that spec), each with its own pg-user/secret/connstring, where “plain systemd service, wide open to the whole filesystem” is more privilege than any of them need, but “full podman container per instance” is real overhead (image management, a container network to wire into the bouncer/LB story, slower iteration) for something that’s really just one Haskell/whatever binary reading one config file and talking to one upstream port.

What “containerization mode” actually means here — three tiers, not one

Firejail and bubblewrap are both namespace-based sandboxing tools, but they’re worth evaluating against a third option that isn’t a separate tool at all: systemd’s own per-unit sandboxing directives, which every recipe in this repo already goes through Systemd.systemdService for.

Tier	Mechanism	New binary needed?	Isolation granularity
A. systemd hardening	`[Service]` directives (`ProtectSystem`, `PrivateTmp`, `NoNewPrivileges`, `CapabilityBoundingSet`, ...)	No — systemd already runs every service	Coarse-to-medium: named policy levels + path allow-lists, no arbitrary bind-mount graph
B. bubblewrap (`bwrap`)	Unprivileged (user namespaces), wraps the command in an argv of explicit `--bind`/`--ro-bind`/`--unshare-*` flags	Yes, `bubblewrap`	Fine-grained: explicit bind-mount list, explicit namespace unshares, nothing implicit
C. firejail	Wraps the command via a setuid-root helper, driven by profile files (built-in per-app profiles or custom `.profile`s)	Yes, `firejail`	Coarse (profile-driven) unless a custom profile is authored

Recommendation: build A first, B second, treat C as documented-but-not- default. Reasoning:

A costs nothing new. Every existing service recipe already routes through Systemd.systemdService — adding hardening fields to Systemd.Service benefits Nginx/PgBouncer/Postgrest/internaltool immediately, with no new dependency to install, detect, or version-pin. For a same-machine “shared tier” deployment where the main worry is one app instance reading another’s secret file or binding another’s port, ProtectSystem=strict + ProtectHome=yes + a ReadWritePaths= allow-list + NoNewPrivileges=yes already closes most of that gap.
B is the natural next step when A’s directive vocabulary isn’t expressive enough — e.g. a precise, arbitrary bind-mount graph (not just “read-only except these paths”), or wanting the same sandboxing outside a systemd context at all (ad hoc test tooling, the qemu test harness in specs/qemu-test-vms.md). bwrap has no daemon, no setuid binary (modern kernels: unprivileged user namespaces), no profile files to keep in sync with a recipe’s actual needs — every flag is explicit on the command line, which matches this project’s “everything explicit, typed” philosophy better than firejail’s profile-file model.
C (firejail) is real but has a worse security-model fit and a worse fit for this codebase’s conventions: it’s a setuid-root binary with a meaningful CVE history specifically in that setuid helper (privilege- escalation bugs recur because the design requires root to set up the sandbox before dropping it) — a bigger trust footprint than bwrap’s unprivileged-by-default model for a project that otherwise runs everything as an explicit, auditable command. Its profile-file model (/etc/firejail/*.profile, whitelist/blacklist directives in their own DSL) also doesn’t compose with typed Haskell values the way this project’s other builtins do — a Firejail node would either shell out to an existing hand-authored .profile (opaque to salmon, un-typed) or reimplement enough of firejail’s DSL as Haskell types to be worth it, neither of which is attractive. Worth supporting if there’s a concrete reason (an existing firejail profile someone already relies on, a specific feature bwrap lacks), but not the default recommendation.

Kernel namespaces in play

Both bwrap and systemd’s hardening directives are front-ends over the same underlying kernel mechanism (unshare(2)/clone(2) with CLONE_NEW* flags) — worth being explicit about each namespace on its own, since “turn on sandboxing” is really several independent, separately-costed decisions, not one knob. ip netns/Salmon.Builtin.Nodes.LinuxBridge (the qemu spec) is the same network-namespace machinery again, one more data point that this project already relies on namespaces elsewhere.

Namespace	Isolates	bwrap flag	systemd directive	Relevant to an app instance here?
Mount (`CLONE_NEWNS`)	The process's view of the filesystem tree	Implicit — bwrap's whole `--bind`/`--ro-bind` model is a mount namespace; there's no "don't use one"	`ProtectSystem=`, `ProtectHome=`, `PrivateTmp=`, `ReadWritePaths=`/`ReadOnlyPaths=`/`InaccessiblePaths=` (all mount-ns-backed)	Yes, the main one. This is what stops one app instance from reading another's secret file — the whole point of §Tier A/B.
PID (`CLONE_NEWPID`)	Process ID space; can't see or signal processes outside the namespace	`--unshare-pid`	`RestrictNamespaces=`+`PrivatePIDs=` isn't a real systemd directive — systemd doesn't PID-namespace a unit's main process by default and has no single directive that does; would need pairing with bwrap or a raw `unshare --pid` wrapper	Mild value (can't `kill -9` a sibling instance, can't read its `/proc/<pid>/environ`), but note the systemd interaction: systemd's own process supervision (knowing whether the service is still alive, restart-on-failure) tracks the unit via cgroups, not PID visibility, so PID-namespacing a systemd-managed process doesn't break `systemctl status`/restart semantics the way it might naively seem to. Still, this is a place where Tier A alone can't reach — a reason Tier B exists.
Network (`CLONE_NEWNET`)	Network interfaces, routing table, port bind space	`--unshare-net`	`PrivateNetwork=yes`	The one with a real tradeoff here — see below.
UTS (`CLONE_NEWUTS`)	Hostname/domainname	`--unshare-uts` (pair with `--hostname`)	`ProtectHostname=yes`	Cosmetic mostly; low priority, cheap to turn on (no functional downside for a service that doesn't self-report its hostname anywhere meaningful).
IPC (`CLONE_NEWIPC`)	System V IPC objects, POSIX message queues	`--unshare-ipc`	`PrivateIPC=yes` (systemd ≥247)	Low relevance for a TCP-speaking app instance; cheap to turn on, no known downside for the recipes in scope.
User (`CLONE_NEWUSER`)	UID/GID mapping — what makes unprivileged bwrap possible at all (root inside the namespace maps to an unprivileged UID outside)	`--unshare-user` (bwrap uses this internally by default on modern kernels even without the flag spelled out, to be able to do the rest unprivileged)	`PrivateUsers=yes` (systemd ≥232+, with caveats around filesystem UID mapping)	This is the mechanism, not really an independent policy choice for Tier B — it's why bwrap doesn't need setuid. Worth calling out precisely because it's the structural reason Tier B was recommended over Tier C (§ above): firejail's setuid helper does its privilege drop a different, riskier way.
Cgroup (`CLONE_NEWCGROUP`)	View of the cgroup hierarchy	`--unshare-cgroup`	Not directly; `ProtectControlGroups=yes` covers the read-only-ness of `/sys/fs/cgroup` via the mount namespace instead	Minor hardening (hides host cgroup layout); low priority.
Time (`CLONE_NEWTIME`, Linux ≥5.6)	Per-namespace offsets for `CLOCK_MONOTONIC`/`CLOCK_BOOTTIME`	`--unshare-all` pulls it in on new-enough bwrap; no dedicated flag on most bwrap versions in the wild yet	No dedicated directive	Essentially never relevant to anything in this repo (it exists for container-migration/checkpoint use cases). Not worth spending design effort on.

The network-namespace tradeoff, specifically

This is the one namespace decision that actually changes behavior for the app-instance use case, so it’s worth its own paragraph rather than just a table row. --unshare-net/PrivateNetwork=yes gives the sandboxed process only a loopback interface with no route out — no connectivity to the bouncer, the LB, or anything else, until something explicitly re-provides it:

Don’t unshare the network at all (this spec’s leaning, restated from the open question below): rely on Tier A’s RestrictAddressFamilies= (still lets a process talk AF_INET/AF_INET6/AF_UNIX freely, just can’t open e.g. AF_PACKET raw sockets) plus ordinary firewall rules (Salmon.Builtin.Nodes.Netfilter) if per-instance port-level restriction is wanted. Simplest, and correct for the diagram’s shape — every app instance’s only real network need is “reach my bouncer’s port” — but it means app instances share the host’s network namespace and can, in principle, reach each other or anything else the host can reach; the actual boundary is the mount namespace (can’t read each other’s secrets) plus whatever the bouncer/Postgres side enforces via its own authentication.
Do unshare the network per instance: real isolation (an instance literally cannot open a socket to anything but what’s explicitly wired in), but needs one of: a veth pair into a bridge per instance (the same LinuxBridge primitives the qemu spec introduces, reused for a very different purpose — process sandboxes instead of VMs), or slirp4netns- style userspace NAT (another new dependency). Meaningfully more plumbing, and — the concrete cost — DNS resolution and any outbound calls an app instance makes beyond its bouncer (a JWKS fetch, an external webhook) would need that plumbing to actually work, not just the bouncer connection.

No change to the recommendation already in the open questions section below: start without network namespacing, revisit only if a concrete case needs it (e.g. a genuinely hostile/untrusted app instance sharing a “shared tier” box with others, where “can’t even attempt to reach a neighbor’s port” becomes a real requirement rather than defense in depth).

Design goals / non-goals

Goals:

Tier A available to every existing Systemd.systemdService-based recipe by construction, not as an opt-in rewrite each recipe author has to remember.
Tier B (Bwrap) usable both as a Systemd.Start-wrapper (for services) and standalone (for ad hoc sandboxed command execution, e.g. from test tooling), since bwrap’s value isn’t systemd-specific.
Every new field/type has a safe, explicit default — no recipe’s behavior changes just from this landing; hardening is opt-in per recipe/service, matching CLAUDE.md’s “don’t add validation/hardening for scenarios that can’t happen” spirit turned the other way: don’t force isolation decisions onto recipes that haven’t asked for them yet.

Non-goals (v1):

A firejail builtin (§ above — documented as an option, not built, unless a concrete need shows up).
Rootless bubblewrap without unprivileged user namespaces available (some hardened kernels disable CLONE_NEWUSER for unprivileged processes) — detecting and falling back is a real concern but not solved here; see open questions.
Seccomp/syscall-filter authoring (SystemCallFilter= in tier A, --seccomp in bwrap) — both mechanisms support it, but hand-authoring a correct filter per service is its own project or template pipeline; starting with namespace/filesystem/capability isolation only.
A per-recipe “which tier” policy/decision engine — the tier a given service uses is picked by whoever writes the seed/recipe wiring, not inferred from a Tier/HardwareProfile value automatically (though see the pg-ha spec’s open questions — this may become relevant there later).

Proposed design

Tier A: `Systemd.Service` hardening fields

Extend Salmon.Builtin.Nodes.Systemd’s Service/render_service (the only change needed to an existing module — everything else here is new modules):

data Hardening
    = Hardening
    { harden_protect_system :: Maybe ProtectSystemLevel  -- ProtectSystem=
    , harden_protect_home :: Bool                         -- ProtectHome=yes|no
    , harden_private_tmp :: Bool                           -- PrivateTmp=
    , harden_no_new_privileges :: Bool                     -- NoNewPrivileges=
    , harden_read_write_paths :: [FilePath]                -- ReadWritePaths=
    , harden_capability_bounding_set :: [Text]              -- CapabilityBoundingSet= (empty list = "" = drop all)
    , harden_restrict_address_families :: [Text]            -- RestrictAddressFamilies=, e.g. ["AF_INET", "AF_UNIX"]
    , harden_protect_hostname :: Bool                        -- ProtectHostname= (UTS namespace; cheap, see "Kernel namespaces in play")
    , harden_private_ipc :: Bool                              -- PrivateIPC= (systemd >=247; IPC namespace)
    , harden_private_network :: Bool                          -- PrivateNetwork= (network namespace) -- see the network-namespace tradeoff discussion; False by default for every recipe in scope today
    }

data ProtectSystemLevel = ProtectStrict | ProtectFull | ProtectReadOnly

noHardening :: Hardening   -- every field off/empty; today's exact behavior

data Service
    = Service
    { -- ...existing fields unchanged...
    , service_hardening :: Hardening   -- new field, defaults to 'noHardening' everywhere existing code constructs a 'Service'
    }

render_service grows the corresponding [Service] lines only for non-default Hardening fields (e.g. omit ReadWritePaths= entirely if the list is empty, rather than emitting an empty directive). Because Service already has every field spelled out positionally at each of its four current call sites (Nginx.setup, PgBouncer.setup, SreBox.Postgrest’s systemd wiring), this is a real, mechanical edit to each — not a source-breaking change hidden behind a default, since Haskell records don’t have optional-with-default construction without deriving extra machinery this codebase doesn’t currently use. Worth doing as its own small PR (touch the 3–4 call sites, thread noHardening through) before building anything Tier-A-consuming on top.

A concrete “shared tier app instance” profile, for the pg-ha spec’s internaltool-instance recipe to use once it exists:

appInstanceHardening :: Hardening
appInstanceHardening =
    noHardening
        { harden_protect_system = Just ProtectStrict
        , harden_protect_home = True
        , harden_private_tmp = True
        , harden_no_new_privileges = True
        , harden_read_write_paths = []  -- nothing; app instances are stateless besides their own connstring/secret, both read-only
        , harden_capability_bounding_set = []  -- drop all
        , harden_restrict_address_families = ["AF_INET", "AF_INET6"]  -- talks to its bouncer over TCP, nothing else
        }

Tier B: `Salmon.Builtin.Nodes.Bwrap`

A new builtin, in the same shape as Salmon.Builtin.Nodes.Netfilter/ Salmon.Builtin.Nodes.LinuxBridge (renders an explicit argv, no persistent state of its own to be idempotent about beyond “the wrapped process is running” — which is exactly what wrapping a Systemd.Start already handles for free):

data BindMount = BindMount { bindHostPath :: FilePath, bindGuestPath :: FilePath, bindReadOnly :: Bool }

data BwrapConfig
    = BwrapConfig
    { bwrap_binds :: [BindMount]           -- explicit allow-list; --ro-bind or --bind per entry
    , bwrap_unshare_net :: Bool             -- --unshare-net; see "the network-namespace tradeoff" -- False by default
    , bwrap_unshare_pid :: Bool             -- --unshare-pid
    , bwrap_unshare_uts :: Bool              -- --unshare-uts
    , bwrap_unshare_ipc :: Bool              -- --unshare-ipc
    , bwrap_unshare_cgroup :: Bool           -- --unshare-cgroup
    , bwrap_die_with_parent :: Bool         -- --die-with-parent (avoid orphaned sandboxed processes)
    , bwrap_new_session :: Bool             -- --new-session
    , bwrap_hostname :: Maybe Text          -- --hostname, only meaningful if bwrap_unshare_uts is set
    }

-- Mount and user namespaces aren't separate boolean fields here: mount-
-- namespacing is inherent to bwrap's whole bind-mount model, and user-
-- namespacing is the privilege-drop mechanism bwrap always relies on to run
-- unprivileged in the first place -- see "Kernel namespaces in play" for why
-- these two aren't optional the way the others are.

-- | Wraps a systemd Start's command+args in a bwrap invocation.
wrapStart :: Track' (Binary "bwrap") -> BwrapConfig -> Systemd.Start -> Systemd.Start
wrapStart bin cfg (Systemd.Start path args) =
    Systemd.Start "/usr/bin/bwrap" (bwrapArgs cfg <> [Text.pack path] <> args)

Every bwrap invocation needs / itself bound (typically --ro-bind / / as the base, then the explicit allow-list layers read-write exceptions on top) — bwrapArgs renders that base plus one --ro-bind/--bind pair per BindMount, --unshare-net/--unshare-pid if requested, --proc /proc --dev /dev (bwrap needs these explicitly, unlike a full container runtime’s defaults). This is the part that most benefits from hand-testing against a real service before finalizing exact flags — sketched here at the “what fields does the type need” level, not “the exact argv,” per this project’s practice of validating shell-invocation shapes for real (see how specs/qemu-test-vms.md’s phased plan puts a hand-validated boot before wrapping in a node — same discipline applies here).

wrapStart composes with Systemd.systemdService (or Qemu-style bare process execution — see specs/qemu-test-vms.md) — PgBouncer.setup/ Nginx.setup/Postgrest’s systemd wiring would build their Start as today, then apply wrapStart bwrapBin appInstanceBwrapConfig before handing it to Systemd.Config. No change needed to Systemd.hs itself for this tier, unlike Tier A.

Also usable standalone (not just for services) — e.g. Bwrap.runSandboxed :: Track' (Binary "bwrap") -> BwrapConfig -> Command sym arg -> Command sym arg as a Command-level wrapper, for anywhere else in this codebase that shells out to something that could benefit from the same treatment without going through Systemd at all.

Tier C: firejail (documented, not built)

If a concrete need shows up: a Firejail builtin would look like Bwrap’s wrapStart (prefix the command with firejail --profile=<path> -- or an explicit --noprofile + flag list), but this project’s convention is typed values driving rendered config, not shelling out to an externally-authored profile file salmon can’t inspect — so if this ever gets built, prefer rendering an explicit firejail flag list (mirroring BwrapConfig’s shape) over accepting a .profile path, keeping the same “everything explicit” property Tier B has. Not scheduled; revisit only if Tier A+B turn out insufficient for some concrete case.

Open questions

Network isolation vs. the bouncer: see “the network-namespace tradeoff” above — leaning towards not unsharing net by default, only revisited if a concrete “shared tier” case needs stronger-than-mount- namespace isolation between instances.
Capability set specifics per existing recipe: Nginx/PgBouncer may need to bind low ports (CAP_NET_BIND_SERVICE) if configured to listen below 1024 — appInstanceHardening’s “drop all capabilities” sketch above needs per-recipe review, not a single one-size-fits-all profile.
Detecting bwrap availability / unprivileged userns support: some hardened kernels (grsecurity-influenced sysctls, some container hosts) disable unprivileged CLONE_NEWUSER. Should Bwrap’s Op have a prelim that checks this and reports something actionable, or is “let bwrap itself fail loudly and let Binary.untrackedExec’s existing non-zero-exit-throws behavior surface it” (CLAUDE.md’s existing “failure must not be swallowed” convention) good enough? Leaning towards the latter — consistent with how every other builtin in this codebase handles a missing precondition, no new mechanism needed.
Does Tier A’s Hardening threading break existing test fixtures? Test.PostgresInitSpec/Test.PodmanSpec construct Systemd.Service values directly in a few places (worth grep-confirming before landing the service_hardening field) — a mechanical noHardening addition at each existing call site, but worth doing carefully in one pass rather than piecemeal to avoid a half-migrated Service type.

Phased plan

Tier A: add Hardening/ProtectSystemLevel/noHardening to Systemd.hs, thread service_hardening through Service’s existing 4-ish call sites with noHardening (behavior-preserving), confirm cabal build+existing tests still pass.
Write appInstanceHardening (or similar) and apply it by hand to one real service (e.g. PgBouncer.setup’s call site, or once it exists, the internaltool-instance recipe from specs/pg-ha-control-plane.md) — hand-test that the service still starts and does its job with ProtectSystem= strict/ReadWritePaths=/etc. actually applied, not just that it compiles.
Salmon.Builtin.Nodes.Bwrap (Tier B): BwrapConfig, wrapStart, bwrapArgs. Hand-validate the rendered argv against a trivial command first (bwrap --ro-bind / / --unshare-net -- echo hi-shaped), the same “hand-validate before wrapping in a node” discipline as the qemu spec.
Apply wrapStart to one real service, confirm it still works under the combination of Tier A and Tier B (they’re not mutually exclusive — systemd’s own directives plus a bwrap-wrapped ExecStart compose, since systemd applies its sandboxing to whatever process ExecStart names, which by then is bwrap itself wrapping the real binary).
Firejail (Tier C), only if a concrete need shows up.

Future work

Seccomp filter authoring/templates (both tiers support the underlying mechanism; authoring correct filters is future work).
A Tier/HardwareProfile-driven policy (from specs/pg-ha-control-plane.md) that picks a sandboxing tier automatically based on shared-vs-dedicated deployment, once real usage shows what the right default actually is.
Rootless-bwrap-unavailable fallback (detect, degrade to Tier A only, warn).

Per-node state machines: a supervised tree over a folded graph

2026-09-25T14:15:36Z

Generated from specs/per-node-state-machines.md — the repository is the canonical source, and may be ahead of this page.

Per-node state machines: a supervised tree over a folded graph

Status: milestones 1 to 9 below are implemented, each marked landed with its deviations recorded in place: check :: IO CheckResult (Builtin/Extension.hs), Salmon.Op.Dag, Salmon.Op.Ledger, both synchronous drivers over them (Actions/UpDown.hs), Salmon.Op.Rewrite, Salmon.Op.Status/Salmon.Op.Mailbox/Salmon.Actions.Concurrent, Salmon.Actions.Upkeep with Salmon.Op.Supervision, Extension.managed with Nodes/Daemon.hs, and supStrategy = RestForOne. What is left — the (R) and (I) items, of which only I2 and I4 are still open — is tracked in specs/per-node-state-machines-remaining.md. Kept as the design record.

This revision follows the deptrack-devops precedent (Devops/Graph.hs), which already solved most of this, and refines its per-node intent stream into a per-node ledger. It supersedes the first draft’s five-state machine.

For where this stands — what shipped, what is left, and which tradeoffs are still reversible — read specs/per-node-state-machines-remaining.md §“Where this stands” rather than this file. This one is the design and the record: every milestone below is marked landed with its departures written in place, which is the right place to read them (next to what they changed) and the wrong place to get an overview. Salmon.Builtin.Nodes.Supervised — a first cut at supervision under the current execution model — has since been removed in favour of this design; Serve.serveWith still exists but has no consumer, and is subsumed here too.

A later revision corrected six things checked against the tree rather than remembered: Ref is location-addressed and not content-addressed; the ledger has to carry edges and retire rather than delete a retracted contribution; the magma holds Extensions and not Ops; the synchronous driver cannot be built on the FSMs because nothing in them is terminal; Completed is a CheckResult; and merging prelim into check is not quite behaviour-free. Each is marked in place.

Problem

Execution in salmon is a traversal. upTreeWith (Actions/UpDown.hs:128) walks an expanded Cofree Graph post-order; downTreeWith (:233) collapses that Cofree into a Ref-level DAG and releases nodes in reverse-dependency order. Both are one-shot: they start, visit each node once, and return IO Bool.

Three consequences:

Nothing is long-lived, so “keep this running” needs apparatus bolted on the side. The removed Supervised carried a pid table, a polling reaper and an STM wakeup channel — 584 lines — for no reason other than that Extension.up :: IO () (Builtin/Extension.hs:39) returns and forgets, leaving a handle nowhere to live.
No dependency-aware restart. A predecessor going away is not an event any dependent can observe; serve’s converge (Actions/Serve.hs:1013) gates on direction and convergence only.
No parallelism. Independent subtrees are walked one at a time, though the DAG already is the parallelism structure.

And one structural gap that (3) hides: the traversal only ever knows a node’s predecessors. Teardown ordering — “free once your last dependent is done” — has to be recovered by collapsing the Cofree and counting dependents (Actions/UpDown.hs:266–:317), which is where the “directory not empty” bug CLAUDE.md records came from.

The model: four structures, one control loop

Everything below is derived from folding declared graphs into state, rather than from walking a graph:

structure	type	what it is
magma	`HashMap Ref Node`	every node ever seen, one representative per `Ref`; `Node` is the `Extension` plus its shorthand, not an `Op`
precedence	`HashMap Ref [Ref]` ×2	dependencies and dependants, derived from the ledger
ledger	`Map DirectiveDigest Contribution`	which declarations still want which nodes and edges
processes	`HashMap Ref (Async (), TVar Status, TBQueue Instruction)`	the machine per node, its observable state, and its mailbox

The magma and precedence come from the comonadic fold: expand already produces the Cofree, and the collapse downTreeWith used to perform internally becomes a first-class reusable step that emits both adjacency directions instead of one — Salmon.Op.Dag, as of milestone 2. Because it is keyed by Ref, folding a second graph into the same structures is a merge, not a replacement — which is what makes the graph dynamic.

`Ref` is location-addressed, not content-addressed

mkRef (salmon-ops/src/Salmon/Op/Ref.hs) hashes a kind tag plus an author-chosen identity key, and that key is deliberately not the node’s behaviour: filecontents keys on the path alone (Nodes/Filesystem.hs:59 — mkRef "file-contents" path), as do dir and bash-run. So an equal Ref means “the same effect site”, not an equal node. An earlier draft of this document said the opposite and leaned on it twice; both leanings are withdrawn.

The consequence for the magma is that a Ref collision between two declarations is a real choice, not a no-op: two live declarations writing different bytes to /etc/foo produce one magma entry, and something has to decide whose.

Decided: last-writer-wins, and the fold reports the conflict. Last-wins because re-declaring is the normal way an operator changes a node, and first-wins would make the second declaration silently inert. Reported because the fold is the first thing in salmon that can notice: upTree dedupes by Ref today (Actions/UpDown.hs:144, first-encountered wins) but per-traversal and discarded at the end, so two declarations fighting over one file is currently invisible. The fold sees both representatives at once.

Comparing representatives needs an equality the magma can compute, and Extension has none — up :: IO () is not Eq. So the conflict test is on the fields that are comparable: help, notes, the shorthand, and the rendering of dynamics. That is a heuristic and will miss a node whose action changed behind an identical description. It is still strictly more than the zero available today.

This still does not need instance Semigroup Extension (Builtin/Extension.hs:58), whose up a <> up b would run both actions: choosing a representative is not combining two.

Two knock-on effects, both real:

A Managed node whose command line changed but whose ref key did not is the same node. Last-writer-wins replaces the magma entry, but the running process belongs to a machine started from the old one, so that machine has to be restarted when its representative is replaced. This is the one case where swapping the Async is right — see the mailbox section, which is where the earlier draft’s second, wrong argument lived.
Tightening a ref key is a per-node fix, not a global one. Putting a content digest in filecontents’ key would make a content change a different node — correct there, but the same move on a long-running service would make every config tweak a new machine. Left to node authors, node by node, and out of scope here.

Storage is bounded by nodes and live declarations, not by history

This is what makes the retention work already shipped (worldEpochs/prune, Actions/Serve.hs:1190) smaller, not absent — an earlier draft said “unnecessary”, which was too strong. Today a retired epoch’s whole Cofree Graph Op is kept because it is the only remaining description of how to tear its nodes down. Here the magma holds each node’s down once, keyed by Ref, and the ledger holds the edges as flat Set (Ref, Ref)s — so nothing needs a per-declaration graph, which is where the saving is. What a retracted declaration does still need is its Contribution: two flat sets, kept only until its nodes settle. That is prune’s existing rule (Actions/Serve.hs:1190: keep an epoch if it is active, or if it still describes a node wanted TurnDown) applied to something orders of magnitude smaller than a graph.

Node is the Extension plus its shorthand, never an Op. Not a detail: Op = OpGraph Identity Actions', whose predecessors field retains the whole expanded closure, so a HashMap Ref Op would retain every graph ever folded and bound nothing at all. All structure lives in precedence; the magma holds only what a node is. Serve.NodeState (Actions/Serve.hs:163) already does exactly this — shorthand and help text, never the Op — which is why worldNodes is affordable today.

A node leaves all four structures when it has dropped out of desired and its machine has settled in Down; a Contribution leaves the ledger once none of its refs is still standing.

Folding declarations into the ledger

Decided: the ledger is a set, not a count — and it carries edges as well as nodes.

type Edge = (Ref, Ref)          -- (dependency, dependant)

data Contribution = Contribution
    { contribRefs  :: !(Set Ref)
    , contribEdges :: !(Set Edge)
    , contribLive  :: !Bool   -- False once retracted, until its nodes settle
    }

type Ledger = Map DirectiveDigest Contribution

desired :: Ledger -> Set Ref
desired = Set.unions . fmap contribRefs . filter contribLive . Map.elems

precedenceOf :: Ledger -> Set Edge
precedenceOf = Set.unions . fmap contribEdges . Map.elems

A declaration is a (graph, direction) pair, and folding it is an insert or a retraction keyed by the encoded directive — which is exactly what worldActive (Actions/Serve.hs) already does, with the ref-set LogEntry.logRefs already carries. A node is wanted up iff it appears in some live contribution.

Worked through the example — g0 declares {A,B} up, g1 retracts g0, g2 declares {A} up:

                       ledger                          desired
  up   g0 {A,B}        {g0: {A,B} live}                {A, B}
  down g0              {g0: {A,B} retiring}            {}       A and B both go down
  (A,B settle Down)    {}                              {}       g0 collected
  up   g2 {A}          {g2: {A} live}                  {A}      A back up, B stays down

Counting was the wrong instinct — mine as much as anyone’s — and every property it needed hand-maintaining, the set gets structurally:

hazard under counting	under sets
a node reached by several paths in one graph double-counts	it is a `Set`; one membership
`down` of something never up drives the count negative	delete of an absent key is a no-op
re-declaring the same seed reaches 2, so one `down` strands it up	same digest, same key: insert replaces
two declarations wanting the same node must not cancel each other	union; the node leaves when the last set does

The one thing counting bought was O(1) lookup per node. That is not worth buying: declarations are rare (a human or a control plane types them), while node state changes are the hot path and do not touch the ledger at all. So recompute desired on declaration change, and memoise only if it ever shows up in a profile.

Storage stays bounded by the live and retiring declarations rather than by history — and by two flat sets each rather than a graph — so the append-only problem worldEpochs had does not come back.

Why edges are in the ledger, and why a retraction retires rather than deletes

Both answers are the same answer: edges have to be retractable, and a retracted declaration’s edges are needed after it is retracted.

An earlier draft kept precedence as a single accumulating structure and said nothing about how an edge leaves it. That is not a harmless omission, because a stale edge is not inert here the way it would be in a walk. Given a stale A → B where B is no longer in desired, B settles Down/TurnDown/Stable, and A’s waitStability TurnUp Stable [B] retries forever: a silent deadlock, no report, no failure — strictly worse than the Blocked a traversal would have produced. So precedence is derived from the ledger on declaration change, exactly as desired is, and for the same reason.

But it cannot be derived from the live contributions alone. Retracting g0 is precisely the moment A → B matters most: both nodes are wanted TurnDown, and that edge is what says B comes down before A does. Delete the contribution outright and the teardown order is gone with it. Hence contribLive: a retraction clears the flag, which removes the contribution from desired while leaving its edges in precedenceOf, and the contribution is collected only once none of its refs is still standing.

That is prune’s rule (Actions/Serve.hs:1190) restated over two flat sets instead of a Cofree Graph Op, which is the whole of the saving claimed above.

The node process

deptrack does not use one five-state machine. It uses two, and that is better: an upkeep machine and a downkeep machine, three states each.

data UpkeepState   = WaitUp   | Upping  | Up
data DownkeepState = WaitDown | Downing | Down

Plus the piece the first draft missed entirely — a node’s observable state is not just which state it is in, but whether it has settled:

data Stability = Stable | Transient
data Status = Status
    { statusCheck      :: !CheckResult
    , statusDirection  :: !Direction
    , statusStability  :: !Stability
    , statusLastActive :: !Word64   -- monotonic ns; see below
    , statusOutput     :: !(Ring Text)
    }

A node’s TVar Status is created Transient, before its machine starts. waitStability below reads direction and stability only, so a status initialised Stable/TurnUp would let every dependant proceed before the node had done anything at all. One line, and otherwise the kind of thing that surfaces as a heisenbug on a wide graph.

Progress, not just settledness

deptrack has exactly two stability values, and two is one short: a node that has been Transient for four seconds because it is building, and a node that has been Transient for four seconds because it is wedged, are the same value. That is precisely the distinction a supervisor’s restart decision needs.

Decided: keep two states, and add a monotonic activity timestamp. Anything observable the node does bumps statusLastActive — a state transition, a check completing, and in particular a line arriving in the output ring. “Wedged” is then a derived predicate rather than a third state:

wedged now st = statusStability st == Transient
             && now - statusLastActive st > watchdog

Decided: watchdog is authored on the node, not guessed globally. A cabal build is legitimately silent for minutes and a web server’s startup is not, and no global or per-kind default can know the difference — the node author does. This is systemd’s WatchdogSec= in all but name, and it lives next to Restart as part of the same per-node policy. A node that declares no watchdog is never considered wedged, which is the right default: silence is only evidence when someone has said what silence would mean.

This keeps Stable/Transient as the thing waitStability blocks on (a timestamp would make that condition unstable and wake dependants constantly) while giving the supervisor what it needs to tell slow from stuck. It also means a chatty build is visibly progressing for free, which is the common case and the one an operator most wants to see.

Output: a bounded ring per node

Decided: each node keeps a bounded ring of its recent output. Ownership (below) makes capturing a Managed node’s stdout/stderr possible for the first time; a ring rather than a buffer is what stops a chatty service from being quietly accumulated into the heap. It pays for itself three times over: it is what an operator wants when a node is Failure, it is what feeds statusLastActive, and it means a crash report can carry the last N lines that preceded it rather than just an exit code.

Size is per-node and small (a few hundred lines); anything wanting real logs should be shipping them somewhere, which is a node of its own.

Ordering is STM, not messages

Dependency ordering is a blocking read of the neighbours’ TVar Status:

waitStability :: Direction -> Stability -> [TVar Status] -> STM ()
waitStability dir stab tvars = do
    sts <- traverse readTVar tvars
    if all (\s -> statusStability s == stab && statusDirection s == dir) sts
        then pure ()
        else retry

A node turning up waits on its dependencies being Stable/TurnUp; a node turning down waits on its dependants being Stable/TurnDown. That single inversion is the whole of the teardown-ordering problem the current downTreeWith spends eighty lines on — and it is only expressible because the fold kept both adjacency directions.

retry also means no polling, no wakeup channel and no scheduler: a node blocks until a neighbour’s state actually changes. Serve.serveWith’s STM (Set Ref) plumbing exists only because nodes today have no state of their own to block on.

…but instructions are messages: one mailbox per node

Neighbour state is pulled with retry. Instructions have to be pushed, because a node must be tellable things that are not derivable from its neighbours at all:

Force — run up even though check says Skipped; the operator knows something the check does not;
Skip — treat as satisfied without acting (Query.forceSkip’s semantics);
Recheck — collapse the adaptive delay to its floor and check now;
Pause/Resume — stop the upkeep FSM without tearing the effect down.

Decided: a bounded mailbox per node, rather than swapping the Async in the magma. Swapping cannot express a transient instruction without killing and restarting the machine — for a Managed node that means killing a healthy process to set a flag.

An earlier draft gave a second reason — that swapping is redundant, since “a node whose definition really changed has a different Ref” — and that reason is withdrawn, for the reason §“Ref is location-addressed” gives: a ref key is an effect site, not a behaviour. So swapping keeps one narrow job after all, and only that one: when the fold replaces a node’s representative under last-writer-wins, the machine started from the old representative is cancelled and restarted. That is a fold-time event, not an instruction. Everything an operator wants to say still goes through the mailbox.

Decoration still has a place, but at fold time rather than run time. A Plan is part of a declaration, so Query.forceSkip is applied as the graph is folded into the magma: the node enters with its check pre-answered and no running machine is disturbed. Declaration-time forcing is decoration; run-time forcing is the mailbox. That split is what keeps run up --plan meaning the same thing it does today while still allowing an operator to force one node in a live serve.

The two inputs compose in STM, which is the point:

atomically $
      (Left  <$> readTBQueue mailbox)
  <|> (Right <$> waitStability TurnUp Stable dependencies)

Bounded, because a control plane can outrun a node busy doing something slow, and an unbounded mailbox turns a wedged node into a memory leak. Overflow drops the oldest — these are statements about current intent, so stale ones are the ones to lose — and a dropped instruction appears in the report stream, or forcing a node becomes silently unreliable.

Provisional on purpose. Whether drop-oldest is right, or whether a coalescing mailbox (at most one pending instruction of each kind) is better, depends on how instructions actually get used, and there is no way to know that before something is driving them.

“Keep this running” is `check` plus re-`up`

The upkeep machine’s steady state is not “hold a process handle”. It is a periodic check with an adaptive delay:

fsm delay Up xyz = threadDelay delay >> do
    newStatus <- checkUpStatus xyz
    case statusCheck newStatus of
        Success -> fsm (increaseDelay delay) Up     xyz   -- ×2, capped 60s
        Skipped -> fsm (increaseDelay delay) Up     xyz
        _       -> fsm (decreaseDelay delay) Upping xyz   -- ÷2, floor 500ms

This is the most important idea to take from deptrack, and it is why Supervised was deleted rather than ported. Supervision becomes:

up — spawn it;
check — is it alive?;
down — kill it;

with the FSM’s adaptive delay being the backoff, and no pid table, no reaper, no getProcessExitCode, no wakeup channel.

Decided: a check may be arbitrarily expensive. Most should be cheap, but there is no reason to forbid one that probes a remote endpoint or runs a real query, and forbidding it would just push authors into writing a worse check. Three consequences follow, and are the whole cost of the decision:

the adaptive value is a delay between checks, not a period, so a slow check reduces its own frequency and the load is self-limiting;
a check in flight must not hold the status TVar or block the mailbox, or a slow check makes the node unresponsive to Force and to teardown — and cancel must be able to interrupt one, which means checks must be interruptible IO rather than a long uninterruptible FFI call;
a check in progress counts as activity for the watchdog above, or every slow check would look exactly like a wedge. It also generalises past what Supervised can do: a node whose process salmon does not own — a systemd unit, a container, something on another host — is supervised exactly the same way, because the model never assumed ownership.

`check` is the field nobody implements

Extension already has check :: IO () (Builtin/Extension.hs:42). It is assigned by zero nodes across salmon-ops and salmon-ops-recipes, and Salmon.Actions.Check.checkTree has zero callers — it is not wired into CommandLine at all. Meanwhile prelim :: IO Requirement (:40) is implemented 23 times and answers very nearly the question the FSM needs: Skippable means “the effect is already in place”.

So the change is a merge, not an addition:

data CheckResult
    = Success
    | Skipped
    | Completed        -- did its work and stopped; see the restart policy
    | Failure !Text
    | Unknown

check :: IO CheckResult     -- replaces both `prelim` and today's dead `check`

Requirement derives from it: Success/Skipped/Completed ⇒ Skippable, Failure/Unknown ⇒ Required. Erring toward Required is the safe direction — a node whose check cannot tell gets re-up’d, and up is required to be idempotent anyway — but it is a choice, and it is the one that keeps a node with a broken check converging rather than stalling.

Of the 23 prelim sites, 22 are nodes and port mechanically. The twenty-third is Query.forceSkip (Actions/Query.hs:146), which rewrites the field rather than implementing it, and it is the one this document later reinterprets as fold-time decoration. Salmon.Actions.Check is deleted rather than fixed.

One behaviour change rides along, and it is an improvement worth naming rather than a regression to hide: today act.extension.prelim is evaluated outside the try @SomeException that wraps up (Actions/UpDown.hs:157 vs :164), so a prelim that throws escapes upTreeWith entirely and kills the whole traversal instead of failing one node. Under check :: IO CheckResult that becomes a Failure value, contained like any other. So milestone 1 is mechanical but not quite “no behaviour change”.

notify :: IO () (:43) is the same story and gets the same treatment: zero implementations, and Salmon.Actions.Notify.notifyTree has zero callers. Decided: removed, along with Salmon.Actions.Notify. Whatever it was reaching for, Broadcast/Reporter is the thing that actually carries per-node events here.

That covers every effect salmon does not own. For one it does own, check is necessary but not sufficient — see the next section.

Recovering process ownership

Polling check is the right answer for an effect salmon did not spawn: a systemd unit, a container, a service on another host. It is a poor answer for a process salmon started itself, and an earlier draft of this document gave ownership up too readily. Three things go with it:

The exit status. check answers alive-or-dead; waitForProcess answers ExitFailure 137. Without it there is no way to express Restart=on-failure — the common and correct policy of not restarting a service that exited cleanly.
Timeliness. The adaptive delay backs off on success, to a 60s cap. A service that dies a second after a successful check stays dead for a minute. For anything user-facing that is the difference between supervision and a cron job.
Identity. A pidfile plus kill -0 cannot survive pid reuse, and cannot tell a live process from a zombie.

The per-node machine recovers all three, and it is precisely the machine that makes it possible: the handle never has to escape, because the node’s own thread is in scope for the effect’s entire lifetime. That was impossible under the traversal, where up :: IO () ran and returned into nothing.

data Lifecycle
    = OneShot (IO ())        -- returns; the effect persists on its own
    | Managed (IO ExitCode)  -- blocks while the effect is up; returns when it stops

A Managed node’s action is the process:

Managed $ withProcess cp $ \ph -> waitForProcess ph

Upping runs the action. A OneShot node reaches Up when it returns; a Managed node is Up for as long as it is still running, and the ExitCode it eventually yields is the reason it stopped. Teardown is cancel on the Async, and the bracket inside withProcess does the killing. This is why no pid table appears anywhere in this design: the pid is a local variable on the owning thread’s stack.

So Up has two exits, and a node may have either or both: the action returning (a managed process died, with its status) or a check failing (an unowned effect went away). Concretely Up races the running action against the adaptive check timer; a node supplying only check behaves exactly as the previous section describes, and one supplying only Managed never polls.

Three consequences worth settling before this is built:

-threaded stops being a recommendation and becomes a requirement. waitForProcess blocks the entire runtime on the non-threaded RTS, and this design has one such call per managed node. The test suite already met this: the (now removed) Test.SupervisedSpec needed ghc-options: -threaded before a blocking read stopped freezing the reaper along with everything else. That flag is still on the test suite for serve’s own reader thread.
cancel alone is not a stop. withCreateProcess’s cleanup sends SIGTERM and waits; a service that ignores it wedges the teardown. The grace-then-SIGKILL escalation Supervised.serviceDown had is worth recovering into the bracket, as is create_group = True so the whole group goes. Both are in git history at f9d7116 rather than lost.
This is waitpid(pid), not waitpid(-1). Supervised polled getProcessExitCode partly to dodge a race with Binary.untrackedExec that, on reflection, never applied to it — waiting on a specific child cannot steal another child’s status. That race is real only for a process obliged to reap orphans it never spawned, i.e. PID 1, which is exactly why specs/salmon-as-init.md keeps a Rust half. Owned children are waited on individually; orphans stay someone else’s problem.

The restart policy reads the exit code

Decided: yes — that is most of what ownership is for, and it is what makes Restart=on-failure expressible at all.

data Restart = Always | OnFailure | Never

With one ordering subtlety that falls out of having both mechanisms: consult check before the policy. A process that exits 0 because it daemonised is still up, and check is the only thing that can say so.

on exit with code c:
    check >>= \case
        Success -> Up              -- it forked; the effect is there regardless
        _       -> apply policy to c

That single line handles the double-fork case for free — the one shape Supervised could not have handled at all, since a handle to a process that has exited tells you nothing about the daemon it left behind.

Default OnFailure. A Managed node that exits cleanly finished on purpose, and restarting it fights its own decision.

Decided: Completed is a CheckResult, not a fourth upkeep state. An earlier draft called it “a terminal Completed condition that is visibly not Up”, which does not typecheck against either structure this document declares: UpkeepState has three constructors, and Status has no field that could hold a fourth. So the machine rests in Up and statusCheck reads Completed rather than Success.

Two things follow, and both are wanted:

It counts as converged. The batch driver treats it as done and it does not hold up quiescence, so a job modelled as Managed lets run up terminate normally.
Its dependants proceed. waitStability reads direction and stability only, so a Completed node is indistinguishable from an Up one to everything downstream — which is exactly right for a migration or a build step, and is the reason Completed belongs in CheckResult rather than somewhere waitStability would have to learn about.

The price is that status must render Up+Completed differently from Up+Success rather than collapsing both to “fine” — a service that quietly completed is exactly the thing an operator needs to see. Always is for services that exit 0 on reload; Never for a one-shot job modelled as Managed only to capture its output.

Note this default differs from systemd’s Restart=no, deliberately: a node declared up that has stopped being up is a convergence gap, and quietly accepting it would make this model weaker than run up is today.

The supervision tree

Framing the node machines as Erlang processes under a supervisor gives two things beyond “an Async per node”.

Let it crash. A node machine that throws is not caught inline. The control process monitors its children (waitCatch on the Async) and applies a restart policy. That splits two kinds of failure the current design conflates: the managed effect stopped (handled inside the upkeep FSM, Up → Upping) versus the machine managing it died (handled by the supervisor). It is the same two-level structure as specs/salmon-as-init.md’s PID-1 floor versus supervisor policy, which is a good sign the shape is right.

Restart strategies map onto the DAG. Erlang’s one_for_one is “restart just this node”; rest_for_one is “restart this node and everything after it”. Along dependency edges rest_for_one is dependency-aware restart — a node leaving Up demotes its dependants to WaitUp, and waitStability already makes them block. It is also today’s Blocked semantics (Actions/UpDown.hs:150) expressed as a supervision policy rather than as a Bool threaded through a walk. Making the strategy per-node is then a natural knob: a config-file node probably wants rest_for_one, a log shipper probably wants one_for_one.

Two drivers over one node model

The first draft treated run up’s batch semantics as a hard problem needing a quiescence predicate. deptrack shows the simpler answer: keep both drivers over the same node definitions.

syncTurnupGraph  :: Broadcast -> OpGraph -> IO ()                 -- topological, one-shot
asyncTurnupGraph :: Broadcast -> Statuses -> Intents -> OpGraph -> IO ()
upkeepGraph      :: ... -> UpkeepFSM -> DownkeepFSM -> IO ()      -- continuous

run up/run down keep the synchronous topological driver and therefore keep returning IO Bool with today’s exact failure containment; serve uses the async supervised driver. What the two share is node definitions, the magma, the ledger and the precedence graph — not the machines; see §“no separate Blocked status” below, which is where the earlier draft left two incompatible answers. No quiescence predicate is needed for the batch case at all, because the synchronous driver still terminates by construction — and with Completed counting as converged, a Managed node that finishes its work does not hold it open either. Broadcast — in deptrack, (OpUniqueId, CheckResult, Stability, Direction) -> IO () — is salmon’s Reporter, so the report stream is where the two drivers stay comparable.

Decided: no separate Blocked status; WaitUp is it — and the synchronous driver does not run the FSMs at all. A node waiting on a predecessor that will never arrive is in the same state as one waiting on a predecessor that is merely slow. What differs is what the driver does about it, and an earlier draft left that in two incompatible halves: one section said the batch driver “still terminates by construction”, another said it “recognises waiting on something terminal”.

The second is not available, because nothing in the node model is terminal. Upping retries on the adaptive delay, and Always/OnFailure loop by construction; only Completed and Never stop, and failure never does. A driver built on the FSMs has no terminality to recognise, so it has no termination condition — which for run up is not a refinement to postpone, it is whether the command returns.

So the first half wins, and sharply: syncTurnupGraph keeps today’s semantics exactly — one pass in topological order, one attempt per node, check then up, termination structural in the finite graph it walks. It reports Blocked for a node whose predecessor failed in this pass, which is a statement about the pass and needs no notion of terminality at all (Actions/UpDown.hs:151). upkeepGraph runs the FSMs and waits indefinitely, correctly so: the predecessor may yet be repaired, and the node should then proceed without anyone re-declaring anything.

Blocked therefore survives as a report emitted by the synchronous driver, rather than as a node state — which is what keeps the existing suites meaningful, since they assert on the report stream, while the node’s own machine stays three-valued.

Bounding concurrency

Decided in two parts, and the first is not a concurrency limit at all.

(i) Collections. The answer to “a graph with two hundred apt-get install nodes” is that it should not have two hundred nodes. Debian.Package already does this today: installAllDebsAtOnceWith (Nodes/Debian/Package.hs) is an Op -> Op rewrite that harvests every Package from the graph’s dynamics and replaces the per-package nodes with a single batched debs node, while removeSinglePackages prunes the originals.

Under this design that rewrite becomes more valuable, not less: the collection is one node with one machine, so 199 Asyncs never exist to need limiting. It also generalises past apt — anything whose underlying tool is dramatically cheaper in bulk (package managers, nft rulesets, a batch of DNS records) wants a collection rewrite rather than a semaphore.

Decided: the rewrite runs after the fold, and is direction-aware. This is the opposite of what an earlier draft of this document guessed, and the reason is decisive: a collection must not sweep together packages that are being installed with packages that are being removed, and nothing before the fold knows which is which. desired is what tells a deb node whether it is wanted up or down, so installAllDebsAtOnceWith has to take that as an input and partition on it, emitting an install batch and a removal batch rather than one blind batch.

The partition is conservative: still-required wins. A node in desired goes to the install batch; only a node absent from desired goes to the removal batch. Two live declarations can disagree — one still wants nginx, another was retracted and would have removed it — and the ledger has already answered that by union. The rewrite inherits that answer rather than deriving its own, so a package any live declaration still wants is never swept into a removal. Erring the other way would let a retraction pull a package out from under something still standing on it, which is the one failure mode here that retrying does not recover.

A batch reports failure for all of its members. apt-get install a b c exiting non-zero says the batch failed, not which package; narrowing it would mean parsing apt’s prose, which is a fragile thing to make a node’s status depend on. So every member goes to Failure with the batch’s exit status, and the batch’s output ring is where the actual message lives. This is the accepted trade — collections buy efficiency and pay in attribution. If it ever bites, the fix is bisection rather than parsing: on a batch failure, fall back to per-node execution for that batch on the next pass. A refinement, not a requirement.

That settles the “do the ledger and the magma disagree about what exists?” worry by drawing the line somewhere else than either draft did:

the ledger is declared intent, and keeps the user’s own per-package nodes. status still reports per package, which is what an operator wants;
the rewrite is an execution-plan detail. The collection node exists only in the process layer, its lifetime derived from the set of package nodes it subsumes, and edges into a collected node redirect to it.

So they never disagree, because they are answering different questions. It also means the rewrite has to redirect precedence edges, not just replace nodes — anything that depended on deb foo now waits on the batch that installs it.

Why a recipe cannot do this itself

The ordering is not a scheduling convenience. It follows from a real expressiveness limit in the pipeline, and naming that limit is the better argument for “after the fold”.

A recipe author supplies Track' directive, i.e. directive -> Op (Op/Track.hs) — a function of one directive in isolation. It cannot see the other live declarations, it cannot see their directions, and it cannot see what was declared before. So “batch this package with the other packages that are also currently wanted up” is not awkward to write in a recipe; it is inexpressible there. Nothing in directive -> Op has the second argument.

That limit is why dynamics :: [Dynamic] (Builtin/Extension.hs:44) exists at all. It is the escape hatch by which a node says “I am a Package” without knowing what will be done about it, so that a later pass can collect the set and act on it — which is exactly what installAllDebsAtOnceWith does. The channel is already the right shape; it is only under-supplied, because today that later pass runs over a single expanded graph and therefore knows no more than the recipe did.

Folding is what enriches it. The same dynamics mechanism, read across the whole magma with each node’s desired direction in hand, gives the later pass the two things the recipe structurally cannot have: other declarations, and directions. “After the fold” is simply where that information first exists.

The tempting alternative is to widen the recipe instead — Ledger -> directive -> Op — and it should be rejected on three counts:

expansion stops being deterministic in the directive. A directive that expands differently depending on what else happens to be declared makes run up irreproducible;
it breaks the plan contract. Query.planDirectiveDigest pins a plan to the digest of the directive it was computed from (Builtin/CommandLine.hs); if the graph is a function of ambient state too, the digest no longer identifies the graph;
it breaks retraction. The ledger’s down assumes what a declaration contributed is a property of that declaration. If expansion depended on the ledger, a declaration’s contribution would depend on the order declarations arrived, and retracting it could not be computed at all.

So the split stands, and gets sharper: recipes stay a pure function of their own directive; cross-declaration knowledge lives in a post-fold pass. The practical consequence is that a rewrite stops being an Op -> Op an application remembers to apply, and becomes a registered phase the control process runs — roughly Set Ref -> Magma -> Precedence -> (Magma, Precedence), with dynamics as its input channel.

Decided: query shows the computed graph by default, and the declared one on request. The default has to be what will actually execute, or query describes a fiction and run up --plan’s excluded refs are written against the wrong node set. The declared view stays available behind a flag, because it is what the operator wrote and what they will edit.

One subtlety this exposes: for run up the computed graph is a function of a single directive, so it is as reproducible as the directive itself and Query.planDirectiveDigest stays meaningful. Under serve it depends on every live declaration, so a plan captured against it is valid only while the rest of the ledger holds still. Plans are a batch-mode affordance and should probably stay one.

The lock the two batches fight over

Splitting by direction creates one new problem: apt-get install and apt-get remove both want the dpkg lock, and as two nodes they may run concurrently. Retries do cover it, and are the fallback, but they are the weakest of the available answers — a retry loop has to distinguish “could not acquire the lock” from “no such package”, and if it cannot, it retries real failures too.

Two better answers, both already available:

An edge. The rewrite emits both batches, so it can also emit the ordering between them, and the DAG then serialises them for free with no retry and no new mechanism. Removals before installs is the right order — it is what one would do by hand to clear conflicts, and it matches the existing converge sequence (downTree pass, then upTree pass).
The bounding primitive of (ii). This is precisely the case that motivates it: a named exclusive resource (dpkg) that several nodes declare they need. When that primitive exists, the edge becomes an unnecessary special case.

Recommend the edge now and the primitive later, with retries as the safety net rather than the design.

(ii) Bounding primitives, later. Some resources are genuinely exclusive and cannot be collected away: the dpkg lock, a single-writer migration, a shared build directory. Those want an explicit primitive — a named bounded resource a node declares it needs, with the control process holding the semaphore — rather than a global thread cap. Deferred deliberately, because (i) removes most of the pressure and because the right shape is much easier to see once real graphs have run under this model.

The ordering matters: a global concurrency cap is the easy answer and the wrong one, since it slows every unrelated node in the graph in order to protect one contended resource.

What this removes

shipped	fate
`Nodes/Supervised.hs` (584 lines: pid table, reaper, wakeups, `Policy`)	already removed; `up`/`check`/`down` + adaptive-delay FSM replace it
`Serve.serveWith` + `STM (Set Ref)` wakeups	deleted; `waitStability` is the event source
`Serve`'s `worldEpochs`/`prune` retention	shrunk, not deleted; a retracted declaration keeps two flat sets until its nodes settle, never a graph
`Serve.Convergence` (`:146`)	replaced by `Status` (check + direction + stability)
`Extension.prelim`, `Extension.check`, `Actions/Check.hs`	merged into `check :: IO CheckResult`
`Extension.notify`, `Actions/Notify.hs`	deleted; 0 implementations, 0 callers
`upTree`/`downTree`	kept, re-expressed as the synchronous driver

Nothing is lost by that removal. Process ownership — the one capability Supervised had that a check-and-re-up loop does not — returns as Managed, and stronger, because the owning thread outlives the effect and so the handle never needs a table to live in. Supervised’s genuinely useful parts are not lost either — they sit in f9d7116 waiting to be recovered: create_group = True, the grace-then-SIGKILL escalation, and the requested-exit-versus-crash distinction all move into the Managed bracket and the ExitCode it returns. What goes is the bookkeeping that only existed because up :: IO () had nowhere to put a pid: the table, the reaper, the wakeup channel, and the run_stopping flag.

Suggested milestones

check :: IO CheckResult — landed. Absorbing prelim; delete Salmon.Actions.Check and Salmon.Actions.Notify with the notify field. Mechanical across 22 node call sites plus Query.forceSkip, and a prerequisite for everything else. One deliberate behaviour change comes with it: a throwing check stops killing the traversal.

One departure, added later and after milestone 9, once there was a loop to feel it: the type has a sixth constructor, Immaterial, and it is the default. The sketch above has Unknown doing two jobs — “a check ran and could not tell” and “nobody wrote a check” — which is harmless for the one-shot drivers this milestone was written for and not harmless for a supervisor, which has to decide whether to keep asking. Immaterial says there is nothing here worth asking about: applying the effect costs about what finding out would, which is the same property that makes these nodes idempotent. requirement maps it to Required, so run up is unchanged and the two are indistinguishable to it; Actions/Upkeep parks such a node instead of polling it. See (R1) in the companion document for why this was the half of that item worth doing first.
Salmon.Op.Dag — landed. The comonadic fold to magma + both adjacency directions, lifted out of downTreeWith. Pure and unit-testable, and downTree keeps using it, so it is a refactor with no behaviour change — except that this is also where last-writer-wins and the conflicting-representative report land, so §“Ref is location-addressed” has to be settled before this step rather than during it. Two things the fold does that the inline collapse did not, beyond the report: it keeps the edges of every occurrence of a node rather than only the first one’s (without which mergeDag would silently drop the joining graph’s edges, which is the entire point of the module), and it therefore drops the seen-pruning of the walk — the same full traversal upTree’s postOrderM already does. Conflicting is a new UpDown.Report constructor; only downTreeWith emits it, because only downTreeWith goes through the fold until step 4.
The ledger — landed. Per-declaration Contributions (ref set and edge set), desired as the union over live ones and precedence as the union over live and retiring ones, shrinking worldEpochs/prune in serve to a graph-free equivalent. Still no concurrency; Test.ServeSpec is the regression net, and the retraction cases it already covers are what prove the edge sets retract correctly.

Two things this turned out to need that the section above did not say. First, “graph-free” is only true of the teardown: upTreeWith still walks a Cofree of its own, so an epoch’s graph is still what the up pass reads, and the saving is that an epoch is now dropped the moment its declaration is retired or superseded rather than being held until its nodes are down. Step 4 is what closes that. Second, running a teardown off the magma needs downTreeWith split in two: downDag is the walk over a Dag, and downTreeWith is expand plus the fold plus downDag. The Dag a serve teardown walks is rebuilt from the magma and precedenceOf by Dag.fromMagma, which is dagEdges’ inverse.

worldActive is gone, absorbed: the ledger’s liveness is the only source of truth for what is declared up, and worldEpochs after prune is exactly the live declarations’ newest epochs. NodeState.nodeEpoch is gone too — it was write-only, and there is no longer an epoch to name once a declaration retires.
Sync drivers re-expressed — landed. Over the magma and ledger. run up/run down produce the same Report stream and the same Bool; Test.DownTreeSpec/Test.QuerySpec are the net.

“The same Report stream” turned out to be one constructor too strong. Redundant was a statement about a repeated occurrence in the expanded Cofree, and there are no occurrences left once the fold has run: a node is one node however many paths reach it, which is exactly what the magma means. So Redundant is deleted rather than preserved, and upTree now matches downTree, which never emitted it. Nothing consumed it — Serve.stateWriter ignored it, and Test.QuerySpec’s assertion was that the excluded node reported Skip once, which is now structural rather than incidental. The “how many paths reach this node” question it half answered is length . dependantsOf.

Both drivers are now the same walk — UpDown.walk, parameterised by which adjacency direction a node waits on — so upDag and downDag differ only in direction and in which action they run. That is the shape §“Two drivers over one node model” asks for, one step early: the synchronous driver is now single, and step 6’s async driver joins it over the same structures.

One behaviour change falls out of the merge and is worth having. A Dag built from a flat edge set can describe a cycle — impossible from an expanded Cofree, but reachable now that two declarations can each contribute one leg — and a node on a cycle never becomes ready. Both drivers used to leave it silently unapplied and still report success. The walk now sweeps for unreached nodes at the end and reports them Blocked.

serve loses its last graph-driven pass: worldDag rebuilds one structure from the magma and precedenceOf, and both directions run over it. epochOp is gone with upOps and forest. epochGraph survives for one reason, which is worth stating because it is the residue this milestone did /not/ eliminate: --select resolves path patterns, and a Dag has refs and edges but no paths.
Rewrites as a registered post-fold phase — landed. Taking desired and reading dynamics across the magma. installAllDebsAtOnceWith ports to it and becomes direction-aware: an install batch and a removal batch with an ordering edge between them, and precedence edges redirected onto them. Worth doing before parallelism rather than after, since it is what stops the first wide graph from wanting a concurrency limit — and it is the step where applications stop applying Op -> Op passes by hand.

The phase input needed a second field. desired alone is not enough, because a plan’s excluded refs and a converge --select’s complement are nodes the traversal will not touch — and batching one of those into a collection would run exactly the work the operator asked to skip, under another node’s name. So a phase gets Phase { phaseDesired, phaseIgnored } and must leave the second alone. That also settles how run up --plan composes with collections: exclusion moved from Query.forceSkip to a Gate, so a batch is worth running iff some member of it is — the same membersOf translation serve’s gate does, and the same Skip report either way.

The membership map is the other thing the section above did not name. A collection node has no NodeState and no ledger entry — it exists only here — so both the gate and the convergence recording go through membersOf: a batch is worth touching iff any declared node it stands in for is, and what happened to it happened to all of them. That is what makes “a batch reports failure for all of its members” fall out rather than needing special-casing, and it is why status still reports per package. For a node no rewrite touched, membersOf is the singleton of itself, so nothing else changed.

installAllDebsAtOnce/removeSinglePackages are kept and deprecated rather than deleted: the author’s own out-of-tree apps still call them, and porting is a one-line change they can make when they choose.

Not done, and deliberately so: query still shows the declared graph, not the computed one. The decision above stands, but a rewritten Dag has refs and edges and no paths, and --select matches path patterns — so printing the computed graph needs a renderer that does not exist yet. The fiction the section warns about is narrower than it was, since plan exclusion now composes with collections through membersOf rather than silently missing them.
TVar Status per node and waitStability — landed. Plus the async drivers and the per-node mailbox. Parallelism appears here.

Salmon.Actions.Concurrent is one thread per node, ordering by waitStability rather than by counters, with the same Report stream, the same IO Bool and the same failure containment as the sequential drivers. serve converges through it; run up/run down stay synchronous, per §“Two drivers over one node model”. Convergence is therefore now parallel and unbounded — the protection against contention is the DAG’s own edges plus milestone 5’s collections, exactly as §“Bounding concurrency” argues, and nothing else. Recipes with a hidden shared resource that were safe only because the traversal was sequential are not safe any more.

Four things the sections above did not have to say, because they only arise once nodes run at once:
- Failure containment cannot live in Status. waitStability reads direction and stability only, so a node that settled having failed is indistinguishable from one that settled having succeeded — which the spec wants, since the two drivers answer “proceed past a failure?” differently. So the pass keeps a TVar (Set Ref) of what did not succeed, and recording that failure and settling have to be one transaction, or a dependant can observe Stable before the failure is visible and proceed against a node that in fact failed.
- Reports have to be serialised. The reporter belongs to the caller and cannot be assumed thread-safe; a multi-line report interleaving with another node’s is garbage. One MVar around every runReporter.
- A cycle has to be found before the walk, not after. The sequential drivers discover unreachable nodes by finishing and noticing what they never touched. A thread waiting on a node in a cycle simply never wakes, so Dag.stuck is consulted up front.
- serve’s own bookkeeping had a latent bug that only parallelism could expose: stateWriter recorded convergence with modifyIORef', which is not atomic, so concurrent nodes reporting into one World would silently lose records.
Instruction’s Skip is spelled Satisfy, to stay out of UpDown.Report.Skip’s way. Recheck/Pause/Resume are read and reported but mean nothing to a single-pass driver; they are for the upkeep FSM in step 7. statusOutput is live — a node narrates its own transitions into the ring — but nothing else writes to it until Managed in step 8.
Upkeep/downkeep FSMs with adaptive delay, over OneShot nodes only, plus the authored watchdog — landed. Supervision of unowned effects appears here.

Salmon.Actions.Upkeep is the continuous driver: UpkeepState and DownkeepState exactly as declared above, waitStability for ordering, the adaptive delay as the backoff, one mailbox per node, and a single scanning thread for the watchdogs. Salmon.Op.Supervision is the policy — Restart plus Maybe watchdog, riding dynamics, read back with the same getDynamics the collection rewrite uses, first-wins with the losers reported. Neither one owns a process; that is step 8.

Eight places this differs from what the sections above say, six of them because the sections were written about one pass and this is a loop.
- Unknown restarts nothing. §“Keep this running” reads _ -> fsm (decreaseDelay delay) Upping, which lumps Unknown in with Failure. That is right for the one-shot drivers, where requirement Unknown = Required errs safely over an idempotent action, and a spin loop here: a node with no check answered Unknown forever, so it would re-run up at the 500ms floor for as long as serve lived. “I could not look” is not evidence the effect went away. Only Failure demotes a node out of Up. (A node with no check now answers Immaterial rather than Unknown, and is parked rather than polled — see milestone 1’s departure. The rule stated here is unchanged; it just applies to far fewer nodes than it did when it was written, which is the improvement.)
- A failing up backs off; only a vanished effect tightens. The spec adapts the delay on what the check said and is silent on how often to retry an up that keeps throwing. Tightening there would retry apt-get twice a second, so Upping doubles toward the cap on each failure while Up -> Upping still halves toward the floor.
- The restart policy is consulted before satisfaction, not after. §“The restart policy” wants Always to rerun a Completed node, and Completed is a satisfied verdict — so a look that asks “satisfied?” first can never reach the policy, and Always would be unreachable. Hence Intent: arriving in Upping from WaitUp consults the check, arriving from Up (or from a Force) does not, because the answer already in hand is the reason.
- A supervisor is told what the last pass achieved (Standing). This is not in the spec at all and is load-bearing: almost nothing in this repository implements check, so a supervisor started after a convergence pass would consult every node, get no usable answer, and run every up in the graph a second time. Settled skips the first up and nothing else — the node is still watched, and still put back if its check later says the effect is gone.
- serve supervises only while it is idle, rather than “serve uses the async supervised driver” wholesale (§“Two drivers”). The machines start when nothing is waiting in the input and stand down before any command is handled. Two reasons, and the second is the real one: a piped script has every line, EOF included, queued before the first pass ends, so it is never supervised and serve < script stays a deterministic sequence of passes; and starting machines only to stop them because a command had been sitting in the queue would make “was this node acted on?” depend on thread timing. run up/run down are untouched, as §“Two drivers” wants.
- A restricted converge --select scopes the pass, not the world. Supervision is unrestricted, so a node a restricted pass skipped is still tended once the loop goes idle. The alternative — carrying a transient flag into the standing watch — would mean a one-off --select silently stopped watching everything else, which is a worse surprise than the one it avoids. supervise off is the way to get a pass that is the only thing touching anything.
- The watchdog reports and does not kill. Interrupting an up needs the teardown-through-a-bracket that owning the process buys, i.e. step 8. Reporting is still most of the value: it is what tells a slow node from a stuck one.
- Down is terminal, Up is not. Nothing in the model answers “is it still gone” — check answers “does my effect need creating” — so a downkeep machine that reaches Down exits, while an upkeep machine that reaches Up has only started. The asymmetry is in the two state names above but its consequence was never stated.
Time is Micros (an Int) rather than DiffTime: salmon-ops has no time dependency and neither consumer of the value wants one — threadDelay takes microseconds and getMonotonicTimeNSec returns an integral nanosecond count.

Three things fell out. serveWakingWith/noWakeups/Woken are gone, which its own todo predicted: the hook existed because a node had no state of its own to block on, and it is not a smaller version of this. Serve’s Direction was a second, identical declaration of Status.Direction and is now that one, re-exported. And Instruction’s Recheck/Pause/ Resume mean something for the first time.

What this does not yet buy, and it is worth being blunt about it. Supervision is exactly as good as nodes’ checks, and at this milestone almost no node had one: filecontents did not, so a managed file deleted behind salmon’s back was not noticed. The engine is here and tested; making it do anything for a real graph is a per-node question — which is the ordering question §“Open questions” already logged against todo, now sharper: it is not “does this shape work”, it is “which nodes get a check”. filecontents comparing its own contents is the obvious first one, and it changes what run up does for every existing caller, so it was deliberately not smuggled in here. (It, and Systemd.systemdService, have since been done — see (R1) in the companion document, including the part where the file node turned out to be what made the unit node’s promise true.)

Nothing demotes a node’s dependants when it stops being up; that is step 9. A node that has actually failed does hold off a dependant still in WaitUp, which is the containment the one-shot drivers have, expressed as a wait rather than as a Blocked.
Managed nodes — landed. Up races the running action against the check timer, cancel tears down through the bracket, and exit statuses reach the restart policy. This is the step that restored what removing Supervised gave up.

Extension.managed :: Maybe (Output -> IO ExitCode) is the action; Salmon.Builtin.Nodes.Daemon is the one builtin that fills it in, and Test/DaemonSpec.hs covers the part only a real subprocess shows. The state machine changes in exactly one place: Up’s nap becomes a four-way STM race (nap, mailbox, the action’s own exit, and — for a machine that holds one — never the halt flag), and the exit is what the policy reads.

Six departures, three of them shapes this document guessed wrong.
- Lifecycle is a field, not a sum. §“Recovering process ownership” declares OneShot (IO ()) | Managed (IO ExitCode). Replacing up’s type would rewrite all 106 up = sites in the tree for a feature a handful of nodes use, so managed sits beside up and Nothing is every existing node, unchanged and uninspected. The sum is the better type; it is not worth that diff until a third lifecycle exists.
- The action takes an Output sink. The declared type is IO ExitCode, which leaves §“Output: a bounded ring per node” unable to deliver what it promised — ownership was the thing that made capturing stdout possible, and with no channel the ring can only ever hold the machine’s own narration. So the action is handed a Text -> IO () that writes into it. The pipes are drained while the process runs rather than after, or one that fills a pipe buffer blocks forever and looks wedged for a reason nobody could see.
- A machine holding a process is Kept, not stopped. This is the largest thing the document did not anticipate, and it follows from milestone 7’s own choice to rebuild the supervisor whenever the loop goes idle. Stopping a supervisor means “stop tending”, and serve stands its machines down before every command — so a supervisor that wound its processes down with it would restart every service every time anybody typed status. Holding machines therefore survive their supervisor and the next one adopts them, on precisely the condition §“Ref is location-addressed” identified as the one legitimate reason to swap a machine: still wanted TurnUp, and its representative unchanged. Everything else it left is released — cancelled, which tears the effect down through the action’s own bracket.
- A managed node is invisible to both convergence passes, and the ordering around that is load-bearing. run up cannot host it (so its up throws, loudly, rather than no-oping into a world that then believes it is up) and there is nothing left for a one-shot down to do. But letting go of the machine has to happen before the down pass, because the down pass is what removes the daemon’s config file and working directory. Serve.settleManaged is that step, and it records those nodes down as it goes — exact rather than optimistic, since for an effect that only exists while something holds it, “nothing holds it” is what being down is.
- The restart policy needed two more fields. Restart alone cannot express a crash loop, and the module deleted at f9d7116 already knew it: supStableAfter (having been up this long forgets the earlier failures) and supGiveUpAfter. Without the first the second latches off any long-lived node eventually — a service that falls over once a day reaches any finite limit in that many days, having never been in a crash loop. A node that gives up is parked, not gone: its dependants must keep seeing it settled-and-failing, and an operator has to be able to change their mind (Force or Recheck).
- A managed node whose check already says the effect is there is not spawned. It is treated as an unowned effect and polled, because starting a second copy of something already running is worse than not owning what is running. Worth naming because it means a process left behind by a serve that has since exited is never re-adopted — the pidfile problem §“Non-goals” excludes, showing up in the one place it still bites.
Settled is never claimed about a managed node, whatever the caller believes: a Settled claim is about an effect that persists on its own, and a managed effect does not persist without its machine. withAsync rather than async is what makes the teardown work at all — cancelling the machine cancels the action, and whatever bracket the action is built from does the killing, which is why no pid table appears anywhere here. The grace-then-SIGKILL escalation and create_group = True are recovered from f9d7116 as this document said they should be, not reinvented.

One bug worth recording, because of how it hid: adding Ended to the machine’s Wake type left announce non-exhaustive, so every managed node’s thread died of a pattern-match failure the moment its action returned. The -Wincomplete-patterns sweep that would have caught it reported nothing, because cabal build with different --ghc-options in an up-to-date build directory does not recompile. Sweep in a fresh --builddir or not at all.
rest_for_one: a node leaving Up demotes its dependants — landed. Supervision gains supStrategy :: Strategy (OneForOne/RestForOne, defaulting to OneForOne), a node in Up watches the dependencies that declared RestForOne, and one of them leaving sends it back to WaitUp. Test/UpkeepSpec.hs covers the eight behaviours, Test/ServeSpec.hs the ninth that only exists under serve. Five departures, and the first two are corrections to this document rather than choices:
- A level read of a neighbour’s Stability cannot see a departure at all. §9.1’s sketch — “resting‘s STM choice gains a branch watching its dependencies’ statuses” — misses every transition it is for: a dependency that fell over and recovered between two of a dependant’s waits is Stable at both of them, and STM keeps no queue of what happened in between. A config file rewritten in milliseconds is exactly that shape, so the feature would have worked only for slow failures. The fix is a monotonic Status.statusEpoch, bumped when a settled node unsettles: the dependant remembers the number it last saw and compares. Level-triggered STM, turned into edge detection by remembering.
- The remembered number is only meaningful with the machine it came from. Under serve a dependency gets a new machine on every command — a fresh Status, counting from zero — so a dependant comparing its memory of the old one would read a departure every time an operator typed anything, restarting every service, which is precisely what Kept exists to prevent. The TVar is therefore remembered alongside the epoch, and a dependency whose machine has been replaced is re-armed rather than acted on.
- An adopted machine had to be given its supervisor’s state, not just watched with it. Everything a machine waits on — the statuses, the failure set, the neighbour lists, and the halt flag — belongs to a supervisor, while a machine holding a managed action outlives the one that started it. Before this milestone that was invisible, because such a machine only ever took paths that ignore the halt flag; a demoted one takes standby, which heeds it, and would have read a permanently-set flag and quietly exited, orphaning its process. Hence Upkeep.Under, which startUpkeep writes into every machine it adopts. It also fixes a bug that predates this milestone: an adopted machine was recording its failures in a set no dependant read.
- A dependency that has not been seen up cannot demote anybody. Not in the plan, and without it this milestone would have undone milestone 7’s Standing: a supervisor starting over a graph a pass has just converged would send every opted-in node back to WaitUp before its dependencies’ machines had settled, re-running every up in the cone once per command. A dependency is armed the first time it is seen settled up, and not before.
- supStableAfter is used as a rate limit, not as a settling delay. §9.3 asks for the second and it cannot work: a settling delay swallows the case the feature is for, since the config file a service stands on is back within milliseconds of being rewritten. So an isolated departure is always honoured whenever it comes, and what is dropped is a second demotion inside the node’s own supStableAfter — which is what a flap looks like and a change does not. That bounds a flapping dependency to rebuilding the cone behind it once per interval.
The thundering herd §9.1 worried about does not arise, because the watch is authored on the node that goes away rather than on the ones that get bounced: a machine with no RestForOne dependency subscribes to no statuses at all, and crossing is skipped rather than being a branch that never fires. The cascade needed no code — a demoted node is itself no longer up, which is all a dependant of it that opted in has to see.

Those departures, plus two things found by building a demonstration of this milestone (salmon-ops-serve-fixture --daemon), are written up as wanting another pass in specs/per-node-state-machines-remaining.md §“Landed, but wanting another iteration” — they are decided, not settled. Two of them are not matters of taste: a demoted managed node whose check is satisfied is torn down and never restarted while reporting itself Up (I1), and a convergence pass does nothing about a re-declaration that changed a node’s content, so only a node with a check ever picks one up (I6).

Non-goals (v1)

Cross-machine supervision. Nodes/Self.hs’s remote-op flattening is unaffected.
Replacing Configure/the seed→directive protocol; this is execution only.
Persisting machine state across a serve restart — the pidfile convention above is what makes a restart recoverable, not a state file.
A general actor framework. Two three-state FSMs and a supervisor.

Open questions

Four rounds of these are now settled in the sections above — the ledger shape (nodes and edges, retiring rather than deleted), what Ref equality does and does not mean, what the magma stores, concurrency, notify, the plan machinery, captured output, exit codes, Transient, where the rewrite runs and why, Completed, mailbox overflow, check cost, collection conservatism, batch failure attribution, Blocked versus WaitUp and what separates the two drivers, and what query shows.

The one question the last round left open — how is a node’s watchdog authored? — is settled too, and the answer was already in the tree.

What is not settled is a different list, and it is not in this document: the questions above are the ones this design set itself, while the places where a landed milestone’s shape was decided under that milestone’s pressure and wants a second pass are collected in specs/per-node-state-machines-remaining.md §“Landed, but wanting another iteration” (I1–I5). Read that before acting on the departures recorded in the milestone list, which are written as decisions taken.

Decided: supervision policy rides dynamics, not a new Extension field. dynamics :: [Dynamic] (Builtin/Extension.hs:44) is exactly the channel by which a node states something about itself for a later pass to act on — which is the argument §“Why a recipe cannot do this itself” already makes for the collection rewrite. Package uses it (Nodes/Debian/Package.hs:47) and installAllDebsAtOnceWith reads it back with collectDynamics. So:

data Supervision = Supervision
    { supRestart  :: !Restart
    , supWatchdog :: !(Maybe DiffTime)
    }

-- at the node:
actions{ dynamics = [toDyn (Supervision OnFailure (Just 30))] }

and the process layer reads it with getDynamics, exactly as the post-fold rewrite pass reads Package. Three things this buys:

No new Extension field, so nothing changes for the many nodes with no opinion about how they are supervised.
The default falls out. “A node that declares no watchdog is never considered wedged” is just getDynamics returning [], rather than a Nothing every node has to write.
It is optional and one line to add, which was the bar this question set itself: a watchdog is only as good as authors’ willingness to set one.

The cost is that it is untyped and unenforced — nothing stops two conflicting Supervision dynamics on one node. That is the same weakness Package collection already lives with, and it gets the same treatment as a conflicting magma representative: take one, report the rest.

What remains is an ordering question rather than a design one: the shape above wants exercising on two or three real long-running nodes before milestone 7 hardens it. Tracked in todo.

Relationship to the other specs

specs/per-node-state-machines-remaining.md is the companion plan: what is left of the milestone list below (8 and 9), plus the residue no milestone covers — chiefly that check is implemented by roughly a quarter of nodes, which is what currently limits milestone 7 to supervising almost nothing. Start there if you are picking this work up rather than reading it.
specs/salmon-as-init.md needs this and gets simpler for it: its PID-2 supervisor is this control loop, its restart policy is the upkeep FSM, and its “PID 1 rate-limits, the supervisor decides” split is the same two-level supervision as above. The Rust PID 1 is unaffected — that boundary is about waitpid(-1), not scheduling.
specs/multi-user-privilege-separation.md composes unchanged: an Invoker decorates the CreateProcess a node’s up spawns.
specs/advance-querying.md is the open question above.

How to write and test salmon ops

2026-09-25T14:15:36Z

Generated from resources/howto-ops.md — the repository is the canonical source, and may be ahead of this page.

How to write and test salmon ops

This is a cookbook. It is written to be unambiguous rather than elegant: copy the patterns below, adapt the names/fields, and you will produce a valid Op. If you want the conceptual model instead (why any of this exists), read salmon-core.md first. This document assumes you already know Haskell syntax but not necessarily this codebase.

Every code sample below is a paraphrase of real code in this repository. When in doubt, grep for the cited file/function and copy its shape exactly.

1. The one type you need: `Op`

type Op = OpGraph Identity Actions'

An Op is a single graph node plus everything needed to run it: how to bring it up, how to tear it down, how to know if it’s already up, and what it depends on. You almost never construct one by hand — you call the op helper.

op :: ShortHand -> Identity (Graph Op) -> (Extension -> Extension) -> Op

ShortHand (a Text) — a short, human-readable kind name, e.g. "directory", "file-contents", "pg-user". Used in Ref construction (see below) and in Tree/Dot output. Pick something stable — changing it changes the node’s identity.
Identity (Graph Op) — this node’s dependencies. Almost always built with one of the two helpers below, never by hand:
- nodeps :: Identity (Graph Op) — no dependencies.
- deps :: [Op] -> Identity (Graph Op) — depends on (is preceded by) this list of ops, all co-occurring at the same level (no ordering is implied between them — see §3 for ordering).
(Extension -> Extension) — a function that fills in the actual behavior on top of a blank no-op Extension. You almost always write this as a record update on the actions argument: \actions -> actions { help = ..., up = ... }.

2. The `Extension` record — what you fill in

data Extension = Extension
    { help    :: Text            -- one-line description
    , notes   :: [Text]          -- longer free-form notes
    , ref     :: Ref             -- dedup identity (see §2.1)
    , up      :: IO ()           -- bring this node's own effect into being
    , managed :: Maybe (Output -> IO ExitCode)
                                 -- ...or *be* the effect, for as long as it runs (see §2.2)
    , check   :: IO CheckResult  -- is my effect already in place? (see §4)
    , down    :: IO ()           -- undo this node's own effect
    , dynamics :: [Dynamic]      -- arbitrary typed metadata, see §7
    }

You only ever need to touch help, ref, up, down, and — if the node needs idempotency beyond what up itself can guarantee — check. managed is Nothing for all but a handful of nodes; see §2.2 if yours is one of them.

check answers one question about the node’s own effect, with six possible answers: Success (it is in place), Skipped (someone decided to treat it as satisfied — only Query.forceSkip produces this), Completed (it ran to completion and stopped on purpose), Failure reason (it is not in place — the ordinary answer on a first run, not an error report), Unknown (a check ran and could not tell), and Immaterial (there is nothing here worth asking about — applying the effect costs about what finding out would; this is the default when you write no check). upTree skips the first three and runs up for the last three.

check earns its keep twice, and the second time is easy to miss. In a one-shot run up it is an optimisation: it saves an up you didn’t need. Under run serve, which tends its nodes between commands (Salmon.Actions.Upkeep), it is the only thing that can notice your effect going away — nothing else in the model looks. A node with no check answers Immaterial, and the upkeep FSM parks it: brought up once, then blocked on its mailbox rather than woken on a timer to be told the same thing. (An explicit Unknown is different: the FSM keeps asking, and keeps not acting on it — a loop that treated “I could not tell” as “so run up” would spin.) So if your node is something that can stop being true on its own — a service, a mount, a firewall rule, a file something else might clobber — write a check; it is the difference between a node that gets applied and a node that gets supervised. Netfilter.rule’s skipIfNftRuleExists is the template, Systemd.checkService is the worked example of what a real one costs, and Filesystem.checkFileContents is the one most node authors will want to copy: it compares the bytes it would write with the bytes that are there, which is exact where skipIfFileExists would call a file holding the wrong thing satisfied.

There is one alternative to writing a check, and it is narrow: Salmon.Op.Supervision.supReapply opts a parked node into re-running up on the loop instead of asking. Only sound when up is both genuinely cheap and genuinely idempotent — Filesystem.dir sets it (createDirectoryIfMissing costs about what doesDirectoryExist would, so there’s nothing to compare); a build, a clone, or anything that talks to the network should write a check instead, or leave itself parked.

up and down default to pure () (a no-op) if you don’t set them — this is useful for pure “grouping” nodes (see §3) that only exist to bundle dependencies, with no effect of their own.

2.1 `ref`: dedup identity

ref = mkRef "directory" path

mkRef :: (Hashable key) => Text -> key -> Ref builds a Ref from a kind tag plus anything Hashable that uniquely identifies this instance of the node — usually the primary key of whatever you’re describing (a file path, a database name, a (src, tgt) pair for a copy). Two Ops with the same Ref are treated as the same node by the graph traversal (upTree/downTree): reachable via two different paths or not, it is one node, applied once. Getting ref right is the single most important thing when writing a new node — get it wrong (too coarse, e.g. reusing one Ref for two different files) and you silently skip work; get it wrong the other way (varying per-call for what should be the same resource) and you silently do the work twice. Forgetting ref altogether is the first kind: the default is mkRef "noop" shorthand, so every node sharing a ShortHand collapses into one.

Note what the identity key is not: the node’s behaviour. filecontents keys on the path alone, so an equal Ref means “the same effect site”, not an equal node — two Ops writing different bytes to one path are one node, and only one of them wins. Both traversals go through Salmon.Op.Dag, so the rule is the same in both directions: the last representative wins, and the loser is reported as Conflicting so the collision is at least visible. Tightening a key (putting a content digest in it, say) is a legitimate per-node fix, but think about it per node: it is right for a file and wrong for a long-running service, where every config tweak would become a different node.

2.2 `managed`: when the node is a running process

up :: IO () describes an effect that persists once it has been made: the file stays written, the route stays installed. A long-running process does not — nothing keeps it alive but something watching it. That is what managed is for:

managed :: Maybe (Output -> IO ExitCode)

It blocks for as long as the node is up and returns the reason it stopped, which means the node’s own thread is in scope for the process’s entire lifetime and the handle never has to escape. Three things follow that a check alone cannot give you: an exit status (so “don’t restart a service that exited cleanly” is expressible), promptness (a death is noticed at once rather than at the end of a check delay that may be a minute), and identity that survives pid reuse.

Don’t write one from scratch — use Salmon.Builtin.Nodes.Daemon:

Daemon.daemon reportPrint (Daemon.defaultDaemon "webserver" (proc "nginx" ["-g", "daemon off;"]))

or, if your node needs dependencies or a liveness check of its own, build it around Daemon.runDaemon, which is the same action without the node wrapped round it. Either way you get the teardown: a signal (SIGTERM by default, daemon_stop.stop_signal otherwise) to the process group, a grace period, then SIGKILL. Writing that yourself is easy to get subtly wrong.

Three things to know before reaching for this:

only run serve honours it. run up/run down call up, and a one-pass driver has nowhere to put an action that never returns. So a node like this should throwIO from up (that’s what Daemon.daemon does) rather than no-op into a world that then believes it is up.
prefer systemd where there is systemd. Systemd.systemdService hands the whole problem to an init system that is better at it and survives salmon exiting. managed is for where that is not available: a container, a test harness, or salmon-as-init itself.
the Output argument is where your process’s own lines go — into the node’s bounded ring, which is what an operator reads when it has failed and what tells a watchdog it is still making progress. Daemon.runDaemon handles the piping.

3. The canonical example: `Filesystem.hs`

Read salmon-ops/src/Salmon/Builtin/Nodes/Filesystem.hs end to end — it is the smallest complete example of every pattern in this document. Key excerpts:

A node with no dependencies:

newtype Directory = Directory {directoryPath :: FilePath}

dir :: Directory -> Op
dir directory =
    op "directory" nodeps $ \actions ->
        actions
            { help = Text.pack $ "ensures " <> path <> " exists, including subdirs"
            , ref = mkRef "directory" path
            , up = createDirectoryIfMissing True path
            , down = removeDirectoryIfPresent path
            , dynamics = [supervised defaultSupervision{supReapply = True}]
            }
  where
    path = directory.directoryPath

A node that depends on another node it builds internally:

data FileContents a = FileContents {filePath :: FilePath, contents :: a}

filecontents :: (EncodeFileContents a) => FileContents a -> Op
filecontents fcontents =
    op "file-contents" (deps [enclosingdir]) $ \actions ->
        actions
            { help = Text.pack $ "writes " <> path <> " with some contents"
            , ref = mkRef "file-contents" path
            , check = checkFileContents fcontents
            , up = ByteString.writeFile path =<< encodeFileContents fcontents.contents
            , down = removeFileIfPresent path
            }
  where
    enclosingdir = dir (Directory $ takeDirectory path)
    path = fcontents.filePath

Note both downs tolerate the effect already being gone (removeDirectoryIfPresent/removeFileIfPresent, not removeDirectory/ removeFile): a down that throws leaves its node “still standing” and blocks the teardown of everything under it (§5), so down needs the same run-twice safety as up.

Note the pattern: filecontents doesn’t take a Directory as an argument — it derives the directory it needs (takeDirectory path) and constructs that dependency itself, inline, in a where clause. This is idiomatic: a node should make its own prerequisites, not expect a caller to remember to also build them. As long as the derived dir call produces the same Ref every time it’s reachable, it dedups correctly no matter how many other nodes also depend on “the same” directory.

A node combining several dependencies with explicit ordering (replaceDirectory in the same file):

replaceDirectory :: FilePath -> FilePath -> FilePath -> Op
replaceDirectory src tgt trash =
    op "replace-dir" (deps [delete3 `inject` move2 `inject` move1]) $ \actions ->
        actions
            { help = Text.pack $ "replace " <> src <> " " <> tgt
            , ref = mkRef "replace-dir" (src, tgt)
            }
  where
    move1 = moveDirectory tgt trash $ \actions -> actions{check = skipIfDirectoryIsMissing tgt}
    move2 = moveDirectory src tgt id
    delete3 = destroyDirectory trash

This node has no up/down of its own (they default to no-ops) — it’s a pure orchestration node whose entire job is wiring three other ops together in order. This is a completely normal and common pattern: not every op call needs to do anything itself.

3.1 `deps` vs `inject` vs `overlaid`

deps [a, b, c] — this node is preceded by a, b, and c, each also preceded by whatever they depend on. No ordering is implied among a, b, c themselves.
a `inject` b (from Salmon.Op.OpGraph) — “b must happen before a”: adds b as a predecessor of a via an ordered graph connection. Chain it to express a strict sequence: c `inject` b `inject` a means run a, then b, then c.
a `overlaid` b — “a and b co-occur, no ordering implied between them” (the unordered counterpart to inject).

Use deps [...] for the normal “these are my prerequisites” case. Reach for inject only when two ops need a specific order that the dependency structure alone wouldn’t otherwise guarantee (e.g. “delete the trash dir” must happen after “move src into place”, not just alongside it).

4. Idempotency: making `up` safe to run twice

upTree may run the same Op graph repeatedly (that’s the point — it’s meant to converge a system to a target state, not run once and be thrown away). up must be safe to call when the target state already holds. In rough order of preference:

Use a replace-shaped command if the tool has one. E.g. ip route replace instead of ip route add (see Salmon.Builtin.Nodes.Routes), or SQL ALTER SYSTEM SET instead of an insert.
Use a CREATE ... IF NOT EXISTS / CREATE OR REPLACE form if it exists for what you’re creating.
Guard a bare CREATE with a conditional inside a DO block, when the tool allows a CREATE with no IF NOT EXISTS form to run inside one — e.g. Postgres CREATE ROLE has no IF NOT EXISTS, but can run inside:
```
DO $$
BEGIN
  IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'myuser') THEN
    CREATE ROLE myuser;
  END IF;
END $$;
```
(see Postgres.CreateUser/CreateGroup in salmon-ops/src/Salmon/Builtin/Nodes/Postgres.hs).
Shell-level check-then-act, when even that isn’t available — e.g. Postgres refuses to run CREATE DATABASE inside a transaction/DO block at all, so Postgres.CreateDB does:
```
psql -tAc "SELECT 1 FROM pg_database WHERE datname = 'mydb'" | grep -q 1 \
  || psql -c 'CREATE DATABASE mydb'
```
Append-if-missing, for config file lines with no SQL/CLI equivalent at all: grep -qxF '<line>' file || echo '<line>' >> file (see Postgres.ensureHbaLineScript, for pg_hba.conf).
A check-based skip check, when the underlying tool has no idempotent verb at all — nothing to replace, no IF NOT EXISTS — so up itself cannot be made safe to re-run. Instead, check before running whether the effect already exists, and report Success if so:
```
skipIfNftRuleExists :: Chain -> Rule -> IO CheckResult
skipIfNftRuleExists c rule = do
    (code, out, _err) <- readCreateProcessWithExitCode (proc "nft" ["list", "chain", ...]) ""
    pure $
        if renderedRuleText rule `isInfixOf` out
            then Success
            else Failure "no such rule in the chain"
```
```
op "netfilter-rule" (deps [...]) $ \actions ->
    actions
        { check = skipIfNftRuleExists chain rule
        , up = addRule ...
        , ...
        }
```
This is the same shape as the built-in skipIfFileExists/ skipIfDirectoryIsMissing/skipIfNetworkExists helpers in Salmon.Actions.UpDown/Salmon.Builtin.Nodes.Podman — when you add a new node wrapping a command with no idempotent “set” verb (e.g. nft add rule, podman network create), this is the template to copy: write a skipIfXExists :: ... -> IO CheckResult that shells out to a read-only “does X exist” check, and wire it into check.

check defaults to pure Immaterial — “asking would cost what applying costs”, which upTree reads as “run up” — if you don’t set it, so options 1–5 above need no check at all; only option 6 does. That default is a claim, not an absence, and for options 1–5 it is a true one: the whole reason those nodes are idempotent is that re-applying them is cheap. It is false for anything that can stop being true on its own, which is why such a node needs option 6. Note also that a check that throws is contained: it is read as Failure, so the node is evaluated and the rest of the traversal is unaffected.

5. Failure: how `up`/`down` report errors

up :: IO () and down :: IO () have no return value to signal failure — the only way a failure becomes visible to the graph traversal (upTree/ downTree in Salmon.Actions.UpDown) is if up/down throws. A thrown exception is caught by the traversal, the node is reported Failed, and every node that (transitively) depends on it is reported Blocked and not evaluated (for upTree; downTree blocks the opposite direction — a node’s own predecessors — since teardown walks top-to-bottom). Both upTree and downTree return IO Bool: True iff nothing was Failed/Blocked.

Getting this right is mostly automatic if you build up on top of Salmon.Builtin.Nodes.Binary.untrackedExec (which almost every builtin does via withBinary) — it already checks the subprocess exit code and throws CommandFailed on non-zero, so a normal up = someCommand r' ... needs no extra handling.

Two cases do need you to add explicit handling:

If your node is built on the lower-level withBinaryIO/CommandIO (used when you need to redirect a subprocess’s stdin/stdout yourself, e.g. WireGuard.privateKey/publicKey), you get a raw ProcessHandle back instead of a checked result — call Binary.checkExitCode label on it yourself (it throws CommandFailedSimple on non-zero).
If your node’s up recursively runs its own nested upTree (e.g. to drive a remote machine, see PostgresMigrations.remoteMigrateOpaqueSetup), you must check the returned Bool yourself and throwIO if it’s False — the outer traversal has no other way to learn the nested one failed:
```
up = do
    ok <- upTree r nat nestedGraph
    unless ok (throwIO (userError "nested upTree failed"))
```

Never swallow an exception inside up/down (e.g. via catch/handle) just to “keep going” — that defeats Failed/Blocked propagation and makes a real failure look like success to the traversal and to anything scripting around this binary’s exit code.

All of the above is the one-shot drivers (run up/run down). Under run serve a node is tended rather than applied once (Salmon.Actions.Upkeep): a failing up is retried on a backing-off delay, and a dependant waits for its dependency to recover rather than being reported Blocked for good. The node’s Supervision (§7) says what to do when it keeps failing; see serve-supervision.md. The rule for up itself is the same either way: throw.

6. Composing with `Track`/`Tracked` — when a node needs a value from elsewhere

Sometimes one node’s construction genuinely needs a value that another part of the graph produces or that the caller controls (not just an ordering dependency) — e.g. “grant these rights to this database owner role, which some other part of the setup created.” That’s what Track/Tracked are for:

newtype Track m n a = Track { run :: a -> OpGraph m n }

A Track' a (i.e. Track Identity Actions' a) is “given an a, I know how to build the Op that provisions it.” You’ll see this passed around as a parameter so callers can plug in different strategies for “how do I get an a” without the node itself caring — e.g. Postgres.database takes a Track' (Binary "psql") so callers can point it at a locally-installed psql binary or a differently-provisioned one.

ignoreTrack :: Track' a is a real, always-available Track that discards its input and produces a no-op Op — use it when you have a value already in hand and don’t need the Track machinery to produce anything more, only to satisfy a function signature that expects one (this is extremely common; grep existing recipes for ignoreTrack to see the pattern used dozens of times).

You do not need Track/Tracked for the vast majority of new nodes — only reach for it when you’re deliberately decoupling “how do I build a value” from “what do I do with it,” the same way Postgres.hs and the recipes in salmon-ops-recipes/src/SreBox/ do it. See salmon-core.md §Track for the type-level reasoning.

7. `dynamics` — attaching arbitrary metadata to a node

dynamics :: [Dynamic] lets you stash arbitrary typed metadata on a node that some other part of the codebase can later recover by type, without changing Extension itself. A few real uses:

placeholder (in Salmon.Builtin.Extension) stashes a PlaceHolder Text so dot-graph rendering can show a label without the node needing real up/ down behavior.
getDynamics/collectDynamics walk a graph and pull out every Dynamic of a chosen type — used e.g. to flatten “which of these ops are actually remote calls” out of a graph for special handling.
Debian.Package.deb stashes a Package, which the batchPackages rewrite (Salmon.Op.Rewrite) collects across the whole graph into one apt-get invocation.
A Salmon.Op.Supervision.Supervision states how the node wants to be tended: whether to put it back when its check says the effect has gone (Always/ OnFailure/Never, defaulting to OnFailure), how long its silence may last before somebody should worry, and — supStrategy — what its going away means for the nodes standing on it. Salmon.Actions.Upkeep reads it back with the same getDynamics. One line, and a node with no opinion needs none:
```
dynamics = [supervised defaultSupervision{supWatchdog = Just (seconds 30)}]
```
Amend defaultSupervision rather than spelling out every field: the record has grown three times and will again, and a node that only cares about its watchdog should not have to have an opinion about giving up.

supStrategy is the one field authored for somebody else’s benefit. The default, OneForOne, is that putting this node back is a statement about this node and nothing downstream is disturbed. RestForOne sends every dependant back to WaitUp, to be brought up again on top of whatever this node turns into — Erlang’s strategy of the same name, read along dependency edges. A configuration file is the case for it:
```
-- the services reading this file are bounced when it is rewritten
dynamics = [supervised defaultSupervision{supStrategy = RestForOne}]
```
Note it goes on the config node, not on the services: the file’s author knows their content is load-bearing, while each service reading it would otherwise have to know, separately, that it might change underneath. Only run serve acts on it (a one-shot pass has no “already up” to send back from), and it costs nothing at all until a node opts in. The full field list (supStableAfter, supGiveUpAfter, supReapply, …) is in serve-supervision.md §6.

That last one is what the field is really for, and it’s worth understanding the shape. A node says “I am a Package” without knowing what will be done about it, and a later pass collects the set and acts on it. The later pass is the only place that can act: your Track' directive is a function of one directive in isolation, so it can’t see the other declarations run serve currently holds, or which way each of their nodes is wanted. A Rewrite runs after the graph has been folded to a Ref-keyed DAG, where both of those exist — so it can batch across declarations and split an install batch from a removal batch, neither of which a recipe could express. Register one with CommandLine.execCommandOrSeedWithRewrites rather than applying an Op -> Op inside your Track'.

You will rarely need to add a new dynamic type; if you find yourself wanting one, search for existing toDyn/fromDynamic/getDynamics usages first. The question to ask is whether some later, wider pass needs to know this about your node — if the answer is “only this node cares”, it’s a field on your own value type, not a dynamic.

8. Writing a new builtin node — a worked template

Put a new node under salmon-ops/src/Salmon/Builtin/Nodes/<Concern>.hs (one module per concern — see the existing list: Filesystem, Systemd, Debian.Package, Podman, Postgres, Netfilter, Ssh, WireGuard, etc.). Minimal template:

module Salmon.Builtin.Nodes.Widget where

import qualified Data.Text as Text
import Salmon.Builtin.Extension
import Salmon.Op.Ref (mkRef)
import Salmon.Reporter

-- | Whatever this node needs to do its job.
data Report = CreatedWidget !WidgetName | RemovedWidget !WidgetName
    deriving (Show)

newtype WidgetName = WidgetName Text
    deriving (Eq, Ord, Show)

widget :: Reporter Report -> WidgetName -> Op
widget r name =
    op "widget" nodeps $ \actions ->
        actions
            { help = "creates a widget named " <> getWidgetName name
            , ref = mkRef "widget" name
            , up = createWidget name >> runReporter r (CreatedWidget name)
            , down = removeWidget name >> runReporter r (RemovedWidget name)
            }

Points worth calling out explicitly since they’re easy to get subtly wrong:

Reporter r is the standard way a node emits structured events (not print/logging directly) — every builtin module defines its own Report sum type and takes a Reporter Report as its first argument, then contramaps it into sub-calls (contramap SomeConstructor r) when composing with other nodes’ reporters. Look at any existing module (e.g. PostgresInit .hs’s Report type, which wraps Postgres.Report/Self.Report/etc.) to copy the pattern for a module that composes several builtins.
Don’t forget idempotency (§4) before calling it done — this is the most commonly-missed step for a first-draft node.
Real subprocess calls should go through Salmon.Builtin.Nodes.Binary (withBinary/untrackedExec) rather than raw System.Process calls, so failure propagation (§5) is automatic.

9. Wiring a node into a CLI binary (seed → spec → ops)

If you’re building a whole binary (not just adding one node to an existing recipe), every salmon binary follows the same two-subcommand shape via Salmon.Builtin.CommandLine.execCommandOrSeed:

my-salmon config <seed-args...>   # seed (CLI-friendly) -> JSON directive on stdout
my-salmon run up|down|tree|dag    # reads a JSON directive on stdin, executes/prints it
my-salmon run serve               # reads seed declarations as lines on stdin, converges and tends them

(run serve is the long-running driver; see serve-supervision.md. There is also a query subcommand — query show/query plan/query extract-directive — for looking at or excluding part of the graph; see specs/advance-querying.md.)

You need three things (see salmon-apps/src/Migrator.hs as the worked, complete example):

A Seed type — parsed from CLI args (Options.Generic/optparse-applicative).
A Spec type — FromJSON/ToJSON, the thing that actually gets piped over stdin/stdout between config and run.
A Configure IO Seed Spec (Configure { gen :: Seed -> IO Spec }) plus a Track' Spec (i.e. program :: Track' Spec, program = Track $ \spec -> ... build an Op from spec ...) that turns the Spec into the actual Op graph.

main :: IO ()
main = do
    cmd <- execParser (info parseRecord ...)
    CLI.execCommandOrSeed reportPrint configure program cmd

program :: Track' Spec
program = Track $ \spec -> op "program" (deps [...]) id

configure :: Configure IO Seed Spec
configure = Configure $ \seed -> ... build a Spec from seed ...

Migrator.hs itself calls CLI.execCommandOrSeedWithRewrites, which is the same plus a reporter for run serve’s own reports and a list of Rewrite phases to run over the graph (§7). Use execCommandOrSeed if you have no rewrites.

This split exists so config generation (impure, human-parametrized, runs on the commanding machine) and execution (must be IO/hermetic, meant to run unattended, e.g. piped to a remote box over ssh) stay separate, independently inspectable steps — the JSON Spec is the contract between them.

10. Testing ops

Tests for salmon-ops-recipes live under salmon-ops-recipes/test/Test/ and run via cabal test salmon-ops-recipes. All the plumbing you need is in Test.Harness (salmon-ops-recipes/test/Test/Harness.hs) — read that file; this section is a guide to it, not a replacement for it.

Tests are organized into four tiers by IO cost/blast-radius (Layers 0–3; this section covers the first three, which are what a new node almost always needs). Pick the cheapest tier that actually exercises what you changed.

Layer 0 — structural, no side effects

Assert on the shape of the graph itself, not on any effect. Use Salmon.Builtin.Extension.evalDeps :: Op -> Cofree Graph Op to walk the graph and check what’s in it (e.g. “does this graph contain a node with this Ref”, “how many nodes does it have”). No IO happens at all. Good for checking wiring/composition logic (did I actually connect the deps I meant to) without needing any real environment.

Layer 1 — sandboxed IO, no external services

Run the real up/down for real, but only against something throwaway and local — a temp directory, an in-process value — nothing that needs a running service. Use:

runUp   :: Op -> IO Bool   -- runs upTree, returns True iff everything succeeded
runDown :: Op -> IO Bool   -- runs downTree, same contract
withTempDir :: (FilePath -> IO a) -> IO a  -- auto-cleaned scratch dir
runUpCapturing :: Op -> IO [UpDown.Report Extension]  -- full trace, for asserting Skip/Eval/Blocked directly

Example shape (see JWTSigningSpec.hs for a real one):

testCase "writes the file" $ withTempDir $ \dir -> do
    let op = someNodeThatWritesInto dir
    ok <- runUp op
    assertBool "expected up to succeed" ok
    contents <- readFile (dir </> "expected-file")
    contents @?= "expected contents"

Always check the Bool runUp/runDown return (via assertBool) in addition to any postcondition check — a postcondition alone can’t tell you whether the traversal itself reported Failed/Blocked somewhere it shouldn’t have.

Layer 2 — real system services via disposable podman containers

For anything that needs a real service (postgres, a systemd unit, apt) to verify against, dogfood the project’s own Podman builtins as the sandbox provisioner — do not hand-roll podman run/podman rm shell-outs. Guard the whole test with requireExecutable "podman" so CI/dev machines without podman skip loudly instead of failing:

testCase "creates a real database" $ requireExecutable "podman" $
    withContainer (Podman.Image "debian:bookworm")
                  (Podman.PortMapping "15432" "5432" Podman.TCPPort) $ \cid -> do
        -- one-time sandbox prep that isn't part of the recipe under test:
        podmanExec_ cid ["apt-get", "update", "-qq"]

        withShimmedPath cid ["apt-get", "sudo", "bash"] $ do
            let op = MyRecipe.setupSomething ...
            ok <- runUp op
            assertBool "recipe should fully succeed" ok

        -- postcondition: check it for real, inside the container
        (code, out, err) <- podmanExecCapture cid ["sudo", "-u", "postgres", "psql", "-lqt"]
        assertBool ("expected db in: " <> out) ("mydb" `isInfixOf` out)

Key pieces, all from Test.Harness:

withContainer :: Podman.Image -> Podman.PortMapping -> (String -> IO a) -> IO a — pulls the image, starts a uniquely-named container, hands you its container id, and guarantees teardown (via the real Podman down action, not a manual podman rm) even on exception.
withShimmedPath :: String -> [String] -> IO a -> IO a — for recipes that shell out to real binaries by name (apt-get, sudo, psql, …) with no indirection to mock: writes lookalike wrapper scripts for each named command that forward into the container via podman exec, and prepends them to PATH for the duration of the action. This lets you run the recipe’s actual, unmodified command construction and graph wiring against the sandbox, instead of testing a mocked stand-in. Figure out which commands to shim by reading which binaries the recipe under test actually invokes (see PostgresInitSpec.hs’s shimmedCommands for a worked example and comment explaining the reasoning).
podmanExec_ :: String -> [String] -> IO () — run a setup command inside the container, discard output, error on non-zero exit. For sandbox prep only (e.g. apt-get install sudo), not for the thing under test.
podmanExecCapture :: String -> [String] -> IO (ExitCode, String, String) — same, but for postcondition checks: hands back everything instead of throwing.

A real bug this pattern caught (documented at the top of PostgresInitSpec.hs): a builtin used to hardcode a Postgres cluster version that didn’t match the container’s actual installed version, so the recipe silently failed to start the cluster — a Layer 0 test would never have caught this, because the graph shape was perfectly correct; only running it for real against a real (if disposable) Debian container did.

Layer 3 — a whole machine, via a qemu VM

For what a container can’t exercise well (systemd as PID 1, real network interfaces), Test.Harness also boots a qemu VM from a debootstrapped rootfs (withVm/withVmAt, then sshToVm/scpToVm), dogfooding Salmon.Builtin.Nodes.LinuxBridge/Qemu as the provisioner. It needs root or the one-time setcap grants described on hasVmPrivileges, and skips loudly otherwise; see QemuSmokeSpec.hs for the smallest example and specs/qemu-test-vms.md for the design. Layer 2 and 3 specs contend for process-global resources (PATH shims, one test bridge), so test/Main.hs runs them inside one sequentialTestGroup — add yours there, not to the concurrent list.

Choosing a layer: a quick decision guide

Changed how ops are wired together (deps/ordering/ref dedup)? → Layer 0 is probably enough.
Changed what a node actually writes/does, but it’s pure filesystem/local state? → Layer 1.
Changed anything that shells out to a real service’s CLI (psql, pg_ctlcluster, nft, systemctl, apt-get, podman, …) or depends on that service’s actual behavior? → Layer 2 — a Layer 0/1 test would pass even if the actual command is wrong, wrong-ordered, or the tool’s real idempotency behavior doesn’t match what you assumed.
Needs a real init system, real interfaces, or more than one machine? → Layer 3.

Running the tests

cabal test salmon-ops-recipes

Layer 2 tests are skipped loudly (not failed) if podman isn’t on PATH — check the test output for SKIPPED: lines if you expect Layer 2 coverage and don’t see it exercised.

Postgres with automatic failover: salmon provisions, Patroni decides

2026-09-25T14:15:36Z

Generated from specs/pg-patroni.md — the repository is the canonical source, and may be ahead of this page.

Postgres with automatic failover: salmon provisions, Patroni decides

Status: draft / not implemented. A design sketch to react to, not a committed plan.

Companion: pg-switchover.md is the two-node, operator-driven design for test harnesses and low SLAs. This one is for when nobody should have to be awake for a failover.

Position

Salmon does not decide when to fail over, and does not know who the primary is. Safe automatic failover needs consensus, so that a split network cannot elect two primaries, and a leader that demotes itself when it loses the quorum, which is Patroni’s form of fencing. run serve is one process per machine with no coordination between them. Building consensus into salmon would mean writing a worse etcd.

So the split of work is:

Salmon	Patroni
installs and configures etcd, Patroni, routing	runs Postgres: initdb, start, stop, restart, promote, demote
supervises the Patroni and etcd systemd units	supervises Postgres
states the cluster-wide config it wants	applies it, restarting members when a setting needs it
creates databases, roles, templates, clones, runs migrations, through the leader	holds the leader key in etcd, i.e. the only answer to "who is primary"
backups and archiving	uses the archive to rebuild replicas

The chain of ownership is salmon → systemd → Patroni → Postgres. Salmon’s supervision (Actions/Upkeep.hs) must stop at the Patroni unit: a salmon node that restarts Postgres is fighting its owner.

The consequence for everything that exists

The primary’s location never appears in a directive, a ref, help or notes. This is the opposite of pg-switchover.md, where the operator’s declaration is the truth. Here, a declared primary would be stale after the first automatic failover, and the next run up would try to “correct” it.

Existing builtins that must not run on a Patroni member:

Builtin	Replaced by
`createCluster`, `pgLocalCluster` (initdb, start)	Patroni's bootstrap on the first member, and its replica creation on the others. Debian's `postgresql@` units must be disabled or masked, or two supervisors race for the data directory.
`startCluster` / `stopCluster` / `restartCluster` / `promoteCluster`	nothing; `patronictl` if an operator must
`alterSystemSet`, `reloadConf`	the DCS config node (below). `ALTER SYSTEM` still "works", but Patroni can overwrite what it sets and it does not reach the other members
`hbaLine`, `allowReplicationFrom`, `allowClientCertFrom`	`postgresql.pg_hba` in `patroni.yml`, or the DCS config
`primaryReplicationSetup`, `standbyReplicationSetup`, `replicationSlot`	Patroni: it creates replicas and manages physical slots for members

Builtins that still apply, as long as they reach the leader: database, user, group, grant, adminScript, templates and clones, SreBox.PostgresMigrations. Every one of them runs sudo -u postgres psql -p <port> on the local machine today. On a replica, CREATE DATABASE fails with “cannot execute … in a read-only transaction”. See “Reaching the leader”.

New builtins

`Etcd`

Renders the config and a systemd unit; the check is etcdctl endpoint health against the member itself.
The trap is bootstrap. initial-cluster-state is new exactly once in a cluster’s life. After that, a new member is added by a runtime call (etcdctl member add) followed by starting it with existing, and it cannot be expressed as config alone. A node for “etcd cluster of these three” therefore has a seed phase (all start with new when no member answers) and a join phase (member add, then start). Its check compares the member list, and does not look at a config file.
TLS: peer and client certificates come in pre-provisioned, per the recipes’ secret-transport rule. Certificates can mint them in a recipe that owns that choice; the builtin only takes paths.
Several Patroni clusters can share one etcd (each has its own scope and namespace). This is how pg-ha-control-plane.md’s cross-replicated pair becomes safe: two Postgres machines plus a small third one running only etcd, with three votes between them.

`Patroni`

Renders patroni.yml (an EncodeFileContents value, so its content fingerprint reaches notes and a re-declaration is Stale under run serve), plus a systemd unit.
The check reads the local REST API on port 8008:
- GET /health: is Postgres running under Patroni;
- GET /patroni: state, role, pending_restart, and whether the member is in the cluster at all.
- Role is reported, never judged: a replica is as healthy as a leader.
Debian’s patroni package ships its own integration with the postgresql-common layout (pg_createconfig_patroni and a per-cluster patroni@ unit). Whether to build on it or render our own is an open question below.

The cluster-wide config node

This is ALTER SYSTEM’s replacement. Patroni keeps postgresql.parameters and friends in etcd, applied on every member.

check: GET /config, comparing only the keys this node declares, so that other declarations or operators can own other keys.
up: PATCH /config with those keys.
A second check condition, the same move as checkService reading NeedDaemonReload: after a restart-only setting changes, /patroni on each member says pending_restart. The node is not Success until nothing is pending, and its up can trigger patronictl restart --pending, which restarts replicas before the leader. Whether salmon should restart members at all, or only report, is an open question.

Routing to the leader

Three options, from least to most machinery. The recipe should offer the first and one of the others:

libpq multi-host connection strings: host=a,b,c target_session_attrs=read-write. No new process; works only for libpq clients and only at connect time.
vip-manager: a floating IP that follows the leader key in etcd. Needs one L2 segment.
HAProxy with option httpchk on GET /primary (port 8008): the standard Patroni setup. One port for the leader, one for replicas (GET /replica). It sits in front of, or replaces the upstream of, the existing PgBouncer.

A Haproxy builtin would have the same shape as Nginx: a config value, a renderer, and a systemd unit. Unlike pgbouncer, HAProxy needs no reload on failover: its health checks move the traffic.

Reaching the leader

Admin nodes need a way to reach whichever member is the leader right now, which is not known when the directive is written.

The recommendation is to address the leader by its routed endpoint (the HAProxy primary port, or the VIP) with a connection string, not by sudo -u postgres on a machine. That means connection-string variants of the admin builtins. SreBox.PostgresMigrations.remoteMigrateOpaqueSetup already works this way; database, user, templates and clones do not. This is the largest single piece of work in this spec, and it is independently useful: anything managed (Cloud SQL, see Gcp/PostgrestCloudRun) has the same “no local superuser” shape.

The alternative, running the admin nodes on every member with a check that answers Skipped when this member is not the leader, is smaller but racy:

the leader can change between the check and the up;
templates would be built by whichever member was the leader at the time.

Refused.

Nodes that must run on exactly one member

Backups (from a replica, by preference) and scheduled jobs (CronTask) would otherwise run on every member.

Backups: a Patroni tag such as nofailover or a custom one marks the member dedicated to them. The scheduled command gates itself with curl -sf localhost:8008/replica, skipping when the member is not a replica.
PITR: SreBox.PostgresBackup is pg_dump, which has no point-in-time recovery and no use to Patroni. A continuous archive (pgBackRest or WAL-G, both packaged by Debian) earns its place twice: as the disaster recovery story, and as Patroni’s create_replica_methods, which rebuilds a replica from the archive instead of loading the leader with a base backup.

Switchover, under Patroni

pg-switchover.md’s role node has a counterpart here, and it is opt-in: a preferred leader.

check: GET /leader names the preferred member.
up: POST /switchover.

The default is no preference. With a preference declared and supervision on, salmon would move the leader back after every automatic failover, as soon as the preferred member is healthy. That is sometimes what is wanted (a preferred site or a bigger machine) and sometimes a flap. It must be a choice, never the default.

Disaster scenarios

These reuse pg-switchover.md’s scenario catalogue where the cause is the same, with different expected outcomes, plus the ones only consensus makes meaningful.

Machines: three VMs, each with postgresql, patroni, etcd-server and haproxy in the rootfs (the guests cannot reach the network past boot, see Test.PostgresReplicationSpec). The addresses are testVmAddr/testVmAddr2/testVmAddr3. HAProxy and the client loop run on one of them, or on a fourth VM.

#	Scenario	Assert
T1	Kill the leader's machine	a new leader within the TTL (about 30s by default); writes through HAProxy resume; the old leader rejoins as a replica when it comes back
T2	Partition the leader from the other two	the old leader demotes itself once its lease expires; at no moment do two members accept writes (the client loop writes through each member's own port and records which accepted)
T3	After T1, rerun `run up` on every member and on the controller	nothing changes: no restart, no config write, no role change. This is the salmon-specific assertion: salmon does not fight Patroni
T4	Change a restart-only parameter through the config node	members restart replicas first; `pending_restart` clears; the node's check reaches `Success`
T5	Lose etcd quorum (stop two of three)	with `failsafe_mode` off, the leader goes read-only; with it on, the leader keeps writing while it can reach all members. Pins down which mode the recipe defaults to
T6	Create a database and a clone through the routed endpoint, fail over, repeat	both work on each leader; no admin node ever ran against a replica
T7	Add a fourth member	etcd's join phase runs, not a bootstrap; Patroni builds the replica from the archive if one is configured

T3 is the test the whole design rests on, and the reason “the primary never appears in a directive” is a rule rather than a guideline.

Phased plan

Connection-string admin builtins (database, user, grants, templates, clones). Useful today for managed Postgres, and required here.
Etcd, with the seed and join phases; Layer 3, three VMs, including a restart of one member.
Patroni plus the cluster-wide config node; T1, T3, T4.
Routing: multi-host connection strings first, then HAProxy; T2, T6.
Archive and PITR (pgBackRest or WAL-G), and backups on one member; T7.
The opt-in preferred leader.

Open questions

Debian’s pg_createconfig_patroni and patroni@ units, or our own? Theirs fits postgresql-common’s layout, which the rest of Postgres.hs assumes (pg_lsclusters, /var/lib/postgresql/$version/$cluster). Ours is simpler to reason about, and portable to non-Debian hosts.
Should the config node restart members (patronictl restart --pending), or report pending_restart and leave the restart to an operator? Restarting is what run up means everywhere else. A restart of the leader under load is also a small outage.
failsafe_mode default (see T5).
REST API authentication: Patroni’s unsafe endpoints (/switchover, PATCH /config, /restart) need restapi.authentication. The credentials come in as a pre-provisioned file.
Which etcd API: Patroni’s etcd3: section, the v3 API, is the only sensible one with current etcd. Say so in the builtin and do not offer v2.

HA Postgres + bouncer + app-instance + LB control plane

2026-09-25T14:15:36Z

Generated from specs/pg-ha-control-plane.md — the repository is the canonical source, and may be ahead of this page.

HA Postgres + bouncer + app-instance + LB control plane

Status: draft / not implemented as a whole. Of the phased plan, only the first half of phase 1 exists: the operator-driven pair of specs/pg-switchover.md (SreBox.PostgresPair, salmon-pgpair); the Patroni half is still a sketch, and phases 3–7 (app-instance recipe, LB and DNS wiring, ControlPlaneSeed, hardware profiles, monitoring) have no code. This is a design sketch to react to, not a committed plan.

The database tier (§1–2) has moved to pg-switchover.md (operator-driven, two nodes) and pg-patroni.md (automatic failover). This spec keeps the bouncer/app/LB/DNS/monitoring picture around them.

Problem

We want to host a service (multiple services, eventually) on top of:

a replicated Postgres pair per logical cluster, WAL-streamed both ways across two machines so each machine holds one primary and one standby (“diagonal” replication — machine.ab.0 is primary for pg.a/standby for pg.b, machine.ab.1 is the mirror image),
pgbouncer in front of each cluster, pointed at whichever side currently holds the primary role,
app instances (PostgREST-style, plus the user’s own “internaltool” service) that need their own pg users/roles/secrets/CORS/rate-limit settings, sat behind the bouncers,
a front load balancer (or a small HA pair of them) fronting the bouncers and app instances, with DNS pointed at it,
monitoring/alerting across all of the above,
multiple sizing tiers of this whole shape — shared multi-tenant deployments and dedicated ones on different hardware profiles.

Sketch (as given):

 ┌────────────────────┐        ┌────────────────────┐
 │ pg.a   pg.b         │  wal   │ pg.a   pg.b         │
 │ (primary) (standby) │◄──────►│ (standby) (primary) │
 │ machine.ab.0         │        │ machine.ab.1         │
 └──────────┬──────────┘        └──────────┬──────────┘
      bouncer.a                       bouncer.b
            │  (crossed: every app instance can reach either bouncer)
   internaltool.a.0  internaltool.b.0  internaltool.a.1  internaltool.b.1  postgrest.b  control-plane
            └──────────────────────┬──────────────────────┘
                                   LB ── DNS

Sticky note requirement: control-plane seeds must be unfoldable to machines, pg clusters (certs, users), bouncers, app-instances (pg-users, secrets, roles, cors-settings, rate-limit-settings), load-balancers (DNS configs), PostgREST configs (secrets, pg-users), and monitoring/alerting configs.

What already exists (inventory)

Salmon already has most of the leaf building blocks this needs; nothing below is new work:

Diagram box	Existing code
pg primary/standby, WAL streaming	`Salmon.Builtin.Nodes.Postgres` (`primaryReplicationSetup`/`standbyReplicationSetup`/`replicationUser`/`EnsurePhysicalReplicationSlot`) — exercised today only by the hand-run `salmon-postgres-replication-fixture`, not wired into a real recipe
bouncer.a / bouncer.b	`Salmon.Builtin.Nodes.PgBouncer` — takes plain host/port/dbname/user/password, deliberately decoupled from how the upstream cluster was provisioned
LB	`Salmon.Builtin.Nodes.Nginx` — vhost list → upstream group, reverse proxy. Nothing HA (keepalived/VRRP) for "load-balancer(s)" plural yet
DNS	`SreBox.MicroDNS`, `SreBox.DNSRegistration`
app instance (internaltool/postgrest shape: pg-user, secret, connstring, systemd unit, pushed to a remote box)	`SreBox.Postgrest` is the exact template — build the "internaltool" recipe by mirroring its `PostgrestSetup`/`setupPostgrest` shape, not by generalizing it prematurely (see `[[recipe_key_exchange_agnostic]]`-style convention: pass in pre-provisioned secrets, don't invent a transport)
seed → directive → ops, long-running convergence	`Salmon.Builtin.CommandLine.execCommandOrSeed`, `Salmon.Actions.Serve` (`World`/`Epoch`/`NodeState`)
declaring "which pg cluster/user/db"	`SreBox.PostgresInit`, `Postgres.CreateDB`/`CreateUser`/etc.

What’s genuinely missing:

A recipe for a replicated pair and moving its primary, which is now pg-switchover.md, and for automatic failover, now pg-patroni.md.
(Was: a seed carrying forward “which side is primary” from one config to the next. Neither design needs it; see §1–2.)
Sizing tiers / hardware profiles (shared vs. dedicated) as a first-class concept threaded through machine/cluster seeds.
A control-plane seed that unfolds into the sub-seeds for machines/clusters/bouncers/app-instances/LB/DNS/monitoring, per the sticky note.
Monitoring/alerting nodes — zero Prometheus/alerting builtins exist in the repo today.
An HA front LB story — today’s Nginx.hs is a single reverse proxy config renderer; “load balancer(s)” plural in the diagram implies either DNS-level multi-A-record fanout (already partially available via MicroDNS) or a keepalived/VRRP-style active/standby pair, neither of which has a builtin yet.

Design goals / non-goals

Goals:

Reuse every existing builtin/recipe listed above unchanged; new code should be composition, not rewrites.
Keep the “recipes are key-exchange/secret-transport agnostic” convention: new recipes take pre-provisioned secrets/certs/files, never invent a way to move them (matches the existing constraint on salmon-ops-recipes modules).
Salmon never decides when to fail over. Either an operator declares where the primary is (pg-switchover.md), or Patroni decides and salmon never mentions the primary at all (pg-patroni.md). There is no third mode where salmon guesses.
Sizing tiers should be data (a profile value in the seed), not a code fork — a shared-tier deployment and a dedicated-tier deployment should go through the same recipes with different profile values.

Non-goals (v1):

Automatic failover built into salmon. Salmon has no consensus, so it delegates that to Patroni; see pg-patroni.md.
A generic multi-service control plane. Build this for the pg/bouncer/app/ LB shape in the diagram; generalize later only if a second, differently- shaped service shows the abstraction is right.
Monitoring dashboards/alert rules content — v1 is just “the nodes exist to install and configure an agent,” not a curated set of alerts.

Proposed architecture

1–2. The database tier: superseded by two specs

This section used to sketch a PgClusterPair recipe whose seed carried pair_primary_side, and a state-file convention to carry that side forward from one config to the next. Both are replaced. Which one applies depends on who decides where the primary is:

pg-switchover.md: the operator decides. Two machines, the primary’s location is a per-pair field in the directive, and salmon moves it with a resumable switchover (or a failover the operator vouches for). No state file is needed: without automatic failover, the declaration is the truth. This is the tier for test harnesses, disaster scenarios and low-SLA services.
pg-patroni.md: Patroni decides, with etcd for consensus. The primary’s location must never appear in a directive, or the first automatic failover makes every run up fight it. Admin nodes and bouncers reach the leader through a routed endpoint (HAProxy, a VIP, or libpq multi-host) instead.

The cross-replicated two-machine layout (pg.a primary on machine.ab.0, pg.b primary on machine.ab.1) works for the first, as a data choice rather than a code fork. For the second it needs a third, small machine running only etcd, since two machines cannot form a quorum.

bouncerUpstream survives only in the first: under Patroni the bouncers’ upstream is the routed endpoint, which never changes.

The general idea of a seed that reads back what it last converged to (stable secrets, stable port allocations) is still worth having, but nothing here needs it any more. Build it when the first real case appears.

3. Sizing tiers / hardware profiles

A Tier value threaded through the control-plane seed, data not code:

data Tier
    = Shared     -- multiple logical clusters/instances per machine
    | Dedicated  -- one logical deployment owns the machine
    deriving (Eq, Show, Generic)

data HardwareProfile
    = HardwareProfile
    { profile_tier :: Tier
    , profile_pg_shared_buffers :: Text   -- e.g. postgresql.conf tuning knobs
    , profile_bouncer_max_client_conn :: Int
    , profile_app_instance_count :: Int
    -- extend as real constraints show up; resist modeling resource limits
    -- (cgroups/systemd slices) until a concrete need forces it — nothing in
    -- salmon-ops does resource-limiting today
    }

Postgres.defaultReplicationTuning and PgBouncer.BouncerConfig’s size-shaped fields (bouncer_max_client_conn, bouncer_default_pool_size) already exist as plain values a HardwareProfile can feed — no changes needed to those modules, just don’t hardcode the numbers in the new control-plane recipe.

4. Control-plane seed unfolding

Mirror the existing Migrator.Seed → Migrator.Spec → Migrator.Ops three-file split (salmon-apps/src/Migrator/), one level deeper, so the control-plane binary follows the same seed→spec→ops shape every other salmon binary already uses (per CLAUDE.md’s “seed → spec → ops CLI protocol” section) instead of inventing a new pattern:

-- ControlPlane/Seed.hs — human/CLI-facing, ParseRecord
data ControlPlaneSeed
    = ControlPlaneSeed
    { cp_tier :: Tier
    , cp_pair :: ClusterSeed              -- see pg-switchover.md / pg-patroni.md
    , cp_app_instances :: [AppInstanceSeed]  -- internaltool.a.0, internaltool.b.0, ...
    , cp_postgrest :: [PostgrestSeed]
    , cp_lb :: LbSeed
    , cp_dns :: DnsSeed
    , cp_monitoring :: MonitoringSeed
    }

-- ControlPlane/Spec.hs — FromJSON/ToJSON directive, output of `config`
data ControlPlaneSpec
    = ControlPlaneSpec
    { cps_pair :: Pair               -- SreBox.PostgresPair, or a Patroni cluster
    , cps_bouncers :: [PgBouncer.BouncerConfig]
    , cps_app_instances :: [AppInstanceSetup]     -- mirrors PostgrestSetup
    , cps_postgrest :: [PostgrestSetup]
    , cps_lb :: Nginx.NginxConfig
    , cps_dns :: ...
    , cps_monitoring :: ...
    }

-- ControlPlane/Ops.hs — Track' ControlPlaneSpec -> Op, composing the above
-- recipes exactly as SreBox.Postgrest/SreBox.Initialize already do

gen :: ControlPlaneSeed -> IO ControlPlaneSpec is where all the “unfold one seed into machines/clusters/bouncers/…” fan-out happens — e.g. deriving each internaltool.X.N instance’s pg-user name from X/N deterministically, deriving bouncer upstream lists from cps_pair + bouncerUpstream above, deriving the LB’s vhost upstream list from the concrete app-instance ports. This step is pure fan-out/derivation logic, independently testable without touching any Op/IO machinery (same argument advance-querying.md makes for why directive-shape is worth pinning down separately from execution).

Whether one ControlPlaneSeed should directly enumerate every app instance (as sketched above) or itself be built from a smaller “how many of each tier” description is an open question below — start with explicit enumeration (simplest, matches how Migrator.Seed already just lists fields) and only introduce a generator if the enumeration gets unwieldy.

5. Monitoring/alerting — new builtin, deliberately thin in v1

New Salmon.Builtin.Nodes.Monitoring (naming TBD) module, following the PgBouncer.hs/Nginx.hs shape exactly (a config value type, a render function, justInstall + Systemd.systemdService/restartService). v1 scope: install and configure a metrics agent (node_exporter-style) per machine and a scrape-target list on whatever central collector exists — not alert rule content, dashboards, or the collector itself, all of which need a stack decision first (see open questions).

6. HA front LB

Two independently-shippable pieces, not one:

Multiple LB instances: Nginx.setup already renders a full config from a value — running it on two boxes is just calling it twice with the same NginxConfig. No new code.
Active/standby or DNS-fanout in front of them: either extend SreBox.DNSRegistration/MicroDNS to publish multiple A records (client-side failover/round-robin, simplest, no new node), or add a new Keepalived/VRRP builtin for a floating VIP (real HA, meaningfully more work: needs its own idempotency story per CLAUDE.md’s conventions section since keepalived.conf has no natural “replace” verb, would likely follow the Netfilter.rule-style prelim-based skip). Recommend starting with DNS fanout and only building VRRP support if it’s a hard requirement.

Open questions

Which database tier per deployment tier: does the shared tier run on pg-switchover.md and only the dedicated tier on pg-patroni.md, or does everything that serves real traffic go to Patroni?
Monitoring stack: Prometheus + node_exporter + something for alerts (Alertmanager? a hosted service?) — needs a decision before §5 can be more than a stub.
“Internaltool” vs. generic app-instance: is SreBox.Postgrest close enough to fork/mirror directly, or does internaltool need meaningfully different shape (its own migrations, non-PostgREST HTTP surface, different secret set)? Affects whether §4’s AppInstanceSeed is its own new module or a thin renaming of PostgrestSeed.
Cross-wiring in the diagram (every internaltool.X.N reaching both bouncers, not just its “own” one): intentional (read replicas via the standby-side bouncer, or just redundancy), or a simplification in the sketch? Determines whether AppInstanceSetup takes one connstring or a primary/replica pair.
Enumeration vs. generator for ControlPlaneSeed (§4): revisit once a first real deployment shows how many app instances/tiers actually need representing.

Phased plan

The database tier: pg-switchover.md’s phased plan, then pg-patroni.md’s. Each has its own Layer 3 disaster scenarios.
(Was: the previous-seed state convention; dropped, see §1–2.)
AppInstanceSeed/internaltool recipe (§4, mirroring SreBox.Postgrest), wired to a single pair — no LB/DNS/monitoring yet.
LB + DNS wiring (§6, DNS-fanout option first).
ControlPlaneSeed (§4) tying 1–4 together end to end for one tier.
HardwareProfile/Tier (§3) threaded through, second tier added.
Monitoring (§5), stub scope.

Future work

VRRP/keepalived-based LB HA (§6), if DNS fanout turns out insufficient.
Generalizing the control-plane seed pattern beyond this one service shape, if/when a second differently-shaped service needs the same treatment.

Guides

2026-09-25T12:00:00Z

Guides

These pages are generated straight from the resources/ directory of the salmon repository — the repository is always the canonical source, and may be ahead of what is published here.

The salmon model — the general, domain-independent model that lives in salmon-core (Graph, OpGraph, Track, Eval), why it is shaped this way, and where else it could be pointed. Start here for the why.
How to write and test salmon ops — the cookbook: copy the patterns, adapt the names, and you produce a valid Op. Written to be usable as a reference even by less-context-heavy tooling. Start here for the how.
Salmon ops patterns — recurring shapes worth reusing across recipes, such as a one-time privileged bootstrap.
run serve: convergence and supervision — what the long-running mode gives you for free (convergence, status/force/pause, self-healing on re-declaration), how to try it, the line protocol, the HTTP API and event stream, the web UI, TLS over TCP, pull mode and the status sink, and what to decorate a node with (check, Supervision, managed) to get more out of it.
A Postgres pair whose primary is declared, not discovered — what a switchover does step by step, when the recipe refuses and why, and how to watch a client keep writing across one with the salmon-toy-qemu-pg-ha demo.
Validating the GCP builtins against a real project — the playbook for exercising the Gcp.* builtins in a sandbox you are willing to destroy, with the salmon-gcp-toy binary.

Two more documents exist only in the repository:

CLAUDE.md — the architecture module by module and the repository-level conventions (package layering, idempotency, failure propagation), for anyone, human or AI, working in the codebase.
CHANGELOG.md — release history.

Getting started with salmon

2026-09-25T12:00:00Z

Getting started with salmon

Salmon is a Haskell library and toolkit, not a hosted service: you write a small binary over the recipes you need, and that binary provisions things. This page gets you from a checkout to a running graph.

Build

git clone https://github.com/lucasdicioccio/salmon
cd salmon
cabal build all
cabal build salmon-core salmon-ops salmon-ops-recipes salmon-apps   # individual packages
cabal build salmon-migrator                                          # a single executable

The project is layered strictly bottom-to-top:

salmon-core  <-  salmon-ops  <-  salmon-ops-recipes  <-  salmon-apps
                                        ^
                                        |
                          salmon-ops-recipes-experimental

salmon-core — the algebraic Graph, and the OpGraph/Track/Eval abstractions that let “create a file” and “turn a server up” be the same type. No IO-heavy dependencies, kept minimal on purpose.
salmon-ops — IO-heavy primitives (files, systemd, Debian packages, podman, postgres, wireguard, certificates, ssh, netfilter, cron, rsync, qemu, GCP, …), the drivers that execute a graph (one-shot up/down, and the long-running serve loop with its supervision, HTTP API, web UI and pull mode), and the CLI plumbing every salmon binary shares.
salmon-ops-recipes — higher-level, opinionated compositions of the builtins, where project conventions get enforced. Has the real test suite.
salmon-ops-recipes-experimental — recipes needing heavier or less stable dependencies (kitchen-sink site generation, ACME certificates). Not in the default package set.
salmon-apps — the shipped binaries, each a small Main over a recipe.

Some git dependencies in cabal.project point at the author’s other repositories, pinned by commit; they are not on Hackage.

Test

cabal test salmon-ops-recipes

Tests are tiered by IO cost and blast radius: cheap in-process assertions on graph shape and traversal, up through tests that dogfood the project’s own Podman builtins to run real recipes against disposable containers, and a few that build qemu VMs. The container and VM tiers need podman or qemu on the machine and are skipped loudly otherwise. See Writing ops, §10 for the breakdown.

Using a salmon binary

Every salmon binary speaks the same two-step protocol: config turns human-facing command-line arguments (a seed) into a JSON directive, and run reads that directive on standard input and acts on the graph it expands to.

my-salmon config <seed-args...>                        # seed -> JSON directive on stdout
my-salmon config <seed-args...> | my-salmon run up     # converge the graph
my-salmon config <seed-args...> | my-salmon run down   # tear it down
my-salmon config <seed-args...> | my-salmon run tree   # print the dependency tree (run dag: Graphviz)
my-salmon config <seed-args...> | my-salmon query show --select '/some/path/**'
my-salmon run serve                                    # read seed declarations as commands, keep converging

The split keeps the possibly-impure “decide what I want” step separate from the hermetic “make it so” step, which is meant to run unattended, often on another machine. run up/run down/run serve take --json for one JSON object per report. query targets a subset of the graph: query plan --exclude PATTERN writes a plan that run up --plan FILE honours.

run serve is the long-running mode. It reads up/down/only/clear/ converge/status/… lines, keeps a world of every declared seed, converges after each one, and between commands tends the nodes, re-applying what goes missing under a per-node supervision policy. The same loop can be driven over a unix socket, over HTTP with an event stream and a web UI (or TLS and a token over TCP), from a terminal client, or pull its declarations from a registry — a directory, git, HTTPS, DNS or a bucket, optionally signed — and report back through a status file that salmon-fleet status folds. All of that is run serve: convergence and supervision.

With --http PATH the same process also serves a web page (and a JSON API, and an event stream) on a second socket, which salmon-tui and a browser can attach to. The repository’s fixture binary, salmon-ops-serve-fixture, is the quickest way to see it: run serve --http /tmp/x.http, declare a seed or two, and open the page:

The model, in one paragraph

An Op is a graph node centered on itself, with an effectful recipe for finding its own predecessors — this is what lets a one-line filesystem op and a whole clustered-database-with-replication setup type-check identically. Every op carries up/down (bring the effect into being / undo it), check (a CheckResult answering “is my effect already in place”, which is what makes a node idempotent to re-run), and a Ref that gives it a stable identity so the traversal can dedupe a resource reached via several graph paths. The drivers collapse the materialized graph into a DAG with one node per Ref, walk it in dependency order (or the reverse, for teardown), contain real failures (a thrown exception marks a node Failed and blocks everything depending on it rather than being swallowed), and report exactly what happened, node by node.

Write your own binary

Define a seed type, a directive type (FromJSON/ToJSON), a Configure IO seed directive, and a Track' directive that turns a directive into an Op by composing builtin nodes and recipes; then hand the four to Salmon.Builtin.CommandLine.execCommandOrSeed. The cookbook walks through each piece with a worked example, and salmon-apps/src/Migrator.hs in the repository is a complete one.

salmon

What is left of `specs/per-node-state-machines.md`

What is left of specs/per-node-state-machines.md

Where this stands

Shipped

Left

The tradeoffs, in one place

Landed, but wanting another iteration

I1. A demoted node consults its own check — fixed

I2. The strategy is authored on the dependency, not on the dependant

I3. supStableAfter carried two unrelated meanings — done

I4. The cascade needs opting in at every hop

I5. Adoption refreshes a machine’s supervisor, but not its policy — done

I6. A re-declaration that changes what a node is does not re-apply it — done

Not in any milestone

R1. Nodes have no check, so almost nothing is actually supervised

What Immaterial settled

The behaviour change the remaining two carry

R2. An operator cannot address a node, so the mailbox is unreachable — done

R3. statusOutput has no reader — done

R4. query/run tree/run dag still print the declared graph — done

R5. Supervisor-level restart — done

R6. Bounding concurrency — done

R8. Restart means two different things — done

R7. Two dead bindings — done

R9. “Re-apply me on the loop; it is cheaper than asking” — done

The order I would do it in

How milestones 8 and 9 actually went

Milestone 8: Managed nodes

Milestone 9: rest_for_one

9.1 The exact gap, as the code stands

9.2 It has to be a per-node choice

9.3 The hazard to design against

9.4 What to test

Relationship to the other specs

The salmon model

The salmon model

The problem this is solving

The four core pieces

Graph — the algebraic graph

OpGraph — a self-centered node with an effectful neighbor-finder

Track — “given an a, I know how to build the graph for it”

Eval.expand — materializing the graph

Why a graph, not a plain sequence of steps

Actions/Act/Actionless — the generic “bag of named effects”

Generalizing beyond infrastructure

Vocabulary (from the original salmon-core README)

Where to go next

Terraform integration: consuming (and optionally driving) existing Terraform-managed infra

Terraform integration: consuming (and optionally driving) existing Terraform-managed infra

Problem

What “integration” could mean — two different things

Design goals / non-goals

Proposed design

1. Reading outputs into a seed (Model A)

2. Recommended workflow shape

3. Optional: Salmon.Builtin.Nodes.Terraform as an Op (Model B)

4. Reading Terraform-generated secrets

Open questions

Phased plan

Future work

Salmon as PID 1: an init system whose unit graph is a real DAG

Salmon as PID 1: an init system whose unit graph is a real DAG

What has shipped since this was first written

Problem / goal

The build model: a cabal-built init, not a generic one

Why this is genuinely attractive (the pitch)

What genuinely is not here

1. (Shipped.) “Keep this running” is managed, and the init node is a second Daemon

1b. The one new tension: Daemon’s bracket versus “services are children of PID 1”

2. PID 1 has duties that are not convergence at all

3. The bootstrap paradox

The hard engineering problem: waitpid(-1) versus the GHC runtime

Architecture: a Rust PID 1, a Haskell PID 2

Restart backoff: PID 1 rate-limits, the supervisor decides

How they compose instead of fighting

The supervisor slot is special

The config file

Clock

Repo consequences

What is left of `specs/per-node-state-machines.md`

I1. A demoted node consults its own `check` — fixed

I3. `supStableAfter` carried two unrelated meanings — done

R1. Nodes have no `check`, so almost nothing is actually supervised

What `Immaterial` settled

R3. `statusOutput` has no reader — done

R4. `query`/`run tree`/`run dag` still print the declared graph — done

R8. `Restart` means two different things — done

Milestone 8: `Managed` nodes

Milestone 9: `rest_for_one`

`Graph` — the algebraic graph

`OpGraph` — a self-centered node with an effectful neighbor-finder

`Track` — “given an `a`, I know how to build the graph for it”

`Eval.expand` — materializing the graph

`Actions`/`Act`/`Actionless` — the generic “bag of named effects”

3. Optional: `Salmon.Builtin.Nodes.Terraform` as an `Op` (Model B)

1. (Shipped.) “Keep this running” is `managed`, and the init node is a second `Daemon`

1b. The one new tension: `Daemon`’s bracket versus “services are children of PID 1”

The hard engineering problem: `waitpid(-1)` versus the GHC runtime

The control socket is `specs/generic-server.md`’s socket

State that `World` doesn’t have — now mostly `Upkeep`’s

New modules under `salmon-ops/src/Salmon/Builtin/Nodes/Gcp/`

New recipes under `salmon-ops-recipes/src/SreBox/Gcp/`

3. Shared Core: `Gcp.Core`

4. Compute Engine VMs: `Gcp.Compute`

7. Load Balancers: `Gcp.LoadBalancing`

8. Artifact Registry: `Gcp.ArtifactRegistry`

9. Cloud Storage: `Gcp.Storage`

10. IAM: `Gcp.Iam`

11. CloudRun Services: `Gcp.CloudRun`

Advanced querying: targeting `run` at a subset of nodes

The `Plan` artifact

`forceSkip`

`query show` output sketch

The structural finding: the decoration cannot live on `Op`

L1: `RunAs` + `Invoker` — segmenting the binary calls

The carrier: `Invoker`

`Ref` identity — a correctness requirement, not a nicety