Various follow-ups to milestone 4c#67
Conversation
After the daemon issues a reboot, it now returns early from Reconcile() on subsequent invocations. This prevents the daemon from running bootc status on a node that is mid-shutdown, which could return garbage data. This also makes the Rebooting state durable (it persists until the node actually goes down), so the e2e test can now assert on it as an intermediate lifecycle step. This also matches what the MCO does. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
The `reset()` method only needs to clear the backoff retry counter, so it belongs on `stageOp` rather than `BootcNodeReconciler`. Clearing `rebootIssued` is unnecessary in production; it doesn't make sense to "un-issue" a reboot. The way it gets "reset" is by the daemon naturally getting restarted as part of the reboot. That said, add a `resetDaemon()` test helper to make resetting everything easier. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
After acquiring runMu, check ctx.Err() before calling Stage(). This prevents stale goroutines from briefly starting a bootc switch process with the wrong image after rapid spec changes (e.g. B→C→D). Multiple goroutines may still pile up on the mutex, but each cancelled one exits immediately without spawning a process. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
exec.CommandContext already sends SIGKILL on context cancellation, and bootc has no SIGINT handler, so graceful vs. hard termination is equivalent. Replace the TODO with a brief note about the existing behavior. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
Run bootc switch as a transient systemd unit on the host rather than directly inside the daemon pod's cgroup. It's just cleaner to structure it this way (e.g. we don't have to account for bootc memory requirements in our own, and also the operation transparently survives daemon pod restarts), and will also allow in the future using various systemd knobs, such as those around IO scheduling (see #61). Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
This reverts commit 7c5e31c. The memory bump to 512Mi is no longer needed: bootc switch now runs as a transient systemd unit on the host (outside the pod cgroup), so its memory usage does not count against the daemon pod limit. 128Mi is sufficient for the daemon itself (Go binary, controller-runtime, API watches). Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
jlebon
force-pushed
the
daemon-stage-follow-ups
branch
from
67475de to
53725cb
Compare
| // TODO: use --download-only once available (https://github.com/bootc-dev/bootc/issues/2137) | ||
| cmd := e.nsenterCmd(ctx, "bootc", "switch", image) | ||
| // Stop any stale unit from a previous daemon incarnation. | ||
| e.stopStageUnit() |
There was a problem hiding this comment.
What if there is still an in-progress staging operation functionally working? In this case, we stop it and recreate it right?
Don't we need simply need to re-attach to it or adopting it?
AFAIU, we wanted to run the staging operation with systemd if the daemon was restarted, e.g for upgrades, and not to stop the operation.
There was a problem hiding this comment.
I started looking at what the "adopt" path would look like when the daemon restarts, and it works but it's non-trivial. Hesitating paying for that complexity for an event that should be somewhat rare, when the current behaviour will still eventually give convergence (even if less efficiently than it could be).
This patch still at least buys us decoupling from bootc memory requirements and e.g. eventually doing #61, etc... which I see as more pressing than surviving restarts.
|
The discussion about launching the switch operation via systemd is here. Maybe we should continue it in this PR or create na ad-hoc issue |
|
From the offline discussion, we decided to relay on systemd-run and the memory consumed by the |
|
I was thinking about the idea we discussed of not needing a goroutine at all and thinking of the systemd-run as our goroutine (+ adoption, etc...), but an obvious thing I missed there is that we still need to re-reconcile once the switch is done. And the cleanest way to do that is still through a goroutine + GenericEvent rather than e.g. periodic requeues, etc. |
2 participants
These are most of the things that came up as follow-ups as part of #50.