fix(recovery): require exact progress and preserve blocked stalled work

[rldyourmnd]

Scope

Implements the portable scheduler-recovery portion of the fleet latency/stability audit. This is a code change with regression tests, not a declaration of deployed recovery or throughput acceptance.

• Preserve exact stalled identities when explicit recovery_blockers prevent a manager-wide restart.
• Emit unhealthy for blocked stalled work instead of confusing action suppression with health.
• Require progressed and remaining to form a complete, disjoint partition of the original attempt identities. Reject empty, omitted, foreign, duplicate and ambiguous identities.
• Reject concatenated JSON in command observations/progress.
• Enforce the progress contract in both command adapters and the recovery engine.
• Interrupted attempts may verify progress but cannot replay a restart using stale authorization. Unknown/incomplete proof finishes failed and requires a fresh policy evaluation.
• Add the paired-rollout and acceptance contract in docs/runbooks/recovery-evidence-contract.md.

Verification actually executed

Nine new top-level Go tests, including eleven partition subcases, passed with the race detector. go vet passed for the same focused source set.

The offline environment has Go 1.23.2 and no network dependency resolution. The focused harness compiles the exact changed production files and new tests with only the unchanged Heartbeat type extracted from store.go. It does not execute FileStore, the full repository dependency graph, or the pinned Go toolchain. Full package/repository CI remains required; no live systemd, Incus or GitHub delivery tests were executed.

Deployment ordering

Install this recovery binary before any deployment adapter emitting recovery_blockers; old binaries correctly reject that unknown field. Pause only the recovery timer during the paired binary/adapter/config replacement. Do not stop running workers. Preserve recovery history and take checkpoints.

The matching deployment adapter must provide positive per-identity progression, an attempt-bound immutable checkpoint, and a last-moment restart eligibility check. That check is not an atomic admission fence; admission-fenced or scale-set-local repair remains separate work.

Remaining acceptance

• Full pinned-toolchain CI and integration against the real deployment adapter.
• Independent review of interrupted recovery/cooldown behavior.
• Canary: affected exact job advances, no sibling jobs are lost, and queue/workflow latency is measured on matched cohorts.

No merge, deployment, fleet restart, runner deletion, or runtime-identity record update is included.

[rldyourmnd]

Verification update: full repository CI passed

Head: b3c587eadd3516931112c913736a2e984fcf4016.
The Race job checked out synthetic merge cfc8c5e7bfa92d45564e8af3189b39fb1889175a against base 6030feb7a29337ff21d8e8b2643f84dc11b865ef.

CI run 34066329613 completed successfully. The job list and actual Race log were inspected:

• Static/build: gofmt, go vet ./..., go build -trimpath ./... passed.
• Test: module download/verification and go test ./... passed.
• Race: Go 1.26.7, GOTOOLCHAIN=local, go test -race ./... passed, including internal/schedulerrecovery.
• Derivative-input classification and final Gate passed.
• Reproducible provider/GARM derivative jobs were skipped, not executed.
• CodeQL run 34066329986 completed successfully.

This supersedes the initial PR description's pending full-toolchain verification. The initial offline limitation remains an accurate description of local execution, not the current GitHub CI status.

No merge, runtime installation, paired-adapter deployment or fleet latency acceptance was performed. These checks ran on GitHub-hosted workers and do not establish private-fleet production health.