JobCompleted does not clean up a live EphemeralRunner when the runner process fails to exit #4603

Checks

• I read the ARC troubleshooting guide; this failure mode is not covered.
• I use the official gha-runner-scale-set-controller and gha-runner-scale-set charts.
• This is a bug report, not a support question.
• I reviewed the changelog.

Controller version

0.14.2 (the same no-op HandleJobCompleted behavior is present on current master at a035c5a.)

Deployment method

Helm

Reproduction / observed behavior

1. An ephemeral runner accepted job GUID 85cb98eb-2919-5766-872c-cc997f618c1f from workflow run 31753891150.
2. GitHub made the job terminal (failure) at 2026-08-14T00:30:37Z.
3. The runner process did not exit. Its Pod and EphemeralRunner (runnerId: 2402) stayed Running until an external reconciler deleted them at 2026-08-14T07:56:21Z—more than seven hours later.
4. The occupied runner counted toward the scale set replica target, so the capacity slot could not serve another job.

The listener did receive and persist the matching JobStarted metadata. ARC controller logs continued to report Runner container is still running; there was no Pod restart, node pressure, or OOM for this runner.

The trigger appears to have been an exceptional runner-process hang, not a generally broken runtime: a controlled cancellation using the same ARC 0.14.2 controller, official runner 2.336.0 entrypoint (/home/runner/run.sh), image, Pod template, and Kata Firecracker RuntimeClass completed cleanup normally. The cancellation reached the runner at 08:31:36Z; runner cleanup completed, ARC observed exit 0 at 08:31:42Z, and removed registration/Pod/ER by 08:31:43Z.

The ARC robustness gap is independent of that trigger: HandleJobCompleted currently only sets dirty = true and discards the terminal event. If an ephemeral runner ever fails to exit, ARC has no terminal-job fallback and can lose a capacity slot indefinitely.

A deterministic RED test that reproduces the discarded terminal event is here:
robinbraemer@ec1febe

Expected behavior

Normal behavior should remain runner-driven: the ephemeral runner exits after its one job and the controller cleans it up.

As a fallback, when ARC receives JobCompleted, it should associate the event with the exact EphemeralRunner using runner ID, runner name, job ID, and workflow-run ID. If the runner remains alive after a short grace period, ARC should issue deletion through its existing finalizer path. Duplicate, late, mismatched, and already-deleted events must be harmless.

Additional context

• Runtime: Kata Containers with Firecracker; restartPolicy: Never; terminationGracePeriodSeconds: 30.
• Runner command: /home/runner/run.sh (no custom PID-1 or signal wrapper).
• A successful controlled cancel shows that the local Firecracker setup normally delivers the cancellation and allows runner exit.
• The original runner logs were gone after emergency cleanup, so the reason that one runner process failed to exit is not proven. This issue is specifically about ARC lacking bounded self-healing once GitHub already knows the job is terminal.
• Related but distinct from Runner stuck in "Running" state after job failure, jobs are stuck in Queued #4203, which addressed an EphemeralRunner remaining after its Pod had already exited. Here the Pod and runner container remained live.

Relevant controller/listener evidence

00:17:47 listener: Updating job info for runner ... runnerId=2402 jobId=85cb... workflowRunId=31753891150
00:30:37 GitHub job terminal: failure
07:52:10 controller: Runner container is still running
07:56:21 external safety reconciler: deleted verified stale EphemeralRunner
07:56:23 controller finalizers removed runner registration, Pod, and JIT secret

I have a tested patch ready and will open it referencing this issue.