playbooks: run nvidia-smi tasks outside the ssh cgroup so they survive the login GPU guard

[100-JM]

Problem

With slurm_login_on_compute: true, roles/slurm/tasks/login-compute-setup.yml sets DeviceAllow=/dev/nvidiactl on the ssh service so users cannot touch GPUs outside a Slurm job. Ansible connects through that same ssh service, so its own processes land in the restricted cgroup (/system.slice/ssh.service) and become does not change that — the device filter is per cgroup, not per user. Every task that runs nvidia-smi after the guard is applied therefore sees no GPUs:

$ nvidia-smi; echo rc=$?                      # in an ssh session on the node
No devices were found
rc=6

This breaks the second Ansible run that docs/slurm-cluster/slurm-single-node.md asks for ("reboot manually when prompted and then run Ansible again"):

task	effect once the guard is active
playbooks/utilities/gpu-clocks.yml — nvidia-smi -acp UNRESTRICTED	rc=6 → play fails (when allow_user_set_gpu_clocks: yes)
playbooks/nvidia-software/nvidia-cuda.yml — test nvidia-smi	rc=6 → play fails (default slurm_cluster_install_cuda: yes)
playbooks/nvidia-software/nvidia-mig.yml — … | grep -v 'N/A'	rc=0 with No devices were found → treated as a MIG-capable node
roles/nvidia-mig-manager/tasks/main.yml — same probe, same filter	same misdetection inside the role
playbooks/utilities/nvidia-set-gpu-clocks.yml — nvidia-smi -lgc/-rgc	rc=6 → play fails

scripts/validation/validate_slurm.py already documents the phenomenon for the validator and uses srun as the authoritative test; the playbook tasks above were still running plainly over ssh.

Observed on a DGX B300, DGX OS 7.5.0, driver 580.126.20, DeepOps master.

Fix

Run every one of those nvidia-smi invocations through systemd-run --wait --pipe --collect --quiet. The command executes in a transient unit under system.slice, outside the ssh cgroup, so the check still runs for real instead of being skipped. Both MIG probes (playbook and role) now accept only Enabled/Disabled and get an explicit become: true (the rest of that play/role already requires root). Behaviour on nodes without the guard is unchanged.

nvidia-driver.yml carries the same test nvidia-smi; it is deliberately not touched here to avoid conflicting with #1396, and can take the same wrapper once that lands.

Side note: nvidia-smi -acp is deprecated in current drivers ("This option is deprecated … Treating as warning and moving on") and has no effect on Blackwell; a comment records that, the task is kept as-is otherwise.

Verification

On the affected node with the guard active, same ssh session:

$ nvidia-smi -L; echo rc=$?
No devices were found
rc=6
$ sudo systemd-run --wait --pipe --collect --quiet nvidia-smi -L; echo rc=$?
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-364baba8-…)
…
GPU 7: NVIDIA B300 SXM6 AC (UUID: GPU-95bf3df7-…)
rc=0

ansible-playbook --syntax-check passes for the four playbooks; ansible-lint 26.1.1 with the project config reports 0 failures. systemd-run --wait/--pipe/--collect need systemd ≥ 236, which every supported Ubuntu/RHEL release ships.