Suspend deadlock on Turing: nv_drm_mmap spins in kgmmuInvalidateTlb after BAR2 teardown; fbsr restore then fails on resume

[silicon905]

NVIDIA Open GPU Kernel Modules Version

610.57.04 (open kernel modules, locally built via DKMS from the distro nvidia-open-dkms package).

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

EndeavourOS (Arch-based, rolling release).

Kernel Release

Linux 6.18.48-1-lts #1 SMP PREEMPT_DYNAMIC (stable Arch LTS package, not self-built), x86_64. (Hostname redacted from the uname output.)

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA T1200 Laptop GPU (Turing, TU117) — UUID redacted. PCI 01:00.0. Single internal eDP panel, no external displays. Laptop firmware is in "Discrete Graphics" mode: no iGPU present, the dGPU drives the panel directly. PRIME / render offload is not in use.

Describe the bug

During system suspend, a userspace client that is inside nv_drm_mmap() gets
stuck in an unbounded register-read loop. The BAR2 CPU mapping appears to be torn
down by the suspend path while the client is still walking the GMMU, so the
pending TLB invalidation can never be acknowledged. The thread spins at 100%
system time while holding an rw-semaphore, which blocks nvidia-modeset and
systemd-sleep.

That corrupts the framebuffer save, so on resume memmgrRestorePowerMgmtState()
fails and the display pipeline never returns (Flip event timeout on head 0,
black screen), with nvidia-modeset blocked indefinitely until a hard power
cycle.

Related issues in the same __nv_drm_gem_nvkms_map / mapping_reuse.c code
region, but with different triggers (not suspend-specific): #1132 and #1134
(mapping exceeds a PCI BAR under steady-state/long-uptime load), and #945
(resume-side memmgrRestorePowerMgmtState black screen on Blackwell + iGPU).
This report is distinct in being suspend-triggered (BAR2 teardown mid-mmap),
in the soft-lockup spinning thread, and in the fbsr_gm107.c save corruption —
fbsrEnd_HAL currently has no other hits in this tracker.

Spinning thread (from the soft-lockup backtrace):

RIP: 0010:_regRead.isra.0+0x13/0x170 [nvidia]
kgmmuCheckPendingInvalidates_TU102+0x93/0xf0  [nvidia]
kgmmuInvalidateTlb_GM107+0x179/0x4d0          [nvidia]
gvaspaceAlloc_IMPL / dmaAllocMapping_GM107    [nvidia]
_kbusMapAperture_GM107 / _kbusInternalBar1Map [nvidia]
reusemappingdbMap / kbusMapFbAperture_GM107   [nvidia]
memMap_IMPL / serverMap                       [nvidia]
rmapiMapToCpuWithSecInfoV2 / _nv04MapMemory   [nvidia]
nvkms_call_rm / nvRmApiMapMemory / MapMemory  [nvidia_modeset]
__nv_drm_gem_nvkms_map / __nv_drm_gem_nvkms_mmap / nv_drm_mmap [nvidia_drm]
drm_gem_prime_mmap

BAR2 mapping gone mid-walk:

NVRM: Assertion failed:
  (pKernelBus->pReadToFlush != NULL ||
   pKernelBus->virtualBar2[GPU_GFID_PF].pCpuMapping != NULL)
  @ kern_bus_gv100.c:388
NVRM: kgmmuInvalidateTlb_GM107: TLB invalidation failed waiting for prior
      invalidate (status=0x00000065)
NVRM: dmaAllocMapping_GM107: can't alloc VA space for mapping.
NVRM: ... [NV_ERR_INVALID_STATE] (0x00000040)
      returned from reusemappingdbMap(...) @ kern_bus_gm107.c:3152

Lock chain reported by the kernel (PIDs are illustrative):

nvidia-modeset  <reader> blocked on rw-semaphore owned by systemd-sleep
systemd-sleep   <writer> blocked on rw-semaphore owned by Media (spinning)
watchdog: BUG: soft lockup - CPU#N stuck for 26s! [Media:<tid>]  (100% system)

Resume failure:

NVRM: Assertion failed: 0 @ fbsr_gm107.c:1053
NVRM: [NV_ERR_GENERIC] from fbsrEnd_HAL @ mem_mgr_pwr_mgmt.c:270
NVRM: [NV_ERR_GENERIC] from memmgrRestorePowerMgmtState @ kern_mem_sys.c:1192
NVRM: [NV_ERR_GENERIC] from gpuStateLoad(... GPU_STATE_FLAGS_PRESERVING)
      @ gpu_suspend.c:281
[drm:nv_drm_atomic_commit] *ERROR* Flip event timeout on head 0

A relevant behavioural detail: the outcome depends on
NVreg_UseKernelSuspendNotifiers. The hang occurs either way, but recovery
differs consistently (4 occurrences, cleanly split):

UseKernelSuspendNotifiers	sleep services	outcome
1 (kernel notifiers)	disabled	self-recovers after ~3.5 min
0 (classic suspend.sh)	enabled	never recovers, hard power cycle

This suggests the ordering of the BAR2 teardown relative to freezing userspace
matters, and that the classic path leaves the framebuffer save in a state the
restore cannot handle at all.

To Reproduce

• Preconditions: single Turing dGPU driving the panel directly (discrete-graphics
  firmware mode, no iGPU), Wayland (KWin), driver 610.57.04 open modules.
• Run a Chromium-based application that keeps an active Media worker thread
  (reproduced with an Electron app, and separately with a Qt WebEngine app).
  Playing video raises the hit rate.
• Suspend the system (systemctl suspend or lid close) while such a client is
  active at the moment the display blanks.
• Confirmed the culprit by snapshotting all threads named Media immediately
  before suspend (a systemd system-sleep pre-hook) and matching the TID against
  the one named in the soft-lockup watchdog line.

Bug Incidence

Sometimes

nvidia-bug-report.log.gz

Not attached. I would prefer to provide targeted excerpts rather than the full
archive, which collects a large amount of unrelated system data. Please tell me
exactly which sections you need and I will supply them. (Noting the field is
marked required — happy to discuss what is genuinely necessary for triage.)

More Info

Expected behaviour: the suspend path should quiesce or reject new RM client
mapping operations before tearing down the BAR2 CPU mapping, so an in-flight
nv_drm_mmap() cannot end up spinning on a TLB invalidation that can no longer
be acknowledged. Failing that, kgmmuCheckPendingInvalidates_* should bail out
when the BAR2 mapping is absent instead of spinning until the soft-lockup
watchdog fires.

[Henaneitor]

I hit what appears to be the same suspend-side deadlock on Blackwell, so this does not seem to be Turing-specific.

System

• Arch Linux (Omarchy), Wayland / Hyprland 0.56.2
• Linux 7.1.9-arch1-2 (stable Arch kernel)
• NVIDIA GeForce RTX 5070 Ti, GB203, PCI ID 10de:2c05 rev a1; this is the only VGA/3D adapter enumerated
• nvidia-open-dkms 610.57.04-1, nvidia-utils 610.57.04-1
• Suspend mode: deep (S3)
• NVreg_UseKernelSuspendNotifiers=1
• NVreg_PreserveVideoMemoryAllocations=1
• nvidia-suspend.service / nvidia-resume.service disabled

A Brave/Chromium process tree was active. The task that refused to freeze was its Media thread. The same boot had eight earlier PM: suspend exit markers before this final failure.

Failure sequence

22:03:58.822887 kernel: PM: suspend entry (deep)
22:04:00.955906 kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (pKernelBus->pReadToFlush != NULL || pKernelBus->virtualBar2[GPU_GFID_PF].pCpuMapping != NULL) @ kern_bus_gv100.c:388
22:04:20.975945 kernel: Freezing user space processes failed after 20.001 seconds (1 tasks refusing to freeze, wq_busy=0):
22:04:20.976452 kernel: task:Media state:R running task pid:549683 tgid:547762 ppid:529365

The refusing task's stack matches the report:

os_get_monotonic_time_ns [nvidia]
_checkTimeout [nvidia]
_regRead.isra.0 [nvidia]
timeoutCheck [nvidia]
kgmmuCheckPendingInvalidates_TU102 [nvidia]
kgmmuInvalidateTlb_GM107 [nvidia]
gvaspaceInvalidateTlb_IMPL [nvidia]
gvaspaceAlloc_IMPL [nvidia]
dmaAllocMapping_GM107 [nvidia]
_kbusMapAperture_GM107 [nvidia]
_kbusInternalBar1Map [nvidia]
reusemappingdbMap [nvidia]
kbusMapFbAperture_GM107 [nvidia]
memMap_IMPL [nvidia]
serverMap [nvidia]
rmapiMapToCpuWithSecInfoV2 [nvidia]
nvRmApiMapMemory [nvidia_modeset]
MapMemory [nvidia_modeset]
__nv_drm_gem_nvkms_map [nvidia_drm]
__nv_drm_gem_nvkms_mmap [nvidia_drm]
nv_drm_mmap [nvidia_drm]
drm_gem_prime_mmap

Ten seconds after the freezer aborted, the same TLB/VA-space failure appeared:

22:04:30.956911 kernel: NVRM: kgmmuInvalidateTlb_GM107: TLB invalidation failed waiting for prior invalidate (status=0x00000065), vaspaceFlags 0x4081081, scope 0x2, GFID 0
22:04:30.957049 kernel: NVRM: ... Invalid state [NV_ERR_INVALID_STATE] (0x00000040) ... mmuWalkProcessPdes(...) @ mmu_walk_map.c:142
22:04:30.957083 kernel: NVRM: mmuWalkMap: Failed to map VA Range 0x7400000 to 0x79fffff. Status = 0x00000040
22:04:30.957204 kernel: NVRM: mmuWalkUnmap: Failed to unmap VA Range 0x7400000 to 0x79fffff. Status = 0x00000040
22:04:30.957303 kernel: NVRM: ... (pKernelBus->pReadToFlush != NULL || pKernelBus->virtualBar2[GPU_GFID_PF].pCpuMapping != NULL) @ kern_bus_gv100.c:388

The kernel aborted suspend and restarted tasks, but the display did not recover. The machine continued producing journal entries until I forced a hardware reset. There was no OOM kill or system-memory pressure. I have not tested the proprietary module.

Disclosure: the quoted data is direct output from my machine; I used an AI assistant to correlate the journal timeline and format this comment.