Reconcile Blackwell physical-window allocation with validated layer usage

[zaclanzon]

My RTX 5090 could not drive the following four-display configuration on Ubuntu with overlay layers enabled. The same configuration works in Windows, and disabling overlay layers at Linux driver initialization also made it work. This patch makes it work with overlays enabled.

Display	Connection	Simultaneous target mode
MSI MAG321UX OLED	HDMI	3840 × 2160 at 240 Hz, HDR
Samsung Odyssey G93SC	DisplayPort	5120 × 1440 at 240 Hz, HDR
LG HDR 4K	DisplayPort	3840 × 2160 at 60 Hz, HDR
HYTE Y70ti	DisplayPort	2560 × 682 at 73.778 Hz, SDR

Reproduction

1. Connect all four displays to the RTX 5090 and use the unmodified NVIDIA 610.43.02 driver with overlay layers enabled (nvidia-modeset.enable_overlay_layers=1).
2. In the GNOME Wayland session, request all four modes in the table simultaneously, with HDR enabled on MSI, Samsung, and LG.
3. The unmodified driver rejects the configuration. As a control, disabling overlay layers at driver initialization (enable_overlay_layers=0) allows the same target configuration. With this patch, the target works with overlays enabled.

The patch also preserves transitions back to modes where overlay resources are available; that behavior is exercised separately by the active-overlay test below.

Cause and change

These modes require six hardware tiles. Reserving two layers per tile requires twelve physical windows on a device with eight, even when validated usage marks the extra layers unusable.

The patch allocates physical windows according to validated layer usage and reconciles hardware window ownership with those assignments. It preserves exposed overlay planes and software mappings. Assignment and ownership changes use the head-shutdown path before reprogramming; the flip/core split decision moves after PreUpdate so it includes changes discovered there. Existing usage validation prevents ordinary flips from activating a released layer.

I reached this implementation after testing and rolling back earlier allocation and sequencing changes that produced black screens and Xid 56. Tracking hardware ownership alongside physical assignments led to the working combination. The subsequent review added allocation, sequencing, ownership, and queued-initialization regressions.

Two additional allocator defects are fixed in a separate hardening commit:

• Retry the failed head after reclaiming resources, or return failure; the previous path could report success with an incomplete allocation.
• Use a 64-bit intermediate when constructing the 32-window mask to avoid an undefined shift. This boundary defect did not cause the failure on my eight-window GPU.

Validation

Tested on Ubuntu with NVIDIA 610.43.02, kernel 7.0.0-30-generic, and overlay layers enabled. The production changes apply to that driver version; the PR is based on 610.57.04.

• Full shutdown/power-on loaded the hardened module successfully. All four target modes and PQ HDR on the three HDR monitors matched readback and looked correct. Automatic return to the conservative baseline also matched.
• Two active-overlay runs completed twenty commits each: enable and update overlay content at baseline, release resources for the target modes, then restore baseline overlay operation. Both returned to the maximum desktop without matching Xid, lost-notification, or flip-timeout errors. All four unavailable target-overlay TEST_ONLY probes were rejected in each run.
• MSI overlay visibility was confirmed on the physical monitor. Earlier observations included checkerboards across the other screens, but a complete per-screen visual sequence was not recorded.
• ASan/UBSan tests passed, including 2,359,296 allocator combinations. Comparison with the base preserved 15,598 complete allocations and 199,212 unchanged-head replays. Another 12,482 incomplete upstream results were evaluated separately against the allocation-retry fix. These are software-model results, not equivalent hardware coverage.

The overlay tests exercise release and reacquisition; they do not claim simultaneous active overlays at maximum modes where validation makes those layers unavailable. Broader hardware coverage, extended use, and suspend/resume remain unvalidated. A recurring platform startup issue also prevents claiming warm-reboot reliability.

Tested source: 5077cded3506880f3146c1dd1643efe361da7564 · loaded module build ID: cba99c0c70d11b5ef2097934c433dfc31c65efe1.

The standalone regression harness and run instructions are preserved on a separate fork branch as supporting evidence. This PR changes only the two production source files.

This replaces the earlier, closed upstream PR #1337.

Earlier diagnostic work is preserved in fork PR #1.

Developed and tested through a joint investigation with GPT-6 Astra.

[CLAassistant]

All committers have signed the CLA.