CPU restart does not reproduce the continuous run bit for bit (v8.4.1, 240 km, dt=300); same signature with MYNN and YSU, so core/restart-I/O path

[mordisdevone]

Summary

On a pristine v8.4.1 (91c5eac) CPU build, a run continued from a restart file does not reproduce the uninterrupted run bit for bit. The difference is independent of the physics suite (same signature with MYNN PBL + MYNN sfclayer and with YSU + Monin-Obukhov), so it appears to be in the core / restart I/O path rather than in a scheme. We have no fix; reporting the evidence.

Protocol

240 km x1.10242, config_dt = 300, single MPI rank, NVHPC 25.11, flags -O4 -Mnofma, OPENACC=false. Three legs per configuration: (1) continuous 00:00→01:00 with history and diagnostics at 00:30 and 01:00 and a restart written at 00:30; (2) 00:00→00:30 writing the same restart; (3) restart leg 00:30→01:00 reading it (restart.2014-09-10_00.30.00.nc, restart_timestamp). All legs exit 0; clocks validated from the namelists. Comparison with nccmp -d -s on the 01:00 history and diagnostics files.

Result

Suite	01:00 history bitwise?	first differing variable	differing history vars	differing diagnostics vars
MYNN PBL + MYNN sfclayer	no	qv	45	95
YSU + Monin-Obukhov	no	qv	45	95

The differing-variable lists are identical in content and order for both suites (qv, qc, qr, qi, qs, qg, u, w, pressure, surface_pressure, rho, theta, relhum, divergence, vorticity, ke, uReconstructZonal, uReconstructMeridional, rainnc, precipw, ...; diagnostics starting with mslp, relhum_50hPa, ...). Example first difference on an earlier run of the same protocol: qv : POSITION [0,0,0] : 0.0166578 <> 0.016656.

Same build, same executable, continuous vs continuous (two cold runs) is bitwise identical, so this is not run-to-run nondeterminism.

Notes

• Restart cadence chosen so the restart write aligns with the timestep (dt=300 divides 1800 s; with dt=288 the write is misaligned, which we excluded separately).
• We did not bisect which field or step in the restart write/read introduces the difference; a likely place to look is any state that is diagnosed or recomputed at restart-read rather than restored (e.g. qv derived from scalars/theta_m pathways, or fields not in the restart stream). Happy to run further protocol variants if useful.
• Related but distinct: Fix for GPU-enabled restart runs #1503 (GPU build segfault in RRTMG SW on the restart leg), which we also reproduce on the pristine OpenACC build.