HDDS-16273. OM bootstrap never completes when a single SST file is larger than ozone.om.ratis.snapshot.max.total.sst.size

[Eason09053360]

What changes were proposed in this pull request?

OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir() checks the per-request budget
(ozone.om.ratis.snapshot.max.total.sst.size) before recording a file. When a single SST file is larger than
the whole budget it is the first non-excluded file of every request, so the leader answers with an empty
tarball each time. The follower's exclude list never changes, and RDBSnapshotProvider.downloadDBSnapshotFromLeader()
loops forever while holding the OzoneManager monitor: no error, no bound, no progress. The comparison was also
<= 0, so a file exactly equal to the remaining budget was treated as over budget.

Leader side: when a file does not fit and nothing has been collected for the request yet, record it anyway and
end the part, so every request transfers at least one file. The budget check is now < 0, so an exact fit is
not over budget. The v1 OMDBCheckpointServlet records before checking and was never affected.

Follower side: RDBSnapshotProvider counts consecutive parts that bring no new file into the candidate dir and
aborts with a diagnosable IOException after two instead of looping forever. One no-progress part is tolerated
for transient causes such as a truncated exclude list. The part : N log is now 1-based.

The ozone-default.xml description now states that an oversized file is still sent.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16273

How was this patch tested?

• New TestOMDbCheckpointServletInodeBasedXfer#testCollectFilesFromDirTransfersFileLargerThanMaxSize: two
  files larger than the budget are sent one per request, the limit is enforced again once the tarball holds a
  file, and a file exactly filling the budget is accepted.
• testTarballBatching assertion changed to <= to match the exact-fit semantics; both parameterizations pass.
• New TestRDBSnapshotProvider#testDownloadAbortsWhenPartsBringNoNewFile: the first empty part is tolerated,
  the second aborts with the new message.
• checkstyle.sh — 0 violations.

Generated-by: Claude Code (Claude Fable 5.1)

[Copilot]

🟢 Approval recommended

The functional fix is well-scoped, matches the PR description, and is covered by new targeted tests; remaining feedback is non-blocking.

Pull request overview

Fixes an OM HA bootstrap livelock where incremental checkpoint download never completes if a single SST exceeds ozone.om.ratis.snapshot.max.total.sst.size, by ensuring leader transfers always make progress and follower transfers abort instead of looping forever.

Changes:

• Leader: allow a single oversized SST to be transferred alone (and treat exact-fit as in-budget).
• Follower: detect consecutive no-progress parts during snapshot download and fail fast with a diagnosable IOException.
• Tests/docs: add targeted regression tests and clarify the config description in ozone-default.xml.

File summaries

File	Description
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java	Ensures oversized SSTs are still sent (one per part) and fixes exact-fit budget semantics.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java	Adds coverage for oversized-file transfer progress and updates batching assertion for exact-fit behavior.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/RDBSnapshotProvider.java	Adds “no progress” part detection to prevent infinite looping during snapshot download.
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestRDBSnapshotProvider.java	Adds regression test validating abort after consecutive no-progress parts.
hadoop-hdds/common/src/main/resources/ozone-default.xml	Documents that an oversized SST is still transferred to ensure progress.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.