HDDS-16084. Add ErrorProne to Ozone CI by ivandika3 · Pull Request #11056 · apache/ozone

ivandika3 · 2026-08-19T01:44:30Z

What changes were proposed in this pull request?

ErrorProne is a well-known static analysis tool used in other ASF projects such as Celeborn, Druid, HBase, Solr, and Beam to catch common Java coding errors. It is lightweight compared with fbinfer (HDDS-15560).

This patch adds an ErrorProne CI check that fails when it finds any ERROR-severity bug patterns (https://errorprone.info/bugpatterns). All ErrorProne diagnostics are retained in diagnostics.txt; summary.txt contains only ERROR diagnostics or an explicit fallback for a non-diagnostic Maven failure, which is what the CI summary presents.

This patch also fixes all current ERROR-severity bug patterns in Ozone. The fixes are split into multiple commits. I have attached errorprone-error-fixes.md for the bug patterns and the possible risks. This caught real defects in CompactionDag and no-op assertions.

Behavior fixes included

CompactionDag pruning now removes pruned SST entries from compactionNodeMap; previously Map#remove received a node instead of its file-name key and therefore did nothing.
The valueOf(name()) conversions remove ordinal coupling. They are behavior-preserving for the currently matching protocol/error-code names; parity tests guard future enum additions, with the SCM security protocol’s two explicit historical exceptions covered separately.

WARNING diagnostics are intentionally not fixed in this patch; they can be handled by follow-up work. The ErrorProne ThreadSafe annotation is also deferred so it can be coordinated with fbinfer HDDS-15560.

Note for reviewer:

The possible risk is at the ProtobufBufferOrdinal rule (https://errorprone.info/bugpattern/ProtocolBufferOrdinal), but the current ResultCodes.valueOf(protoStatus.name()) mapping is covered by enum name-parity tests.
If we merge this, ongoing PRs may need to merge master because ErrorProne can reject existing code in their commits.

Generated by: GPT 5.6 Sol

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16084

How was this patch tested?

CI (Clean CI: https://github.com/ivandika3/ozone/actions/runs/32216855338).

rich7420

The ErrorProne gate looks well-built and CI is green. One thing: a few "cleanup" fixes are actually behavior changes / real bug fixes bundled in a lint PR worth a test and a description note. Two comments beloww

rich7420 · 2026-08-20T12:14:55Z

-              omKeyLocationInfoGroup.getLocationList()
-                  .stream().map(omKeyLocationInfo -> pipelines.add(
-                      omKeyLocationInfo.getPipeline())));
+      omKeyInfo.getKeyLocationVersions().forEach(omKeyLocationInfoGroup ->


Not a test-lint fix — getPipelines() returned empty before (the old stream().map(...add...) was never consumed), so this actually changes what Recon reports. Please add a test for a non-empty pipeline list and call the behavior fix out in the description instead of under "Fix tests".

Thanks for the review, added a test for this.

Added the non-empty pipeline regression test and updated the PR description under “Behavior fixes included” to explicitly call out the changed /api/v1/containers payload.

rich7420 · 2026-08-20T12:14:55Z

    if (resp.getStatus() != OK) {
      throw new OMException(resp.getMessage(),
-          ResultCodes.values()[resp.getStatus().ordinal()]);
+          ResultCodes.valueOf(resp.getStatus().name()));


valueOf(name()) is the right pattern, but this changes error-code translation for any status where the Status and ResultCodes ordinals differ. Is this an intended behavior fix? If so, a small test (a status past the first divergence) + a note in the description would help.

error-code translation for any status where the Status and ResultCodes ordinals differ

I'm not sure I follow this, this patch does not use Status ordinal.

Verified this is behavior-preserving for the current enums: TestResultCodes asserts equal sizes, name parity, and valueOf(name()) conversion for every value. I also clarified the removal of ordinal coupling in the PR description.

rich7420 · 2026-08-20T12:15:17Z

@ivandika3 thanks for the patch!

smengcl

Thanks @ivandika3 for the addition.

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>

# Conflicts: # hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnExceptions.java # hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerMetadataManagerImpl.java

# Conflicts: # hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnExceptions.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestOMClientRequestWithUserInfo.java

peterxcli

Nice work — adding Error Prone is clearly worth it, and the ERROR-level fixes catch several genuine bugs (CompactionDag, ReconContainerMetadataManagerImpl, the dropped includeNonStandardContainers row in ContainerBalancerConfiguration.toString(), and a good number of no-op assertions).

I went through the whole diff. Things I verified and am happy with:

The -Perrorprone profile really does apply to default-compile as well as default-testCompile — I ran mvn -Perrorprone -X -pl hadoop-hdds/config -am test-compile locally and confirmed combine.children="append" merges the Error Prone args with the existing -AartifactId=${project.artifactId} arg rather than replacing it. Error Prone diagnostics are emitted for both main and test sources.
SCMException.ResultCodes and OMException.ResultCodes: the values()[ordinal()] -> valueOf(name()) conversions are safe, because TestSCMExceptionResultCodes and TestResultCodes both assert equal enum length and exact name equality. SCMSecretKeyException.ErrorCode also matches the proto exactly. On your ProtocolBufferOrdinal note in the description — agreed, those three are fine; the fourth one is the odd one out (inline comment below).
SCM_ROOT_CA_PREFIX is SCM_ROOT_CA + "@" with no format specifier, so dropping String.format(...) is a no-op and now matches HASecurityUtils#initializeSecurity (SCM_ROOT_CA_PREFIX + hostname).
OmRangerSyncArgs.newServiceVersion is a primitive long, so the removed Objects.requireNonNull could never fire.
DelegatingProperties#contains -> containsValue: Hashtable#contains already has value semantics, so this is behaviour-preserving and just removes the confusion.
BucketEndpoint / VolumeEndpoint: both keep the @Inject constructor that assigns the field, so removing the redundant field annotation is safe.
ContainerBalancerConfiguration#toString: I counted — 17 %-50s %X%n rows = 34 conversions against 34 arguments after the patch. Correct.
TestECBlockInputStream: 5000 * ONEMB overflowed int to ~904 MB; 5000L restores the intended "very large block", and hasSufficientLocations() still holds.
TestReconContainerManager: State.{UNHEALTHY,INVALID,DELETED}.getNumber() = 5/6/7 vs ordinals 4/5/6, so the container IDs shift to 125/126/127 — no collision with the other IDs used in that class.
The errorprone check is wired correctly (#checks:basic marker, check_needs_bats picks up .bats$), and the check took ~10.5 min in your run, comfortably inside the 30-minute job timeout.

Seven things I'd like your thoughts on, left as inline comments. Only the first is something I'd consider blocking; the rest are follow-ups or nits.

errorprone.sh bypasses _post_process.sh, so a Maven failure that isn't an Error Prone ERROR reports failures=0 with a misleading summary.
The CompactionDag prune fix is a real behaviour change with no regression test.
SCMSecurityProtocolProtos.Status -> SCMSecurityException.ErrorCode is the one name mapping with no parity test, and it already has a name with no counterpart.
TestHSync now asserts hasSize() on a table scanned across the whole shared MiniOzoneCluster.
The ReconContainerMetadataManagerImpl fix changes a public Recon REST payload — worth a release note.
TestRocksDiffUtils drops a map entry rather than de-duplicating it.
Two @SuppressWarnings without a justification comment.

On your merge-order note: agreed, this will need master merged into in-flight PRs, and it might be worth landing the CI plumbing and the ERROR fixes as separate commits on master so a revert of one doesn't drag the other along.

peterxcli

@ivandika3 Thanks for the update, other LGTM.

rich7420 · 2026-09-04T14:17:56Z

+    ReconOMMetadataManager omMetadataManager = mock(ReconOMMetadataManager.class);
+    Table<String, OmKeyInfo> keyTable = mock(Table.class);
+    when(omMetadataManager.getKeyTable(BucketLayout.LEGACY)).thenReturn(keyTable);
+    when(keyTable.getSkipCache(key1)).thenReturn(new OmKeyInfo.Builder()


getSkipCache() reads persisted OM rows, whose OmKeyInfo codec omits pipeline data via getProtobuf(true, ...). This mock bypasses that codec, so current OM rows still produce an empty pipeline list. Please test through the real table and source the pipeline from data that is actually retained.

Good catch. The mocked getSkipCache() bypassed OM metadata serialization. Persisted OmKeyInfo omits pipeline information, so a real table cannot supply pipelines on this path. I reverted the pipeline population change and its mock-only test rather than asserting behavior that does not exist in production.

thanks for the update

rich7420 · 2026-09-04T14:17:56Z

+DIAGNOSTIC_FILE="$REPORT_DIR/diagnostics.txt"
+OUTPUT_LOG=$(mktemp)
+
+MAVEN_OPTIONS='-B -fae --no-transfer-progress -Perrorprone -DskipDocs -DskipRecon -DskipShade'


Please cover ozonefs-hadoop2 and ozonefs-hadoop3: -DskipShade removes both from this reactor, so this check still passes with an Error Prone ERROR in those sources. Hadoop2 copied sources also match the global -XepExcludedPaths exclusion.

Addressed. The check no longer sets -DskipShade, so the build-with-ozonefs profile includes both OzoneFS compatibility modules. The excluded-path regex now keeps ozonefs-hadoop2/target/generated-sources/java in scope while continuing to exclude other generated sources.

thanks for the update

rich7420 · 2026-09-07T04:48:50Z

 - bats: [Checks](../hadoop-ozone/dev-support/checks/bats.sh) bash scripts, (using the [Bash Automated Testing System](https://github.com/bats-core/bats-core#bats-core-bash-automated-testing-system-2018))
 - checkstyle: [Runs](../hadoop-ozone/dev-support/checks/checkstyle.sh) 'mvn checkstyle' plugin to confirm Java source abides by Ozone coding conventions
 - docs: [Builds](../hadoop-ozone/dev-support/checks/docs.sh) website with [Hugo](https://gohugo.io/)
+- errorprone: [Runs](../hadoop-ozone/dev-support/checks/errorprone.sh) Error Prone static analysis during Java compilation. The check reports all warning and error diagnostics in `summary.txt` and fails for errors.


nit: please update this to match errorprone.sh: all diagnostics go to diagnostics.txt, while summary.txt contains only ERROR/fallback.

ivandika3 added 9 commits August 18, 2026 22:42

HDDS-16084. Add Error Prone CI check

b7a93ca

HDDS-16084. Fix Error Prone errors in HDDS client and common

2ac75a1

HDDS-16084. Fix Error Prone errors in HDDS server modules

136355b

HDDS-16084. Fix Error Prone errors in checkpoint differ

9e0afd5

HDDS-16084. Fix Error Prone errors in Ozone client and common

facc7cf

HDDS-16084. Fix Error Prone errors in Ozone Manager

38d568b

HDDS-16084. Fix Error Prone errors in Recon

e6d888d

HDDS-16084. Fix Error Prone errors in integration tests

638b8bf

Fix PMD

5351fd1

ivandika3 requested a review from Tejaskriya August 19, 2026 01:54

ivandika3 changed the title ~~HDDS-16084. Add ErrorProne into Ozone verification pipeline~~ HDDS-16084. Add ErrorProne into Ozone CI Aug 19, 2026

HDDS-16084. Fix tests exposed by Error Prone

7fe4d5b

ivandika3 changed the title ~~HDDS-16084. Add ErrorProne into Ozone CI~~ HDDS-16084. Add ErrorProne to Ozone CI Aug 19, 2026

ivandika3 added test build Pull request that modifies the build process labels Aug 19, 2026

ivandika3 requested a review from smengcl August 19, 2026 09:44

ivandika3 self-assigned this Aug 19, 2026

ivandika3 requested a review from adoroszlai August 19, 2026 10:24

ivandika3 marked this pull request as ready for review August 19, 2026 16:06

rich7420 reviewed Aug 20, 2026

View reviewed changes

smengcl reviewed Aug 21, 2026

View reviewed changes

Comment thread hadoop-ozone/dev-support/checks/errorprone.sh Outdated

Comment thread hadoop-ozone/dev-support/checks/errorprone.sh Outdated

ivandika3 and others added 6 commits August 24, 2026 18:20

Add clean to mvn test-compile

8b3f56e

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>

Merge branch 'master' into HDDS-16084

5ca184b

# Conflicts: # hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnExceptions.java # hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerMetadataManagerImpl.java

Make compiler to output US-en

942dc9b

Add Recon pipeline regression test

e2fa77f

HDDS-16084. Capture all Error Prone diagnostics

bb1924d

Fix checkstyle

de4b9e3

ivandika3 mentioned this pull request Aug 26, 2026

fix(ozone): add isTrue() to bare assertThat in testCaching #11119

Closed

Empty commit

0ac149b

ivandika3 added 2 commits August 28, 2026 18:15

Merge remote-tracking branch 'origin/master'

04eaed5

Merge branch 'master' into HDDS-16084

f5c4a85

# Conflicts: # hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnExceptions.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestOMClientRequestWithUserInfo.java

peterxcli reviewed Aug 28, 2026

View reviewed changes

ivandika3 added 2 commits September 3, 2026 12:04

HDDS-16084. Address ErrorProne review feedback

049c639

Fix bats

8b8d0ae

peterxcli approved these changes Sep 3, 2026

View reviewed changes

Comment thread hadoop-ozone/dev-support/checks/errorprone.sh Outdated

Comment thread hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestHSync.java Outdated

ivandika3 added 2 commits September 4, 2026 09:21

Address reviews

8a50d00

Merge branch 'master' into HDDS-16084

b4d9f49

rich7420 reviewed Sep 4, 2026

View reviewed changes

ivandika3 added 2 commits September 7, 2026 09:44

HDDS-16084. Address review comments

18dfcef

HDDS-16084. Fix ErrorProne CI failures

f2081f0

rich7420 reviewed Sep 7, 2026

View reviewed changes

Update ci.md

9546876

Uh oh!

Conversation

ivandika3 commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Behavior fixes included

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

rich7420 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rich7420 commented Aug 20, 2026

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

peterxcli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

peterxcli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ivandika3 commented Aug 19, 2026 •

edited

Loading

rich7420 left a comment •

edited

Loading