pdp: re-fetch receipt when piece add extraction finds no logs

[TippyFlitsUK]

Priority

Not urgent. This is not a v1.28.5 regression, it has no observed proving impact, and it is unrelated to the instability behind the current release. It predates that work and can wait behind it. Raising it now only because the diagnosis is complete while it is fresh.

What happens

eth_getTransactionReceipt sometimes returns a successful receipt carrying no logs for a transaction that did emit events. watch_eth.go marshals whatever it gets and stores it, and nothing re-checks. PiecesFromReceipt then fails on every watcher pass, the row keeps pieces_added = FALSE, and the same transactions are retried indefinitely while their pieces stay absent from local state.

Two shapes, both confirmed against chain state via EventsRoot and ChainGetEvents:

• The receipt comes back under the queried hash with logs missing, and the same query returns all of them shortly after.
• The receipt comes back keyed to a different hash for the same message, same nonce and same block, and the logs are never populated under the hash that was queried. Re-querying with the returned hash returns them.

Numbers

On my mainnet node this stranded 10 transactions and 220 rows across two data sets, and the count grew through the day. Every one of them had its 27 events present on chain.

Two SPs shipping logs show the symptom in an eight day window: mine at 244,276 warnings and Mongo2Stor Mainnet at 24,253,337. Those are retry-loop warnings rather than distinct affected transactions, so they indicate how long rows sit rather than how many there are. I cannot see the stranded count on the other node.

The change

On extraction failure, re-fetch the receipt via the eth client and follow the returned TxHash when it differs from the one queried, then retry extraction. The corrected receipt is written back to message_waits_eth so the fetch is not repeated on later passes. If the re-fetch itself fails, the original error is returned and behaviour is unchanged.

processDataSetPieceAdd already received an ethClient it did not use, so no new plumbing was needed.

One or two RPC calls per stuck transaction, and no contract calls.

Verification

Running on a mainnet node since 15:01 UTC on 28 August. All 220 stranded rows materialised, including the three that required following the returned hash. Zero duplicate or overlapping piece_id values afterwards, and the Failed to process piece add loop stopped.