test(fixtures): a shrunken fixture passed green in all four ports - #29
Merged
Conversation
- replace every emptiness check on a shared fixture with a floor on its count - cover merge 18, mutations 21, rule 23, tiling 14, probes 3, dilate 12, kernels 5 - state why a floor and not equality, in each language - re-run every suite, and mutation-test each port against its own fixture Every consumer asserted the fixture was non-empty and none asserted how much of it was there. A fixture rebuilt holding three of its eighteen cases is not empty, so web, iOS, Android and Rust would all have gone on passing while testing a sixth of what they advertise. That is the defect the fixture header warns about, running in the checks meant to prevent it. Floors rather than equality, and the reason is in each file. A fixture GAINING a case is the normal and wanted direction, and pinning equality would make every addition an edit in four languages. What a floor cannot catch is a swap, one case removed and another added, and equality would not catch that either. Mutation-tested per port on exit code, each against its own fixture and each restored byte-identically afterwards. Rust 18 to 3 panics with the count in the message, web 23 to 2 exits 1, Android 14 to 2 exits 1, and iOS 18 to 4 exits 1. The iOS arm was worth running separately because a Swift mutation earlier in this work trapped rather than failing, and a harness reading reported failures scored it as a survivor. Green afterwards across all three suites, at 107 web tests, 73 iOS tests in twelve suites, and 106 Android tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every consumer asserted the fixture was non-empty and none asserted how much of it was there. A fixture rebuilt holding three of its eighteen cases is not empty, so web, iOS, Android and Rust would all have gone on passing while testing a sixth of what they advertise. That is the defect the fixture header warns about, running in the checks meant to prevent it.
Floors rather than equality, and the reason is in each file. A fixture GAINING a case is the normal and wanted direction, and pinning equality would make every addition an edit in four languages. What a floor cannot catch is a swap, one case removed and another added, and equality would not catch that either.
Mutation-tested per port on exit code, each against its own fixture and each restored byte-identically afterwards. Rust 18 to 3 panics with the count in the message, web 23 to 2 exits 1, Android 14 to 2 exits 1, and iOS 18 to 4 exits 1. The iOS arm was worth running separately because a Swift mutation earlier in this work trapped rather than failing, and a harness reading reported failures scored it as a survivor.
Green afterwards across all three suites, at 107 web tests, 73 iOS tests in twelve suites, and 106 Android tests.
What changed
Why
How it was verified
pnpm --filter ./apps/web testpnpm --filter ./apps/web run lintcd services/feedback && go vet ./... && go test ./...(if Go changed)cd apps/android && ./gradlew testDebugUnitTest(needs
JAVA_HOMEon Android Studio's JBR; see apps/android/README.md).CI does not run them, so this is the only gate on that suite.
MonOcrCoretests, if iOS changed:cd apps/ios && sh Scripts/swift-test.sh.The
ios-corejob runs these on every push, so this is a faster copy of agate that does exist. The app target is still not built anywhere.
Claims