Conversation
The clean test run executed the whole suite on every run, even right after a full stats collection had just run every test unmutated, and even when every mutant already had a verdict. The forced-fail check collected the whole suite to make one test fail. 1. The clean run is skipped when nothing needs it. Right after a full stats collection, which ran every test unmutated and stops on a failure, and when no mutant without a verdict has tests. Each skip is printed with its reason. collect_or_load_stats now returns whether a full collection ran. 2. Otherwise the clean run covers only the relevant tests. tests_for_mutants unions the tests stats collection associated with the mutants about to be tested (those without a verdict, or the ones named explicitly). 3. Listed test ids give way to the configured selection once they are a majority. Collecting many ids is slower than collecting the configured selection, so clean_run_test_selection returns the whole selection when the relevant tests are at least half of the known tests. 4. The forced-fail check probes a few tests. probe_tests_for_forced_fail ranks tests by how many mutated functions they reached during stats collection, since a test with a single association may only be linked through an import-time hit and pass, takes up to eight from different files (pytest stops at the first failure), and falls back to the whole suite if they unexpectedly pass. 5. TestRunner.run_forced_fail takes the tests to run, for both runners.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skip the clean run when the suite is already known to pass, either because stats collection just ran every test or because nothing is left to test, and otherwise run only the tests the pending mutants use.
The clean test run executed the whole suite on every run, even right after a full stats collection had just run every test unmutated, and even when every mutant already had a verdict. The forced-fail check collected the whole suite to make one test fail.
The clean run is skipped when nothing needs it. Right after a full stats collection, which ran every test unmutated and stops on a failure, and when no mutant without a verdict has tests. Each skip is printed with its reason. collect_or_load_stats now returns whether a full collection ran.
Otherwise the clean run covers only the relevant tests. tests_for_mutants unions the tests stats collection associated with the mutants about to be tested (those without a verdict, or the ones named explicitly).
Listed test ids give way to the configured selection once they are a majority. Collecting many ids is slower than collecting the configured selection, so clean_run_test_selection returns the whole selection when the relevant tests are at least half of the known tests.
The forced-fail check probes a few tests. probe_tests_for_forced_fail ranks tests by how many mutated functions they reached during stats collection, since a test with a single association may only be linked through an import-time hit and pass, takes up to eight from different files (pytest stops at the first failure), and falls back to the whole suite if they unexpectedly pass.
TestRunner.run_forced_fail takes the tests to run, for both runners.