Skip to content

Skip the clean run when the suite is already known to pass. - #572

Open
sknigh wants to merge 1 commit into
boxed:mainfrom
sknigh:perf/clean-run-scoping
Open

sknigh wants to merge 1 commit into
boxed:mainfrom
sknigh:perf/clean-run-scoping

Conversation

@sknigh

@sknigh sknigh commented Sep 8, 2026

Copy link
Copy Markdown

Skip the clean run when the suite is already known to pass, either because stats collection just ran every test or because nothing is left to test, and otherwise run only the tests the pending mutants use.

The clean test run executed the whole suite on every run, even right after a full stats collection had just run every test unmutated, and even when every mutant already had a verdict. The forced-fail check collected the whole suite to make one test fail.

  1. The clean run is skipped when nothing needs it. Right after a full stats collection, which ran every test unmutated and stops on a failure, and when no mutant without a verdict has tests. Each skip is printed with its reason. collect_or_load_stats now returns whether a full collection ran.

  2. Otherwise the clean run covers only the relevant tests. tests_for_mutants unions the tests stats collection associated with the mutants about to be tested (those without a verdict, or the ones named explicitly).

  3. Listed test ids give way to the configured selection once they are a majority. Collecting many ids is slower than collecting the configured selection, so clean_run_test_selection returns the whole selection when the relevant tests are at least half of the known tests.

  4. The forced-fail check probes a few tests. probe_tests_for_forced_fail ranks tests by how many mutated functions they reached during stats collection, since a test with a single association may only be linked through an import-time hit and pass, takes up to eight from different files (pytest stops at the first failure), and falls back to the whole suite if they unexpectedly pass.

  5. TestRunner.run_forced_fail takes the tests to run, for both runners.

The clean test run executed the whole suite on every run, even right
after a full stats collection had just run every test unmutated, and
even when every mutant already had a verdict. The forced-fail check
collected the whole suite to make one test fail.

1. The clean run is skipped when nothing needs it. Right after a full
   stats collection, which ran every test unmutated and stops on a
   failure, and when no mutant without a verdict has tests. Each skip is
   printed with its reason. collect_or_load_stats now returns whether a
   full collection ran.

2. Otherwise the clean run covers only the relevant tests. tests_for_mutants
   unions the tests stats collection associated with the mutants about to
   be tested (those without a verdict, or the ones named explicitly).

3. Listed test ids give way to the configured selection once they are a
   majority. Collecting many ids is slower than collecting the configured
   selection, so clean_run_test_selection returns the whole selection
   when the relevant tests are at least half of the known tests.

4. The forced-fail check probes a few tests. probe_tests_for_forced_fail
   ranks tests by how many mutated functions they reached during stats
   collection, since a test with a single association may only be linked
   through an import-time hit and pass, takes up to eight from different
   files (pytest stops at the first failure), and falls back to the whole
   suite if they unexpectedly pass.

5. TestRunner.run_forced_fail takes the tests to run, for both runners.
@sknigh sknigh changed the title Skip the clean run when the suite is already known to pass, either because stats collection just ran every test or because nothing is left to test, and otherwise run only the tests the pending mutants use. Skip the clean run when the suite is already known to pass. Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant