[boxed/mutmut] run_stats records the TEARDOWN duration per test (missing call.when filter) → near-zero estimates starve the per-mutant budget
Summary
The stats-collection plugin's pytest_runtest_makereport hook stores call.duration without filtering on call.when. The hook fires three times per test (setup / call / teardown) and the storage is last-write-wins, so the recorded per-test duration is the teardown figure — typically well under 10 ms regardless of how expensive the actual test call was.
Measured on our suite (mutants/mutmut-stats.json): 210 of 219 tests recorded under 0.01 s, median 0.0039 s — while the real wall time of the covering set for our hottest mutation targets is ~25–31 s.
Why it matters
timeout_checker derives the per-mutant budget from these stats:
- wall deadline:
(estimated_time_of_tests + 1) * 15
RLIMIT_CPU: (estimated_time_of_tests + 1) * 30
With estimated_time_of_tests ≈ 0, every mutant gets a ~15 s wall budget no matter what the covering tests really cost. Mutants whose covering set genuinely needs more are scored ⏰ (timeout) instead of 🎉/🙁 — and a timed-out mutant is not scored at all, so heavy, well-tested code paths systematically drop out of the mutation score. No suite-level speedup can compensate, because the estimate never reflects the call phase in the first place.
On our project this clipped 366–418 mutants per run as spurious timeouts until we worked around it by shrinking the covering set's real cost (hypothesis max_examples gating on MUTANT_UNDER_TEST).
Suggested fix
In the makereport hook, record only the call phase (or sum the phases):
def pytest_runtest_makereport(self, item, call):
if call.when != "call":
return
...Environment
[boxed/mutmut] run_stats records the TEARDOWN duration per test (missing
call.whenfilter) → near-zero estimates starve the per-mutant budgetSummary
The stats-collection plugin's
pytest_runtest_makereporthook storescall.durationwithout filtering oncall.when. The hook fires three times per test (setup / call / teardown) and the storage is last-write-wins, so the recorded per-test duration is the teardown figure — typically well under 10 ms regardless of how expensive the actual test call was.Measured on our suite (
mutants/mutmut-stats.json): 210 of 219 tests recorded under 0.01 s, median 0.0039 s — while the real wall time of the covering set for our hottest mutation targets is ~25–31 s.Why it matters
timeout_checkerderives the per-mutant budget from these stats:(estimated_time_of_tests + 1) * 15RLIMIT_CPU:(estimated_time_of_tests + 1) * 30With
estimated_time_of_tests ≈ 0, every mutant gets a ~15 s wall budget no matter what the covering tests really cost. Mutants whose covering set genuinely needs more are scored ⏰ (timeout) instead of 🎉/🙁 — and a timed-out mutant is not scored at all, so heavy, well-tested code paths systematically drop out of the mutation score. No suite-level speedup can compensate, because the estimate never reflects the call phase in the first place.On our project this clipped 366–418 mutants per run as spurious timeouts until we worked around it by shrinking the covering set's real cost (hypothesis
max_examplesgating onMUTANT_UNDER_TEST).Suggested fix
In the makereport hook, record only the call phase (or sum the phases):
Environment
[tool.mutmut],paths_to_mutate)mutmut runtimeout_checkeruses outer-loopmutant_namefor the est-lookup, causing spurious 15s timeouts for unrelated mutants when any "no tests" mutant exists #518 (est-lookup uses the wrong key); this report is about the recorded value itself.