[MNT] Mark tsfresh estimators non-deterministic for multithreading tests - #3770
[MNT] Mark tsfresh estimators non-deterministic for multithreading tests#3770Amiirhosseini wants to merge 1 commit into
Conversation
tsfresh parallel feature extraction can produce slightly different floating-point results between n_jobs=1 and n_jobs>1. Use the non_deterministic tag instead of excluding the multithreading check entirely, so n_jobs plumbing is still validated. Fixes aeon-toolkit#3326 Co-authored-by: Cursor <cursoragent@cursor.com>
Thank you for contributing to
|
There was a problem hiding this comment.
Pull request overview
This PR addresses intermittent check_estimator_multithreading failures for tsfresh-based estimators by marking them as non_deterministic and removing prior explicit exclusions, so the multithreading checks still exercise n_jobs / _n_jobs plumbing without requiring identical single- vs multi-thread outputs.
Changes:
- Add
non_deterministic: Truetags to tsfresh-based transformers/estimators (TSFresh, TSFreshRelevant, TSFresh* wrappers, FreshPRINCE*). - Remove
check_estimator_multithreadingexclusions for tsfresh estimators from the globalEXCLUDED_TESTSconfig.
This pull request includes code written with the assistance of AI.
The code has not yet been reviewed by a human.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| aeon/transformations/collection/feature_based/_tsfresh.py | Marks the base TSFresh transformer adapter as non_deterministic. |
| aeon/testing/testing_config.py | Removes prior explicit multithreading-check exclusions for tsfresh estimators. |
| aeon/regression/feature_based/_tsfresh.py | Marks TSFreshRegressor as non_deterministic. |
| aeon/regression/feature_based/_fresh_prince.py | Marks FreshPRINCERegressor as non_deterministic. |
| aeon/clustering/feature_based/_tsfresh.py | Marks TSFreshClusterer as non_deterministic. |
| aeon/classification/feature_based/_tsfresh.py | Marks TSFreshClassifier as non_deterministic. |
| aeon/classification/feature_based/_fresh_prince.py | Marks FreshPRINCEClassifier as non_deterministic. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| "capability:multithreading": True, | ||
| "capability:unequal_length": True, | ||
| "fit_is_empty": True, | ||
| "non_deterministic": True, | ||
| "python_dependencies": "tsfresh", |
| # broken by 0.63.0 numba update, see #3307 attempt to fix | ||
| "HIVECOTEV2": ["check_classifier_against_expected_results"], | ||
| "TemporalDictionaryEnsemble": ["check_classifier_against_expected_results"], | ||
| # multithreading issue, sometimes produces different results between single | ||
| # and multithreading | ||
| "FreshPRINCEClassifier": ["check_estimator_multithreading"], | ||
| "FreshPRINCERegressor": ["check_estimator_multithreading"], | ||
| "TSFreshClassifier": ["check_estimator_multithreading"], | ||
| "TSFreshRegressor": ["check_estimator_multithreading"], | ||
| "TSFreshClusterer": ["check_estimator_multithreading"], | ||
| "TSFreshRelevant": ["check_estimator_multithreading"], | ||
| "TSFresh": ["check_estimator_multithreading"], | ||
| } |
|
Still waiting on maintainer review — pytest and pre-commit are green on my side. Happy to adjust if anything is needed for #3326. |
Summary
check_estimator_multithreadingfailures for tsfresh-related estimators ([MNT] tsfresh multithreaded testing failures #3326).tsfreshparallel feature extraction (n_jobs > 1) can produce slightly different floating-point results vsn_jobs=1— this is transform-side, not downstream sklearn estimators.non_deterministic: Trueand remove the blanketEXCLUDED_TESTSskips so the multithreading check still validatesn_jobs/_n_jobsplumbing without asserting identical outputs.Investigation notes
_clone_estimator+set_params(n_jobs=2)+ fit/predict) onEqualLengthUnivariate-Classification-numpy3Dtest data — no mismatches in 3 runs withefficient/minimalfeature sets, but CI flakes are intermittent (PR [MNT] Skip tsfresh multithreading test #3325 previously skipped these tests for that reason).TSFreshtransformer atfittime with the correctn_jobs, so the divergence originates intsfresh.extract_features, not aeon's sklearn wrappers.Test plan
pytest aeon/transformations/collection/feature_based/tests/test_tsfresh.py— passpytest aeon/classification/feature_based/tests/test_tsfresh.py— passtest_all_estimatorswith--enablethreading True(tsfresh soft dependency)