Skip to content

[MNT] Mark tsfresh estimators non-deterministic for multithreading tests - #3770

Open
Amiirhosseini wants to merge 1 commit into
aeon-toolkit:mainfrom
Amiirhosseini:mnt/tsfresh-multithreading-investigation
Open

[MNT] Mark tsfresh estimators non-deterministic for multithreading tests#3770
Amiirhosseini wants to merge 1 commit into
aeon-toolkit:mainfrom
Amiirhosseini:mnt/tsfresh-multithreading-investigation

Conversation

@Amiirhosseini

Copy link
Copy Markdown

Summary

  • Investigated intermittent check_estimator_multithreading failures for tsfresh-related estimators ([MNT] tsfresh multithreaded testing failures #3326).
  • Root cause: tsfresh parallel feature extraction (n_jobs > 1) can produce slightly different floating-point results vs n_jobs=1 — this is transform-side, not downstream sklearn estimators.
  • Mark all tsfresh-based estimators with non_deterministic: True and remove the blanket EXCLUDED_TESTS skips so the multithreading check still validates n_jobs / _n_jobs plumbing without asserting identical outputs.

Investigation notes

  • Reproduced the CI test path locally (_clone_estimator + set_params(n_jobs=2) + fit/predict) on EqualLengthUnivariate-Classification-numpy3D test data — no mismatches in 3 runs with efficient/minimal feature sets, but CI flakes are intermittent (PR [MNT] Skip tsfresh multithreading test #3325 previously skipped these tests for that reason).
  • Downstream classifiers/regressors (FreshPRINCE, TSFreshClassifier) create a fresh TSFresh transformer at fit time with the correct n_jobs, so the divergence originates in tsfresh.extract_features, not aeon's sklearn wrappers.

Test plan

  • pytest aeon/transformations/collection/feature_based/tests/test_tsfresh.py — pass
  • pytest aeon/classification/feature_based/tests/test_tsfresh.py — pass
  • CI test_all_estimators with --enablethreading True (tsfresh soft dependency)

tsfresh parallel feature extraction can produce slightly different
floating-point results between n_jobs=1 and n_jobs>1. Use the
non_deterministic tag instead of excluding the multithreading check
entirely, so n_jobs plumbing is still validated.

Fixes aeon-toolkit#3326

Co-authored-by: Cursor <cursoragent@cursor.com>
Copilot AI lite review requested due to automatic review settings August 29, 2026 17:48
@aeon-actions-bot aeon-actions-bot Bot added the maintenance Continuous integration, unit testing & package distribution label Aug 29, 2026
@aeon-actions-bot

Copy link
Copy Markdown
Contributor

Thank you for contributing to aeon

I have added the following labels to this PR based on the title: [ maintenance ].
This PR changes too many different packages (>3) for automatic addition of labels, please manually add package labels if relevant.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, please run pre-commit locally and push the fixes to your PR branch.

Don't hesitate to ask questions on the aeon Discord channel if you have any.

PR CI actions

These checkboxes will add labels to enable or disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

  • Run pre-commit checks for all files
  • Run mypy typecheck tests
  • Run all pytest tests and configurations
  • Run all notebook example tests
  • Run numba-disabled codecov tests
  • Disable numba cache loading
  • Regenerate expected results for testing
  • Push an empty commit to re-run CI checks

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses intermittent check_estimator_multithreading failures for tsfresh-based estimators by marking them as non_deterministic and removing prior explicit exclusions, so the multithreading checks still exercise n_jobs / _n_jobs plumbing without requiring identical single- vs multi-thread outputs.

Changes:

  • Add non_deterministic: True tags to tsfresh-based transformers/estimators (TSFresh, TSFreshRelevant, TSFresh* wrappers, FreshPRINCE*).
  • Remove check_estimator_multithreading exclusions for tsfresh estimators from the global EXCLUDED_TESTS config.

This pull request includes code written with the assistance of AI.
The code has not yet been reviewed by a human.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
aeon/transformations/collection/feature_based/_tsfresh.py Marks the base TSFresh transformer adapter as non_deterministic.
aeon/testing/testing_config.py Removes prior explicit multithreading-check exclusions for tsfresh estimators.
aeon/regression/feature_based/_tsfresh.py Marks TSFreshRegressor as non_deterministic.
aeon/regression/feature_based/_fresh_prince.py Marks FreshPRINCERegressor as non_deterministic.
aeon/clustering/feature_based/_tsfresh.py Marks TSFreshClusterer as non_deterministic.
aeon/classification/feature_based/_tsfresh.py Marks TSFreshClassifier as non_deterministic.
aeon/classification/feature_based/_fresh_prince.py Marks FreshPRINCEClassifier as non_deterministic.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 44 to 48
"capability:multithreading": True,
"capability:unequal_length": True,
"fit_is_empty": True,
"non_deterministic": True,
"python_dependencies": "tsfresh",
Comment on lines 61 to 64
# broken by 0.63.0 numba update, see #3307 attempt to fix
"HIVECOTEV2": ["check_classifier_against_expected_results"],
"TemporalDictionaryEnsemble": ["check_classifier_against_expected_results"],
# multithreading issue, sometimes produces different results between single
# and multithreading
"FreshPRINCEClassifier": ["check_estimator_multithreading"],
"FreshPRINCERegressor": ["check_estimator_multithreading"],
"TSFreshClassifier": ["check_estimator_multithreading"],
"TSFreshRegressor": ["check_estimator_multithreading"],
"TSFreshClusterer": ["check_estimator_multithreading"],
"TSFreshRelevant": ["check_estimator_multithreading"],
"TSFresh": ["check_estimator_multithreading"],
}
@Amiirhosseini

Copy link
Copy Markdown
Author

Still waiting on maintainer review — pytest and pre-commit are green on my side. Happy to adjust if anything is needed for #3326.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Continuous integration, unit testing & package distribution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants