Repository navigation
Closed
Labels
MetricsMetric Component.Metric Component.More Info RequestedMore information requested from the userMore information requested from the user
Description
Describe the bug
Greetings again!
I am trying to utilize BinaryAccuracy from torchmetrics in test evaluator.
I used anomalib.metrics.create_anomalib_metric to wrap Metric class into AnomalibMetric class.
When Engine.trainer is trying to create checkpoint, it raised an error that the wrapped class cannot be pickled.
What should I do to avoid this error?
Thank you!
Dataset
MVTecAD
Model
PatchCore
Steps to reproduce the behavior
- Create an
Evaluatorby:Evaluator(test_metrics=[anomalib.metrics.create_anomalib_metric(torchmetrics.classification.BinaryAccuracy)(fields=["pred_score", "gt_label"], prefix="image_")])
- Initialize
Patchcorewith the evaluator created in step 1 - Run model training with
Engine.fit() - I will raise
_pickle.PicklingError.
OS information
OS information:
- OS: Ubuntu 24.04 (docker container)
- Python version: 3.11.12
- Anomalib version: 2.0.0
- PyTorch version: 2.1.2
- CUDA/cuDNN version: 12.6; driver version: 560.76
- GPU models and configuration: 1x GeForce RTX 3070 Ti Laptop
- Any other relevant information:
- Using a sub-dataset from MVTec dataset, the hazelnut dataset.
- Evaluator setup:
Evaluator(test_metrics=[anomalib.metrics.create_anomalib_metric(torchmetrics.classification.BinaryAccuracy)(fields=["pred_score", "gt_label"], prefix="image_")])
Expected behavior
Evaluate the model with BinaryAccuracy successfully.
Screenshots
No response
Pip/GitHub
pip
What version/branch did you use?
No response
Configuration YAML
I did not utilize yaml configs.Logs
File "/src/anomalib_test.py", line 553, in <module>
engine.fit(datamodule=datamodule, model=lightning_model)
File "/usr/local/lib/python3.11/dist-packages/anomalib/engine/engine.py", line 416, in fit
self.trainer.fit(model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 538, in fit
call._call_and_handle_interrupt(
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 574, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 981, in _run
results = self._run_stage()
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 1025, in _run_stage
self.fit_loop.run()
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/fit_loop.py", line 206, in run
self.on_advance_end()
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/fit_loop.py", line 378, in on_advance_end
call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/call.py", line 218, in _call_callback_hooks
fn(trainer, trainer.lightning_module, *args, **kwargs)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 325, in on_train_epoch_end
self._save_topk_checkpoint(trainer, monitor_candidates)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 387, in _save_topk_checkpoint
self._save_none_monitor_checkpoint(trainer, monitor_candidates)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 715, in _save_none_monitor_checkpoint
self._save_checkpoint(trainer, filepath)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 390, in _save_checkpoint
trainer.save_checkpoint(filepath, self.save_weights_only)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 1365, in save_checkpoint
self.strategy.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/strategies/strategy.py", line 490, in save_checkpoint
self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
File "/usr/local/lib/python3.11/dist-packages/lightning/fabric/plugins/io/torch_io.py", line 58, in save_checkpoint
_atomic_save(checkpoint, path)
File "/usr/local/lib/python3.11/dist-packages/lightning/fabric/utilities/cloud_io.py", line 85, in _atomic_save
torch.save(checkpoint, bytesbuffer)
File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 619, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 831, in _save
pickler.dump(obj)
_pickle.PicklingError: Can't pickle <class 'abc.BinaryAccuracy'>: attribute lookup BinaryAccuracy on abc failedCode of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
MetricsMetric Component.Metric Component.More Info RequestedMore information requested from the userMore information requested from the user
Activity
samet-akcay commentedon Jun 6, 2025
Thanks for reporting @Narc17. We'll investigate this
samet-akcay commentedon Jun 9, 2025
@Narc17, I'm unable to reproduce this issue. The following script works on my side:
Here are the terminal results when you run this:
Here are the package versions on which I tested this
[-][Bug]: Evaluator formed with torchmetrics metrics cannot be saved after model training[/-][+]Evaluator formed with torchmetrics metrics cannot be saved after model training[/+]samet-akcay commentedon Jun 10, 2025
I'm moving this to a Q&A. Feel free to convert it back to a bug report if you still think there is something wrong. Cheers!