You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Question]: How to work with winCLIP on custom dataset? (also a call for better examples) #2444
Following the recent advance in Anomaly classification/segmentation, I wanted to try the new winCLIP model, of which the anomalib library also has an implementation. As this is a zero-shot model, i would assume that this would easily work out of the box. Anyone have any experience of how to test zero-shot or to give a couple "normal/healthy" images for few-shot? I can't get it working. I would assume that (as it's zero-shot) i would be able to just input some custom defect images? I also can't figure out the few-shot using some normal/healthy images from my own dataset.
Here is my current code:
fromanomalib.models.imageimportWinClipfromanomalib.engineimportEngine# Import the datamodulefromanomalib.dataimportFolderfromanomalib.data.utilsimportTestSplitMode# Create the datamoduledatamodule=Folder(
name="lasercut_plank",
root="./DATA_0shot",
normal_dir="normal",
test_split_mode=TestSplitMode.NONE
)
# Setup the datamoduledatamodule.setup()
# Access the datasetstrain_dataset=datamodule.train_data# Access the dataloaderstrain_dataloader=datamodule.train_dataloader()
# Create the model and enginemodel=WinClip(class_name="lasercut_plank")
engine=Engine(task="segmentation")
# Train a Patchcore model on the given datamoduleengine.train(datamodule=datamodule, model=model)
Hi, you are right, WinClip is a zero-shot model, so in theory, you can use it out-of-the box to generate some predictions on local images.
However, the full answer is a bit more nuanced. WinClip, like many other anomaly detection models, does not produce binary normal-vs-anomalous classification labels or segmentation masks, but instead produces real-valued anomaly scores (image-level) and anomaly maps (pixel-level). To convert the anomaly scores to labels, we need to compare them to some known threshold value (and assign a normal label to all scores below the threshold, and an anomalous label to all scores above the threshold).
When using WinClip out-of-the-box to generate predictions on a new dataset, we don't know which threshold value we should use. This is why we usually run a validation sequence on a small subset of our data which contains some normal and anomalous samples. During this validation sequence, we adaptively compute the optimal threshold value by maximizing the F1 score over the validation set. The validation sequence also collects some statistics used for normalizing the raw anomaly scores and maps to the [0, 1] range.
Now let's get back to your use-case. If retrieving just the raw anomaly scores is sufficient for your use-case, you could use WinClip to generate predictions by directly calling the engine's predict method, which allows passing a path to an image or folder of images. The following would be sufficient to yield the predictions:
You can access the raw predictions by inspecting the predictions variable.
When running predict, the engine also writes visualizations of the predictions to the results folder in your working directory. By looking at the visualizations, you can easily confirm that the model is not able to make sense of the raw anomaly scores. The labels will likely be inaccurate and the heatmaps flat. This is because of the lack of adequate threshold and normalization statistics needed to convert the raw predictions to meaningful results.
To improve this, you will have to provide Anomalib with some normal and anomalous validation images that can be used for normalization and thresholding. Here's some example code illustrating how this could be achieved.
fromanomalib.modelsimportWinClipfromanomalib.engineimportEnginefromanomalib.dataimportFolderengine=Engine(task="classification")
model=WinClip(class_name="lasercut_plank")
datamodule=Folder(
name="lasercut_plank",
root="./DATA_0shot",
normal_dir="normal",
abnormal_dir="abnormal", # or wherever your abnormal images are locatedtask="classification", # we don't have ground-truth masks, so we can't use the segmentation task type
)
engine.validate(model, datamodule=datamodule)
engine.predict(model, data_path="path/to/your/query/images")
Few-shot mode
To run WinClip in few-shot mode, simply set the k_shot parameter to the desired number of reference images (e.g. 1 for 1-shot or 2 for 2-shot), and pass a path to a folder of normal reference images using the few_shot_source parameter. Here's an example how to run the model in 2-shot mode:
Activity
djdameln commentedon Dec 2, 2024
Hi, you are right, WinClip is a zero-shot model, so in theory, you can use it out-of-the box to generate some predictions on local images.
However, the full answer is a bit more nuanced. WinClip, like many other anomaly detection models, does not produce binary normal-vs-anomalous classification labels or segmentation masks, but instead produces real-valued anomaly scores (image-level) and anomaly maps (pixel-level). To convert the anomaly scores to labels, we need to compare them to some known threshold value (and assign a normal label to all scores below the threshold, and an anomalous label to all scores above the threshold).
When using WinClip out-of-the-box to generate predictions on a new dataset, we don't know which threshold value we should use. This is why we usually run a validation sequence on a small subset of our data which contains some normal and anomalous samples. During this validation sequence, we adaptively compute the optimal threshold value by maximizing the F1 score over the validation set. The validation sequence also collects some statistics used for normalizing the raw anomaly scores and maps to the [0, 1] range.
Now let's get back to your use-case. If retrieving just the raw anomaly scores is sufficient for your use-case, you could use WinClip to generate predictions by directly calling the engine's
predictmethod, which allows passing a path to an image or folder of images. The following would be sufficient to yield the predictions:You can access the raw predictions by inspecting the
predictionsvariable.When running
predict, the engine also writes visualizations of the predictions to theresultsfolder in your working directory. By looking at the visualizations, you can easily confirm that the model is not able to make sense of the raw anomaly scores. The labels will likely be inaccurate and the heatmaps flat. This is because of the lack of adequate threshold and normalization statistics needed to convert the raw predictions to meaningful results.To improve this, you will have to provide Anomalib with some normal and anomalous validation images that can be used for normalization and thresholding. Here's some example code illustrating how this could be achieved.
Few-shot mode
To run WinClip in few-shot mode, simply set the
k_shotparameter to the desired number of reference images (e.g.1for 1-shot or2for 2-shot), and pass a path to a folder of normal reference images using thefew_shot_sourceparameter. Here's an example how to run the model in 2-shot mode:samet-akcay commentedon Dec 3, 2024
Thanks @djdameln for the nice explanation. Would it be an idea to rename
few_shot_sourcetok_shot_sourcefor consistency?