Skip to content

[Question]: How to work with winCLIP on custom dataset? (also a call for better examples)  #2444

Closed

Description

@brechtBDCK

Describe the bug

Following the recent advance in Anomaly classification/segmentation, I wanted to try the new winCLIP model, of which the anomalib library also has an implementation. As this is a zero-shot model, i would assume that this would easily work out of the box. Anyone have any experience of how to test zero-shot or to give a couple "normal/healthy" images for few-shot? I can't get it working. I would assume that (as it's zero-shot) i would be able to just input some custom defect images? I also can't figure out the few-shot using some normal/healthy images from my own dataset.

Here is my current code:

from anomalib.models.image import WinClip
from anomalib.engine import Engine


# Import the datamodule
from anomalib.data import Folder
from anomalib.data.utils import TestSplitMode

# Create the datamodule
datamodule = Folder(
    name="lasercut_plank",
    root="./DATA_0shot",
    normal_dir="normal",
    test_split_mode=TestSplitMode.NONE
)

# Setup the datamodule
datamodule.setup()

# Access the datasets
train_dataset = datamodule.train_data

# Access the dataloaders
train_dataloader = datamodule.train_dataloader()

# Create the model and engine
model = WinClip(class_name="lasercut_plank")
engine = Engine(task="segmentation")

# Train a Patchcore model on the given datamodule
engine.train(datamodule=datamodule, model=model)

Dataset

Other (please specify in the text field below)

Model

Other (please specify in the field below)

Steps to reproduce the behavior

None

OS information

OS information:
WSL ubuntu on windows 11

Name: anomalib
Version: 1.2.0

Expected behavior

none

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

none

Logs

none

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

  1. djdameln commented on Dec 2, 2024

    @djdameln
    Contributor

    Hi, you are right, WinClip is a zero-shot model, so in theory, you can use it out-of-the box to generate some predictions on local images.

    However, the full answer is a bit more nuanced. WinClip, like many other anomaly detection models, does not produce binary normal-vs-anomalous classification labels or segmentation masks, but instead produces real-valued anomaly scores (image-level) and anomaly maps (pixel-level). To convert the anomaly scores to labels, we need to compare them to some known threshold value (and assign a normal label to all scores below the threshold, and an anomalous label to all scores above the threshold).

    When using WinClip out-of-the-box to generate predictions on a new dataset, we don't know which threshold value we should use. This is why we usually run a validation sequence on a small subset of our data which contains some normal and anomalous samples. During this validation sequence, we adaptively compute the optimal threshold value by maximizing the F1 score over the validation set. The validation sequence also collects some statistics used for normalizing the raw anomaly scores and maps to the [0, 1] range.

    Now let's get back to your use-case. If retrieving just the raw anomaly scores is sufficient for your use-case, you could use WinClip to generate predictions by directly calling the engine's predict method, which allows passing a path to an image or folder of images. The following would be sufficient to yield the predictions:

    from anomalib.models import WinClip
    from anomalib.engine import Engine
    
    engine = Engine(task="classification")
    model = WinClip(class_name="lasercut_plank")
    
    predictions = engine.predict(model, data_path="./DATA_0shot/normal")

    You can access the raw predictions by inspecting the predictions variable.

    When running predict, the engine also writes visualizations of the predictions to the results folder in your working directory. By looking at the visualizations, you can easily confirm that the model is not able to make sense of the raw anomaly scores. The labels will likely be inaccurate and the heatmaps flat. This is because of the lack of adequate threshold and normalization statistics needed to convert the raw predictions to meaningful results.

    To improve this, you will have to provide Anomalib with some normal and anomalous validation images that can be used for normalization and thresholding. Here's some example code illustrating how this could be achieved.

    from anomalib.models import WinClip
    from anomalib.engine import Engine
    from anomalib.data import Folder
    
    engine = Engine(task="classification")
    model = WinClip(class_name="lasercut_plank")
    
    datamodule = Folder(
        name="lasercut_plank",
        root="./DATA_0shot",
        normal_dir="normal",
        abnormal_dir="abnormal",  # or wherever your abnormal images are located
        task="classification",   # we don't have ground-truth masks, so we can't use the segmentation task type
    )
    
    engine.validate(model, datamodule=datamodule)
    engine.predict(model, data_path="path/to/your/query/images")

    Few-shot mode

    To run WinClip in few-shot mode, simply set the k_shot parameter to the desired number of reference images (e.g. 1 for 1-shot or 2 for 2-shot), and pass a path to a folder of normal reference images using the few_shot_source parameter. Here's an example how to run the model in 2-shot mode:

    model = WinClip(
        class_name="lasercut_plank",
        k_shot=2,
        few_shot_source="path/to/normal/reference/images",
    )
  2. samet-akcay commented on Dec 3, 2024

    @samet-akcay
    Contributor

    Thanks @djdameln for the nice explanation. Would it be an idea to rename few_shot_source to k_shot_source for consistency?

  3. locked and limited conversation to collaborators on Dec 3, 2024
  4. converted this issue into a discussion #2446 on Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions