Skip to content

[Bug]: EfficientAD - RuntimeError: Calculated padded input size per channel: (7 x 7). Kernel size: (8 x 8). #1148

Closed

Description

@simon-ami

Describe the bug

I'm trying to train an EfficientAD model (tried medium and small) but am running into this issue. Have used similar config for other models successfuly (except model specific configs).
If my interpretation is correct, the output size of one of the Encoder layers doesnt match the expected input size of the following (last) layer.

Dataset

Folder

Model

Other (please specify in the field below)

Steps to reproduce the behavior

train efficientAD using my config in anomalib 0.6

OS information

OS information:

  • OS: win 10
  • Python version: 3.8.1.6
  • Anomalib version: 0.6.0
  • PyTorch version:
  • CUDA/cuDNN version: 11.6
  • GPU models and configuration: rtx3050ti
  • Any other relevant information: custom data

Expected behavior

Training should continue normally

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

0.6.0

Configuration YAML

dataset:
  abnormal_dir: bad
  center_crop: null
  extensions: null
  format: folder
  image_size: 256  # mandatory
  mask: ground_truth
  name: vespa
  normal_dir: good
  normal_test_dir: null
  normalization: imagenet
  num_workers: 8
  path: ./../data/images/data_structure_anomalib
  split_ratio: 0.2
  task: classification
  test_batch_size: 16
  test_split_mode: from_dir
  test_split_ratio: 0.2
  tiling:
    apply: false
    random_tile_count: 16
    remove_border_count: 0
    stride: null
    tile_size: null
    use_random_tiling: false
  train_batch_size: 16
  transform_config:
    eval: c:\Users\benedict\dev\VespaUseCase\machine_learning\src\d01_config\transform_config_1.yaml
    train: null
  val_split_mode: from_test
  val_split_ratio: 0.5
logging:
  log_graph: false
  logger: []
metrics:
  image:
  - F1Score
  - AUROC
  pixel:
  - F1Score
  - AUROC
  threshold:
    manual_image: null
    manual_pixel: null
    method: adaptive
model:
  lr: 0.0001
  model_size: small
  name: efficientad
  normalization_method: min_max
  padding: false
  teacher_out_channels: 384
  weight_decay: 1.0e-05
optimization:
  export_mode: onnx
project:
  path: ./results
  seed: 42
trainer:
  accelerator: auto
  accumulate_grad_batches: 1
  auto_lr_find: false
  auto_scale_batch_size: false
  benchmark: false
  check_val_every_n_epoch: 1
  default_root_dir: null
  detect_anomaly: false
  deterministic: false
  devices: 1
  enable_checkpointing: true
  enable_model_summary: true
  enable_progress_bar: true
  fast_dev_run: false
  gradient_clip_algorithm: norm
  gradient_clip_val: 0
  limit_predict_batches: 1.0
  limit_test_batches: 1.0
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  log_every_n_steps: 50
  max_epochs: 200
  max_steps: -1
  max_time: null
  min_epochs: null
  min_steps: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle
  num_nodes: 1
  num_sanity_val_steps: 0
  overfit_batches: 0.0
  plugins: null
  precision: 32
  profiler: null
  reload_dataloaders_every_n_epochs: 0
  replace_sampler_ddp: true
  strategy: null
  sync_batchnorm: false
  track_grad_norm: -1
  val_check_interval: 1.0
visualization:
  image_save_path: null
  log_images: false
  mode: full
  save_images: true
  show_images: false

Albumentations Config YAML

__version__: 1.3.1
transform:
  __class_fullname__: Compose
  additional_targets: {}
  bbox_params: null
  is_check_shapes: true
  keypoint_params: null
  p: 1.0
  transforms:
  - __class_fullname__: Resize
    always_apply: true
    height: 256
    interpolation: 1
    p: 1
    width: 256
  - __class_fullname__: RandomBrightnessContrast
    always_apply: false
    brightness_by_max: true
    brightness_limit:
    - -0.3
    - 0.3
    contrast_limit:
    - -0.3
    - 0.3
    p: 0.5
  - __class_fullname__: RandomSunFlare
    always_apply: false
    angle_lower: 0
    angle_upper: 1
    flare_roi:
    - 0.3
    - 0.4
    - 0.7
    - 0.6
    num_flare_circles_lower: 1
    num_flare_circles_upper: 2
    p: 0.3
    src_color:
    - 255
    - 255
    - 255
    src_radius: 100
  - __class_fullname__: Rotate
    always_apply: false
    border_mode: 4
    crop_border: false
    interpolation: 1
    limit:
    - -10
    - 10
    mask_value: null
    p: 0.5
    rotate_method: largest_box
    value: null
  - __class_fullname__: Affine
    always_apply: false
    cval: 0
    cval_mask: 0
    fit_output: false
    interpolation: 1
    keep_ratio: false
    mask_interpolation: 0
    mode: 0
    p: 0.3
    rotate:
    - 0.0
    - 0.0
    rotate_method: largest_box
    scale:
      x:
      - 1.0
      - 1.0
      y:
      - 1.0
      - 1.0
    shear:
      x: &id001
      - -5
      - 5
      y: *id001
    translate_percent: null
    translate_px:
      x:
      - 0
      - 0
      y:
      - 0
      - 0
  - __class_fullname__: Normalize
    always_apply: true
    max_pixel_value: 255.0
    mean:
    - 0.485
    - 0.456
    - 0.406
    p: 1.0
    std:
    - 0.229
    - 0.224
    - 0.225
  - __class_fullname__: ToTensorV2
    always_apply: true
    p: 1.0
    transpose_mask: false

Logs

RuntimeError                              Traceback (most recent call last)
Cell In[16], line 1
----> 1 train_results = trainer.fit(model=model, datamodule=datamodule)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py:608, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    606 model = self._maybe_unwrap_optimized(model)
    607 self.strategy._lightning_module = model
--> 608 call._call_and_handle_interrupt(
    609     self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    610 )

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\call.py:38, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
     36         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
     37     else:
---> 38         return trainer_fn(*args, **kwargs)
     40 except _TunerExitException:
     41     trainer._call_teardown_hook()

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py:650, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    643 ckpt_path = ckpt_path or self.resume_from_checkpoint
    644 self._ckpt_path = self._checkpoint_connector._set_ckpt_path(
    645     self.state.fn,
    646     ckpt_path,  # type: ignore[arg-type]
    647     model_provided=True,
    648     model_connected=self.lightning_module is not None,
    649 )
--> 650 self._run(model, ckpt_path=self.ckpt_path)
    652 assert self.state.stopped
    653 self.training = False

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py:1112, in Trainer._run(self, model, ckpt_path)
   1108 self._checkpoint_connector.restore_training_state()
   1110 self._checkpoint_connector.resume_end()
-> 1112 results = self._run_stage()
   1114 log.detail(f"{self.__class__.__name__}: trainer tearing down")
   1115 self._teardown()

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py:1191, in Trainer._run_stage(self)
   1189 if self.predicting:
   1190     return self._run_predict()
-> 1191 self._run_train()

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py:1214, in Trainer._run_train(self)
   1211 self.fit_loop.trainer = self
   1213 with torch.autograd.set_detect_anomaly(self._detect_anomaly):
-> 1214     self.fit_loop.run()

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\loop.py:199, in Loop.run(self, *args, **kwargs)
    197 try:
    198     self.on_advance_start(*args, **kwargs)
--> 199     self.advance(*args, **kwargs)
    200     self.on_advance_end()
    201     self._restarting = False

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\fit_loop.py:267, in FitLoop.advance(self)
    265 self._data_fetcher.setup(dataloader, batch_to_device=batch_to_device)
    266 with self.trainer.profiler.profile("run_training_epoch"):
--> 267     self._outputs = self.epoch_loop.run(self._data_fetcher)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\loop.py:199, in Loop.run(self, *args, **kwargs)
    197 try:
    198     self.on_advance_start(*args, **kwargs)
--> 199     self.advance(*args, **kwargs)
    200     self.on_advance_end()
    201     self._restarting = False

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py:213, in TrainingEpochLoop.advance(self, data_fetcher)
    210     self.batch_progress.increment_started()
    212     with self.trainer.profiler.profile("run_training_batch"):
--> 213         batch_output = self.batch_loop.run(kwargs)
    215 self.batch_progress.increment_processed()
    217 # update non-plateau LR schedulers
    218 # update epoch-interval ones only when we are at the end of training epoch

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\loop.py:199, in Loop.run(self, *args, **kwargs)
    197 try:
    198     self.on_advance_start(*args, **kwargs)
--> 199     self.advance(*args, **kwargs)
    200     self.on_advance_end()
    201     self._restarting = False

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py:88, in TrainingBatchLoop.advance(self, kwargs)
     84 if self.trainer.lightning_module.automatic_optimization:
     85     optimizers = _get_active_optimizers(
     86         self.trainer.optimizers, self.trainer.optimizer_frequencies, kwargs.get("batch_idx", 0)
     87     )
---> 88     outputs = self.optimizer_loop.run(optimizers, kwargs)
     89 else:
     90     outputs = self.manual_loop.run(kwargs)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\loop.py:199, in Loop.run(self, *args, **kwargs)
    197 try:
    198     self.on_advance_start(*args, **kwargs)
--> 199     self.advance(*args, **kwargs)
    200     self.on_advance_end()
    201     self._restarting = False

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py:202, in OptimizerLoop.advance(self, optimizers, kwargs)
    199 def advance(self, optimizers: List[Tuple[int, Optimizer]], kwargs: OrderedDict) -> None:
    200     kwargs = self._build_kwargs(kwargs, self.optimizer_idx, self._hiddens)
--> 202     result = self._run_optimization(kwargs, self._optimizers[self.optim_progress.optimizer_position])
    203     if result.loss is not None:
    204         # automatic optimization assumes a loss needs to be returned for extras to be considered as the batch
    205         # would be skipped otherwise
    206         self._outputs[self.optimizer_idx] = result.asdict()

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py:249, in OptimizerLoop._run_optimization(self, kwargs, optimizer)
    241         closure()
    243 # ------------------------------
    244 # BACKWARD PASS
    245 # ------------------------------
    246 # gradient update with accumulated gradients
    247 else:
    248     # the `batch_idx` is optional with inter-batch parallelism
--> 249     self._optimizer_step(optimizer, opt_idx, kwargs.get("batch_idx", 0), closure)
    251 result = closure.consume_result()
    253 if result.loss is not None:
    254     # if no result, user decided to skip optimization
    255     # otherwise update running loss + reset accumulated loss
    256     # TODO: find proper way to handle updating running loss

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py:370, in OptimizerLoop._optimizer_step(self, optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
    362     rank_zero_deprecation(
    363         "The NVIDIA/apex AMP implementation has been deprecated upstream. Consequently, its integration inside"
    364         " PyTorch Lightning has been deprecated in v1.9.0 and will be removed in v2.0.0."
   (...)
    367         " return True."
    368     )
    369     kwargs["using_native_amp"] = isinstance(self.trainer.precision_plugin, MixedPrecisionPlugin)
--> 370 self.trainer._call_lightning_module_hook(
    371     "optimizer_step",
    372     self.trainer.current_epoch,
    373     batch_idx,
    374     optimizer,
    375     opt_idx,
    376     train_step_and_backward_closure,
    377     on_tpu=isinstance(self.trainer.accelerator, TPUAccelerator),
    378     **kwargs,  # type: ignore[arg-type]
    379     using_lbfgs=is_lbfgs,
    380 )
    382 if not should_accumulate:
    383     self.optim_progress.optimizer.step.increment_completed()

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py:1356, in Trainer._call_lightning_module_hook(self, hook_name, pl_module, *args, **kwargs)
   1353 pl_module._current_fx_name = hook_name
   1355 with self.profiler.profile(f"[LightningModule]{pl_module.__class__.__name__}.{hook_name}"):
-> 1356     output = fn(*args, **kwargs)
   1358 # restore current_fx when nested context
   1359 pl_module._current_fx_name = prev_fx_name

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\core\module.py:1754, in LightningModule.optimizer_step(self, epoch, batch_idx, optimizer, optimizer_idx, optimizer_closure, on_tpu, using_lbfgs)
   1675 def optimizer_step(
   1676     self,
   1677     epoch: int,
   (...)
   1683     using_lbfgs: bool = False,
   1684 ) -> None:
   1685     r"""
   1686     Override this method to adjust the default way the :class:`~pytorch_lightning.trainer.trainer.Trainer` calls
   1687     each optimizer.
   (...)
   1752 
   1753     """
-> 1754     optimizer.step(closure=optimizer_closure)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\core\optimizer.py:169, in LightningOptimizer.step(self, closure, **kwargs)
    166     raise MisconfigurationException("When `optimizer.step(closure)` is called, the closure should be callable")
    168 assert self._strategy is not None
--> 169 step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
    171 self._on_after_step()
    173 return step_output

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\strategies\strategy.py:234, in Strategy.optimizer_step(self, optimizer, opt_idx, closure, model, **kwargs)
    232 # TODO(fabric): remove assertion once strategy's optimizer_step typing is fixed
    233 assert isinstance(model, pl.LightningModule)
--> 234 return self.precision_plugin.optimizer_step(
    235     optimizer, model=model, optimizer_idx=opt_idx, closure=closure, **kwargs
    236 )

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\plugins\precision\precision_plugin.py:119, in PrecisionPlugin.optimizer_step(self, optimizer, model, optimizer_idx, closure, **kwargs)
    117 """Hook to run the optimizer step."""
    118 closure = partial(self._wrap_closure, model, optimizer, optimizer_idx, closure)
--> 119 return optimizer.step(closure=closure, **kwargs)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\optim\lr_scheduler.py:69, in LRScheduler.__init__..with_counter..wrapper(*args, **kwargs)
     67 instance._step_count += 1
     68 wrapped = func.__get__(instance, cls)
---> 69 return wrapped(*args, **kwargs)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\optim\optimizer.py:280, in Optimizer.profile_hook_step..wrapper(*args, **kwargs)
    276         else:
    277             raise RuntimeError(f"{func} must return None or a tuple of (new_args, new_kwargs),"
    278                                f"but got {result}.")
--> 280 out = func(*args, **kwargs)
    281 self._optimizer_step_code()
    283 # call optimizer step post hooks

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\optim\optimizer.py:33, in _use_grad_for_differentiable.._use_grad(self, *args, **kwargs)
     31 try:
     32     torch.set_grad_enabled(self.defaults['differentiable'])
---> 33     ret = func(self, *args, **kwargs)
     34 finally:
     35     torch.set_grad_enabled(prev_grad)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\optim\adamw.py:148, in AdamW.step(self, closure)
    146 if closure is not None:
    147     with torch.enable_grad():
--> 148         loss = closure()
    150 for group in self.param_groups:
    151     params_with_grad = []

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\plugins\precision\precision_plugin.py:105, in PrecisionPlugin._wrap_closure(self, model, optimizer, optimizer_idx, closure)
     92 def _wrap_closure(
     93     self,
     94     model: "pl.LightningModule",
   (...)
     97     closure: Callable[[], Any],
     98 ) -> Any:
     99     """This double-closure allows makes sure the ``closure`` is executed before the
    100     ``on_before_optimizer_step`` hook is called.
    101 
    102     The closure (generally) runs ``backward`` so this allows inspecting gradients in this hook. This structure is
    103     consistent with the ``PrecisionPlugin`` subclasses that cannot pass ``optimizer.step(closure)`` directly.
    104     """
--> 105     closure_result = closure()
    106     self._after_closure(model, optimizer, optimizer_idx)
    107     return closure_result

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py:149, in Closure.__call__(self, *args, **kwargs)
    148 def __call__(self, *args: Any, **kwargs: Any) -> Optional[Tensor]:
--> 149     self._result = self.closure(*args, **kwargs)
    150     return self._result.loss

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py:135, in Closure.closure(self, *args, **kwargs)
    134 def closure(self, *args: Any, **kwargs: Any) -> ClosureResult:
--> 135     step_output = self._step_fn()
    137     if step_output.closure_loss is None:
    138         self.warning_cache.warn("`training_step` returned `None`. If this was on purpose, ignore this warning...")

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py:419, in OptimizerLoop._training_step(self, kwargs)
    410 """Performs the actual train step with the tied hooks.
    411 
    412 Args:
   (...)
    416     A ``ClosureResult`` containing the training step output.
    417 """
    418 # manually capture logged metrics
--> 419 training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())
    420 self.trainer.strategy.post_training_step()
    422 model_output = self.trainer._call_lightning_module_hook("training_step_end", training_step_output)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py:1494, in Trainer._call_strategy_hook(self, hook_name, *args, **kwargs)
   1491     return
   1493 with self.profiler.profile(f"[Strategy]{self.strategy.__class__.__name__}.{hook_name}"):
-> 1494     output = fn(*args, **kwargs)
   1496 # restore current_fx when nested context
   1497 pl_module._current_fx_name = prev_fx_name

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\strategies\strategy.py:378, in Strategy.training_step(self, *args, **kwargs)
    376 with self.precision_plugin.train_step_context():
    377     assert isinstance(self.model, TrainingStep)
--> 378     return self.model.training_step(*args, **kwargs)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\anomalib\models\efficientad\lightning_model.py:217, in EfficientAD.training_step(***failed resolving arguments***)
    214     self.imagenet_iterator = iter(self.imagenet_loader)
    215     batch_imagenet = next(self.imagenet_iterator)[0]["image"].to(self.device)
--> 217 loss_st, loss_ae, loss_stae = self.model(batch=batch["image"], batch_imagenet=batch_imagenet)
    219 loss = loss_st + loss_ae + loss_stae
    220 self.log("train_st", loss_st.item(), on_epoch=True, prog_bar=True, logger=True)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\nn\modules\module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\anomalib\models\efficientad\torch_model.py:286, in EfficientADModel.forward(self, batch, batch_imagenet)
    284 # Autoencoder and Student AE Loss
    285 aug_img = self.choose_random_aug_image(batch)
--> 286 ae_output_aug = self.ae(aug_img)
    288 with torch.no_grad():
    289     teacher_output_aug = self.teacher(aug_img)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\nn\modules\module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\anomalib\models\efficientad\torch_model.py:182, in AutoEncoder.forward(self, x)
    180 def forward(self, x):
    181     x = imagenet_norm_batch(x)
--> 182     x = self.encoder(x)
    183     x = self.decoder(x)
    184     return x

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\nn\modules\module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\anomalib\models\efficientad\torch_model.py:112, in Encoder.forward(self, x)
    110 x = F.relu(self.enconv4(x))
    111 x = F.relu(self.enconv5(x))
--> 112 x = self.enconv6(x)
    113 return x

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\nn\modules\module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\nn\modules\conv.py:463, in Conv2d.forward(self, input)
    462 def forward(self, input: Tensor) -> Tensor:
--> 463     return self._conv_forward(input, self.weight, self.bias)

File c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\nn\modules\conv.py:459, in Conv2d._conv_forward(self, input, weight, bias)
    455 if self.padding_mode != 'zeros':
    456     return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
    457                     weight, bias, self.stride,
    458                     _pair(0), self.dilation, self.groups)
--> 459 return F.conv2d(input, weight, bias, self.stride,
    460                 self.padding, self.dilation, self.groups)

RuntimeError: Calculated padded input size per channel: (7 x 7). Kernel size: (8 x 8). Kernel size can't be greater than actual input size

Code of Conduct

  • I agree to follow this project's Code of Conduct

Edit:
added mask dir in config which was missing

Edit 2:
Added content of transform_config_1.yaml below config section

Activity

  1. alexriedel1 commented on Jun 22, 2023

    @alexriedel1
    Contributor

    Whats inside c:\Users\benedict\dev\VespaUseCase\machine_learning\src\d01_config\transform_config_1.yaml ?

  2. simon-ami commented on Jun 22, 2023

    @simon-ami
    Author

    Whats inside c:\Users\benedict\dev\VespaUseCase\machine_learning\src\d01_config\transform_config_1.yaml ?

    i've added the contents of it below the config section

  3. simon-ami commented on Jun 22, 2023

    @simon-ami
    Author

    sorry i've copied the wrong one, i'll update it in a minute

    Edit: Done

  4. alexriedel1 commented on Jun 22, 2023

    @alexriedel1
    Contributor

    In the Configuration YAML you set image_size 256 and in the transform_config_1 you set CenterCrop to 224. Both should have the same value and both should be at least 256.
    Please also set normalization: null in the Configuration YAML and remove Normalization in the transform_config_1. Normalization is already done inside of EfficientAD by default

  5. simon-ami commented on Jun 22, 2023

    @simon-ami
    Author

    In the Configuration YAML you set image_size 256 and in the transform_config_1 you set CenterCrop to 224. Both should have the same value and both should be at least 256. Please also set normalization: null in the Configuration YAML and remove Normalization in the transform_config_1. Normalization is already done inside of EfficientAD by default

    I think you're right, i didn't remove the center cropping correctly. I'll try it out again with the correct size of 256x256.

    Will respond whether it works now later, but thanks already!

  6. simon-ami commented on Jun 22, 2023

    @simon-ami
    Author

    the previous issue seems resolved, but now i'm facing another issue:

    RuntimeError                              Traceback (most recent call last)
    Cell In[16], line 1
    ----> 1 train_results = trainer.fit(model=model, datamodule=datamodule)
    
    File [c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\pytorch_lightning\trainer\trainer.py:608](file:///C:/Users/benedict/anaconda3/envs/anomalib_env/lib/site-packages/pytorch_lightning/trainer/trainer.py:608), in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
        606 model = self._maybe_unwrap_optimized(model)
        607 self.strategy._lightning_module = model
    --> 608 call._call_and_handle_interrupt(
        609     self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
        610 )
    
    
    ....
    ....
    ....
    
    File [c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\anomalib\models\efficientad\lightning_model.py:217](file:///C:/Users/benedict/anaconda3/envs/anomalib_env/lib/site-packages/anomalib/models/efficientad/lightning_model.py:217), in EfficientAD.training_step(***failed resolving arguments***)
        214     self.imagenet_iterator = iter(self.imagenet_loader)
        215     batch_imagenet = next(self.imagenet_iterator)[0]["image"].to(self.device)
    --> 217 loss_st, loss_ae, loss_stae = self.model(batch=batch["image"], batch_imagenet=batch_imagenet)
        219 loss = loss_st + loss_ae + loss_stae
        220 self.log("train_st", loss_st.item(), on_epoch=True, prog_bar=True, logger=True)
    
    File [c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\torch\nn\modules\module.py:1130](file:///C:/Users/benedict/anaconda3/envs/anomalib_env/lib/site-packages/torch/nn/modules/module.py:1130), in Module._call_impl(self, *input, **kwargs)
       1126 # If we don't have any hooks, we want to skip the rest of the logic in
       1127 # this function, and just call forward.
       1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
       1129         or _global_forward_hooks or _global_forward_pre_hooks):
    -> 1130     return forward_call(*input, **kwargs)
       1131 # Do not call functions when jit is used
       1132 full_backward_hooks, non_full_backward_hooks = [], []
    
    File [c:\Users\benedict\anaconda3\envs\anomalib_env\lib\site-packages\anomalib\models\efficientad\torch_model.py:278](file:///C:/Users/benedict/anaconda3/envs/anomalib_env/lib/site-packages/anomalib/models/efficientad/torch_model.py:278), in EfficientADModel.forward(self, batch, batch_imagenet)
        274 distance_st = torch.pow(teacher_output - student_output[:, : self.teacher_out_channels, :, :], 2)
        276 if self.training:
        277     # Student loss
    --> 278     d_hard = torch.quantile(distance_st, 0.999)
        279     loss_hard = torch.mean(distance_st[distance_st >= d_hard])
        280     student_output_penalty = self.student(batch_imagenet)[:, : self.teacher_out_channels, :, :]
    
    RuntimeError: quantile() input tensor is too large
    

    In the Configuration YAML you set image_size 256 and in the transform_config_1 you set CenterCrop to 224. Both should have the same value and both should be at least 256. Please also set normalization: null in the Configuration YAML and remove Normalization in the transform_config_1. Normalization is already done inside of EfficientAD by default

    I think you're right, i didn't remove the center cropping correctly. I'll try it out again with the correct size of 256x256.

    Will respond whether it works now later, but thanks already!

  7. simon-ami commented on Jun 22, 2023

    @simon-ami
    Author

    In the Configuration YAML you set image_size 256 and in the transform_config_1 you set CenterCrop to 224. Both should have the same value and both should be at least 256. Please also set normalization: null in the Configuration YAML and remove Normalization in the transform_config_1. Normalization is already done inside of EfficientAD by default

    when i set dataset.normalization to null i get this error:
    ValueError: None is not a valid InputNormalizationMethod

  8. alexriedel1 commented on Jun 22, 2023

    @alexriedel1
    Contributor

    In the Configuration YAML you set image_size 256 and in the transform_config_1 you set CenterCrop to 224. Both should have the same value and both should be at least 256. Please also set normalization: null in the Configuration YAML and remove Normalization in the transform_config_1. Normalization is already done inside of EfficientAD by default

    when i set dataset.normalization to null i get this error: ValueError: None is not a valid InputNormalizationMethod

    sorry, you have to set it to none.

    RuntimeError: quantile() input tensor is too large is a known issue for larger datasets. This PR provides a fix for it #1143

  9. simon-ami commented on Jun 22, 2023

    @simon-ami
    Author

    In the Configuration YAML you set image_size 256 and in the transform_config_1 you set CenterCrop to 224. Both should have the same value and both should be at least 256. Please also set normalization: null in the Configuration YAML and remove Normalization in the transform_config_1. Normalization is already done inside of EfficientAD by default

    when i set dataset.normalization to null i get this error: ValueError: None is not a valid InputNormalizationMethod

    sorry, you have to set it to none.

    RuntimeError: quantile() input tensor is too large is a known issue for larger datasets. This PR provides a fix for it #1143

    i have applied the pr and set the config but i'm still facing the same issue sadly.

    Thanky you, anyways!

  10. alexriedel1 commented on Jun 22, 2023

    @alexriedel1
    Contributor

    You are still seeing RuntimeError: quantile() input tensor is too large after pulling the PR?

    Thats the important part, is it really in your local repo?
    https://github.com/openvinotoolkit/anomalib/blob/efc7e608d4481fb0d4af8507c6f1b249eb10453c/src/anomalib/models/efficientad/lightning_model.py#L178-L206

  11. d3tk commented on Jun 22, 2023

    @d3tk
    Contributor

    I get the error "RuntimeError: quantile() input tensor is too large" from the forward method of torch_model.py when I run it.

  12. simon-ami commented on Jun 22, 2023

    @simon-ami
    Author

    You are still seeing RuntimeError: quantile() input tensor is too large after pulling the PR?

    Thats the important part, is it really in your local repo?

    https://github.com/openvinotoolkit/anomalib/blob/efc7e608d4481fb0d4af8507c6f1b249eb10453c/src/anomalib/models/efficientad/lightning_model.py#L178-L206

    yeah exactly, but i applied the fixes manually. double checked for completeness, but will try once more with cloned repo to make sure it's applied correctly.

  13. simon-ami commented on Jun 22, 2023

    @simon-ami
    Author

    You are still seeing RuntimeError: quantile() input tensor is too large after pulling the PR?

    Thats the important part, is it really in your local repo?

    https://github.com/openvinotoolkit/anomalib/blob/efc7e608d4481fb0d4af8507c6f1b249eb10453c/src/anomalib/models/efficientad/lightning_model.py#L178-L206

    Tried again with the cloned and merged repo install but still the same error.

  14. alexriedel1 commented on Jun 22, 2023

    @alexriedel1
    Contributor

    Hm strange. Can you do some debugging like printing len(maps_flat) in that particular function and check its value?

  15. simon-ami commented on Jun 23, 2023

    @simon-ami
    Author

    Just tried some print statements that aren't printed, both in the respective function itself and right before it's call, but they aren't printed.
    Not sure tho if some ide setting might be blocking the output of the prints, used vs code and pycharm.
    Currently trying to inspect these values via debugger...

    Edit:
    It seems like the breakpoints set aren't accessed in pycharm, it doesn't stop when i set them like this (stopping on breakpoints outside my code is enabled and functional)
    image

  16. 5 remaining items

  17. simon-ami commented on Jun 23, 2023

    @simon-ami
    Author

    I installed it like you mentioned but i'm still getting the error RuntimeError: quantile() input tensor is too large.

    Currently trying the print statements and debugging again

  18. simon-ami commented on Jun 23, 2023

    @simon-ami
    Author

    The debugger still doesn't stop at my set breakpoints and the print statements aren't printed..

    C:\Users\benedict\anaconda3\envs\anomalib_edit_env\python.exe "C:/Program Files/JetBrains/PyCharm Community Edition 2022.3.3/plugins/python-ce/helpers/pydev/pydevd.py" --multiprocess --qt-support=auto --client 127.0.0.1 --port 52014 --file C:\Users\benedict\dev\VespaUseCase\machine_learning\src\09_EfficientAD.py 
    Connected to pydev debugger (build 231.8109.197)
    C:\Users\benedict\dev\VespaUseCase\machine_learning\src
    C:\Users\benedict\dev\VespaUseCase\machine_learning\src/../../../VespaUseCase
    To use wandb logger install it using `pip install wandb`
    C:\Users\benedict\dev\VespaUseCase\machine_learning\src\d05_visualization
    path to the python executable: C:\Users\benedict\anaconda3\envs\anomalib_edit_env\python.exe
    path to the pytorch installation: C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\__init__.py
    path to the anomalib installation: C:\Users\benedict\anomalib\src\anomalib\__init__.py
    anomalib version: 1.0.0dev
    torch: 2.0.1+cpu
    torch cuda: None
    torch cudnn: None
    is cuda available: False
    n available cuda devices: 0
    ===== IMPORTANT =====
    File 'C:\Users\benedict\dev\VespaUseCase\machine_learning\src\d01_config\config_anomalib_efficientad.yaml' already exists. Skipping copy.
    C:\Users\benedict\dev\VespaUseCase\machine_learning\src\d01_config\config_anomalib_efficientad.yaml
    path exists
    C:\Users\benedict\anomalib\src\anomalib\config\config.py:275: UserWarning: config.project.unique_dir is set to False. This does not ensure that your results will be written in an empty directory and you may overwrite files.
      warn(
    ===== IMPORTANT =====
    WARNING: Transform path 'C:\Users\benedict\dev\VespaUseCase\machine_learning\src\d01_config\transform_config_1.yaml' already exists. Using the existing configuration.
    C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\torchmetrics\utilities\prints.py:36: UserWarning: Metric `PrecisionRecallCurve` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
      warnings.warn(*args, **kwargs)
    GPU available: False, used: False
    TPU available: False, using: 0 TPU cores
    IPU available: False, using: 0 IPUs
    HPU available: False, using: 0 HPUs
    `Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
    `Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..
    `Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..
    `Trainer(limit_predict_batches=1.0)` was configured so 100% of the batches will be used..
    `Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
    C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\torchmetrics\utilities\prints.py:36: UserWarning: Metric `ROC` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
      warnings.warn(*args, **kwargs)
    C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\torchmetrics\utilities\prints.py:36: UserWarning: Metric `PrecisionRecallCurve` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
      warnings.warn(*args, **kwargs)
    C:\Users\benedict\anomalib\src\anomalib\utils\metrics\__init__.py:54: UserWarning: No metric with name MatthewsCorrcoef found in Anomalib metrics or TorchMetrics.
      warnings.warn(f"No metric with name {metric_name} found in Anomalib metrics or TorchMetrics.")
    
      | Name                  | Type                     | Params
    -------------------------------------------------------------------
    0 | image_threshold       | AnomalyScoreThreshold    | 0     
    1 | pixel_threshold       | AnomalyScoreThreshold    | 0     
    2 | model                 | EfficientADModel         | 8.1 M 
    3 | image_metrics         | AnomalibMetricCollection | 0     
    4 | pixel_metrics         | AnomalibMetricCollection | 0     
    5 | normalization_metrics | MinMax                   | 0     
    -------------------------------------------------------------------
    8.1 M     Trainable params
    0         Non-trainable params
    8.1 M     Total params
    32.235    Total estimated model params size (MB)
    C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:224: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
      rank_zero_warn(
    C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:224: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
      rank_zero_warn(
    Training: 0it [00:00, ?it/s]C:\Users\benedict\dev\VespaUseCase\machine_learning\src
    C:\Users\benedict\dev\VespaUseCase\machine_learning\src/../../../VespaUseCase
    To use wandb logger install it using `pip install wandb`
    C:\Users\benedict\dev\VespaUseCase\machine_learning\src\d05_visualization
    path to the python executable: C:\Users\benedict\anaconda3\envs\anomalib_edit_env\python.exe
    path to the pytorch installation: C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\__init__.py
    path to the anomalib installation: C:\Users\benedict\anomalib\src\anomalib\__init__.py
    anomalib version: 1.0.0dev
    torch: 2.0.1+cpu
    torch cuda: None
    torch cudnn: None
    is cuda available: False
    n available cuda devices: 0
    Calculate teacher channel mean: 100%|██████████| 131/131 [06:55<00:00,  3.17s/it]
    Calculate teacher channel std: 100%|██████████| 131/131 [00:04<00:00, 31.32it/s]
    Epoch 0:   0%|          | 0/136 [00:00<?, ?it/s] C:\Users\benedict\dev\VespaUseCase\machine_learning\src
    C:\Users\benedict\dev\VespaUseCase\machine_learning\src/../../../VespaUseCase
    To use wandb logger install it using `pip install wandb`
    C:\Users\benedict\dev\VespaUseCase\machine_learning\src\d05_visualization
    path to the python executable: C:\Users\benedict\anaconda3\envs\anomalib_edit_env\python.exe
    path to the pytorch installation: C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\__init__.py
    path to the anomalib installation: C:\Users\benedict\anomalib\src\anomalib\__init__.py
    anomalib version: 1.0.0dev
    torch: 2.0.1+cpu
    torch cuda: None
    torch cudnn: None
    is cuda available: False
    n available cuda devices: 0
    Traceback (most recent call last):
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\call.py", line 38, in _call_and_handle_interrupt
        return trainer_fn(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 650, in _fit_impl
        self._run(model, ckpt_path=self.ckpt_path)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1112, in _run
        results = self._run_stage()
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1191, in _run_stage
        self._run_train()
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1214, in _run_train
        self.fit_loop.run()
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
        self.advance(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 267, in advance
        self._outputs = self.epoch_loop.run(self._data_fetcher)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
        self.advance(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 213, in advance
        batch_output = self.batch_loop.run(kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
        self.advance(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 88, in advance
        outputs = self.optimizer_loop.run(optimizers, kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
        self.advance(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py", line 202, in advance
        result = self._run_optimization(kwargs, self._optimizers[self.optim_progress.optimizer_position])
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py", line 249, in _run_optimization
        self._optimizer_step(optimizer, opt_idx, kwargs.get("batch_idx", 0), closure)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py", line 370, in _optimizer_step
        self.trainer._call_lightning_module_hook(
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1356, in _call_lightning_module_hook
        output = fn(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\core\module.py", line 1754, in optimizer_step
        optimizer.step(closure=optimizer_closure)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\core\optimizer.py", line 169, in step
        step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 234, in optimizer_step
        return self.precision_plugin.optimizer_step(
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\plugins\precision\precision_plugin.py", line 119, in optimizer_step
        return optimizer.step(closure=closure, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\torch\optim\lr_scheduler.py", line 69, in wrapper
        return wrapped(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\torch\optim\optimizer.py", line 280, in wrapper
        out = func(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\torch\optim\optimizer.py", line 33, in _use_grad
        ret = func(self, *args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\torch\optim\adam.py", line 121, in step
        loss = closure()
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\plugins\precision\precision_plugin.py", line 105, in _wrap_closure
        closure_result = closure()
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py", line 149, in __call__
        self._result = self.closure(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py", line 135, in closure
        step_output = self._step_fn()
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\loops\optimization\optimizer_loop.py", line 419, in _training_step
        training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1494, in _call_strategy_hook
        output = fn(*args, **kwargs)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 378, in training_step
        return self.model.training_step(*args, **kwargs)
      File "C:\Users\benedict\anomalib\src\anomalib\models\efficientad\lightning_model.py", line 247, in training_step
        loss_st, loss_ae, loss_stae = self.model(batch=batch["image"], batch_imagenet=batch_imagenet)
      File "C:\Users\benedict\anaconda3\envs\anomalib_edit_env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Users\benedict\anomalib\src\anomalib\models\efficientad\torch_model.py", line 278, in forward
        d_hard = torch.quantile(distance_st, 0.999)
    RuntimeError: quantile() input tensor is too large
    Epoch 0:   0%|          | 0/136 [00:26<?, ?it/s]
    
  19. alexriedel1 commented on Jun 23, 2023

    @alexriedel1
    Contributor

    Oh sorry I just now realized that the issue is in torch_model.py and not in the normalization process in lightning_model.py. The fix you did only resolve the issue in the lightning_model.py. Can you show your training configs again?

    I guess the problem is now caused by your train_batch_size: 16 in the config which should be only 1

  20. simon-ami commented on Jun 23, 2023

    @simon-ami
    Author

    this is my config, with the batch_size fix already applied:

    dataset:
      abnormal_dir: C:/Users/benedict/dev/VespaUseCase/machine_learning/data/images/data_structure_anomalib_REAL_17_05/bad
      center_crop: null
      extensions: null
      format: folder
      image_size: 256  # mandatory
      mask_dir: C:/Users/benedict/dev/VespaUseCase/machine_learning/data/images/data_structure_anomalib_REAL_17_05/ground_truth_bad_resized
      name: vespa
      normal_dir: C:/Users/benedict/dev/VespaUseCase/machine_learning/data/images/data_structure_anomalib_REAL_17_05/good
      normal_test_dir: null
      normalization: none   # imagenet
      num_workers: 1
      path: C:/Users/benedict/dev/VespaUseCase/machine_learning/data/images/data_structure_anomalib_REAL_17_05
      task: segmentation
      test_batch_size: 1
      test_split_mode: from_dir
      test_split_ratio: 78.0/2160.0
      tiling:
        apply: false
        random_tile_count: 16
        remove_border_count: 0
        stride: null
        tile_size: null
        use_random_tiling: false
      train_batch_size: 1
      transform_config:
        eval: c:\Users\benedict\dev\VespaUseCase\machine_learning\src\d01_config\transform_config_1.yaml
        train: null
      val_split_mode: from_test
      val_split_ratio: 0.5
    logging:
      log_graph: false
      logger: tensorboard
    metrics:
      image:
      - F1Score
      - AUROC
      - AUPR
      - MatthewsCorrcoef
      pixel:
      - F1Score
      - AUROC
      - AUPR
      - MatthewsCorrcoef
      threshold:
        manual_image: null
        manual_pixel: null
        method: adaptive
    model:
      lr: 0.0001
      model_size: small
      name: efficientad
      normalization_method: min_max
      padding: false
      teacher_out_channels: 384
      weight_decay: 1.0e-05
    optimization:
      export_mode: onnx
    project:
      path: C:/Users/benedict/dev/VespaUseCase/machine_learning/data/results/efficientad/vespa/run
      seed: 42
    trainer:
      accelerator: auto
      accumulate_grad_batches: 1
      auto_lr_find: false
      auto_scale_batch_size: false
      benchmark: false
      check_val_every_n_epoch: 1
      default_root_dir: null
      detect_anomaly: false
      deterministic: false
      devices: 1
      enable_checkpointing: true
      enable_model_summary: true
      enable_progress_bar: true
      fast_dev_run: false
      gradient_clip_algorithm: norm
      gradient_clip_val: 0
      limit_predict_batches: 1.0
      limit_test_batches: 1.0
      limit_train_batches: 1.0
      limit_val_batches: 1.0
      log_every_n_steps: 5
      max_epochs: 150
      max_steps: 70000
      max_time: null
      min_epochs: null
      min_steps: null
      move_metrics_to_cpu: false
      multiple_trainloader_mode: max_size_cycle
      num_nodes: 1
      num_sanity_val_steps: 0
      overfit_batches: 0.0
      plugins: null
      precision: 32
      profiler: null
      reload_dataloaders_every_n_epochs: 0
      replace_sampler_ddp: true
      strategy: null
      sync_batchnorm: false
      track_grad_norm: -1
      val_check_interval: 1.0
    visualization:
      image_save_path: null
      log_images: false
      mode: full
      save_images: true
      show_images: false
    

    currently running and waiting for results

  21. simon-ami commented on Jun 23, 2023

    @simon-ami
    Author

    Epoch 0: 1%| | 22/2160 [01:06<1:47:10, 3.01s/it, loss=601, v_num=30, train_st_step=57.90, train_ae_step=0.881, train_stae_step=4.980, train_loss_step=63.80]

    Changing the

    train_batch_size = 1
    test_batch_size = 1
    

    seems to have fixed the issue.

    Thank you very much for helping me out here!

    Btw, greetings from berlin :)

  22. vermouthber commented on Jun 27, 2023

    @vermouthber

    Can the image size be set to something else?not 256?

  23. d3tk commented on Jun 28, 2023

    @d3tk
    Contributor

    Can the image size be set to something else?not 256?

    I've run this model with 256x256 and 640x640 and had no issues.

  24. vermouthber commented on Jun 29, 2023

    @vermouthber

    I have a question that when I export to openvino, the metadata.json show :
    "image_threshold": 1.9160085916519165,
    "pixel_threshold": 1.4764410257339478,
    "min": -0.06203848123550415,
    "max": 38.532230377197266
    Is this normal?
    I want to use openvino model for inference by C++, but don't know how to use the pixel/image threshold and min/max value.

  25. alexriedel1 commented on Jun 30, 2023

    @alexriedel1
    Contributor

    The model outputs smaller and larger numbers for normal and anomalous images. with the min and max values you can normalize these numbers. With the threshold you center the normalized numbers so that 0.5 becomes the boundary for normal/anomalous

    https://github.com/openvinotoolkit/anomalib/blob/main/src/anomalib/post_processing/normalization/min_max.py#L13

  26. simon-ami commented on Jun 30, 2023

    @simon-ami
    Author

    I'm interested in this too.

    I'm getting noisy anomaly maps for normal images but great looking ones for anomalous ones.
    Is it possible to make the model "ignore" lower initial anomaly scores by changing the normalization's min and max?
    And if so, could you give a short example of how to change them during inference?

    Thanks!

  27. alexriedel1 commented on Jul 1, 2023

    @alexriedel1
    Contributor

    What do you mean by "noisy" and "great"? Are you interested in anomaly segmentation or just anomaly image classification? Do you train the model with segmenation ground truths of the anomalies?

  28. simon-ami commented on Jul 1, 2023

    @simon-ami
    Author

    By "noisy" I mean that for normal images it seems to be much more sensible to light distractions/ changes in contrast etc.
    In contrast, for anomalous images I'm getting super clean "heat spots" where actual anomalies are.

    (Btw., I'm currently not talking about results from efficient ad, but I guess this is applicable to all models (the normalization tweak).)

    I'm kinda interested in segmentation, I'm training it with ground truth masks on segmentation task. But I find the heat map to be visually more appealing (for demonstration purposes, under the presumption that the "noise" could be eliminated), that's why I would like to fix it, if possible.

  29. alexriedel1 commented on Jul 3, 2023

    @alexriedel1
    Contributor

    The colors of the heat map are coming from here: https://github.com/openvinotoolkit/anomalib/blob/17efdb56fab4a94a74bb9ba9f85d2234fda52800/src/anomalib/post_processing/post_process.py#L85-L103

    If you use the task segmentation (which I'm not 100% sure because in your initial config.yaml you define task: classification) the heatmap should be normalized already to you anomalous samples. So please make sure to set task to segmentation. Otherwise you can play around with the colorMap function to get better results..

  30. locked and limited conversation to collaborators on Jul 21, 2023
  31. converted this issue into a discussion #1208 on Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions