Unable to train, getting the following error.
/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:557: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/kaggle/working/vall-e/vall_e/train.py", line 128, in
main()
File "/kaggle/working/vall-e/vall_e/train.py", line 119, in main
trainer.train(
File "/kaggle/working/vall-e/vall_e/utils/trainer.py", line 125, in train
engines = engines_loader()
File "/kaggle/working/vall-e/vall_e/train.py", line 21, in load_engines
model=trainer.Engine(
File "/kaggle/working/vall-e/vall_e/utils/engines.py", line 22, in init
super().init(None, *args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 240, in init
self._do_sanity_check()
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1039, in _do_sanity_check
if self.fp16_enabled() and not get_accelerator().is_fp16_supported():
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 796, in fp16_enabled
return self._config.fp16_enabled
AttributeError: 'NoneType' object has no attribute 'fp16_enabled'
[2024-05-23 10:34:39,680] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[WARNING] NVIDIA Inference is only supported on Ampere and newer architectures
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
[WARNING] please install triton==1.0.0 if you want to use sparse attention
1222it [00:00, 62626.79it/s]
/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:557: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/kaggle/working/vall-e/vall_e/train.py", line 128, in
main()
File "/kaggle/working/vall-e/vall_e/train.py", line 119, in main
trainer.train(
File "/kaggle/working/vall-e/vall_e/utils/trainer.py", line 125, in train
engines = engines_loader()
File "/kaggle/working/vall-e/vall_e/train.py", line 21, in load_engines
model=trainer.Engine(
File "/kaggle/working/vall-e/vall_e/utils/engines.py", line 22, in init
super().init(None, *args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 240, in init
self._do_sanity_check()
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1039, in _do_sanity_check
if self.fp16_enabled() and not get_accelerator().is_fp16_supported():
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 796, in fp16_enabled
return self._config.fp16_enabled
AttributeError: 'NoneType' object has no attribute 'fp16_enabled'
Unable to train, getting the following error.
/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:557: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/kaggle/working/vall-e/vall_e/train.py", line 128, in
main()
File "/kaggle/working/vall-e/vall_e/train.py", line 119, in main
trainer.train(
File "/kaggle/working/vall-e/vall_e/utils/trainer.py", line 125, in train
engines = engines_loader()
File "/kaggle/working/vall-e/vall_e/train.py", line 21, in load_engines
model=trainer.Engine(
File "/kaggle/working/vall-e/vall_e/utils/engines.py", line 22, in init
super().init(None, *args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 240, in init
self._do_sanity_check()
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1039, in _do_sanity_check
if self.fp16_enabled() and not get_accelerator().is_fp16_supported():
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 796, in fp16_enabled
return self._config.fp16_enabled
AttributeError: 'NoneType' object has no attribute 'fp16_enabled'
[2024-05-23 10:34:39,680] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[WARNING] NVIDIA Inference is only supported on Ampere and newer architectures
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
[WARNING] please install triton==1.0.0 if you want to use sparse attention
1222it [00:00, 62626.79it/s]
/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:557: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/kaggle/working/vall-e/vall_e/train.py", line 128, in
main()
File "/kaggle/working/vall-e/vall_e/train.py", line 119, in main
trainer.train(
File "/kaggle/working/vall-e/vall_e/utils/trainer.py", line 125, in train
engines = engines_loader()
File "/kaggle/working/vall-e/vall_e/train.py", line 21, in load_engines
model=trainer.Engine(
File "/kaggle/working/vall-e/vall_e/utils/engines.py", line 22, in init
super().init(None, *args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 240, in init
self._do_sanity_check()
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1039, in _do_sanity_check
if self.fp16_enabled() and not get_accelerator().is_fp16_supported():
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 796, in fp16_enabled
return self._config.fp16_enabled
AttributeError: 'NoneType' object has no attribute 'fp16_enabled'