Description
The Python backend unpacks each model's custom execution environment (EXECUTION_ENV_PATH, the conda-pack archive) into a temporary directory (/tmp/python_env_<random>/<N>).
When a model is unloaded — or reloaded pointing at a different environment archive path — the previously extracted environment directory is never deleted. There is no cleanup of unpacked environments on model unload. Every new model version that ships a new environment archive path creates a new entry in the map and a new /tmp/python_env_*/X directory, and the old ones accumulate indefinitely.
In our deployment, each new model revision is delivered under a unique path, so the archive path changes on every update. Models are reloaded frequently, which causes the temp directory to grow without bound until the disk fills up. See also the related discussion: #6629
Triton Information
Triton 25.06 (NGC nvcr.io/nvidia/tritonserver, used via a custom base image built on top of it).
To Reproduce
Minimal steps to reproduce:
- Start Triton with
--model-control-mode=explicit --load-model=* and a model X whose config.pbtxt sets EXECUTION_ENV_PATH to /path/to/env1 (a conda-pack archive).
- Verify the environment was extracted to
/tmp/python_env_<random_string>/0.
- Change the model's
EXECUTION_ENV_PATH to a different archive /path/to/env2.
- Reload model
X:
- update the modification time of the model files:
touch config.pbtxt, touch 1/model.py
curl -X POST localhost:8000/v2/repository/models/X/load
- Observe that a new env appears at
/tmp/python_env_<random_string>/1, while the old /tmp/python_env_<random_string>/0 is not removed.
Repeating steps 3–4 across model versions leaves one orphaned /tmp/python_env_*/x directory per reload, eventually exhausting disk space.
Expected behavior
When a model is unloaded (or reloaded with a different EXECUTION_ENV_PATH), the Python backend should remove the extracted execution environment directory that is no longer in use, so disk usage stays bounded across reloads.
Solution
triton-inference-server/python_backend#442
Description
The Python backend unpacks each model's custom execution environment (
EXECUTION_ENV_PATH, the conda-pack archive) into a temporary directory (/tmp/python_env_<random>/<N>).When a model is unloaded — or reloaded pointing at a different environment archive path — the previously extracted environment directory is never deleted. There is no cleanup of unpacked environments on model unload. Every new model version that ships a new environment archive path creates a new entry in the map and a new
/tmp/python_env_*/Xdirectory, and the old ones accumulate indefinitely.In our deployment, each new model revision is delivered under a unique path, so the archive path changes on every update. Models are reloaded frequently, which causes the temp directory to grow without bound until the disk fills up. See also the related discussion: #6629
Triton Information
Triton 25.06 (NGC nvcr.io/nvidia/tritonserver, used via a custom base image built on top of it).
To Reproduce
Minimal steps to reproduce:
--model-control-mode=explicit --load-model=*and a modelXwhoseconfig.pbtxtsetsEXECUTION_ENV_PATHto/path/to/env1(a conda-pack archive)./tmp/python_env_<random_string>/0.EXECUTION_ENV_PATHto a different archive/path/to/env2.X:touch config.pbtxt,touch 1/model.pycurl -X POST localhost:8000/v2/repository/models/X/load/tmp/python_env_<random_string>/1, while the old/tmp/python_env_<random_string>/0is not removed.Repeating steps 3–4 across model versions leaves one orphaned
/tmp/python_env_*/xdirectory per reload, eventually exhausting disk space.Expected behavior
When a model is unloaded (or reloaded with a different
EXECUTION_ENV_PATH), the Python backend should remove the extracted execution environment directory that is no longer in use, so disk usage stays bounded across reloads.Solution
triton-inference-server/python_backend#442