HuggingFace models fail on Python 3.13: NameError from exec/eval in download_model (PEP 667)

[amd-nishsoni]

Summary

Every hf_* model fails to instantiate on Python 3.13 with
NameError: name '<XConfig>' is not defined, raised from
torchbenchmark/util/framework/huggingface/basic_configs.py::download_model.
python install.py fails outright, so the models cannot even be installed.

Python 3.12 and earlier are unaffected. I understand utils/python_utils.py
currently targets 3.10-3.12, so this is filed as a Python 3.13 support blocker
rather than a regression on a supported config.

Repro

conda create -n tb_py313 python=3.13 -y && conda activate tb_py313
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
git clone https://github.com/pytorch/benchmark.git && cd benchmark
python install.py --torch hf_Bert_large   # already fails here

python test.py -k "test_hf_Bert_large_train_cpu" fails with the same
NameError if you force past the install.

Observed

$ python install.py --torch hf_Bert_large
checking packages numpy, torch are installed, generating constaints...OK
running setup for .../torchbenchmark/models/hf_Bert_large...FAIL

Traceback (most recent call last):
  File ".../torchbenchmark/models/hf_Bert_large/install.py", line 13, in <module>
    cache_model(model_name)
    ~~~~~~~~~~~^^^^^^^^^^^^
  File ".../torchbenchmark/util/framework/huggingface/patch_hf.py", line 15, in cache_model
    download_model(name)
    ~~~~~~~~~~~~~~^^^^^^
  File ".../torchbenchmark/util/framework/huggingface/basic_configs.py", line 304, in download_model
    config = eval(HUGGINGFACE_MODELS[model_name][2])
  File "<string>", line 1, in <module>
NameError: name 'BertConfig' is not defined

RuntimeError: Failed to complete setup

Same failure via python test.py -k "test_hf_Bert_large_train_cpu"

torchbenchmark._components._impl.workers.subprocess_rpc.ChildTraceException: Traceback (most recent call last):
  File ".../torchbenchmark/_components/_impl/workers/subprocess_rpc.py", line 510, in _run_block
    exec(compile(cmd, "<subprocess-worker>", "exec"), globals_dict)  # noqa: P204
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<subprocess-worker>", line 39, in <module>
  File "<subprocess-worker>", line 11, in _run_in_worker_f
  File ".../torchbenchmark/util/model.py", line 43, in __call__
    obj = type.__call__(cls, *args, **kwargs)
  File ".../torchbenchmark/models/hf_Bert_large/__init__.py", line 11, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
      name="hf_Bert_large",
      ^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
      extra_args=extra_args,
      ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File ".../torchbenchmark/util/framework/huggingface/model_factory.py", line 51, in __init__
    self.model_cls, self.model = download_model(name)
                             ~~~~~~~~~~~~~~^^^^^^
  File ".../torchbenchmark/util/framework/huggingface/basic_configs.py", line 304, in download_model
    config = eval(HUGGINGFACE_MODELS[model_name][2])
  File "<string>", line 1, in <module>
NameError: name 'BertConfig' is not defined

Root cause

config_cls_name = _extract_config_cls_name(HUGGINGFACE_MODELS[model_name][2])
exec(f"from transformers import {config_cls_name}")
config = eval(HUGGINGFACE_MODELS[model_name][2])

exec() with no explicit namespace binds the imported class into the snapshot
of download_model's locals. Through 3.12 that snapshot was a cached dict that
the following eval() also read from, so the name was visible. Python 3.13
implements PEP 667, where locals() in an optimized scope returns a fresh
independent snapshot per call, so the binding is discarded before eval() runs.

Minimal repro, no torch or torchbench required:

def download_model():
    exec("from transformers import BertConfig")
    return eval("BertConfig()")

download_model()   # OK on 3.12, NameError on 3.13

This affects all entries in HUGGINGFACE_MODELS, including the AutoConfig
ones, since the config class is always exec-imported.

Suggested fix

Pass an explicit namespace to eval instead of relying on exec leaking into
locals. Works identically on 3.9-3.13:

config_cls_name = _extract_config_cls_name(HUGGINGFACE_MODELS[model_name][2])
eval_globals = {
    "transformers": transformers,
    config_cls_name: getattr(transformers, config_cls_name),
}
config = eval(HUGGINGFACE_MODELS[model_name][2], eval_globals)

The similar exec at extended_configs.py:41 is at module scope and is not
affected.

Happy to send a PR if that fix looks right.

Environment

• pytorch/benchmark main @ fcfbbc8 (2026-08-01), working tree clean
• Python 3.13.14 -> fails, Python 3.12.13 -> passes
• Same commit, same torch 2.13.0+cpu, same transformers 4.57.3 in both envs
• CPU-only, Linux x86_64 (RHEL 9.5, kernel 5.14)