I've been running the benchmark against the pinned repos and went through some of the misses by hand. Four gold labels don't hold up once you open the files they point at:
model2vec — "how vocabulary is pruned during distillation"
The primary gold is model2vec/distill/utils.py, but that file contains a single function, select_optimal_device() — it picks cuda/mps/cpu and has nothing to do with vocabulary or pruning. The actual pruning happens via prune_added_tokens(), called from model2vec/tokenizer/tokenizer.py:71 and model2vec/distill/distillation.py:81. The latter is already the secondary gold here, so the fix might just be dropping distill/utils.py (or swapping in tokenizer/tokenizer.py).
model2vec — "tokenizer construction and vocabulary building"
Same file again as secondary gold. Still just device selection, no tokenizer or vocabulary content.
pydantic — "custom field and model validators"
The secondary gold pydantic/class_validators.py is a 5-line V1 migration shim (__getattr__ = getattr_migration(__name__)). If the intent was to credit the deprecated V1-style validators, that file is pydantic/deprecated/class_validators.py.
aiohttp — "WebSocket client implementation"
The secondary gold aiohttp/_websocket/reader.py is a 31-line import switch that picks the C or Python reader at import time. The implementation it re-exports lives in aiohttp/_websocket/reader_py.py.
All four checked at the SHAs pinned in repos.json and still present in the annotations on main. Happy to put up a PR with whichever corrections you'd prefer — dropping the entries vs. redirecting them changes scores a little either way, so I didn't want to guess at intent.
I've been running the benchmark against the pinned repos and went through some of the misses by hand. Four gold labels don't hold up once you open the files they point at:
model2vec — "how vocabulary is pruned during distillation"
The primary gold is
model2vec/distill/utils.py, but that file contains a single function,select_optimal_device()— it picks cuda/mps/cpu and has nothing to do with vocabulary or pruning. The actual pruning happens viaprune_added_tokens(), called frommodel2vec/tokenizer/tokenizer.py:71andmodel2vec/distill/distillation.py:81. The latter is already the secondary gold here, so the fix might just be droppingdistill/utils.py(or swapping intokenizer/tokenizer.py).model2vec — "tokenizer construction and vocabulary building"
Same file again as secondary gold. Still just device selection, no tokenizer or vocabulary content.
pydantic — "custom field and model validators"
The secondary gold
pydantic/class_validators.pyis a 5-line V1 migration shim (__getattr__ = getattr_migration(__name__)). If the intent was to credit the deprecated V1-style validators, that file ispydantic/deprecated/class_validators.py.aiohttp — "WebSocket client implementation"
The secondary gold
aiohttp/_websocket/reader.pyis a 31-line import switch that picks the C or Python reader at import time. The implementation it re-exports lives inaiohttp/_websocket/reader_py.py.All four checked at the SHAs pinned in repos.json and still present in the annotations on main. Happy to put up a PR with whichever corrections you'd prefer — dropping the entries vs. redirecting them changes scores a little either way, so I didn't want to guess at intent.