[feat] SID: add offline codebook collision-prevention tool

[WhiteSwan1]

Summary

Adds tzrec/tools/sid/collision_prevention.py, an offline tool that bounds how many items may share the same Semantic ID (SID) and relocates the overflow — producing a capacity-safe SID map for downstream generative retrieval. It
complements the sid_quality_report tool (#596), which measures collisions: this one prevents them.

Given a SID table (the codes column from a SID model's tzrec.predict output), it caps each SID bucket at --max_items_per_codebook and reassigns over-capacity items to a free code in the last layer only, keeping every prefix layer — so an item is only ever nudged within its own SID neighborhood.

What it does

• Two reassignment strategies
  • candidate (default): walk each item's model-provided top-k candidate SIDs (candidate_codes), best-first, and take the first with a free slot.
  • random: deterministic pseudo-random last-layer draws (SplitMix64).
• Deterministic and order-independent: tie-breaks use a seeded hash of item_id, so the result is reproducible regardless of input row order or read batch size.
• Vectorized: numpy/pyarrow throughout; the only Python loop is the greedy placement over the overflow items. Chunked writes respect Arrow's 2^31 list-offset limit. --rate_only computes and logs metrics without writing.
• TorchEasyRec-native I/O: reads/writes via create_reader / create_writer (CSV / Parquet / ODPS). Single-process, CPU-only — launch with python -m (no torchrun / process group).

Inputs and outputs

Input — one table:

• item_id
• codes — list<int> of length n_layers
• candidate_codes — list<list<int>> (uniform top-k), for the candidate strategy

Outputs:

• Item map — item_id, origin_codebook, codebook, index — where index is the 1..cap slot within the item's final SID bucket (the de-duplication digit that makes each item's (codebook, index) unique for downstream generative retrieval).
• Grouped SID -> item-ID tables, for both the original and the resolved SIDs.

Test plan

• collision_resolution_test.py covers the pure NumPy core: golden resolution, empty / single-layer / no-overflow inputs, deterministic random draws, chunk-boundary bucket ranking, and range/shape validation.
• collision_prevention_test.py covers the end-to-end runner over Parquet, CSV, and (mocked) ODPS writers: within-band reassignment, fallback behavior, cross-batch candidate alignment, group/slot ordering, determinism & order-independence, Arrow offset-limit chunking, item-id type preservation, and --rate_only.

All green; ruff (lint + format) and pyre clean.

[github-actions]

Review summary

Reviewed statically across code quality, performance, tests, docs, and input-safety. Overall this is a well-engineered PR: clean separation between the I/O runner (collision_prevention.py) and the pure-NumPy core (collision_resolution.py), thorough Google-style docstrings that accurately match the code, deliberate memory management (chunked writes, del of large arrays, Arrow offset-limit guards), and a genuinely strong test suite (golden values, determinism/order-independence, CSV/Parquet/mocked-ODPS backends, writer-cleanup-on-failure, chunk boundaries). The sid_model.py flatten(1) change is consistent with the updated tests and no stale 3D consumers remain.

I left inline comments on the findings worth addressing. The theme running through the top ones is input validation — the core trusts that codes are in range, unique, and rectangular, and several realistic malformed inputs corrupt the output silently instead of erroring:

• codes not range-validated against --codebook → a mismatched codebook silently merges buckets (highest priority; candidates are checked, base codes aren't).
• int64 overflow of the prefix key for deep/large codebooks — the sibling sid_quality_report guards this and this tool dropped it.
• slot_counts dict scales with total SIDs, not overflow count — the main performance concern at the stated millions-of-items scale.
• Duplicate item_id not detected — breaks the advertised order-independence and is handled inconsistently.

Lower-priority follow-ups (not inlined)

• Ragged codes can pass silently: _codes_matrix guards only divisibility, so a ragged-but-divisible list column (e.g. rows of width 2 and 4 with 3 rows → reshaped (3,2)) mis-groups with no error. Consider requiring a single list width / fixed_size_list.
• Cryptic errors on bad field names / codebook: a typo in --item_id_field/--code_field surfaces as a bare KeyError deep in the stream (the sibling tool validates against reader.schema.names), and --codebook "256,,256" silently parses to (256, 256) (sibling rejects empty tokens).
• Test gaps: (a) the --rate_only over-capacity stats branch (collect_grouping=False with unplaceable overflow) is never exercised — yet those are the exact numbers --rate_only exists to report; (b) no test uses ≥3 layers, even though the canonical config in the module docstring is 3-layer (256,256,256), so the middle-layer band fold is uncovered.
• Docs: like the sibling sid_quality_report tool, this has no user-facing doc coverage; a short "SID collision post-processing" note in the SID model docs (which already discuss collision rate) would help discoverability. Consistent with existing precedent, so non-blocking.

[github-actions]

Static review summary

Static review only; no tests or builds were run.

The I/O runner and NumPy core are cleanly separated, the deterministic grouping logic is well documented, and the existing tests cover the main resolution and output paths well.

I left five inline comments on the noteworthy remaining issues:

• reject input/output overlap before a writer can overwrite source data;
• enforce the documented single-process execution invariant;
• reject floating-point SID lists instead of silently truncating them;
• remove a redundant full-table sort from prefix-band construction;
• clarify that unplaceable items can leave buckets over capacity.

I omitted lower-impact suggestions and findings already covered by the earlier review.