README.md

Data processing

These scripts build the data and embeddings needed before training any models.

Required raw inputs

Every script derives its paths from a single dataset name, defaulting to Beauty_and_Personal_Care. Override it for the whole pipeline with the UNIREC_DATASET environment variable (e.g. export UNIREC_DATASET=Baby_Products). With the default, the scripts read three raw files:

• data_rec/temp/meta_Beauty_and_Personal_Care.jsonl – item metadata (one JSON object per line, including parent_asin).
• data_rec/temp/Beauty_and_Personal_Care.jsonl – user reviews (one JSON object per line, including user_id and parent_asin). Only needed if you use reviews.
• data_rec/Amazon_Beauty_and_Personal_Care.inter – tab-separated interaction file. The first line is a header and is skipped; the loader reads the first four columns as user_id, item_id, rating, timestamp.

These files are available from the UniRec HuggingFace dataset under amazon_beauty/. Everything under data_rec/dict/, data_rec/data/, and data_rec/embeddings/ is generated by the scripts below.

Scripts

• create_item_dict.py

  • Reads data_rec/temp/meta_Beauty_and_Personal_Care.jsonl; writes data_rec/dict/Beauty_and_Personal_Care_item_dict.json.
  • Builds a JSON dictionary mapping parent_asin → item metadata (title, image URLs, etc.).
  • Run once per dataset; other scripts read this file to know the fields available for each item.

• create_review_dict.py

  • Reads data_rec/temp/Beauty_and_Personal_Care.jsonl; writes data_rec/dict/Beauty_and_Personal_Care_review_dict.json.
  • Builds a dictionary keyed by user_id|parent_asin mapping to review text (and metadata).
  • Used by review embedding and item encoders that incorporate review text.

• create_triplet_dict.py

  • Reads data_rec/dict/Beauty_and_Personal_Care_item_dict.json; writes data_rec/dict/Beauty_and_Personal_Care_item_triplet_dict.json.
  • Filters item metadata down to the fields declared in config/triplet_config.yaml.
  • Used by item encoders and Item Q-Former training scripts that rely on triplet-based objectives.

• process_rec_new_user.py / process_rec_old_user.py

  • Read data_rec/Amazon_Beauty_and_Personal_Care.inter; write train/test JSON under data_rec/data/....
  • Generate train/test JSON files for recommendation:
    • New users vs. existing users.
    • Include user history, ground-truth target items, and candidate sets.
  • Outputs (for example) Amazon_Beauty_and_Personal_Care_all_train_LRanker.json and ..._20_train.json.

• item_embedding_clip.py

  • Computes CLIP embeddings for each item using its text and/or image fields.
  • Inputs: item dict + train/test data (to know which item_ids appear).
  • Outputs: a JSON file under data_rec/embeddings/... mapping item_id → CLIP embedding.

• review_embedding_clip.py

  • Computes CLIP embeddings over review text (if you incorporate reviews into item representation).
  • Inputs: review dict + item dict + train data.
  • Outputs: JSON file of review embeddings and/or aggregated item-level review features.

• qformer_inference.py

  • Loads a trained Item Q-Former checkpoint and runs fast inference over cached item field embeddings.
  • Outputs item-level query tokens (for example, num_query_tokens × hidden_dim) for each item_id.

• generate_all_item_embeddings.py

  • Orchestrates batch generation of item query tokens for all items:
    • Loads the item dict.
    • Uses qformer_inference.py to run the Item Q-Former in batches.
    • Saves a JSON / pickle of item_id → query tokens.
  • Run this after training the Item Q-Former; the resulting query-token cache is consumed by user/Qwen training.