Hateful Memes Detection — EE-559 EPFL

Authors: Zacharie Bourlard, Théodore Decaux, Elias Bourgon

Binary classification of hateful memes (text + image) using the Hateful Memes dataset from Facebook AI.

---

Models

We explore three families of models:

Family	Description
Text baseline	Five encoders compared: BERT, RoBERTa, Twitter-RoBERTa-hate, MPNet, RoBERTa-large — each with a classification head
Image baseline	ResNet-50 (CNN) and ViT-B/16, both ImageNet-pretrained, with a two-layer MLP head and weighted cross-entropy
CLIP (main model)	Personalized CLIP architecture (ViT-L/14) with a custom multimodal interaction module
Late Fusion	MLP trained on top of frozen text + image embeddings

We do not include model checkpoints in this repository. Pre-trained backbone weights are downloaded automatically from HuggingFace at runtime.

---

Dataset

We use the Hateful Memes dataset (Kiela et al., 2020). It is not included in this repository. Place the data as follows:

data/
├── img/                  # meme images
├── train_split.jsonl     # 7 650 training samples
├── val_split.jsonl       # 850 validation samples
├── dev.jsonl             # 500 held-out test samples
└── test.jsonl            # unlabeled (challenge test set)

Each JSONL line: {"id": 42, "img": "img/42.png", "text": "...", "label": 0|1}

---

Project structure

deep-learning/
├── models/                        # Shared model and dataset classes
│   ├── text.py                    # TextClassifier, HatefulMemesTextDataset
│   ├── image.py                   # ImageClassifier, CNNClassifier, HatefulMemesImageDataset
│   ├── clip.py                    # CLIPHatefulMemesClassifier, CLIPTrainDataset, CLIPEvalDataset
│   └── fusion.py                  # LateFusionMLP, FusionDataset, load_embeddings
│
├── train/                         # Training scripts
│   ├── text_baseline.py
│   ├── image_baseline.py          # --backbone_type vit|cnn
│   ├── clip_hateful_memes.py
│   └── late_fusion.py
│
├── eval/                          # Evaluation and threshold tuning
│   ├── threshold_tuning.py        # Threshold sweep for Late Fusion
│   ├── threshold_tuning_clip.py   # Threshold sweep for CLIP
│   ├── predict_val_image.py
│   └── eval_metrics.py
│
├── plots/                         # Figure generation
│   ├── generate_results_plots.py
│   └── threshold_plots.py
│
└── scripts/                       # RunAI cluster job submission
    ├── submit_text.sh
    ├── submit_image.sh
    └── submit_clip.sh

---

Installation

pip install -r requirements.txt

Main dependencies: torch, transformers, torchvision, scikit-learn, Pillow, matplotlib

---

Usage

All scripts are run from the repository root. The models/ package is imported via sys.path automatically.

Text baseline

python3 train/text_baseline.py \
  --data_dir /path/to/data \
  --results_dir /path/to/results/text \
  --model_name cardiffnlp/twitter-roberta-base-hate \
  --epochs 5

Image baseline

# ViT
python3 train/image_baseline.py \
  --data_dir /path/to/data \
  --results_dir /path/to/results/image \
  --backbone_type vit \
  --epochs 5

# CNN
python3 train/image_baseline.py \
  --data_dir /path/to/data \
  --results_dir /path/to/results/image \
  --backbone_type cnn \
  --cnn_backbone resnet50 \
  --epochs 5

CLIP

python3 train/clip_hateful_memes.py \
  --data_dir /path/to/data \
  --results_dir /path/to/results/clip \
  --model_name openai/clip-vit-large-patch14 \
  --epochs 15

Late Fusion

Requires text_embeddings.pt and image_embeddings.pt saved by the text and image training scripts.

python3 train/late_fusion.py \
  --data_dir /path/to/data \
  --text_emb /path/to/results/text/text_embeddings.pt \
  --image_emb /path/to/results/image/image_embeddings.pt \
  --results_dir /path/to/results/fusion \
  --fusion concat_hadamard \
  --epochs 30

Threshold tuning

# CLIP
python3 eval/threshold_tuning_clip.py \
  --checkpoint /path/to/results/clip/best_model.pt \
  --data_dir /path/to/data \
  --results_dir /path/to/results/thresholds

# Late Fusion
python3 eval/threshold_tuning.py \
  --checkpoint /path/to/results/fusion/best_model.pt \
  --text_emb /path/to/text_embeddings.pt \
  --image_emb /path/to/image_embeddings.pt \
  --data_dir /path/to/data \
  --results_dir /path/to/results/thresholds

Sweeps τ ∈ [0.05, 0.95] and reports optimal thresholds for F1, F2, F3, and G-mean on both balanced and imbalanced (10% harmful) test sets.

---

RunAI (EPFL RCP cluster)

bash scripts/submit_text.sh [EPOCHS]
bash scripts/submit_image.sh [EPOCHS]
bash scripts/submit_clip.sh [EPOCHS] [MODEL_NAME]

Results are written to /scratch/results/. Logs: runai logs <job-name> -f