PROSKILL

Segment-Level Skill Assessment in Procedural Videos

Authors

Michele Mazzamuto¹², Daniele Di Mauro², Gianpiero Francesca³, Giovanni Maria Farinella¹, Antonino Furnari¹

Affiliations

Co-first authors
¹ IPLAB, University of Catania
² Next Vision s.r.l., Italy
³ Toyota Motor Europe, Belgium

Directory Structure

data/

Contains the annotation data and metadata for five procedural video datasets:

• taxonomy.yaml: A YAML file mapping each dataset to its list of procedural tasks and skill categories
• assembly101/: Assembly101 dataset containing assembly and disassembly task annotations
• egoexo4d/: Ego-Exo4D dataset containing bike repair tasks (chain cleaning/lubrication, wheel installation/removal, flat tire fixing)
• epic-tent/: EPIC-Tent dataset containing tent assembly annotations
• ikea/: IKEA furniture assembly dataset with annotations for various furniture pieces (TV bench, side table, coffee table, shelf drawer)
• meccano/: Meccano toy assembly dataset

Each dataset subdirectory contains:

• taxonomy.json: Task-specific taxonomy and skill definitions
• train.txt, val.txt, test.txt: Video identifiers for train/validation/test splits in the form of winner,loser,global goal->subgoal
• rounds/: Contains YAML files (numbered 1.yaml through 6.yaml) organizing video segment pairs for comparative judgment across six annotation rounds. Each file maps task subtasks to lists of paired video segments to be compared for skill assessment.
• rankings/: Contains skill ranking files (named rank_after_round_{n}.yml, i.e., 1.yml through 6.yml) with Elo ratings, Swiss scores, z-scores, and expert level assessments for each video segment after each annotation round.
• transcripts_coarse/: Coarse-grained action transcripts

AMT_results/

Contains Amazon Mechanical Turk (AMT) crowdsourced comparative skill assessment results across six annotation rounds per task. Each subdirectory corresponds to a dataset:

• assembly101/: 12 JSON files (6 rounds each for assembly and disassembly tasks)
  • Format: assembly_round_X_results.json and disassembly_round_X_results.json
• egoexo4d/: 6 JSON files (one per round for bike repair tasks)
  • Format: bike_round_X_results.json
• epic-tent/: Similar structure with tent assembly results
• ikea/: Similar structure with furniture assembly results
• meccano/: Similar structure with meccano assembly results

Each result file contains comparative judgment data with:

• Input.video1_url and Input.video2_url: Paired video identifiers being compared
• AnswerExperienced1 and AnswerExperienced2: Skill ratings from annotators (0-5 scale)
• winner: The video deemed to demonstrate higher skill level
• agreement_rate: Inter-annotator agreement on the judgment

Videos Dataset Download:

To download each video dataset follow their documentation, in particular:

• assembly101: follow istruction to download here
• egoexo4d: follow istruction to download here
• epic-tent: follow istruction to download here
• ikea: follow istruction to download here
• meccano: follow istruction to download here

Extract clip tool

We share a script (tools/extract_clips.py) to extract the annotated clips, the current implementation expect a videos/ directory, containing the original videos, and a clips/ directory, where the output will be extracted, inside every dataset dir, for Ego-Exo4D you should copy keystep_val.json and keystep_train.json from the original dataset inside the egoexo4d directory.

Generate rounds and rankings

This repository provides two helper scripts to build annotation rounds and compute skill rankings from comparative judgments:

• src/generate_rounds.py: generates the next set of segment-pair comparisons (a "round") for a dataset, optionally reconstructing tournament state from annotated results.
• src/generate_rankings.py: builds final rankings (Elo / Swiss) from pre-ranked matches or by simulating the tournament.

Common CLI options (use --help for full details):

• --dataset: one of meccano, ikea, egoexo4d, epic-tent, assembly101.
• --type: input file type for annotated matches (yaml, json, csv).
• --accepted: path to the accepted clips list (default: filtered_clips.txt).
• --output: output filename (default: rounds.yaml for rounds, rankings.yml for rankings).

Examples

• Generate the next round using AMT JSON results (writes a rounds file to ../data/<dataset>/rounds/ and, if --input is provided, updates ../data/<dataset>/rankings/):

python3 src/generate_rounds.py --dataset assembly101 \
  --input ../AMT_results/assembly101 --type json --output rounds.yaml

• Compute rankings (simulate tournament) and save to a rankings file:

python3 src/generate_rankings.py --dataset assembly101 \
  --output ../data/assembly101/rankings/1.yml --simulate

Notes

• When using generate_rounds.py with --input, the script will reconstruct the tournament state from the provided annotated matches and then produce the next round. The saved round file is placed in ../data/<dataset>/rounds/ and the reconstructed rankings (if any) are written to ../data/<dataset>/rankings/<n>.yml.
• generate_rankings.py without --simulate will reconstruct rankings from existing rounds (it calls reconstruct_from_rounds). Use --simulate to run an internal tournament simulation instead.
• Use --type csv when your annotated matches are CSVs (see src/data_io.py for CSV field expectations).