Add demand allocation scraper

MAINTENANCE.md

@@ -39,6 +39,8 @@ Reference PRs: [PR #3286](https://github.com/nusmodifications/nusmods/pull/3286)
 - [ ] In `app-config.json`, add semester to `examAvailability` to indicate exam information is available for the semester
 - [ ] Update the ModReg schedule in `website/src/data/modreg-schedule.json`, and make sure the correct version is pointed to in `website/src/config/index.ts`
   - Reference PR: [PR #2764](https://github.com/nusmodifications/nusmods/pull/2764)
+- [ ] After each CourseReg round is released, manually upload that round's vacancy, UG demand allocation, and GD demand allocation PDFs under `scrapers/demand-allocation-scraper/archive/pdfs/<acad-year>/semesters/<semester>/`, then run `pnpm dev <semester> <acad-year> --round <round> --pdfDir archive/pdfs/<acad-year>/semesters/<semester>` from `scrapers/demand-allocation-scraper`
+  - Verify that the PDFs contain released data, not "Information not available yet", and that `scrapers/nus-v2/data/<acad-year>/semesters/<semester>/courseRegHistory.json` is generated with non-empty module history for the academic-year archive
 
 ## CPEx
 

scrapers/README.md

@@ -11,6 +11,7 @@ universities. However, we have only implemented an NUS scraper. More details on
 can be found in its folder.
 
 1. [NUS Scraper](nus-v2)
+2. [Demand Allocation Scraper](demand-allocation-scraper)
 
 If you are from another university and would like to implement a scraper for
 your uni, feel free to file an issue or just contact us! We already have some

scrapers/demand-allocation-scraper/.gitignore

@@ -0,0 +1,7 @@
+/archive/pdfs/
+/build/
+/coverage/
+/node_modules/
+/.venv/
+__pycache__/
+*.log

scrapers/demand-allocation-scraper/README.md

@@ -0,0 +1,106 @@
+# Demand Allocation Scraper
+
+This scraper builds CourseReg demand allocation and vacancy history for the NUSMods API.
+
+The scraper is intentionally separate from `scrapers/nus-v2` because it works from CourseReg PDFs rather than the module/timetable APIs. It still writes generated API data into the NUS v2 data tree:
+
+```text
+../nus-v2/data/<acad-year>/semesters/<semester>/courseRegHistory.json
+```
+
+## Setup
+
+Install workspace dependencies from the repository root:
+
+```sh
+pnpm install
+```
+
+PDF conversion requires Java available through `JAVA_HOME` and a Python environment with the pinned dependencies:
+
+```sh
+cd scrapers/demand-allocation-scraper
+python3 -m venv .venv
+.venv/bin/python -m pip install -r requirements.txt
+```
+
+Then pass the pinned environment to scraper runs with `--python .venv/bin/python`.
+
+## Source PDF Archive
+
+NUS CourseReg PDFs are not reliably downloadable with a plain scraper request because NUS may return an HTML bot-protection page instead of PDF bytes. Treat the PDFs as manual/operator input unless NUS provides a supported download path.
+
+Store source PDFs under:
+
+```text
+archive/pdfs/<acad-year>/semesters/<semester>/
+  vacancy/round_<round>.pdf
+  ug/round_<round>.pdf
+  gd/round_<round>.pdf
+```
+
+For example:
+
+```text
+archive/pdfs/2025-2026/semesters/2/vacancy/round_1.pdf
+archive/pdfs/2025-2026/semesters/2/ug/round_1.pdf
+archive/pdfs/2025-2026/semesters/2/gd/round_1.pdf
+```
+
+`archive/pdfs/` is gitignored. Generated JSON belongs under `../nus-v2/data/` and is the only CourseReg artifact intended for the public API sync.
+
+To import an existing CourseRekt checkout into this archive layout:
+
+```sh
+python3 scripts/import_courserekt_pdfs.py /path/to/courserekt
+```
+
+## Run
+
+Build CourseReg history after manually staging one newly released round:
+
+```sh
+JAVA_HOME=/path/to/java \
+pnpm dev 2 2025/2026 \
+  --round 1 \
+  --pdfDir archive/pdfs/2025-2026/semesters/2 \
+  --python .venv/bin/python
+```
+
+By default, output goes to:
+
+```text
+../nus-v2/data/2025-2026/semesters/2/courseRegHistory.json
+```
+
+When `--round` is provided, the scraper updates that round in the existing
+`courseRegHistory.json` if one exists, while preserving previously scraped rounds.
+Classes that existed in earlier rounds but are missing from the newly scraped
+round are recorded as `"notAvailable"` for that round.
+
+To rebuild all available rounds from staged PDFs, omit `--round`:
+
+```sh
+JAVA_HOME=/path/to/java \
+pnpm dev 2 2025/2026 \
+  --pdfDir archive/pdfs/2025-2026/semesters/2 \
+  --python .venv/bin/python
+```
+
+To test against a temporary output directory:
+
+```sh
+JAVA_HOME=/path/to/java \
+pnpm dev 2 2025/2026 \
+  --round 1 \
+  --pdfDir archive/pdfs/2025-2026/semesters/2 \
+  --inputDir /tmp/demand-allocation-csv \
+  --outputDir /tmp/demand-allocation-output \
+  --python .venv/bin/python
+```
+
+## Data Semantics
+
+Class identity is keyed by course/module code plus class code. Titles and departments from the PDFs are not used as identifiers because they can vary or be repeated.
+
+Slot values in the generated JSON are non-negative integers, `"unlimited"`, or `"notAvailable"`. `0` is preserved as a real slot count; `"unlimited"` represents uncapped vacancies; `"notAvailable"` represents a class or student type that is not listed for that round/report.

scrapers/demand-allocation-scraper/archive/README.md

@@ -0,0 +1,16 @@
+# Demand Allocation PDF Archive
+
+This directory is for operator-staged CourseReg source PDFs used by the demand
+allocation scraper.
+
+PDFs should be stored under:
+
+```text
+archive/pdfs/<acad-year>/semesters/<semester>/
+  vacancy/round_<round>.pdf
+  ug/round_<round>.pdf
+  gd/round_<round>.pdf
+```
+
+The PDF files are ignored by git. Generated API data is written separately to
+`../nus-v2/data/<acad-year>/semesters/<semester>/courseRegHistory.json`.

scrapers/demand-allocation-scraper/oxlint.config.mjs

@@ -0,0 +1,34 @@
+import nkzw from '@nkzw/oxlint-config';
+import { defineConfig } from 'oxlint';
+
+const config = { ...nkzw };
+config.jsPlugins = config.jsPlugins?.filter(
+  (p) =>
+    p !== '@nkzw/eslint-plugin' &&
+    p !== 'eslint-plugin-unused-imports' &&
+    !(typeof p === 'object' && p.name === 'react-hooks-js') &&
+    p !== 'eslint-plugin-react-hooks',
+);
+config.rules = Object.fromEntries(
+  Object.entries(config.rules ?? {}).filter(
+    ([key]) =>
+      !key.startsWith('@nkzw/') &&
+      !key.startsWith('@typescript-eslint/') &&
+      !key.startsWith('react-hooks-js/') &&
+      !key.startsWith('react-hooks/') &&
+      !key.startsWith('unused-imports/') &&
+      key !== '@typescript-eslint/no-unused-vars',
+  ),
+);
+
+export default defineConfig({
+  extends: [config],
+  rules: {
+    'import-x/no-namespace': 'off',
+    'no-console': 'off',
+    'perfectionist/sort-object-types': 'off',
+    'perfectionist/sort-objects': 'off',
+    'unicorn/prefer-string-replace-all': 'off',
+    'unicorn/prefer-top-level-await': 'off',
+  },
+});

scrapers/demand-allocation-scraper/package.json

@@ -0,0 +1,30 @@
+{
+  "name": "nusmods-demand-allocation-scraper",
+  "version": "1.0.0",
+  "description": "NUSMods scraper for CourseReg demand allocation and vacancy history",
+  "license": "MIT",
+  "repository": "https://github.com/nusmodifications/nusmods",
+  "main": "src/index.ts",
+  "scripts": {
+    "scrape": "node build/src/index.js",
+    "dev": "pnpm build && node build/src/index.js",
+    "build": "tsc",
+    "lint": "oxlint -c oxlint.config.mjs src",
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run --coverage",
+    "test:watch": "vitest",
+    "check": "run-s lint typecheck test"
+  },
+  "devDependencies": {
+    "@nkzw/eslint-plugin": "catalog:",
+    "@nkzw/oxlint-config": "catalog:",
+    "@types/node": "catalog:",
+    "@vitest/coverage-v8": "catalog:",
+    "eslint-plugin-no-only-tests": "catalog:",
+    "eslint-plugin-perfectionist": "catalog:",
+    "eslint-plugin-unused-imports": "catalog:",
+    "oxlint": "catalog:",
+    "typescript": "catalog:",
+    "vitest": "catalog:"
+  }
+}