From 24803b690f614156d9fac1724dbe68ecffd20596 Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Wed, 5 Aug 2026 13:42:51 -0700 Subject: [PATCH 01/10] docs: draft design for PDP SFTP ingestion DAB Capture automation approach, file discovery modes, school validation, and gcp_config decoupling before implementation. Co-authored-by: Cursor --- pipelines/ingestion/pdp/DESIGN.md | 161 ++++++++++++++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 pipelines/ingestion/pdp/DESIGN.md diff --git a/pipelines/ingestion/pdp/DESIGN.md b/pipelines/ingestion/pdp/DESIGN.md new file mode 100644 index 000000000..d3af818e1 --- /dev/null +++ b/pipelines/ingestion/pdp/DESIGN.md @@ -0,0 +1,161 @@ +# Design: Convert PDP ingestion notebooks into a DAB + +**Goal:** Automate `PIPELINE_pdp_to_databricks.py` (SFTP → filter by institution → bronze volume) as a Databricks Asset Bundle job that can be run on demand or scheduled — same pattern as other Edvise DABs under `pipelines/`. + +**Status:** Design draft. Implementation can reuse / evolve work from `feature/nsc-sftp-scripts-and-dab` and `notebooks/nsc_sftp_automated_data_ingestion/`. + +--- + +## Current state + +### Interactive notebook (`PIPELINE_pdp_to_databricks.py`) + +One-shot, human-driven flow: + +1. Load `gcp_config.yaml` (secret scope, institution ID map, catalog prefixes). +2. Connect to SFTP `./receive`. +3. **Widgets:** pick a remote file + pick a PDP institution. +4. Download (atomic/resumable), normalize columns, filter rows for that institution. +5. Print cohort / cohort_term checks. +6. Write filtered CSV to `/Volumes/{catalog}/{inst}_bronze/bronze_volume/`, skip if file already exists. + +### Existing automation direction + +`notebooks/nsc_sftp_automated_data_ingestion/` and branch `feature/nsc-sftp-scripts-and-dab` already split this into: + +| Stage | Responsibility | State | +|-------|----------------|--------| +| 01 scan + stage | SFTP list/download, upsert `ingestion_manifest`, queue staged paths | Delta + UC `tmp` volume | +| 02 expand | Distinct PDP institution IDs per staged file → `institution_ingest_plan` | Delta | +| 03 bronze ingest | SST API resolve PDP ID → bronze schema/volume; filter + write; update manifest | Bronze volumes | + +DAB sketch lives at `pipelines/ingestion/pdp` on that branch (`nsc_sftp_automated_ingestion` job, git-sourced `spark_python_task` chain). Notebooks are also acceptable as DAB `notebook_task`s — conversion to scripts is optional. + +**Gap for scheduling:** stage 01 still requires explicit `cohort_file_name` / `course_file_name` job params. School registry still conceptually tied to `gcp_config` in the old notebook; automated path prefers API + bronze discovery. + +--- + +## Proposed pipeline shape + +Do **not** wrap the widget notebook as a single scheduled task. Keep the 3-stage DAG: + +```text +sftp_receive_scan → file_institution_expand → per_institution_bronze_ingest +``` + +Shared state: + +- `ingestion_manifest` — file fingerprint, status (`NEW` / `BRONZE_WRITTEN` / `FAILED`), errors, run id +- `pending_ingest_queue` — staged local UC volume path (downstream does not re-hit SFTP) +- `institution_ingest_plan` — `(file × institution_id)` work items + +`max_concurrent_runs: 1` so scheduled runs do not race the same SFTP files / queue rows. + +--- + +## Design decisions + +### 1. File selection (filename unknown until SFTP list) + +NSC cohort/course files share a 14-digit stamp: `..._YYYYMMDDHHMMSS.csv`. + +| Mode | Job params | Behavior | +|------|------------|----------| +| **Manual** | `cohort_file_name` + `course_file_name` | Current behavior; fail if missing on SFTP | +| **Latest pair** | empty / `mode=latest` | List `./receive`, pair by stamp, take newest stamp | +| **Uningested** (default for schedule) | empty / `mode=uningested` | Same pairing; skip fingerprints already `BRONZE_WRITTEN` (or queued); take newest remaining | + +Rules: + +- Pair cohort + course by identical stamp; log / fail on partial pairs. +- Idempotency key = `file_fingerprint` in `ingestion_manifest` (not “path exists in bronze”). +- Log: available stamps, chosen stamp, skip reasons. +- Optional guards: `min_file_date`, `max_age_days`, `dry_run=true` (list only). + +### 2. New schools / `gcp_config.yaml` + +Old notebook uses `gcp_config` for: + +1. SFTP secret scope/keys +2. Allowed PDP institution IDs (dropdown) +3. Institution → `{prefix}_bronze` catalog mapping + +Automated path should **not** require a yaml edit per new school: + +| Concern | Approach | +|---------|----------| +| Secrets | Fixed secret scope (e.g. `nsc-sftp-asset`); not per-institution | +| Which schools | IDs discovered from file; resolve via SST `GET /institutions/pdp-id/{pdp_id}` | +| Where to write | `databricksify_inst_name` + find `{inst}_bronze` schema / bronze volume | + +**Onboarding checklist (outside the ingest job):** + +1. Institution exists in SST/API with correct PDP ID +2. UC schema `{inst}_bronze` + bronze volume exist +3. SFTP secrets already configured in the shared scope + +Optional allowlist (Delta table or job param JSON) if we must not auto-ingest every ID in the dump. Prefer: ingest IDs that resolve **and** have bronze provisioned; log/skip `UNKNOWN_OR_UNPROVISIONED`. + +Treat workspace-local `gcp_config.yaml` school lists as tech debt for this DAB. + +### 3. School checks / logging (replace widget validation) + +Structured logs (and optional summary table) in stages 02/03: + +**After expand (02):** + +- Distinct PDP IDs per file + row counts +- API resolve success vs failure +- Resolve OK but missing bronze schema/volume +- Intersection with optional allowlist + +**Before/after write (03):** + +- Per `(file, institution)`: filtered row count, latest cohort / cohort_term (parity with old notebook prints), destination path, skip-vs-wrote +- End rollup: `ingested`, `skipped_already_present`, `unresolved`, `no_bronze` +- Fail job (or fail-soft + Slack) on policies such as `unresolved > 0` or `ingested == 0` when files were `NEW` + +### 4. Operating modes + +1. **Scheduled** — `mode=uningested`; Slack on failure / zero-ingest +2. **Manual** — pass filenames or stamp for one-off / reprocess +3. **Force reprocess** — `force=true` even if manifest says done (define overwrite policy) +4. **Optional filter** — `institution_ids=...` for targeted runs without dropdown UX + +### 5. Bronze overwrite policy (open) + +Old notebook: skip if destination file exists. Scheduled runs may need: + +- versioned paths under bronze (e.g. stamp subdirectory), or +- overwrite with audit fields on the manifest + +Decide explicitly before enabling cron. + +--- + +## Implementation priorities (vs existing branch) + +1. Auto file discovery (`latest` / `uningested`) when filename params are empty +2. School validation report (API + bronze provisioning) +3. Overwrite / versioning policy for bronze writes +4. Decouple school registry from `gcp_config` (secrets only) +5. Bundle under `pipelines/ingestion/pdp` aligned with other DABs (`git_commit` / `git_tag`, permissions, webhooks) +6. Notebook vs script tasks — either is fine; prefer shared helpers in `src/edvise/ingestion/` + +--- + +## Out of scope (for later) + +- Chaining this DAB into PDP training/inference automatically +- Replacing GCS validated → bronze sync (`pipelines/ingestion/shared`) +- GenAI mapping onboarding + +--- + +## References + +- Behavior source: `PIPELINE_pdp_to_databricks.py` (interactive) +- Automation notebooks: `notebooks/nsc_sftp_automated_data_ingestion/` +- Prior WIP: `origin/feature/nsc-sftp-scripts-and-dab` +- Helpers on `develop`: `src/edvise/ingestion/nsc_sftp_helpers.py`, `src/edvise/utils/sftp.py`, `src/edvise/ingestion/constants.py` +- Sibling DAB pattern: `pipelines/ingestion/shared/` From f9dbd0a55b8b5664234282f0d438961203f2e4b6 Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Wed, 5 Aug 2026 13:58:18 -0700 Subject: [PATCH 02/10] feat: add PDP/NSC SFTP ingestion DAB with auto file selection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the interactive PIPELINE_pdp_to_databricks flow with a schedulable 01→02→03 job under pipelines/ingestion/pdp, including uningested/latest cohort-course discovery and school-check logging. Co-authored-by: Cursor --- .../01_sftp_receive_scan.ipynb | 4 +- .../02_file_institution_expand.ipynb | 4 +- .../03_per_institution_bronze_ingest.ipynb | 10 +- pipelines/ingestion/pdp/.gitignore | 1 + pipelines/ingestion/pdp/DESIGN.md | 161 ------ pipelines/ingestion/pdp/databricks.yml | 59 +++ .../pdp/resources/nsc_sftp_ingestion.yml | 136 +++++ src/edvise/ingestion/constants.py | 91 ---- src/edvise/ingestion/nsc_sftp/__init__.py | 5 + src/edvise/ingestion/nsc_sftp/constants.py | 124 +++++ .../ingestion/nsc_sftp/file_selection.py | 193 +++++++ .../helpers.py} | 4 +- .../nsc_sftp/scripts/01_sftp_receive_scan.py | 244 +++++++++ .../scripts/02_file_institution_expand.py | 203 +++++++ .../03_per_institution_bronze_ingest.py | 497 ++++++++++++++++++ .../ingestion/nsc_sftp/scripts/__init__.py | 1 + tests/ingestion/test_file_selection.py | 117 +++++ tests/ingestion/test_nsc_sftp_helper.py | 2 +- 18 files changed, 1592 insertions(+), 264 deletions(-) create mode 100644 pipelines/ingestion/pdp/.gitignore delete mode 100644 pipelines/ingestion/pdp/DESIGN.md create mode 100644 pipelines/ingestion/pdp/databricks.yml create mode 100644 pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml delete mode 100644 src/edvise/ingestion/constants.py create mode 100644 src/edvise/ingestion/nsc_sftp/__init__.py create mode 100644 src/edvise/ingestion/nsc_sftp/constants.py create mode 100644 src/edvise/ingestion/nsc_sftp/file_selection.py rename src/edvise/ingestion/{nsc_sftp_helpers.py => nsc_sftp/helpers.py} (99%) create mode 100644 src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py create mode 100644 src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py create mode 100644 src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py create mode 100644 src/edvise/ingestion/nsc_sftp/scripts/__init__.py create mode 100644 tests/ingestion/test_file_selection.py diff --git a/notebooks/nsc_sftp_automated_data_ingestion/01_sftp_receive_scan.ipynb b/notebooks/nsc_sftp_automated_data_ingestion/01_sftp_receive_scan.ipynb index 6a9e361ec..3093c2306 100644 --- a/notebooks/nsc_sftp_automated_data_ingestion/01_sftp_receive_scan.ipynb +++ b/notebooks/nsc_sftp_automated_data_ingestion/01_sftp_receive_scan.ipynb @@ -112,14 +112,14 @@ "from pyspark.sql import functions as F\n", "\n", "from edvise.utils.sftp import connect_sftp, list_receive_files\n", - "from edvise.ingestion.constants import (\n", + "from edvise.ingestion.nsc_sftp.constants import (\n", " MANIFEST_TABLE_PATH,\n", " QUEUE_TABLE_PATH,\n", " SFTP_REMOTE_FOLDER,\n", " SFTP_SOURCE_SYSTEM,\n", " SFTP_TMP_DIR,\n", ")\n", - "from edvise.ingestion.nsc_sftp_helpers import (\n", + "from edvise.ingestion.nsc_sftp.helpers import (\n", " build_listing_df,\n", " download_new_files_and_queue,\n", " ensure_manifest_and_queue_tables,\n", diff --git a/notebooks/nsc_sftp_automated_data_ingestion/02_file_institution_expand.ipynb b/notebooks/nsc_sftp_automated_data_ingestion/02_file_institution_expand.ipynb index 9e3c409c0..5cd78beb4 100644 --- a/notebooks/nsc_sftp_automated_data_ingestion/02_file_institution_expand.ipynb +++ b/notebooks/nsc_sftp_automated_data_ingestion/02_file_institution_expand.ipynb @@ -80,8 +80,8 @@ "from pyspark.sql import types as T\n", "from databricks.connect import DatabricksSession\n", "\n", - "from edvise.ingestion.nsc_sftp_helpers import ensure_plan_table, extract_institution_ids\n", - "from edvise.ingestion.constants import (\n", + "from edvise.ingestion.nsc_sftp.helpers import ensure_plan_table, extract_institution_ids\n", + "from edvise.ingestion.nsc_sftp.constants import (\n", " QUEUE_TABLE_PATH,\n", " PLAN_TABLE_PATH,\n", " COLUMN_RENAMES,\n", diff --git a/notebooks/nsc_sftp_automated_data_ingestion/03_per_institution_bronze_ingest.ipynb b/notebooks/nsc_sftp_automated_data_ingestion/03_per_institution_bronze_ingest.ipynb index 3cdc865ca..898b46e00 100644 --- a/notebooks/nsc_sftp_automated_data_ingestion/03_per_institution_bronze_ingest.ipynb +++ b/notebooks/nsc_sftp_automated_data_ingestion/03_per_institution_bronze_ingest.ipynb @@ -19,10 +19,10 @@ "outputs": [], "source": [ "# Databricks notebook source\n", - "# Script 4 — 04_per_institution_bronze_ingest\n", + "# Script 4 \u2014 04_per_institution_bronze_ingest\n", "#\n", "# Purpose:\n", - "# Consume institution_ingest_plan (created by Script 3), and for each (file × institution):\n", + "# Consume institution_ingest_plan (created by Script 3), and for each (file \u00d7 institution):\n", "# - get bearer token from SST staging using X-API-KEY (from Databricks secrets)\n", "# - call /api/v1/institutions/pdp-id/{pdp_id} to resolve institution name\n", "# - map name -> schema prefix via databricksify_inst_name()\n", @@ -104,14 +104,14 @@ "from edvise.utils.databricks import (\n", " find_bronze_schema,\n", " find_bronze_volume_name,\n", - " databricksify_inst_name,\n", ")\n", + "from edvise.utils.institution_naming import databricksify_inst_name\n", "from edvise.utils.sftp import output_file_name_from_sftp\n", - "from edvise.ingestion.nsc_sftp_helpers import (\n", + "from edvise.ingestion.nsc_sftp.helpers import (\n", " process_and_save_file,\n", " update_manifest,\n", ")\n", - "from edvise.ingestion.constants import (\n", + "from edvise.ingestion.nsc_sftp.constants import (\n", " CATALOG,\n", " PLAN_TABLE_PATH,\n", " MANIFEST_TABLE_PATH,\n", diff --git a/pipelines/ingestion/pdp/.gitignore b/pipelines/ingestion/pdp/.gitignore new file mode 100644 index 000000000..15bcc6dd0 --- /dev/null +++ b/pipelines/ingestion/pdp/.gitignore @@ -0,0 +1 @@ +.databricks diff --git a/pipelines/ingestion/pdp/DESIGN.md b/pipelines/ingestion/pdp/DESIGN.md deleted file mode 100644 index d3af818e1..000000000 --- a/pipelines/ingestion/pdp/DESIGN.md +++ /dev/null @@ -1,161 +0,0 @@ -# Design: Convert PDP ingestion notebooks into a DAB - -**Goal:** Automate `PIPELINE_pdp_to_databricks.py` (SFTP → filter by institution → bronze volume) as a Databricks Asset Bundle job that can be run on demand or scheduled — same pattern as other Edvise DABs under `pipelines/`. - -**Status:** Design draft. Implementation can reuse / evolve work from `feature/nsc-sftp-scripts-and-dab` and `notebooks/nsc_sftp_automated_data_ingestion/`. - ---- - -## Current state - -### Interactive notebook (`PIPELINE_pdp_to_databricks.py`) - -One-shot, human-driven flow: - -1. Load `gcp_config.yaml` (secret scope, institution ID map, catalog prefixes). -2. Connect to SFTP `./receive`. -3. **Widgets:** pick a remote file + pick a PDP institution. -4. Download (atomic/resumable), normalize columns, filter rows for that institution. -5. Print cohort / cohort_term checks. -6. Write filtered CSV to `/Volumes/{catalog}/{inst}_bronze/bronze_volume/`, skip if file already exists. - -### Existing automation direction - -`notebooks/nsc_sftp_automated_data_ingestion/` and branch `feature/nsc-sftp-scripts-and-dab` already split this into: - -| Stage | Responsibility | State | -|-------|----------------|--------| -| 01 scan + stage | SFTP list/download, upsert `ingestion_manifest`, queue staged paths | Delta + UC `tmp` volume | -| 02 expand | Distinct PDP institution IDs per staged file → `institution_ingest_plan` | Delta | -| 03 bronze ingest | SST API resolve PDP ID → bronze schema/volume; filter + write; update manifest | Bronze volumes | - -DAB sketch lives at `pipelines/ingestion/pdp` on that branch (`nsc_sftp_automated_ingestion` job, git-sourced `spark_python_task` chain). Notebooks are also acceptable as DAB `notebook_task`s — conversion to scripts is optional. - -**Gap for scheduling:** stage 01 still requires explicit `cohort_file_name` / `course_file_name` job params. School registry still conceptually tied to `gcp_config` in the old notebook; automated path prefers API + bronze discovery. - ---- - -## Proposed pipeline shape - -Do **not** wrap the widget notebook as a single scheduled task. Keep the 3-stage DAG: - -```text -sftp_receive_scan → file_institution_expand → per_institution_bronze_ingest -``` - -Shared state: - -- `ingestion_manifest` — file fingerprint, status (`NEW` / `BRONZE_WRITTEN` / `FAILED`), errors, run id -- `pending_ingest_queue` — staged local UC volume path (downstream does not re-hit SFTP) -- `institution_ingest_plan` — `(file × institution_id)` work items - -`max_concurrent_runs: 1` so scheduled runs do not race the same SFTP files / queue rows. - ---- - -## Design decisions - -### 1. File selection (filename unknown until SFTP list) - -NSC cohort/course files share a 14-digit stamp: `..._YYYYMMDDHHMMSS.csv`. - -| Mode | Job params | Behavior | -|------|------------|----------| -| **Manual** | `cohort_file_name` + `course_file_name` | Current behavior; fail if missing on SFTP | -| **Latest pair** | empty / `mode=latest` | List `./receive`, pair by stamp, take newest stamp | -| **Uningested** (default for schedule) | empty / `mode=uningested` | Same pairing; skip fingerprints already `BRONZE_WRITTEN` (or queued); take newest remaining | - -Rules: - -- Pair cohort + course by identical stamp; log / fail on partial pairs. -- Idempotency key = `file_fingerprint` in `ingestion_manifest` (not “path exists in bronze”). -- Log: available stamps, chosen stamp, skip reasons. -- Optional guards: `min_file_date`, `max_age_days`, `dry_run=true` (list only). - -### 2. New schools / `gcp_config.yaml` - -Old notebook uses `gcp_config` for: - -1. SFTP secret scope/keys -2. Allowed PDP institution IDs (dropdown) -3. Institution → `{prefix}_bronze` catalog mapping - -Automated path should **not** require a yaml edit per new school: - -| Concern | Approach | -|---------|----------| -| Secrets | Fixed secret scope (e.g. `nsc-sftp-asset`); not per-institution | -| Which schools | IDs discovered from file; resolve via SST `GET /institutions/pdp-id/{pdp_id}` | -| Where to write | `databricksify_inst_name` + find `{inst}_bronze` schema / bronze volume | - -**Onboarding checklist (outside the ingest job):** - -1. Institution exists in SST/API with correct PDP ID -2. UC schema `{inst}_bronze` + bronze volume exist -3. SFTP secrets already configured in the shared scope - -Optional allowlist (Delta table or job param JSON) if we must not auto-ingest every ID in the dump. Prefer: ingest IDs that resolve **and** have bronze provisioned; log/skip `UNKNOWN_OR_UNPROVISIONED`. - -Treat workspace-local `gcp_config.yaml` school lists as tech debt for this DAB. - -### 3. School checks / logging (replace widget validation) - -Structured logs (and optional summary table) in stages 02/03: - -**After expand (02):** - -- Distinct PDP IDs per file + row counts -- API resolve success vs failure -- Resolve OK but missing bronze schema/volume -- Intersection with optional allowlist - -**Before/after write (03):** - -- Per `(file, institution)`: filtered row count, latest cohort / cohort_term (parity with old notebook prints), destination path, skip-vs-wrote -- End rollup: `ingested`, `skipped_already_present`, `unresolved`, `no_bronze` -- Fail job (or fail-soft + Slack) on policies such as `unresolved > 0` or `ingested == 0` when files were `NEW` - -### 4. Operating modes - -1. **Scheduled** — `mode=uningested`; Slack on failure / zero-ingest -2. **Manual** — pass filenames or stamp for one-off / reprocess -3. **Force reprocess** — `force=true` even if manifest says done (define overwrite policy) -4. **Optional filter** — `institution_ids=...` for targeted runs without dropdown UX - -### 5. Bronze overwrite policy (open) - -Old notebook: skip if destination file exists. Scheduled runs may need: - -- versioned paths under bronze (e.g. stamp subdirectory), or -- overwrite with audit fields on the manifest - -Decide explicitly before enabling cron. - ---- - -## Implementation priorities (vs existing branch) - -1. Auto file discovery (`latest` / `uningested`) when filename params are empty -2. School validation report (API + bronze provisioning) -3. Overwrite / versioning policy for bronze writes -4. Decouple school registry from `gcp_config` (secrets only) -5. Bundle under `pipelines/ingestion/pdp` aligned with other DABs (`git_commit` / `git_tag`, permissions, webhooks) -6. Notebook vs script tasks — either is fine; prefer shared helpers in `src/edvise/ingestion/` - ---- - -## Out of scope (for later) - -- Chaining this DAB into PDP training/inference automatically -- Replacing GCS validated → bronze sync (`pipelines/ingestion/shared`) -- GenAI mapping onboarding - ---- - -## References - -- Behavior source: `PIPELINE_pdp_to_databricks.py` (interactive) -- Automation notebooks: `notebooks/nsc_sftp_automated_data_ingestion/` -- Prior WIP: `origin/feature/nsc-sftp-scripts-and-dab` -- Helpers on `develop`: `src/edvise/ingestion/nsc_sftp_helpers.py`, `src/edvise/utils/sftp.py`, `src/edvise/ingestion/constants.py` -- Sibling DAB pattern: `pipelines/ingestion/shared/` diff --git a/pipelines/ingestion/pdp/databricks.yml b/pipelines/ingestion/pdp/databricks.yml new file mode 100644 index 000000000..c74121196 --- /dev/null +++ b/pipelines/ingestion/pdp/databricks.yml @@ -0,0 +1,59 @@ +# Dedicated bundle folder: pipelines/ingestion/pdp/ +# Automates NSC/PDP SFTP → bronze ingestion (replaces interactive PIPELINE_pdp_to_databricks). +bundle: + name: Edvise NSC SFTP ingestion + uuid: b8f4e2c1-9a3d-4f1e-b7c2-1d4e8f9a0b2c + +include: + - resources/nsc_sftp_ingestion.yml + +variables: + git_commit: + description: "Commit SHA to fetch (dev/CI)" + default: "" + git_tag: + description: "Release tag to fetch for deployment (prod)" + default: "" + DB_workspace: + description: "Unity Catalog name for NSC ingestion (tables and staging volume)" + ds_run_as: + description: "Service principal / app ID used as run_as for the job" + service_account_executer: + description: "Human or technical user granted CAN_MANAGE on the job" + datakind_group_to_manage_workflow: + description: "Workspace group with CAN_MANAGE on the job" + ingestion_slack_webhook_id: + description: "Slack webhook ID for ingestion pipeline failure notifications" + +run_as: + service_principal_name: ${var.ds_run_as} + +targets: + dev: + mode: development + default: true + variables: + DB_workspace: "dev_sst_02" + ingestion_slack_webhook_id: "1e30ca91-8d95-4324-be05-9d20bffd747b" # edvise-data-crew on dev_sst_02 + resources: + jobs: + nsc_sftp_automated_ingestion: + git_source: + git_url: https://github.com/datakind/edvise + git_provider: gitHub + git_commit: ${var.git_commit} + + prod: + mode: production + variables: + DB_workspace: "staging_sst_01" + ingestion_slack_webhook_id: "c51b737e-988d-45dd-ad9b-6f9024a6b56b" # edvise-support on staging_sst_01 + workspace: + root_path: /Workspace/Shared/bundles/${bundle.name}/${bundle.target} + resources: + jobs: + nsc_sftp_automated_ingestion: + git_source: + git_url: https://github.com/datakind/edvise + git_provider: gitHub + git_tag: ${var.git_tag} diff --git a/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml b/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml new file mode 100644 index 000000000..f89859f57 --- /dev/null +++ b/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml @@ -0,0 +1,136 @@ +# NSC/PDP SFTP automated ingestion: Git-sourced spark_python_task chain (01→02→03). +# Prerequisites: Job parameter DB_workspace (Unity Catalog), matching UC tables/volumes, +# secret scope nsc-sftp-asset (SFTP + SST API key), and cluster egress for SFTP/APIs. +# +# File selection: +# - Set cohort_file_name + course_file_name for a manual run, OR +# - Leave them empty and use file_selection_mode=uningested|latest (default: uningested). + +resources: + jobs: + nsc_sftp_automated_ingestion: + name: nsc_sftp_automated_ingestion + max_concurrent_runs: 1 + queue: + enabled: true + webhook_notifications: + on_failure: + - id: ${var.ingestion_slack_webhook_id} + + parameters: + - name: DB_workspace + default: ${var.DB_workspace} + - name: file_selection_mode + default: uningested + - name: cohort_file_name + default: "" + - name: course_file_name + default: "" + + tasks: + - task_key: sftp_receive_scan + spark_python_task: + python_file: src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py + source: GIT + parameters: + - --DB_workspace + - "{{job.parameters.DB_workspace}}" + - --file_selection_mode + - "{{job.parameters.file_selection_mode}}" + - --cohort_file_name + - "{{job.parameters.cohort_file_name}}" + - --course_file_name + - "{{job.parameters.course_file_name}}" + job_cluster_key: nsc-sftp-ingestion-cluster + libraries: + - pypi: + package: pandas==2.2.3 + - pypi: + package: numpy==1.26.4 + - pypi: + package: pyarrow>=17.0.0 + - pypi: + package: requests==2.32.5 + - pypi: + package: pyyaml~=6.0 + - pypi: + package: pydantic~=2.10 + - pypi: + package: paramiko~=3.5 + + - task_key: file_institution_expand + depends_on: + - task_key: sftp_receive_scan + spark_python_task: + python_file: src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py + source: GIT + parameters: + - --DB_workspace + - "{{job.parameters.DB_workspace}}" + job_cluster_key: nsc-sftp-ingestion-cluster + libraries: + - pypi: + package: pandas==2.2.3 + - pypi: + package: numpy==1.26.4 + - pypi: + package: pyarrow>=17.0.0 + - pypi: + package: requests==2.32.5 + - pypi: + package: pyyaml~=6.0 + - pypi: + package: pydantic~=2.10 + + - task_key: per_institution_bronze_ingest + depends_on: + - task_key: file_institution_expand + spark_python_task: + python_file: src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py + source: GIT + parameters: + - --DB_workspace + - "{{job.parameters.DB_workspace}}" + job_cluster_key: nsc-sftp-ingestion-cluster + libraries: + - pypi: + package: pandas==2.2.3 + - pypi: + package: numpy==1.26.4 + - pypi: + package: pyarrow>=17.0.0 + - pypi: + package: requests==2.32.5 + - pypi: + package: pyyaml~=6.0 + - pypi: + package: pydantic~=2.10 + + job_clusters: + - job_cluster_key: nsc-sftp-ingestion-cluster + new_cluster: + cluster_name: "" + spark_version: 15.4.x-cpu-ml-scala2.12 + spark_conf: + spark.master: local[*, 4] + spark.databricks.cluster.profile: singleNode + gcp_attributes: + use_preemptible_executors: false + availability: ON_DEMAND_GCP + zone_id: HA + node_type_id: n2-standard-16 + custom_tags: + ResourceClass: SingleNode + x-databricks-nextgen-cluster: "true" + enable_elastic_disk: true + data_security_mode: SINGLE_USER + runtime_engine: STANDARD + num_workers: 0 + + permissions: + - group_name: ${var.datakind_group_to_manage_workflow} + level: CAN_MANAGE + - service_principal_name: ${var.ds_run_as} + level: CAN_MANAGE + - user_name: ${var.service_account_executer} + level: CAN_MANAGE diff --git a/src/edvise/ingestion/constants.py b/src/edvise/ingestion/constants.py deleted file mode 100644 index 8eef55f54..000000000 --- a/src/edvise/ingestion/constants.py +++ /dev/null @@ -1,91 +0,0 @@ -""" -Constants for NSC SFTP ingestion pipeline. - -These values are fixed and don't vary between runs or environments. -For environment-specific values (like secret scope names), see gcp_config.yaml. -""" - -from typing import Any -from unittest.mock import MagicMock - -dbutils: Any - -# Databricks catalog and schema -try: - from databricks.sdk.runtime import dbutils as _dbutils -except Exception: - # Local/offline context: allow imports/tests to run without Databricks. - dbutils = MagicMock() - CATALOG = "dev_sst_02" -else: - dbutils = _dbutils - try: - workspace_id = str( - dbutils.notebook.entry_point.getDbutils() - .notebook() - .getContext() - .workspaceId() - .get() - ) - except Exception: - # Databricks SDK is importable, but we're not running in a notebook/runtime - # context where workspace ID is available. - dbutils = MagicMock() - CATALOG = "dev_sst_02" - else: - if workspace_id == "4437281602191762": - CATALOG = "dev_sst_02" - elif workspace_id == "2052166062819251": - CATALOG = "staging_sst_01" - else: - raise RuntimeError( - f"Unsupported Databricks workspace_id={workspace_id!r} for NSC ingestion. " - "Add a mapping in src/edvise/ingestion/constants.py." - ) -DEFAULT_SCHEMA = "default" - -# Table names (without catalog.schema prefix) -MANIFEST_TABLE = "ingestion_manifest" -QUEUE_TABLE = "pending_ingest_queue" -PLAN_TABLE = "institution_ingest_plan" - -# Full table paths -MANIFEST_TABLE_PATH = f"{CATALOG}.{DEFAULT_SCHEMA}.{MANIFEST_TABLE}" -QUEUE_TABLE_PATH = f"{CATALOG}.{DEFAULT_SCHEMA}.{QUEUE_TABLE}" -PLAN_TABLE_PATH = f"{CATALOG}.{DEFAULT_SCHEMA}.{PLAN_TABLE}" - -# SFTP settings -SFTP_REMOTE_FOLDER = "./receive" -SFTP_SOURCE_SYSTEM = "NSC" -SFTP_PORT = 22 -SFTP_TMP_VOLUME_NAME = "tmp" -SFTP_TMP_VOLUME_FQN = f"{CATALOG}.{DEFAULT_SCHEMA}.{SFTP_TMP_VOLUME_NAME}" -SFTP_TMP_DIR = f"/Volumes/{CATALOG}/{DEFAULT_SCHEMA}/{SFTP_TMP_VOLUME_NAME}" -SFTP_DOWNLOAD_CHUNK_MB = 150 -SFTP_VERIFY_DOWNLOAD = "size" # Options: "size", "sha256", "md5", "none" - -# Edvise API settings -SST_BASE_URL = "https://staging-sst.datakind.org" -SST_TOKEN_ENDPOINT = f"{SST_BASE_URL}/api/v1/token-from-api-key" -INSTITUTION_LOOKUP_PATH = "/api/v1/institutions/pdp-id/{pdp_id}" -SST_API_KEY_SECRET_KEY = "sst_staging_api_key" # Key name in Databricks secrets - -# File processing settings -INSTITUTION_COLUMN_PATTERN = r"(?=.*institution)(?=.*id)" - -# Column name mappings (mangled -> normalized) -# Applied after snake_case conversion -COLUMN_RENAMES = { - # NOTE: convert_to_snake_case splits trailing digit groups with an underscore, - # e.g. "attemptedgatewaymathyear1" -> "attemptedgatewaymathyear_1". - "attemptedgatewaymathyear_1": "attempted_gateway_math_year_1", - "attemptedgatewayenglishyear_1": "attempted_gateway_english_year_1", - "completedgatewaymathyear_1": "completed_gateway_math_year_1", - "completedgatewayenglishyear_1": "completed_gateway_english_year_1", - "gatewaymathgradey_1": "gateway_math_grade_y_1", - "gatewayenglishgradey_1": "gateway_english_grade_y_1", - "attempteddevmathy_1": "attempted_dev_math_y_1", - "attempteddevenglishy_1": "attempted_dev_english_y_1", - "completeddevmathy_1": "completed_dev_math_y_1", - "completeddevenglishy_1": "completed_dev_english_y_1", -} diff --git a/src/edvise/ingestion/nsc_sftp/__init__.py b/src/edvise/ingestion/nsc_sftp/__init__.py new file mode 100644 index 000000000..48f8ed28e --- /dev/null +++ b/src/edvise/ingestion/nsc_sftp/__init__.py @@ -0,0 +1,5 @@ +"""NSC SFTP automated ingestion (constants, helpers, job scripts).""" + +from edvise.ingestion.nsc_sftp import constants, file_selection, helpers + +__all__ = ["constants", "file_selection", "helpers"] diff --git a/src/edvise/ingestion/nsc_sftp/constants.py b/src/edvise/ingestion/nsc_sftp/constants.py new file mode 100644 index 000000000..b1f6d3b95 --- /dev/null +++ b/src/edvise/ingestion/nsc_sftp/constants.py @@ -0,0 +1,124 @@ +""" +Constants for NSC SFTP ingestion pipeline. + +Unity Catalog name must match the job's DB_workspace parameter (see +configure_nsc_catalog / resolve_nsc_catalog). Other values here are fixed or +scoped to default schema. +""" + +from __future__ import annotations + +import os +import sys + +# Unity Catalog name — set by configure_nsc_catalog (usually from job parameter DB_workspace). +DEFAULT_CATALOG_FOR_LOCAL = "dev_sst_02" +DEFAULT_SCHEMA = "default" + +# Table names (without catalog.schema prefix) +MANIFEST_TABLE = "ingestion_manifest" +QUEUE_TABLE = "pending_ingest_queue" +PLAN_TABLE = "institution_ingest_plan" + +SFTP_TMP_VOLUME_NAME = "tmp" + +CATALOG: str +MANIFEST_TABLE_PATH: str +QUEUE_TABLE_PATH: str +PLAN_TABLE_PATH: str +SFTP_TMP_VOLUME_FQN: str +SFTP_TMP_DIR: str + + +def configure_nsc_catalog(catalog: str) -> None: + """Set Unity Catalog name and derived table/volume paths (once per process).""" + global CATALOG, MANIFEST_TABLE_PATH, QUEUE_TABLE_PATH, PLAN_TABLE_PATH + global SFTP_TMP_VOLUME_FQN, SFTP_TMP_DIR + cat = str(catalog).strip() + if not cat: + raise ValueError( + "NSC ingestion catalog is empty. Pass job parameter DB_workspace " + "(Unity Catalog name), set widget DB_workspace, or NSC_DB_WORKSPACE." + ) + CATALOG = cat + MANIFEST_TABLE_PATH = f"{CATALOG}.{DEFAULT_SCHEMA}.{MANIFEST_TABLE}" + QUEUE_TABLE_PATH = f"{CATALOG}.{DEFAULT_SCHEMA}.{QUEUE_TABLE}" + PLAN_TABLE_PATH = f"{CATALOG}.{DEFAULT_SCHEMA}.{PLAN_TABLE}" + SFTP_TMP_VOLUME_FQN = f"{CATALOG}.{DEFAULT_SCHEMA}.{SFTP_TMP_VOLUME_NAME}" + SFTP_TMP_DIR = f"/Volumes/{CATALOG}/{DEFAULT_SCHEMA}/{SFTP_TMP_VOLUME_NAME}" + + +def parse_spark_python_task_params(argv: list[str] | None = None) -> dict[str, str]: + """Parse ``--key value`` pairs from ``spark_python_task.parameters``.""" + if argv is None: + argv = sys.argv + out: dict[str, str] = {} + i = 1 + while i < len(argv): + a = argv[i] + if a.startswith("--") and i + 1 < len(argv): + out[a[2:].replace("-", "_")] = argv[i + 1] + i += 2 + else: + i += 1 + return out + + +def resolve_nsc_catalog(argv: list[str] | None = None) -> str: + """ + Resolve Unity Catalog name in order: task argv ``--DB_workspace``, notebook widget + ``DB_workspace``, env ``NSC_DB_WORKSPACE``, else DEFAULT_CATALOG_FOR_LOCAL. + """ + argv = sys.argv if argv is None else argv + pairs = parse_spark_python_task_params(argv) + raw = pairs.get("DB_workspace", "").strip() + if raw: + return raw + try: + from edvise.utils.databricks import get_db_widget_param + + w = get_db_widget_param("DB_workspace", default="") + if str(w).strip(): + return str(w).strip() + except Exception: + pass + env = os.environ.get("NSC_DB_WORKSPACE", "").strip() + if env: + return env + return DEFAULT_CATALOG_FOR_LOCAL + + +# SFTP settings +SFTP_REMOTE_FOLDER = "./receive" +SFTP_SOURCE_SYSTEM = "NSC" +SFTP_PORT = 22 +SFTP_DOWNLOAD_CHUNK_MB = 150 +SFTP_VERIFY_DOWNLOAD = "size" # Options: "size", "sha256", "md5", "none" + +# Edvise API settings +SST_BASE_URL = "https://staging-sst.datakind.org" +SST_TOKEN_ENDPOINT = f"{SST_BASE_URL}/api/v1/token-from-api-key" +INSTITUTION_LOOKUP_PATH = "/api/v1/institutions/pdp-id/{pdp_id}" +SST_API_KEY_SECRET_KEY = "sst_staging_api_key" # Key name in Databricks secrets + +# File processing settings +INSTITUTION_COLUMN_PATTERN = r"(?=.*institution)(?=.*id)" + +# Column name mappings (mangled -> normalized) +# Applied after snake_case conversion +COLUMN_RENAMES = { + # NOTE: convert_to_snake_case splits trailing digit groups with an underscore, + # e.g. "attemptedgatewaymathyear1" -> "attemptedgatewaymathyear_1". + "attemptedgatewaymathyear_1": "attempted_gateway_math_year_1", + "attemptedgatewayenglishyear_1": "attempted_gateway_english_year_1", + "completedgatewaymathyear_1": "completed_gateway_math_year_1", + "completedgatewayenglishyear_1": "completed_gateway_english_year_1", + "gatewaymathgradey_1": "gateway_math_grade_y_1", + "gatewayenglishgradey_1": "gateway_english_grade_y_1", + "attempteddevmathy_1": "attempted_dev_math_y_1", + "attempteddevenglishy_1": "attempted_dev_english_y_1", + "completeddevmathy_1": "completed_dev_math_y_1", + "completeddevenglishy_1": "completed_dev_english_y_1", +} + +configure_nsc_catalog(DEFAULT_CATALOG_FOR_LOCAL) diff --git a/src/edvise/ingestion/nsc_sftp/file_selection.py b/src/edvise/ingestion/nsc_sftp/file_selection.py new file mode 100644 index 000000000..2c81dfe89 --- /dev/null +++ b/src/edvise/ingestion/nsc_sftp/file_selection.py @@ -0,0 +1,193 @@ +""" +Select cohort/course SFTP file pairs for NSC PDP ingestion. + +Files are expected to end with a shared 14-digit stamp ``_YYYYMMDDHHMMSS`` and +to contain ``cohort`` or ``course`` in the basename (case-insensitive). +""" + +from __future__ import annotations + +import os +import re +from dataclasses import dataclass +from typing import Any, Iterable, Literal, Mapping, Optional + +FILE_STAMP_RE = re.compile(r"_(\d{14})(?:\.[^.]+)?$", re.IGNORECASE) + +FileSelectionMode = Literal["manual", "latest", "uningested"] + +# Manifest statuses that mean "do not auto-pick this file again". +_INGESTED_STATUSES = frozenset({"BRONZE_WRITTEN"}) + + +@dataclass(frozen=True) +class FilePair: + stamp: str + cohort_file_name: str + course_file_name: str + cohort_row: dict[str, Any] + course_row: dict[str, Any] + + +def extract_file_stamp(file_name: str) -> str: + """Return the 14-digit trailing stamp from a file name.""" + base = os.path.basename(file_name) + m = FILE_STAMP_RE.search(base) + if not m: + raise ValueError( + "Expected file name to end with a 14-digit file stamp, e.g. " + f"'..._YYYYMMDDHHMMSS.csv'. Got: {file_name}" + ) + return m.group(1) + + +def try_extract_file_stamp(file_name: str) -> Optional[str]: + """Like extract_file_stamp but returns None when the stamp is missing.""" + try: + return extract_file_stamp(file_name) + except ValueError: + return None + + +def classify_pdp_file_role(file_name: str) -> Optional[Literal["cohort", "course"]]: + """ + Classify an SFTP file as cohort or course from its basename. + + Returns None when the name is ambiguous or does not match either role. + """ + base = os.path.basename(file_name).lower() + has_cohort = "cohort" in base + has_course = "course" in base + if has_cohort and not has_course: + return "cohort" + if has_course and not has_cohort: + return "course" + return None + + +def discover_file_pairs(file_rows: Iterable[Mapping[str, Any]]) -> list[FilePair]: + """ + Group SFTP listing rows into complete cohort+course pairs by stamp. + + Incomplete pairs (missing cohort or course) are omitted. + """ + by_stamp: dict[str, dict[str, dict[str, Any]]] = {} + for row in file_rows: + name = str(row.get("file_name") or "") + stamp = try_extract_file_stamp(name) + if not stamp: + continue + role = classify_pdp_file_role(name) + if role is None: + continue + by_stamp.setdefault(stamp, {})[role] = dict(row) + + pairs: list[FilePair] = [] + for stamp, roles in sorted(by_stamp.items(), key=lambda item: item[0]): + cohort_row = roles.get("cohort") + course_row = roles.get("course") + if not cohort_row or not course_row: + continue + pairs.append( + FilePair( + stamp=stamp, + cohort_file_name=str(cohort_row["file_name"]), + course_file_name=str(course_row["file_name"]), + cohort_row=cohort_row, + course_row=course_row, + ) + ) + return pairs + + +def _pair_is_fully_ingested( + pair: FilePair, + fingerprint_by_name: Mapping[str, str], + status_by_fingerprint: Mapping[str, str], +) -> bool: + fps = [ + fingerprint_by_name.get(pair.cohort_file_name), + fingerprint_by_name.get(pair.course_file_name), + ] + if not all(fps): + return False + statuses = [status_by_fingerprint.get(fp, "") for fp in fps if fp] + return bool(statuses) and all(s in _INGESTED_STATUSES for s in statuses) + + +def select_file_pair( + file_rows: list[Mapping[str, Any]], + *, + mode: str, + cohort_file_name: str = "", + course_file_name: str = "", + fingerprint_by_name: Optional[Mapping[str, str]] = None, + status_by_fingerprint: Optional[Mapping[str, str]] = None, +) -> tuple[str, str, str]: + """ + Resolve cohort/course file names for an ingestion run. + + Returns: + (cohort_file_name, course_file_name, selection_mode_used) + + Raises: + ValueError / FileNotFoundError when selection cannot be resolved. + """ + cohort_file_name = (cohort_file_name or "").strip() + course_file_name = (course_file_name or "").strip() + mode_norm = (mode or "uningested").strip().lower() + + if cohort_file_name and course_file_name: + cohort_stamp = extract_file_stamp(cohort_file_name) + course_stamp = extract_file_stamp(course_file_name) + if cohort_stamp != course_stamp: + raise ValueError( + "cohort_file_name and course_file_name must end with the same file stamp. " + f"Got cohort stamp={cohort_stamp}, course stamp={course_stamp}." + ) + return cohort_file_name, course_file_name, "manual" + + if mode_norm == "manual": + raise ValueError( + "file_selection_mode=manual requires both cohort_file_name and " + "course_file_name job parameters." + ) + + if mode_norm not in {"latest", "uningested"}: + raise ValueError( + f"Unsupported file_selection_mode={mode!r}. " + "Use 'manual', 'latest', or 'uningested'." + ) + + pairs = discover_file_pairs(file_rows) + if not pairs: + available = sorted( + {str(r.get("file_name")) for r in file_rows if r.get("file_name")} + ) + raise FileNotFoundError( + "No complete cohort/course pairs found on SFTP (need both roles sharing a " + f"14-digit stamp). Available file count={len(available)}; " + f"first 25={available[:25]}" + ) + + if mode_norm == "uningested": + fingerprint_by_name = fingerprint_by_name or {} + status_by_fingerprint = status_by_fingerprint or {} + eligible = [ + p + for p in pairs + if not _pair_is_fully_ingested( + p, fingerprint_by_name, status_by_fingerprint + ) + ] + if not eligible: + stamps = [p.stamp for p in pairs] + raise FileNotFoundError( + "All discovered cohort/course pairs are already BRONZE_WRITTEN in " + f"ingestion_manifest. stamps={stamps}" + ) + chosen = max(eligible, key=lambda p: p.stamp) + else: + chosen = max(pairs, key=lambda p: p.stamp) + + return chosen.cohort_file_name, chosen.course_file_name, mode_norm diff --git a/src/edvise/ingestion/nsc_sftp_helpers.py b/src/edvise/ingestion/nsc_sftp/helpers.py similarity index 99% rename from src/edvise/ingestion/nsc_sftp_helpers.py rename to src/edvise/ingestion/nsc_sftp/helpers.py index 2e1f567a0..ccf11c299 100644 --- a/src/edvise/ingestion/nsc_sftp_helpers.py +++ b/src/edvise/ingestion/nsc_sftp/helpers.py @@ -1,5 +1,5 @@ """ -NSC SFTP ingestion helpers. +NSC SFTP ingestion helpers (manifest, queue, plan, staging, bronze writes). NSC-specific utilities for processing SFTP files, extracting institution IDs, managing ingestion manifests, and working with Databricks schemas/volumes. @@ -22,7 +22,7 @@ from pyspark.sql import functions as F from pyspark.sql import types as T -from edvise.ingestion.constants import ( +from edvise.ingestion.nsc_sftp.constants import ( CATALOG, DEFAULT_SCHEMA, MANIFEST_TABLE_PATH, diff --git a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py new file mode 100644 index 000000000..a1c48481d --- /dev/null +++ b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py @@ -0,0 +1,244 @@ +""" +Connect to SFTP, select cohort/course files, upsert unseen files into +ingestion_manifest, and stage NEW files into pending_ingest_queue. + +File selection: + - If both ``cohort_file_name`` and ``course_file_name`` are set → manual. + - Else ``file_selection_mode``: + - ``uningested`` (default): newest stamp pair not fully BRONZE_WRITTEN + - ``latest``: newest stamp pair on SFTP + - ``manual``: requires both file name params + +Outputs: + - Delta: ingestion_manifest, pending_ingest_queue + - Staged files under UC volume path from nsc_sftp.constants.SFTP_TMP_DIR +""" + +from __future__ import annotations + +import logging +import sys + +from edvise.ingestion.nsc_sftp.constants import ( + configure_nsc_catalog, + parse_spark_python_task_params, + resolve_nsc_catalog, +) + +configure_nsc_catalog(resolve_nsc_catalog(sys.argv)) + +from databricks.connect import DatabricksSession +from pyspark.sql import functions as F + +from edvise import utils +from edvise.ingestion.nsc_sftp.constants import ( + MANIFEST_TABLE_PATH, + QUEUE_TABLE_PATH, + SFTP_REMOTE_FOLDER, + SFTP_SOURCE_SYSTEM, + SFTP_TMP_DIR, +) +from edvise.ingestion.nsc_sftp.file_selection import select_file_pair +from edvise.ingestion.nsc_sftp.helpers import ( + build_listing_df, + download_new_files_and_queue, + ensure_manifest_and_queue_tables, + get_files_to_queue, + upsert_new_to_manifest, +) +from edvise.utils.sftp import connect_sftp, list_receive_files + + +try: + dbutils # noqa: F821 +except NameError: + from unittest.mock import MagicMock + + dbutils = MagicMock() + +spark = DatabricksSession.builder.getOrCreate() + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", +) +logger = logging.getLogger(__name__) + +asset_scope = "nsc-sftp-asset" + +host = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-host") +user = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-user") +password = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-password") + +_argv = parse_spark_python_task_params(sys.argv) +cohort_file_name = str( + utils.databricks.get_db_widget_param( + "cohort_file_name", default=_argv.get("cohort_file_name", "") + ) +).strip() +course_file_name = str( + utils.databricks.get_db_widget_param( + "course_file_name", default=_argv.get("course_file_name", "") + ) +).strip() +file_selection_mode = ( + str( + utils.databricks.get_db_widget_param( + "file_selection_mode", + default=_argv.get("file_selection_mode", "uningested"), + ) + ) + .strip() + .lower() + or "uningested" +) + +logger.info("SFTP secured assets loaded successfully.") +logger.info(f"Staging to UC volume path: {SFTP_TMP_DIR}") +logger.info( + "Selection inputs: mode=%s cohort_file_name=%r course_file_name=%r", + file_selection_mode, + cohort_file_name, + course_file_name, +) + +transport = None +sftp = None + +try: + ensure_manifest_and_queue_tables(spark) + + transport, sftp = connect_sftp(host, user, password) + logger.info( + f"Connected to SFTP host={host} and scanning folder={SFTP_REMOTE_FOLDER}" + ) + + file_rows_all = list_receive_files(sftp, SFTP_REMOTE_FOLDER, SFTP_SOURCE_SYSTEM) + if not file_rows_all: + logger.info( + f"No files found in SFTP folder: {SFTP_REMOTE_FOLDER}. Exiting (no-op)." + ) + dbutils.notebook.exit("NO_FILES") + + available = sorted({r.get("file_name") for r in file_rows_all}) + logger.info( + f"Found {len(file_rows_all)} file(s) on SFTP in folder={SFTP_REMOTE_FOLDER}; " + f"first 25={available[:25]}" + ) + + # Fingerprints/statuses needed for uningested selection (full listing). + df_all_listing = build_listing_df(spark, file_rows_all) + fingerprint_by_name = { + r["file_name"]: r["file_fingerprint"] + for r in df_all_listing.select("file_name", "file_fingerprint").collect() + } + status_by_fingerprint: dict[str, str] = {} + if spark.catalog.tableExists(MANIFEST_TABLE_PATH): + status_by_fingerprint = { + r["file_fingerprint"]: r["status"] + for r in spark.table(MANIFEST_TABLE_PATH) + .select("file_fingerprint", "status") + .collect() + if r["file_fingerprint"] and r["status"] + } + + cohort_file_name, course_file_name, mode_used = select_file_pair( + file_rows_all, + mode=file_selection_mode, + cohort_file_name=cohort_file_name, + course_file_name=course_file_name, + fingerprint_by_name=fingerprint_by_name, + status_by_fingerprint=status_by_fingerprint, + ) + logger.info( + "Selected files via mode=%s: cohort=%s course=%s", + mode_used, + cohort_file_name, + course_file_name, + ) + + requested_names = {cohort_file_name, course_file_name} + file_rows = [r for r in file_rows_all if r.get("file_name") in requested_names] + + found_names = {r.get("file_name") for r in file_rows} + missing_names = sorted(requested_names - found_names) + if missing_names: + raise FileNotFoundError( + f"Requested file(s) not found on SFTP in folder '{SFTP_REMOTE_FOLDER}': " + f"{missing_names}. Available file count={len(available)}; " + f"first 25={available[:25]}" + ) + + for r in file_rows: + logger.info( + f"Selected SFTP file: name={r.get('file_name')} size={r.get('file_size')} " + f"modified={r.get('file_modified_time')}" + ) + + df_listing = build_listing_df(spark, file_rows) + fingerprints = [ + r["file_fingerprint"] for r in df_listing.select("file_fingerprint").collect() + ] + + logger.info("SFTP listing (selected files):") + df_listing.select( + "file_name", "file_size", "file_modified_time", "file_fingerprint" + ).show(truncate=False) + + upsert_new_to_manifest(spark, df_listing) + + logger.info("Manifest rows (selected files):") + spark.table(MANIFEST_TABLE_PATH).where( + F.col("file_fingerprint").isin(fingerprints) + ).select( + "file_name", + "file_fingerprint", + "status", + "processed_at", + "error_message", + ).show(truncate=False) + + df_to_queue = get_files_to_queue(spark, df_listing) + + to_queue_count = df_to_queue.count() + if to_queue_count == 0: + logger.info( + "No files to queue: either nothing is NEW, or NEW files are already queued. " + "Exiting (no-op)." + ) + dbutils.notebook.exit("QUEUED_FILES=0") + + logger.info("Files eligible to queue:") + df_to_queue.select( + "file_name", "file_size", "file_modified_time", "file_fingerprint" + ).show(truncate=False) + + logger.info( + f"Queuing {to_queue_count} NEW-unqueued file(s) to {QUEUE_TABLE_PATH} " + "and staging to UC volume." + ) + queued_count = download_new_files_and_queue(spark, sftp, df_to_queue, logger) + + logger.info("Queue rows (selected files):") + spark.table(QUEUE_TABLE_PATH).where( + F.col("file_fingerprint").isin(fingerprints) + ).select("file_name", "file_fingerprint", "local_tmp_path", "queued_at").show( + truncate=False + ) + + logger.info( + f"Queued {queued_count} file(s) for downstream processing in {QUEUE_TABLE_PATH}." + ) + dbutils.notebook.exit(f"QUEUED_FILES={queued_count}") + +finally: + try: + if sftp is not None: + sftp.close() + except Exception: + pass + try: + if transport is not None: + transport.close() + except Exception: + pass diff --git a/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py b/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py new file mode 100644 index 000000000..4e067c320 --- /dev/null +++ b/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py @@ -0,0 +1,203 @@ +""" +Read staged files from pending_ingest_queue, detect institution ID column, +expand to per-institution rows, and MERGE into institution_ingest_plan. + +No SFTP, no API calls, no volume writes beyond reading staged paths. +""" + +from __future__ import annotations + +import logging +import os +import re +import sys +from datetime import datetime, timezone + +from edvise.ingestion.nsc_sftp.constants import ( + configure_nsc_catalog, + resolve_nsc_catalog, +) + +configure_nsc_catalog(resolve_nsc_catalog(sys.argv)) + +from databricks.connect import DatabricksSession +from pyspark.sql import functions as F +from pyspark.sql import types as T + +from edvise.ingestion.nsc_sftp.constants import ( + COLUMN_RENAMES, + INSTITUTION_COLUMN_PATTERN, + PLAN_TABLE_PATH, + QUEUE_TABLE_PATH, +) +from edvise.ingestion.nsc_sftp.helpers import ensure_plan_table, extract_institution_ids + +try: + dbutils # noqa: F821 +except NameError: + from unittest.mock import MagicMock + + dbutils = MagicMock() + +spark = DatabricksSession.builder.getOrCreate() + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", +) +logger = logging.getLogger(__name__) + +INST_COL_PATTERN = re.compile(INSTITUTION_COLUMN_PATTERN, re.IGNORECASE) + +ensure_plan_table(spark, PLAN_TABLE_PATH) + +if not spark.catalog.tableExists(QUEUE_TABLE_PATH): + logger.info(f"Queue table {QUEUE_TABLE_PATH} not found. Exiting (no-op).") + dbutils.notebook.exit("NO_QUEUE_TABLE") + +queue_df = spark.read.table(QUEUE_TABLE_PATH) + +if queue_df.limit(1).count() == 0: + logger.info("pending_ingest_queue is empty. Exiting (no-op).") + dbutils.notebook.exit("NO_QUEUED_FILES") + +existing_fp = ( + spark.table(PLAN_TABLE_PATH).select("file_fingerprint").distinct() + if spark.catalog.tableExists(PLAN_TABLE_PATH) + else None +) +if existing_fp is not None: + queue_df = queue_df.join(existing_fp, on="file_fingerprint", how="left_anti") + +if queue_df.limit(1).count() == 0: + logger.info( + "All queued files have already been expanded into institution work items. Exiting (no-op)." + ) + dbutils.notebook.exit("NO_NEW_EXPANSION_WORK") + +logger.info("Queued files to expand preview (after excluding already-expanded):") +queue_df.select("file_fingerprint", "file_name", "local_tmp_path", "queued_at").show( + 25, truncate=False +) + +queued_files = queue_df.select( + "file_fingerprint", + "file_name", + F.col("local_tmp_path").alias("local_path"), + "file_size", + "file_modified_time", +).collect() + +logger.info( + f"Expanding {len(queued_files)} staged file(s) into per-institution work items..." +) + +work_items = [] +missing_files = [] + +for r in queued_files: + fp = r["file_fingerprint"] + file_name = r["file_name"] + local_path = r["local_path"] + + if not local_path or not os.path.exists(local_path): + missing_files.append((fp, file_name, local_path)) + continue + + try: + inst_col, inst_ids = extract_institution_ids( + local_path, renames=COLUMN_RENAMES, inst_col_pattern=INST_COL_PATTERN + ) + if inst_col is None: + logger.warning( + f"No institution id column found for file={file_name} fp={fp}. Skipping this file." + ) + continue + + if not inst_ids: + logger.warning( + f"Institution column found but no IDs present for file={file_name} fp={fp}. Skipping." + ) + continue + + now_ts = datetime.now(timezone.utc) + for inst_id in inst_ids: + work_items.append( + { + "file_fingerprint": fp, + "file_name": file_name, + "local_path": local_path, + "institution_id": inst_id, + "inst_col": inst_col, + "file_size": r["file_size"], + "file_modified_time": r["file_modified_time"], + "planned_at": now_ts, + } + ) + + preview_ids = inst_ids[:10] + logger.info( + f"file={file_name} fp={fp}: found {len(inst_ids)} institution id(s) using column '{inst_col}'. " + f"Preview first 10 IDs={preview_ids}" + ) + + except Exception as e: + logger.exception(f"Failed expanding file={file_name} fp={fp}: {e}") + raise + +if missing_files: + msg = ( + "Some staged files are missing on disk (staging path missing/inaccessible). " + + "; ".join([f"fp={fp} file={fn} path={lp}" for fp, fn, lp in missing_files]) + ) + logger.error(msg) + raise FileNotFoundError(msg) + +if not work_items: + logger.info("No work items generated from staged files. Exiting (no-op).") + dbutils.notebook.exit("NO_WORK_ITEMS") + +schema = T.StructType( + [ + T.StructField("file_fingerprint", T.StringType(), False), + T.StructField("file_name", T.StringType(), False), + T.StructField("local_path", T.StringType(), False), + T.StructField("institution_id", T.StringType(), False), + T.StructField("inst_col", T.StringType(), False), + T.StructField("file_size", T.LongType(), True), + T.StructField("file_modified_time", T.TimestampType(), True), + T.StructField("planned_at", T.TimestampType(), False), + ] +) + +df_plan = spark.createDataFrame(work_items, schema=schema) + +logger.info("Work items summary by file (distinct institutions):") +df_plan.groupBy("file_name").agg( + F.countDistinct("institution_id").alias("institution_count") +).orderBy("file_name").show(truncate=False) + +df_plan.createOrReplaceTempView("incoming_plan_rows") + +spark.sql( + f""" + MERGE INTO {PLAN_TABLE_PATH} AS t + USING incoming_plan_rows AS s + ON t.file_fingerprint = s.file_fingerprint + AND t.institution_id = s.institution_id + WHEN MATCHED THEN UPDATE SET + t.file_name = s.file_name, + t.local_path = s.local_path, + t.inst_col = s.inst_col, + t.file_size = s.file_size, + t.file_modified_time = s.file_modified_time, + t.planned_at = s.planned_at + WHEN NOT MATCHED THEN INSERT * + """ +) + +count_out = df_plan.count() +logger.info( + f"Wrote/updated {count_out} institution work item(s) into {PLAN_TABLE_PATH}." +) +dbutils.notebook.exit(f"WORK_ITEMS={count_out}") diff --git a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py new file mode 100644 index 000000000..f7e27655e --- /dev/null +++ b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py @@ -0,0 +1,497 @@ +""" +Consume institution_ingest_plan for manifest status=NEW; resolve institutions via SST API, +write filtered CSVs to per-institution bronze volumes, and update ingestion_manifest. + +No SFTP — uses staged local paths from prior steps. +""" + +from __future__ import annotations + +import logging +import os +import sys + +from edvise.ingestion.nsc_sftp.constants import ( + configure_nsc_catalog, + resolve_nsc_catalog, +) + +configure_nsc_catalog(resolve_nsc_catalog(sys.argv)) + +import pandas as pd +from databricks.connect import DatabricksSession +from pyspark.sql import functions as F + +from edvise.ingestion.nsc_sftp.constants import ( + CATALOG, + COLUMN_RENAMES, + INSTITUTION_LOOKUP_PATH, + MANIFEST_TABLE_PATH, + PLAN_TABLE_PATH, + SST_API_KEY_SECRET_KEY, + SST_BASE_URL, + SST_TOKEN_ENDPOINT, +) +from edvise.ingestion.nsc_sftp.helpers import ( + process_and_save_file, + update_manifest, +) +from edvise.utils.api_requests import ( + EdviseAPIClient, + fetch_institution_by_pdp_id, +) +from edvise.utils.data_cleaning import convert_to_snake_case +from edvise.utils.databricks import ( + find_bronze_schema, + find_bronze_volume_name, +) +from edvise.utils.institution_naming import databricksify_inst_name +from edvise.utils.sftp import output_file_name_from_sftp + +try: + dbutils # noqa: F821 +except NameError: + from unittest.mock import MagicMock + + dbutils = MagicMock() + +try: + display # noqa: F821 +except NameError: + + def display(x): + return x + + +spark = DatabricksSession.builder.getOrCreate() + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", +) +logger = logging.getLogger(__name__) + +asset_scope = "nsc-sftp-asset" +SST_API_KEY = dbutils.secrets.get(scope=asset_scope, key=SST_API_KEY_SECRET_KEY).strip() +if not SST_API_KEY: + raise RuntimeError( + f"Empty SST API key from secrets: scope={asset_scope} key={SST_API_KEY_SECRET_KEY}" + ) + +api_client = EdviseAPIClient( + api_key=SST_API_KEY, + base_url=SST_BASE_URL, + token_endpoint=SST_TOKEN_ENDPOINT, + institution_lookup_path=INSTITUTION_LOOKUP_PATH, +) + + +def _get_workflow_run_id(): + try: + ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() + tags = ctx.tags() + for k in ("jobRunId", "runId"): + try: + v = tags.apply(k) + if v: + return str(v) + except Exception: + pass + try: + v = ctx.currentRunId().get() + if v: + return str(v) + except Exception: + pass + except Exception: + pass + return None + + +if not spark.catalog.tableExists(PLAN_TABLE_PATH): + logger.info(f"Plan table not found: {PLAN_TABLE_PATH}. Exiting (no-op).") + dbutils.notebook.exit("NO_PLAN_TABLE") + +if not spark.catalog.tableExists(MANIFEST_TABLE_PATH): + raise RuntimeError(f"Manifest table missing: {MANIFEST_TABLE_PATH}") + +plan_df = spark.table(PLAN_TABLE_PATH) +if plan_df.limit(1).count() == 0: + logger.info("institution_ingest_plan is empty. Exiting (no-op).") + dbutils.notebook.exit("NO_WORK_ITEMS") + +manifest_df = spark.table(MANIFEST_TABLE_PATH).select("file_fingerprint", "status") +plan_new_df = plan_df.join(manifest_df, on="file_fingerprint", how="inner").where( + F.col("status") == F.lit("NEW") +) +if plan_new_df.limit(1).count() == 0: + logger.info("No planned work items where manifest status=NEW. Exiting (no-op).") + dbutils.notebook.exit("NO_NEW_TO_INGEST") + +plan_summary_df = ( + plan_new_df.groupBy("file_name", "inst_col", "local_path") + .agg(F.countDistinct("institution_id").alias("institution_count")) + .orderBy("file_name") +) +logger.info("Planned work summary (manifest status=NEW):") +display(plan_summary_df) + +file_groups = ( + plan_new_df.select( + "file_fingerprint", + "file_name", + "local_path", + "inst_col", + "file_size", + "file_modified_time", + ) + .distinct() + .collect() +) + +logger.info(f"Preparing to ingest {len(file_groups)} NEW file(s).") + +workflow_run_id = _get_workflow_run_id() +logger.info(f"Workflow run_id: {workflow_run_id}") + +processed_files = 0 +failed_files = 0 +skipped_files = 0 +institutions_written = 0 +institutions_skipped_existing = 0 +institutions_unresolved = 0 +institutions_no_bronze = 0 +institutions_empty = 0 + +for fg in file_groups: + fp = fg["file_fingerprint"] + sftp_file_name = fg["file_name"] + local_path = fg["local_path"] + inst_col = fg["inst_col"] + + if not local_path or not os.path.exists(local_path): + err = f"Staged local file missing for fp={fp}: {local_path}" + logger.error(err) + update_manifest( + spark, + MANIFEST_TABLE_PATH, + fp, + status="FAILED", + error_message=err[:8000], + run_id=workflow_run_id, + ) + failed_files += 1 + continue + + try: + header_cols = pd.read_csv(local_path, nrows=0).columns.tolist() + raw_inst_col = next( + ( + c + for c in header_cols + if COLUMN_RENAMES.get( + convert_to_snake_case(c), convert_to_snake_case(c) + ) + == inst_col + ), + None, + ) + dtype = {raw_inst_col: str} if raw_inst_col else None + df_full = pd.read_csv(local_path, on_bad_lines="warn", dtype=dtype) + df_full = df_full.rename( + columns={c: convert_to_snake_case(c) for c in df_full.columns} + ) + df_full = df_full.rename(columns=COLUMN_RENAMES) + + file_student_count = None + try: + student_col = next( + ( + c + for c in ("student_id", "study_id", "student_guid") + if c in df_full.columns + ), + None, + ) + if student_col: + file_student_count = int(df_full[student_col].nunique(dropna=True)) + except Exception: + file_student_count = None + + file_cohort = None + try: + if "cohort" in df_full.columns: + vals = ( + df_full["cohort"] + .dropna() + .astype(str) + .map(lambda x: x.strip()) + .tolist() + ) + vals = [ + v for v in vals if v and v.lower() not in {"nan", "none", "null"} + ] + file_cohort = sorted(set(vals)) or None + except Exception: + file_cohort = None + + file_cohort_term_pairs = None + try: + if {"cohort", "cohort_term"}.issubset(df_full.columns): + tmp = df_full[["cohort", "cohort_term"]].dropna() + tmp = tmp.assign( + cohort=tmp["cohort"].astype(str).map(lambda x: x.strip()), + cohort_term=tmp["cohort_term"] + .astype(str) + .map(lambda x: x.strip().upper()), + ) + tmp = tmp[ + (tmp["cohort"] != "") + & (tmp["cohort_term"] != "") + & (~tmp["cohort"].str.lower().isin({"nan", "none", "null"})) + & (~tmp["cohort_term"].str.lower().isin({"nan", "none", "null"})) + ] + tmp = tmp.drop_duplicates().sort_values(by=["cohort", "cohort_term"]) + pairs = [ + {"cohort": r.cohort, "cohort_term": r.cohort_term} + for r in tmp.itertuples(index=False) + ] + file_cohort_term_pairs = pairs or None + except Exception: + file_cohort_term_pairs = None + + logger.info( + "file=%s fp=%s: student_count=%s cohort_count=%s", + sftp_file_name, + fp, + file_student_count, + (len(file_cohort) if file_cohort else 0), + ) + + if inst_col not in df_full.columns: + err = f"Expected institution column '{inst_col}' not found after normalization/renames for file={sftp_file_name} fp={fp}" + logger.error(err) + update_manifest( + spark, + MANIFEST_TABLE_PATH, + fp, + status="FAILED", + error_message=err[:8000], + run_id=workflow_run_id, + cohort=file_cohort, + cohort_term_pairs=file_cohort_term_pairs, + student_count=file_student_count, + ) + failed_files += 1 + continue + + inst_ids = ( + plan_new_df.where(F.col("file_fingerprint") == fp) + .select("institution_id") + .distinct() + .collect() + ) + inst_ids = [r["institution_id"] for r in inst_ids] + + if not inst_ids: + logger.info( + f"No institution_ids in plan for file={sftp_file_name} fp={fp}. Marking BRONZE_WRITTEN (no-op)." + ) + update_manifest( + spark, + MANIFEST_TABLE_PATH, + fp, + status="BRONZE_WRITTEN", + error_message=None, + run_id=workflow_run_id, + cohort=file_cohort, + cohort_term_pairs=file_cohort_term_pairs, + student_count=file_student_count, + ) + skipped_files += 1 + continue + + preview_inst_ids = inst_ids[:10] + logger.info( + f"file={sftp_file_name} fp={fp}: ingesting {len(inst_ids)} institution(s) " + f"using inst_col='{inst_col}'. Preview first 10 IDs={preview_inst_ids}" + ) + + file_errors = [] + + for inst_id in inst_ids: + try: + target_inst_id = str(inst_id) + filtered_df = df_full[df_full[inst_col] == target_inst_id].reset_index( + drop=True + ) + + if filtered_df.empty: + institutions_empty += 1 + logger.info( + f"file={sftp_file_name} fp={fp}: institution {inst_id} has 0 rows; skipping." + ) + continue + + # Parity with interactive PIPELINE_pdp_to_databricks checks. + if {"cohort", "cohort_term"}.issubset(filtered_df.columns): + latest_cohort = filtered_df["cohort"].max() + latest_cohort_terms = ( + filtered_df.loc[ + filtered_df["cohort"] == latest_cohort, "cohort_term" + ] + .dropna() + .astype(str) + .unique() + .tolist() + ) + logger.info( + "School check file=%s inst=%s rows=%s latest_cohort=%s " + "latest_cohort_terms=%s", + sftp_file_name, + inst_id, + len(filtered_df), + latest_cohort, + latest_cohort_terms, + ) + else: + logger.info( + "School check file=%s inst=%s rows=%s " + "(no cohort/cohort_term columns)", + sftp_file_name, + inst_id, + len(filtered_df), + ) + + try: + inst_info = fetch_institution_by_pdp_id(api_client, inst_id) + except Exception as api_err: + institutions_unresolved += 1 + raise ValueError( + f"SST API lookup failed for pdp_id={inst_id}: {api_err}" + ) from api_err + + inst_name = inst_info.get("name") + if not inst_name: + institutions_unresolved += 1 + raise ValueError( + f"SST API returned no 'name' for pdp_id={inst_id}. " + f"Response={inst_info}" + ) + + inst_prefix = databricksify_inst_name(inst_name) + logger.info( + "Resolved school file=%s pdp_id=%s name=%r prefix=%s", + sftp_file_name, + inst_id, + inst_name, + inst_prefix, + ) + + try: + bronze_schema = find_bronze_schema(spark, CATALOG, inst_prefix) + bronze_volume_name = find_bronze_volume_name( + spark, CATALOG, bronze_schema + ) + except ValueError as bronze_err: + institutions_no_bronze += 1 + raise ValueError( + f"Bronze not provisioned for pdp_id={inst_id} " + f"name={inst_name!r} prefix={inst_prefix}: {bronze_err}" + ) from bronze_err + + volume_dir = f"/Volumes/{CATALOG}/{bronze_schema}/{bronze_volume_name}" + + out_file_name = output_file_name_from_sftp(sftp_file_name) + full_path = os.path.join(volume_dir, out_file_name) + + if os.path.exists(full_path): + institutions_skipped_existing += 1 + logger.info( + f"file={sftp_file_name} inst={inst_id}: already exists in " + f"{volume_dir}; skipping write." + ) + continue + + logger.info( + f"file={sftp_file_name} inst={inst_id}: writing to {volume_dir} " + f"as {out_file_name}" + ) + process_and_save_file( + volume_dir=volume_dir, file_name=out_file_name, df=filtered_df + ) + institutions_written += 1 + logger.info(f"file={sftp_file_name} inst={inst_id}: write complete.") + + except Exception as e: + msg = ( + f"inst_ingest_failed file={sftp_file_name} fp={fp} " + f"inst={inst_id}: {e}" + ) + logger.exception(msg) + file_errors.append(msg) + + if file_errors: + err = " | ".join(file_errors)[:8000] + update_manifest( + spark, + MANIFEST_TABLE_PATH, + fp, + status="FAILED", + error_message=err, + run_id=workflow_run_id, + cohort=file_cohort, + cohort_term_pairs=file_cohort_term_pairs, + student_count=file_student_count, + ) + failed_files += 1 + else: + update_manifest( + spark, + MANIFEST_TABLE_PATH, + fp, + status="BRONZE_WRITTEN", + error_message=None, + run_id=workflow_run_id, + cohort=file_cohort, + cohort_term_pairs=file_cohort_term_pairs, + student_count=file_student_count, + ) + processed_files += 1 + + except Exception as e: + msg = f"fatal_file_error file={sftp_file_name} fp={fp}: {e}" + logger.exception(msg) + update_manifest( + spark, + MANIFEST_TABLE_PATH, + fp, + status="FAILED", + error_message=msg[:8000], + run_id=workflow_run_id, + ) + failed_files += 1 + +logger.info( + "Done. processed_files=%s failed_files=%s skipped_files=%s " + "institutions_written=%s institutions_skipped_existing=%s " + "institutions_unresolved=%s institutions_no_bronze=%s institutions_empty=%s", + processed_files, + failed_files, + skipped_files, + institutions_written, + institutions_skipped_existing, + institutions_unresolved, + institutions_no_bronze, + institutions_empty, +) +if institutions_unresolved or institutions_no_bronze: + logger.warning( + "Some institutions were not fully ingestible (API unresolved or missing bronze). " + "See per-institution errors above; file-level manifest status reflects failures." + ) +dbutils.notebook.exit( + f"PROCESSED={processed_files};FAILED={failed_files};SKIPPED={skipped_files};" + f"WRITTEN={institutions_written};EXISTING={institutions_skipped_existing};" + f"UNRESOLVED={institutions_unresolved};NO_BRONZE={institutions_no_bronze}" +) diff --git a/src/edvise/ingestion/nsc_sftp/scripts/__init__.py b/src/edvise/ingestion/nsc_sftp/scripts/__init__.py new file mode 100644 index 000000000..96604c985 --- /dev/null +++ b/src/edvise/ingestion/nsc_sftp/scripts/__init__.py @@ -0,0 +1 @@ +"""Entrypoint scripts for NSC SFTP ingestion (run as Databricks job tasks or locally).""" diff --git a/tests/ingestion/test_file_selection.py b/tests/ingestion/test_file_selection.py new file mode 100644 index 000000000..a5fd7f8bd --- /dev/null +++ b/tests/ingestion/test_file_selection.py @@ -0,0 +1,117 @@ +import pytest + +from edvise.ingestion.nsc_sftp.file_selection import ( + classify_pdp_file_role, + discover_file_pairs, + extract_file_stamp, + select_file_pair, +) + + +def _row(name: str, size: int = 10) -> dict: + return { + "source_system": "NSC", + "sftp_path": "./receive", + "file_name": name, + "file_size": size, + "file_modified_time": None, + } + + +def test_extract_file_stamp(): + assert extract_file_stamp("PDP_Cohort_File_20240115123045.csv") == "20240115123045" + + +def test_classify_pdp_file_role(): + assert classify_pdp_file_role("School_Cohort_20240115123045.csv") == "cohort" + assert classify_pdp_file_role("School_Course_20240115123045.csv") == "course" + assert classify_pdp_file_role("readme.txt") is None + + +def test_discover_file_pairs_requires_both_roles(): + rows = [ + _row("A_Cohort_20240115123045.csv"), + _row("A_Course_20240115123045.csv"), + _row("B_Cohort_20240201101010.csv"), # incomplete pair + _row("noise_20240301111111.csv"), + ] + pairs = discover_file_pairs(rows) + assert len(pairs) == 1 + assert pairs[0].stamp == "20240115123045" + assert pairs[0].cohort_file_name.endswith("Cohort_20240115123045.csv") + assert pairs[0].course_file_name.endswith("Course_20240115123045.csv") + + +def test_select_file_pair_manual(): + cohort = "A_Cohort_20240115123045.csv" + course = "A_Course_20240115123045.csv" + c, o, mode = select_file_pair( + [], + mode="uningested", + cohort_file_name=cohort, + course_file_name=course, + ) + assert (c, o, mode) == (cohort, course, "manual") + + +def test_select_file_pair_latest(): + rows = [ + _row("A_Cohort_20240115123045.csv"), + _row("A_Course_20240115123045.csv"), + _row("B_Cohort_20240201101010.csv"), + _row("B_Course_20240201101010.csv"), + ] + c, o, mode = select_file_pair(rows, mode="latest") + assert mode == "latest" + assert c == "B_Cohort_20240201101010.csv" + assert o == "B_Course_20240201101010.csv" + + +def test_select_file_pair_uningested_skips_bronze_written(): + rows = [ + _row("A_Cohort_20240115123045.csv"), + _row("A_Course_20240115123045.csv"), + _row("B_Cohort_20240201101010.csv"), + _row("B_Course_20240201101010.csv"), + ] + fingerprint_by_name = { + "B_Cohort_20240201101010.csv": "fp_b_cohort", + "B_Course_20240201101010.csv": "fp_b_course", + "A_Cohort_20240115123045.csv": "fp_a_cohort", + "A_Course_20240115123045.csv": "fp_a_course", + } + status_by_fingerprint = { + "fp_b_cohort": "BRONZE_WRITTEN", + "fp_b_course": "BRONZE_WRITTEN", + } + c, o, mode = select_file_pair( + rows, + mode="uningested", + fingerprint_by_name=fingerprint_by_name, + status_by_fingerprint=status_by_fingerprint, + ) + assert mode == "uningested" + assert c == "A_Cohort_20240115123045.csv" + assert o == "A_Course_20240115123045.csv" + + +def test_select_file_pair_uningested_all_done_raises(): + rows = [ + _row("A_Cohort_20240115123045.csv"), + _row("A_Course_20240115123045.csv"), + ] + fingerprint_by_name = { + "A_Cohort_20240115123045.csv": "fp_a_cohort", + "A_Course_20240115123045.csv": "fp_a_course", + } + status_by_fingerprint = { + "fp_a_cohort": "BRONZE_WRITTEN", + "fp_a_course": "BRONZE_WRITTEN", + } + with pytest.raises(FileNotFoundError, match="already BRONZE_WRITTEN"): + select_file_pair( + rows, + mode="uningested", + fingerprint_by_name=fingerprint_by_name, + status_by_fingerprint=status_by_fingerprint, + ) diff --git a/tests/ingestion/test_nsc_sftp_helper.py b/tests/ingestion/test_nsc_sftp_helper.py index 4a35a4ed0..740f1af05 100644 --- a/tests/ingestion/test_nsc_sftp_helper.py +++ b/tests/ingestion/test_nsc_sftp_helper.py @@ -1,6 +1,6 @@ import re -from edvise.ingestion.nsc_sftp_helpers import ( +from edvise.ingestion.nsc_sftp.helpers import ( detect_institution_column, extract_institution_ids, ) From 655078548f069e2c9170dc68c3b62b39c8564f5b Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Wed, 5 Aug 2026 14:03:39 -0700 Subject: [PATCH 03/10] refactor: condense NSC SFTP ingestion scripts for efficiency Avoid full-listing Spark fingerprinting for file selection, read only the institution column during expand, and groupby institutions once per staged file during bronze writes. Co-authored-by: Cursor --- src/edvise/ingestion/nsc_sftp/__init__.py | 4 +- .../ingestion/nsc_sftp/file_selection.py | 103 +--- src/edvise/ingestion/nsc_sftp/helpers.py | 205 ++++--- src/edvise/ingestion/nsc_sftp/runtime.py | 72 +++ .../nsc_sftp/scripts/01_sftp_receive_scan.py | 204 ++----- .../scripts/02_file_institution_expand.py | 185 ++----- .../03_per_institution_bronze_ingest.py | 514 ++++++------------ tests/ingestion/test_file_selection.py | 30 +- 8 files changed, 506 insertions(+), 811 deletions(-) create mode 100644 src/edvise/ingestion/nsc_sftp/runtime.py diff --git a/src/edvise/ingestion/nsc_sftp/__init__.py b/src/edvise/ingestion/nsc_sftp/__init__.py index 48f8ed28e..3b03d8162 100644 --- a/src/edvise/ingestion/nsc_sftp/__init__.py +++ b/src/edvise/ingestion/nsc_sftp/__init__.py @@ -1,5 +1,5 @@ """NSC SFTP automated ingestion (constants, helpers, job scripts).""" -from edvise.ingestion.nsc_sftp import constants, file_selection, helpers +from edvise.ingestion.nsc_sftp import constants, file_selection, helpers, runtime -__all__ = ["constants", "file_selection", "helpers"] +__all__ = ["constants", "file_selection", "helpers", "runtime"] diff --git a/src/edvise/ingestion/nsc_sftp/file_selection.py b/src/edvise/ingestion/nsc_sftp/file_selection.py index 2c81dfe89..fbbf74bbb 100644 --- a/src/edvise/ingestion/nsc_sftp/file_selection.py +++ b/src/edvise/ingestion/nsc_sftp/file_selection.py @@ -13,26 +13,19 @@ from typing import Any, Iterable, Literal, Mapping, Optional FILE_STAMP_RE = re.compile(r"_(\d{14})(?:\.[^.]+)?$", re.IGNORECASE) - FileSelectionMode = Literal["manual", "latest", "uningested"] -# Manifest statuses that mean "do not auto-pick this file again". -_INGESTED_STATUSES = frozenset({"BRONZE_WRITTEN"}) - @dataclass(frozen=True) class FilePair: stamp: str cohort_file_name: str course_file_name: str - cohort_row: dict[str, Any] - course_row: dict[str, Any] def extract_file_stamp(file_name: str) -> str: """Return the 14-digit trailing stamp from a file name.""" - base = os.path.basename(file_name) - m = FILE_STAMP_RE.search(base) + m = FILE_STAMP_RE.search(os.path.basename(file_name)) if not m: raise ValueError( "Expected file name to end with a 14-digit file stamp, e.g. " @@ -42,7 +35,6 @@ def extract_file_stamp(file_name: str) -> str: def try_extract_file_stamp(file_name: str) -> Optional[str]: - """Like extract_file_stamp but returns None when the stamp is missing.""" try: return extract_file_stamp(file_name) except ValueError: @@ -50,11 +42,6 @@ def try_extract_file_stamp(file_name: str) -> Optional[str]: def classify_pdp_file_role(file_name: str) -> Optional[Literal["cohort", "course"]]: - """ - Classify an SFTP file as cohort or course from its basename. - - Returns None when the name is ambiguous or does not match either role. - """ base = os.path.basename(file_name).lower() has_cohort = "cohort" in base has_course = "course" in base @@ -66,12 +53,8 @@ def classify_pdp_file_role(file_name: str) -> Optional[Literal["cohort", "course def discover_file_pairs(file_rows: Iterable[Mapping[str, Any]]) -> list[FilePair]: - """ - Group SFTP listing rows into complete cohort+course pairs by stamp. - - Incomplete pairs (missing cohort or course) are omitted. - """ - by_stamp: dict[str, dict[str, dict[str, Any]]] = {} + """Group SFTP listing rows into complete cohort+course pairs by stamp.""" + by_stamp: dict[str, dict[str, str]] = {} for row in file_rows: name = str(row.get("file_name") or "") stamp = try_extract_file_stamp(name) @@ -80,39 +63,17 @@ def discover_file_pairs(file_rows: Iterable[Mapping[str, Any]]) -> list[FilePair role = classify_pdp_file_role(name) if role is None: continue - by_stamp.setdefault(stamp, {})[role] = dict(row) + by_stamp.setdefault(stamp, {})[role] = name - pairs: list[FilePair] = [] - for stamp, roles in sorted(by_stamp.items(), key=lambda item: item[0]): - cohort_row = roles.get("cohort") - course_row = roles.get("course") - if not cohort_row or not course_row: - continue - pairs.append( - FilePair( - stamp=stamp, - cohort_file_name=str(cohort_row["file_name"]), - course_file_name=str(course_row["file_name"]), - cohort_row=cohort_row, - course_row=course_row, - ) + return [ + FilePair( + stamp=stamp, + cohort_file_name=roles["cohort"], + course_file_name=roles["course"], ) - return pairs - - -def _pair_is_fully_ingested( - pair: FilePair, - fingerprint_by_name: Mapping[str, str], - status_by_fingerprint: Mapping[str, str], -) -> bool: - fps = [ - fingerprint_by_name.get(pair.cohort_file_name), - fingerprint_by_name.get(pair.course_file_name), + for stamp, roles in sorted(by_stamp.items()) + if "cohort" in roles and "course" in roles ] - if not all(fps): - return False - statuses = [status_by_fingerprint.get(fp, "") for fp in fps if fp] - return bool(statuses) and all(s in _INGESTED_STATUSES for s in statuses) def select_file_pair( @@ -121,17 +82,14 @@ def select_file_pair( mode: str, cohort_file_name: str = "", course_file_name: str = "", - fingerprint_by_name: Optional[Mapping[str, str]] = None, - status_by_fingerprint: Optional[Mapping[str, str]] = None, + ingested_file_names: Optional[Iterable[str]] = None, ) -> tuple[str, str, str]: """ Resolve cohort/course file names for an ingestion run. - Returns: - (cohort_file_name, course_file_name, selection_mode_used) - - Raises: - ValueError / FileNotFoundError when selection cannot be resolved. + ``uningested`` skips pairs whose cohort and course names are both present in + ``ingested_file_names`` (typically BRONZE_WRITTEN file_name values). Stamp-based + NSC names make file_name a stable version key without Spark fingerprinting. """ cohort_file_name = (cohort_file_name or "").strip() course_file_name = (course_file_name or "").strip() @@ -152,7 +110,6 @@ def select_file_pair( "file_selection_mode=manual requires both cohort_file_name and " "course_file_name job parameters." ) - if mode_norm not in {"latest", "uningested"}: raise ValueError( f"Unsupported file_selection_mode={mode!r}. " @@ -170,24 +127,18 @@ def select_file_pair( f"first 25={available[:25]}" ) - if mode_norm == "uningested": - fingerprint_by_name = fingerprint_by_name or {} - status_by_fingerprint = status_by_fingerprint or {} - eligible = [ - p - for p in pairs - if not _pair_is_fully_ingested( - p, fingerprint_by_name, status_by_fingerprint - ) - ] - if not eligible: - stamps = [p.stamp for p in pairs] - raise FileNotFoundError( - "All discovered cohort/course pairs are already BRONZE_WRITTEN in " - f"ingestion_manifest. stamps={stamps}" - ) - chosen = max(eligible, key=lambda p: p.stamp) - else: + if mode_norm == "latest": chosen = max(pairs, key=lambda p: p.stamp) + return chosen.cohort_file_name, chosen.course_file_name, mode_norm + done = set(ingested_file_names or ()) + eligible = [ + p for p in pairs if not ({p.cohort_file_name, p.course_file_name} <= done) + ] + if not eligible: + raise FileNotFoundError( + "All discovered cohort/course pairs are already BRONZE_WRITTEN in " + f"ingestion_manifest. stamps={[p.stamp for p in pairs]}" + ) + chosen = max(eligible, key=lambda p: p.stamp) return chosen.cohort_file_name, chosen.course_file_name, mode_norm diff --git a/src/edvise/ingestion/nsc_sftp/helpers.py b/src/edvise/ingestion/nsc_sftp/helpers.py index ccf11c299..1ff492707 100644 --- a/src/edvise/ingestion/nsc_sftp/helpers.py +++ b/src/edvise/ingestion/nsc_sftp/helpers.py @@ -413,88 +413,161 @@ def ensure_plan_table(spark: pyspark.sql.SparkSession, plan_table: str) -> None: ) -def extract_institution_ids( +def _normalize_header_map( + header_cols: list[str], renames: dict[str, str] +) -> dict[str, str]: + """Map raw CSV header -> normalized/renamed column name.""" + out: dict[str, str] = {} + for raw in header_cols: + normalized = convert_to_snake_case(raw) + out[raw] = renames.get(normalized, normalized) + return out + + +def normalize_staged_frame( + df: pd.DataFrame, *, renames: dict[str, str] +) -> pd.DataFrame: + """Apply snake_case + COLUMN_RENAMES to a staged PDP frame.""" + return df.rename( + columns={ + c: renames.get(convert_to_snake_case(c), convert_to_snake_case(c)) + for c in df.columns + } + ) + + +def load_staged_csv( local_path: str, *, renames: dict[str, str], - inst_col_pattern: re.Pattern, -) -> tuple[Optional[str], list[str]]: + inst_col: Optional[str] = None, +) -> pd.DataFrame: + """ + Load a staged CSV once with institution IDs forced to string when possible. """ - Extract unique institution IDs from a staged CSV file. + header_cols = pd.read_csv(local_path, nrows=0).columns.tolist() + header_map = _normalize_header_map(header_cols, renames) + dtype = None + if inst_col: + raw_inst_col = next( + (raw for raw, norm in header_map.items() if norm == inst_col), None + ) + if raw_inst_col: + dtype = {raw_inst_col: str} + df = pd.read_csv(local_path, on_bad_lines="warn", dtype=dtype) + return normalize_staged_frame(df, renames=renames) + + +def summarize_file_metrics( + df: pd.DataFrame, +) -> tuple[Optional[int], Optional[list[str]], Optional[list[dict[str, str]]]]: + """Cheap file-level metrics for manifest updates / logging.""" + student_count = None + student_col = next( + (c for c in ("student_id", "study_id", "student_guid") if c in df.columns), + None, + ) + if student_col: + student_count = int(df[student_col].nunique(dropna=True)) + + file_cohort = None + if "cohort" in df.columns: + vals = df["cohort"].dropna().astype(str).str.strip() + vals = vals[~vals.str.lower().isin({"", "nan", "none", "null"})] + uniq = sorted(vals.unique().tolist()) + file_cohort = uniq or None + + file_cohort_term_pairs = None + if {"cohort", "cohort_term"}.issubset(df.columns): + tmp = df.loc[:, ["cohort", "cohort_term"]].dropna().copy() + tmp["cohort"] = tmp["cohort"].astype(str).str.strip() + tmp["cohort_term"] = tmp["cohort_term"].astype(str).str.strip().str.upper() + bad = {"", "nan", "none", "null"} + tmp = tmp[ + ~tmp["cohort"].str.lower().isin(bad) + & ~tmp["cohort_term"].str.lower().isin(bad) + ] + tmp = tmp.drop_duplicates().sort_values(["cohort", "cohort_term"]) + pairs = [ + {"cohort": r.cohort, "cohort_term": r.cohort_term} + for r in tmp.itertuples(index=False) + ] + file_cohort_term_pairs = pairs or None - Reads file, normalizes/renames columns, detects institution column, - and returns unique institution IDs. + return student_count, file_cohort, file_cohort_term_pairs - Args: - local_path: Path to local CSV file - renames: Dictionary mapping old column names to new names - inst_col_pattern: Compiled regex pattern to match institution column - Returns: - Tuple of (institution_column_name, sorted_list_of_unique_ids). - Returns (None, []) if no institution column found. - - Example: - >>> pattern = re.compile(r"(?=.*institution)(?=.*id)", re.IGNORECASE) - >>> renames = {"inst_id": "institution_id"} - >>> col, ids = extract_institution_ids( - ... "/tmp/file.csv", renames=renames, inst_col_pattern=pattern - ... ) - >>> print(col, ids) - 'institution_id' ['12345', '67890'] +def _normalize_institution_id(value: object) -> Optional[str]: + try: + if isinstance(value, bool): + return None + if isinstance(value, int): + return str(value) + if isinstance(value, float): + if not math.isfinite(value): + return None + return str(int(value)) if value.is_integer() else str(value).strip() + except Exception: + pass + + s = str(value).strip() + if s == "" or s.lower() in { + "nan", + "inf", + "+inf", + "-inf", + "infinity", + "+infinity", + "-infinity", + }: + return None + if re.fullmatch(r"\d+\.0+", s): + return s.split(".", 1)[0] + return s + + +def extract_institution_ids( + local_path: str, + *, + renames: dict[str, str], + inst_col_pattern: re.Pattern, +) -> tuple[Optional[str], list[str]]: """ - df = pd.read_csv(local_path, on_bad_lines="warn") - # Use convert_to_snake_case from utils instead of normalize_col - df = df.rename(columns={c: convert_to_snake_case(c) for c in df.columns}) - df = df.rename(columns=renames) + Extract unique institution IDs from a staged CSV. - inst_col = detect_institution_column(df.columns.tolist(), inst_col_pattern) + Only the institution column is fully read (header scan first), which keeps + stage-02 cheap for wide PDP files. + """ + header_cols = pd.read_csv(local_path, nrows=0).columns.tolist() + header_map = _normalize_header_map(header_cols, renames) + inst_col = detect_institution_column(list(header_map.values()), inst_col_pattern) if inst_col is None: return None, [] - # Make IDs robust: drop nulls, strip whitespace, keep as string - series = df[inst_col].dropna() - - # Some files store as numeric; normalize to integer-like strings when possible - ids = set() - for v in series.tolist(): - # Handle pandas/numpy numeric types - try: - if isinstance(v, int): - ids.add(str(v)) - continue - if isinstance(v, float): - # Treat +/-inf as invalid IDs - if not math.isfinite(v): - continue - # If 323100.0 -> "323100" - if v.is_integer(): - ids.add(str(int(v))) - else: - ids.add(str(v).strip()) - continue - except Exception: - pass - - s = str(v).strip() - if s == "" or s.lower() in { - "nan", - "inf", - "+inf", - "-inf", - "infinity", - "+infinity", - "-infinity", - }: - continue - # If it's "323100.0" as string, coerce safely - if re.fullmatch(r"\d+\.0+", s): - s = s.split(".")[0] - ids.add(s) + raw_inst_col = next(raw for raw, norm in header_map.items() if norm == inst_col) + series = pd.read_csv(local_path, usecols=[raw_inst_col], on_bad_lines="warn")[ + raw_inst_col + ].dropna() + ids: set[str] = set() + for value in series.tolist(): + normalized = _normalize_institution_id(value) + if normalized is not None: + ids.add(normalized) return inst_col, sorted(ids) +def resolve_bronze_volume_dir( + spark: pyspark.sql.SparkSession, catalog: str, inst_prefix: str +) -> str: + """Return `/Volumes/{catalog}/{schema}/{volume}` for an institution prefix.""" + from edvise.utils.databricks import find_bronze_schema, find_bronze_volume_name + + bronze_schema = find_bronze_schema(spark, catalog, inst_prefix) + bronze_volume_name = find_bronze_volume_name(spark, catalog, bronze_schema) + return f"/Volumes/{catalog}/{bronze_schema}/{bronze_volume_name}" + + def update_manifest( spark: pyspark.sql.SparkSession, manifest_table: str, diff --git a/src/edvise/ingestion/nsc_sftp/runtime.py b/src/edvise/ingestion/nsc_sftp/runtime.py new file mode 100644 index 000000000..978be365e --- /dev/null +++ b/src/edvise/ingestion/nsc_sftp/runtime.py @@ -0,0 +1,72 @@ +"""Shared bootstrap for NSC SFTP spark_python_task scripts.""" + +from __future__ import annotations + +import logging +import sys +from typing import Any +from unittest.mock import MagicMock + +from edvise.ingestion.nsc_sftp.constants import ( + configure_nsc_catalog, + parse_spark_python_task_params, + resolve_nsc_catalog, +) + + +def bootstrap_catalog(argv: list[str] | None = None) -> None: + """Configure UC catalog paths from job argv / widgets / env.""" + configure_nsc_catalog(resolve_nsc_catalog(sys.argv if argv is None else argv)) + + +def get_dbutils() -> Any: + try: + return dbutils # type: ignore[name-defined] # noqa: F821 + except NameError: + return MagicMock() + + +def get_spark(): + from databricks.connect import DatabricksSession + + return DatabricksSession.builder.getOrCreate() + + +def get_logger(name: str) -> logging.Logger: + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", + ) + return logging.getLogger(name) + + +def job_param(name: str, default: str = "", *, argv: list[str] | None = None) -> str: + """Resolve a job/widget parameter as a stripped string.""" + from edvise import utils + + pairs = parse_spark_python_task_params(sys.argv if argv is None else argv) + return str( + utils.databricks.get_db_widget_param(name, default=pairs.get(name, default)) + ).strip() + + +def workflow_run_id(dbutils_obj: Any) -> str | None: + try: + ctx = dbutils_obj.notebook.entry_point.getDbutils().notebook().getContext() + tags = ctx.tags() + for key in ("jobRunId", "runId"): + try: + value = tags.apply(key) + if value: + return str(value) + except Exception: + pass + try: + value = ctx.currentRunId().get() + if value: + return str(value) + except Exception: + pass + except Exception: + pass + return None diff --git a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py index a1c48481d..12c06d008 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py @@ -1,36 +1,15 @@ """ -Connect to SFTP, select cohort/course files, upsert unseen files into -ingestion_manifest, and stage NEW files into pending_ingest_queue. - -File selection: - - If both ``cohort_file_name`` and ``course_file_name`` are set → manual. - - Else ``file_selection_mode``: - - ``uningested`` (default): newest stamp pair not fully BRONZE_WRITTEN - - ``latest``: newest stamp pair on SFTP - - ``manual``: requires both file name params - -Outputs: - - Delta: ingestion_manifest, pending_ingest_queue - - Staged files under UC volume path from nsc_sftp.constants.SFTP_TMP_DIR +SFTP scan → select cohort/course pair → stage NEW files into pending_ingest_queue. """ from __future__ import annotations -import logging -import sys +from edvise.ingestion.nsc_sftp import runtime -from edvise.ingestion.nsc_sftp.constants import ( - configure_nsc_catalog, - parse_spark_python_task_params, - resolve_nsc_catalog, -) - -configure_nsc_catalog(resolve_nsc_catalog(sys.argv)) +runtime.bootstrap_catalog() -from databricks.connect import DatabricksSession from pyspark.sql import functions as F -from edvise import utils from edvise.ingestion.nsc_sftp.constants import ( MANIFEST_TABLE_PATH, QUEUE_TABLE_PATH, @@ -48,98 +27,52 @@ ) from edvise.utils.sftp import connect_sftp, list_receive_files - -try: - dbutils # noqa: F821 -except NameError: - from unittest.mock import MagicMock - - dbutils = MagicMock() - -spark = DatabricksSession.builder.getOrCreate() - -logging.basicConfig( - level=logging.INFO, - format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", -) -logger = logging.getLogger(__name__) +dbutils = runtime.get_dbutils() +spark = runtime.get_spark() +logger = runtime.get_logger(__name__) asset_scope = "nsc-sftp-asset" - host = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-host") user = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-user") password = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-password") -_argv = parse_spark_python_task_params(sys.argv) -cohort_file_name = str( - utils.databricks.get_db_widget_param( - "cohort_file_name", default=_argv.get("cohort_file_name", "") - ) -).strip() -course_file_name = str( - utils.databricks.get_db_widget_param( - "course_file_name", default=_argv.get("course_file_name", "") - ) -).strip() +cohort_file_name = runtime.job_param("cohort_file_name") +course_file_name = runtime.job_param("course_file_name") file_selection_mode = ( - str( - utils.databricks.get_db_widget_param( - "file_selection_mode", - default=_argv.get("file_selection_mode", "uningested"), - ) - ) - .strip() - .lower() - or "uningested" + runtime.job_param("file_selection_mode", "uningested").lower() or "uningested" ) -logger.info("SFTP secured assets loaded successfully.") -logger.info(f"Staging to UC volume path: {SFTP_TMP_DIR}") logger.info( - "Selection inputs: mode=%s cohort_file_name=%r course_file_name=%r", + "Selection inputs: mode=%s cohort=%r course=%r staging=%s", file_selection_mode, cohort_file_name, course_file_name, + SFTP_TMP_DIR, ) -transport = None -sftp = None - +transport = sftp = None try: ensure_manifest_and_queue_tables(spark) - transport, sftp = connect_sftp(host, user, password) - logger.info( - f"Connected to SFTP host={host} and scanning folder={SFTP_REMOTE_FOLDER}" - ) file_rows_all = list_receive_files(sftp, SFTP_REMOTE_FOLDER, SFTP_SOURCE_SYSTEM) if not file_rows_all: - logger.info( - f"No files found in SFTP folder: {SFTP_REMOTE_FOLDER}. Exiting (no-op)." - ) + logger.info("No files in %s; exiting.", SFTP_REMOTE_FOLDER) dbutils.notebook.exit("NO_FILES") - available = sorted({r.get("file_name") for r in file_rows_all}) - logger.info( - f"Found {len(file_rows_all)} file(s) on SFTP in folder={SFTP_REMOTE_FOLDER}; " - f"first 25={available[:25]}" - ) + available = sorted({r["file_name"] for r in file_rows_all if r.get("file_name")}) + logger.info("SFTP files=%s preview=%s", len(available), available[:25]) - # Fingerprints/statuses needed for uningested selection (full listing). - df_all_listing = build_listing_df(spark, file_rows_all) - fingerprint_by_name = { - r["file_name"]: r["file_fingerprint"] - for r in df_all_listing.select("file_name", "file_fingerprint").collect() - } - status_by_fingerprint: dict[str, str] = {} + # Cheap uningested check: BRONZE_WRITTEN file names only (no full Spark fingerprint pass). + ingested_names: set[str] = set() if spark.catalog.tableExists(MANIFEST_TABLE_PATH): - status_by_fingerprint = { - r["file_fingerprint"]: r["status"] + ingested_names = { + r["file_name"] for r in spark.table(MANIFEST_TABLE_PATH) - .select("file_fingerprint", "status") + .where(F.col("status") == F.lit("BRONZE_WRITTEN")) + .select("file_name") .collect() - if r["file_fingerprint"] and r["status"] + if r["file_name"] } cohort_file_name, course_file_name, mode_used = select_file_pair( @@ -147,98 +80,47 @@ mode=file_selection_mode, cohort_file_name=cohort_file_name, course_file_name=course_file_name, - fingerprint_by_name=fingerprint_by_name, - status_by_fingerprint=status_by_fingerprint, + ingested_file_names=ingested_names, ) logger.info( - "Selected files via mode=%s: cohort=%s course=%s", + "Selected via %s: cohort=%s course=%s", mode_used, cohort_file_name, course_file_name, ) - requested_names = {cohort_file_name, course_file_name} - file_rows = [r for r in file_rows_all if r.get("file_name") in requested_names] - - found_names = {r.get("file_name") for r in file_rows} - missing_names = sorted(requested_names - found_names) - if missing_names: + requested = {cohort_file_name, course_file_name} + file_rows = [r for r in file_rows_all if r.get("file_name") in requested] + missing = sorted(requested - {r.get("file_name") for r in file_rows}) + if missing: raise FileNotFoundError( - f"Requested file(s) not found on SFTP in folder '{SFTP_REMOTE_FOLDER}': " - f"{missing_names}. Available file count={len(available)}; " - f"first 25={available[:25]}" - ) - - for r in file_rows: - logger.info( - f"Selected SFTP file: name={r.get('file_name')} size={r.get('file_size')} " - f"modified={r.get('file_modified_time')}" + f"Requested file(s) missing from {SFTP_REMOTE_FOLDER}: {missing}. " + f"Available preview={available[:25]}" ) df_listing = build_listing_df(spark, file_rows) fingerprints = [ - r["file_fingerprint"] for r in df_listing.select("file_fingerprint").collect() + r.file_fingerprint for r in df_listing.select("file_fingerprint").collect() ] - - logger.info("SFTP listing (selected files):") - df_listing.select( - "file_name", "file_size", "file_modified_time", "file_fingerprint" - ).show(truncate=False) - upsert_new_to_manifest(spark, df_listing) - logger.info("Manifest rows (selected files):") - spark.table(MANIFEST_TABLE_PATH).where( - F.col("file_fingerprint").isin(fingerprints) - ).select( - "file_name", - "file_fingerprint", - "status", - "processed_at", - "error_message", - ).show(truncate=False) - df_to_queue = get_files_to_queue(spark, df_listing) - - to_queue_count = df_to_queue.count() - if to_queue_count == 0: - logger.info( - "No files to queue: either nothing is NEW, or NEW files are already queued. " - "Exiting (no-op)." - ) + if df_to_queue.limit(1).count() == 0: + logger.info("Nothing NEW to queue; exiting.") dbutils.notebook.exit("QUEUED_FILES=0") - logger.info("Files eligible to queue:") - df_to_queue.select( - "file_name", "file_size", "file_modified_time", "file_fingerprint" - ).show(truncate=False) - - logger.info( - f"Queuing {to_queue_count} NEW-unqueued file(s) to {QUEUE_TABLE_PATH} " - "and staging to UC volume." - ) queued_count = download_new_files_and_queue(spark, sftp, df_to_queue, logger) - - logger.info("Queue rows (selected files):") - spark.table(QUEUE_TABLE_PATH).where( - F.col("file_fingerprint").isin(fingerprints) - ).select("file_name", "file_fingerprint", "local_tmp_path", "queued_at").show( - truncate=False - ) - logger.info( - f"Queued {queued_count} file(s) for downstream processing in {QUEUE_TABLE_PATH}." + "Queued %s file(s). fingerprints=%s table=%s", + queued_count, + fingerprints, + QUEUE_TABLE_PATH, ) dbutils.notebook.exit(f"QUEUED_FILES={queued_count}") - finally: - try: - if sftp is not None: - sftp.close() - except Exception: - pass - try: - if transport is not None: - transport.close() - except Exception: - pass + for closer in (sftp, transport): + try: + if closer is not None: + closer.close() + except Exception: + pass diff --git a/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py b/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py index 4e067c320..8b2573ae4 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py @@ -1,26 +1,17 @@ """ -Read staged files from pending_ingest_queue, detect institution ID column, -expand to per-institution rows, and MERGE into institution_ingest_plan. - -No SFTP, no API calls, no volume writes beyond reading staged paths. +Expand staged queue files into per-institution rows in institution_ingest_plan. """ from __future__ import annotations -import logging import os import re -import sys from datetime import datetime, timezone -from edvise.ingestion.nsc_sftp.constants import ( - configure_nsc_catalog, - resolve_nsc_catalog, -) +from edvise.ingestion.nsc_sftp import runtime -configure_nsc_catalog(resolve_nsc_catalog(sys.argv)) +runtime.bootstrap_catalog() -from databricks.connect import DatabricksSession from pyspark.sql import functions as F from pyspark.sql import types as T @@ -32,54 +23,28 @@ ) from edvise.ingestion.nsc_sftp.helpers import ensure_plan_table, extract_institution_ids -try: - dbutils # noqa: F821 -except NameError: - from unittest.mock import MagicMock - - dbutils = MagicMock() - -spark = DatabricksSession.builder.getOrCreate() - -logging.basicConfig( - level=logging.INFO, - format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", -) -logger = logging.getLogger(__name__) - +dbutils = runtime.get_dbutils() +spark = runtime.get_spark() +logger = runtime.get_logger(__name__) INST_COL_PATTERN = re.compile(INSTITUTION_COLUMN_PATTERN, re.IGNORECASE) ensure_plan_table(spark, PLAN_TABLE_PATH) - if not spark.catalog.tableExists(QUEUE_TABLE_PATH): - logger.info(f"Queue table {QUEUE_TABLE_PATH} not found. Exiting (no-op).") dbutils.notebook.exit("NO_QUEUE_TABLE") -queue_df = spark.read.table(QUEUE_TABLE_PATH) - +queue_df = spark.table(QUEUE_TABLE_PATH) if queue_df.limit(1).count() == 0: - logger.info("pending_ingest_queue is empty. Exiting (no-op).") dbutils.notebook.exit("NO_QUEUED_FILES") -existing_fp = ( - spark.table(PLAN_TABLE_PATH).select("file_fingerprint").distinct() - if spark.catalog.tableExists(PLAN_TABLE_PATH) - else None +# Skip fingerprints already expanded. +queue_df = queue_df.join( + spark.table(PLAN_TABLE_PATH).select("file_fingerprint").distinct(), + on="file_fingerprint", + how="left_anti", ) -if existing_fp is not None: - queue_df = queue_df.join(existing_fp, on="file_fingerprint", how="left_anti") - if queue_df.limit(1).count() == 0: - logger.info( - "All queued files have already been expanded into institution work items. Exiting (no-op)." - ) dbutils.notebook.exit("NO_NEW_EXPANSION_WORK") -logger.info("Queued files to expand preview (after excluding already-expanded):") -queue_df.select("file_fingerprint", "file_name", "local_tmp_path", "queued_at").show( - 25, truncate=False -) - queued_files = queue_df.select( "file_fingerprint", "file_name", @@ -88,73 +53,51 @@ "file_modified_time", ).collect() -logger.info( - f"Expanding {len(queued_files)} staged file(s) into per-institution work items..." -) - -work_items = [] -missing_files = [] - -for r in queued_files: - fp = r["file_fingerprint"] - file_name = r["file_name"] - local_path = r["local_path"] +work_items: list[dict] = [] +missing: list[str] = [] +now_ts = datetime.now(timezone.utc) +for row in queued_files: + fp, file_name, local_path = ( + row["file_fingerprint"], + row["file_name"], + row["local_path"], + ) if not local_path or not os.path.exists(local_path): - missing_files.append((fp, file_name, local_path)) + missing.append(f"fp={fp} file={file_name} path={local_path}") continue - try: - inst_col, inst_ids = extract_institution_ids( - local_path, renames=COLUMN_RENAMES, inst_col_pattern=INST_COL_PATTERN - ) - if inst_col is None: - logger.warning( - f"No institution id column found for file={file_name} fp={fp}. Skipping this file." - ) - continue - - if not inst_ids: - logger.warning( - f"Institution column found but no IDs present for file={file_name} fp={fp}. Skipping." - ) - continue - - now_ts = datetime.now(timezone.utc) - for inst_id in inst_ids: - work_items.append( - { - "file_fingerprint": fp, - "file_name": file_name, - "local_path": local_path, - "institution_id": inst_id, - "inst_col": inst_col, - "file_size": r["file_size"], - "file_modified_time": r["file_modified_time"], - "planned_at": now_ts, - } - ) - - preview_ids = inst_ids[:10] - logger.info( - f"file={file_name} fp={fp}: found {len(inst_ids)} institution id(s) using column '{inst_col}'. " - f"Preview first 10 IDs={preview_ids}" - ) - - except Exception as e: - logger.exception(f"Failed expanding file={file_name} fp={fp}: {e}") - raise - -if missing_files: - msg = ( - "Some staged files are missing on disk (staging path missing/inaccessible). " - + "; ".join([f"fp={fp} file={fn} path={lp}" for fp, fn, lp in missing_files]) + inst_col, inst_ids = extract_institution_ids( + local_path, renames=COLUMN_RENAMES, inst_col_pattern=INST_COL_PATTERN ) - logger.error(msg) - raise FileNotFoundError(msg) + if not inst_col or not inst_ids: + logger.warning("No institution IDs for file=%s fp=%s; skipping.", file_name, fp) + continue + work_items.extend( + { + "file_fingerprint": fp, + "file_name": file_name, + "local_path": local_path, + "institution_id": inst_id, + "inst_col": inst_col, + "file_size": row["file_size"], + "file_modified_time": row["file_modified_time"], + "planned_at": now_ts, + } + for inst_id in inst_ids + ) + logger.info( + "file=%s: %s institution(s) via %s preview=%s", + file_name, + len(inst_ids), + inst_col, + inst_ids[:10], + ) + +if missing: + raise FileNotFoundError("Missing staged files: " + "; ".join(missing)) if not work_items: - logger.info("No work items generated from staged files. Exiting (no-op).") dbutils.notebook.exit("NO_WORK_ITEMS") schema = T.StructType( @@ -169,35 +112,23 @@ T.StructField("planned_at", T.TimestampType(), False), ] ) - df_plan = spark.createDataFrame(work_items, schema=schema) - -logger.info("Work items summary by file (distinct institutions):") -df_plan.groupBy("file_name").agg( - F.countDistinct("institution_id").alias("institution_count") -).orderBy("file_name").show(truncate=False) - df_plan.createOrReplaceTempView("incoming_plan_rows") - spark.sql( f""" MERGE INTO {PLAN_TABLE_PATH} AS t USING incoming_plan_rows AS s - ON t.file_fingerprint = s.file_fingerprint - AND t.institution_id = s.institution_id + ON t.file_fingerprint = s.file_fingerprint AND t.institution_id = s.institution_id WHEN MATCHED THEN UPDATE SET - t.file_name = s.file_name, - t.local_path = s.local_path, - t.inst_col = s.inst_col, - t.file_size = s.file_size, + t.file_name = s.file_name, + t.local_path = s.local_path, + t.inst_col = s.inst_col, + t.file_size = s.file_size, t.file_modified_time = s.file_modified_time, - t.planned_at = s.planned_at + t.planned_at = s.planned_at WHEN NOT MATCHED THEN INSERT * """ ) - -count_out = df_plan.count() -logger.info( - f"Wrote/updated {count_out} institution work item(s) into {PLAN_TABLE_PATH}." -) +count_out = len(work_items) +logger.info("Wrote/updated %s plan row(s) into %s", count_out, PLAN_TABLE_PATH) dbutils.notebook.exit(f"WORK_ITEMS={count_out}") diff --git a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py index f7e27655e..5ed36095d 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py @@ -1,25 +1,16 @@ """ -Consume institution_ingest_plan for manifest status=NEW; resolve institutions via SST API, -write filtered CSVs to per-institution bronze volumes, and update ingestion_manifest. - -No SFTP — uses staged local paths from prior steps. +Ingest NEW plan rows to per-institution bronze volumes; update ingestion_manifest. """ from __future__ import annotations -import logging import os -import sys +from collections import defaultdict -from edvise.ingestion.nsc_sftp.constants import ( - configure_nsc_catalog, - resolve_nsc_catalog, -) +from edvise.ingestion.nsc_sftp import runtime -configure_nsc_catalog(resolve_nsc_catalog(sys.argv)) +runtime.bootstrap_catalog() -import pandas as pd -from databricks.connect import DatabricksSession from pyspark.sql import functions as F from edvise.ingestion.nsc_sftp.constants import ( @@ -33,418 +24,243 @@ SST_TOKEN_ENDPOINT, ) from edvise.ingestion.nsc_sftp.helpers import ( + load_staged_csv, process_and_save_file, + resolve_bronze_volume_dir, + summarize_file_metrics, update_manifest, ) -from edvise.utils.api_requests import ( - EdviseAPIClient, - fetch_institution_by_pdp_id, -) -from edvise.utils.data_cleaning import convert_to_snake_case -from edvise.utils.databricks import ( - find_bronze_schema, - find_bronze_volume_name, -) +from edvise.utils.api_requests import EdviseAPIClient, fetch_institution_by_pdp_id from edvise.utils.institution_naming import databricksify_inst_name from edvise.utils.sftp import output_file_name_from_sftp -try: - dbutils # noqa: F821 -except NameError: - from unittest.mock import MagicMock - - dbutils = MagicMock() - -try: - display # noqa: F821 -except NameError: - - def display(x): - return x - - -spark = DatabricksSession.builder.getOrCreate() - -logging.basicConfig( - level=logging.INFO, - format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", -) -logger = logging.getLogger(__name__) +dbutils = runtime.get_dbutils() +spark = runtime.get_spark() +logger = runtime.get_logger(__name__) asset_scope = "nsc-sftp-asset" -SST_API_KEY = dbutils.secrets.get(scope=asset_scope, key=SST_API_KEY_SECRET_KEY).strip() -if not SST_API_KEY: +api_key = dbutils.secrets.get(scope=asset_scope, key=SST_API_KEY_SECRET_KEY).strip() +if not api_key: raise RuntimeError( - f"Empty SST API key from secrets: scope={asset_scope} key={SST_API_KEY_SECRET_KEY}" + f"Empty SST API key: scope={asset_scope} key={SST_API_KEY_SECRET_KEY}" ) api_client = EdviseAPIClient( - api_key=SST_API_KEY, + api_key=api_key, base_url=SST_BASE_URL, token_endpoint=SST_TOKEN_ENDPOINT, institution_lookup_path=INSTITUTION_LOOKUP_PATH, ) -def _get_workflow_run_id(): - try: - ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() - tags = ctx.tags() - for k in ("jobRunId", "runId"): - try: - v = tags.apply(k) - if v: - return str(v) - except Exception: - pass - try: - v = ctx.currentRunId().get() - if v: - return str(v) - except Exception: - pass - except Exception: - pass - return None +def _school_check_log(file_name: str, inst_id: str, filtered) -> None: + if {"cohort", "cohort_term"}.issubset(filtered.columns): + latest = filtered["cohort"].max() + terms = ( + filtered.loc[filtered["cohort"] == latest, "cohort_term"] + .dropna() + .astype(str) + .unique() + .tolist() + ) + logger.info( + "School check file=%s inst=%s rows=%s latest_cohort=%s terms=%s", + file_name, + inst_id, + len(filtered), + latest, + terms, + ) + else: + logger.info( + "School check file=%s inst=%s rows=%s", file_name, inst_id, len(filtered) + ) if not spark.catalog.tableExists(PLAN_TABLE_PATH): - logger.info(f"Plan table not found: {PLAN_TABLE_PATH}. Exiting (no-op).") dbutils.notebook.exit("NO_PLAN_TABLE") - if not spark.catalog.tableExists(MANIFEST_TABLE_PATH): raise RuntimeError(f"Manifest table missing: {MANIFEST_TABLE_PATH}") -plan_df = spark.table(PLAN_TABLE_PATH) -if plan_df.limit(1).count() == 0: - logger.info("institution_ingest_plan is empty. Exiting (no-op).") - dbutils.notebook.exit("NO_WORK_ITEMS") - -manifest_df = spark.table(MANIFEST_TABLE_PATH).select("file_fingerprint", "status") -plan_new_df = plan_df.join(manifest_df, on="file_fingerprint", how="inner").where( - F.col("status") == F.lit("NEW") +plan_new_df = ( + spark.table(PLAN_TABLE_PATH) + .join( + spark.table(MANIFEST_TABLE_PATH).select("file_fingerprint", "status"), + on="file_fingerprint", + how="inner", + ) + .where(F.col("status") == F.lit("NEW")) ) if plan_new_df.limit(1).count() == 0: - logger.info("No planned work items where manifest status=NEW. Exiting (no-op).") dbutils.notebook.exit("NO_NEW_TO_INGEST") -plan_summary_df = ( - plan_new_df.groupBy("file_name", "inst_col", "local_path") - .agg(F.countDistinct("institution_id").alias("institution_count")) - .orderBy("file_name") -) -logger.info("Planned work summary (manifest status=NEW):") -display(plan_summary_df) - -file_groups = ( - plan_new_df.select( - "file_fingerprint", - "file_name", - "local_path", - "inst_col", - "file_size", - "file_modified_time", +# One collect: file metadata + institution ids grouped in Python. +plan_rows = plan_new_df.select( + "file_fingerprint", "file_name", "local_path", "inst_col", "institution_id" +).collect() +by_file: dict[str, dict] = {} +inst_ids_by_fp: dict[str, list[str]] = defaultdict(list) +for row in plan_rows: + fp = row["file_fingerprint"] + inst_ids_by_fp[fp].append(row["institution_id"]) + by_file.setdefault( + fp, + { + "file_name": row["file_name"], + "local_path": row["local_path"], + "inst_col": row["inst_col"], + }, ) - .distinct() - .collect() -) - -logger.info(f"Preparing to ingest {len(file_groups)} NEW file(s).") -workflow_run_id = _get_workflow_run_id() -logger.info(f"Workflow run_id: {workflow_run_id}") +run_id = runtime.workflow_run_id(dbutils) +counts = defaultdict(int) +bronze_dir_cache: dict[str, str] = {} -processed_files = 0 -failed_files = 0 -skipped_files = 0 -institutions_written = 0 -institutions_skipped_existing = 0 -institutions_unresolved = 0 -institutions_no_bronze = 0 -institutions_empty = 0 - -for fg in file_groups: - fp = fg["file_fingerprint"] - sftp_file_name = fg["file_name"] - local_path = fg["local_path"] - inst_col = fg["inst_col"] +for fp, meta in by_file.items(): + file_name = meta["file_name"] + local_path = meta["local_path"] + inst_col = meta["inst_col"] + inst_ids = sorted(set(inst_ids_by_fp[fp])) if not local_path or not os.path.exists(local_path): - err = f"Staged local file missing for fp={fp}: {local_path}" - logger.error(err) update_manifest( spark, MANIFEST_TABLE_PATH, fp, status="FAILED", - error_message=err[:8000], - run_id=workflow_run_id, + error_message=f"Staged local file missing: {local_path}"[:8000], + run_id=run_id, ) - failed_files += 1 + counts["failed_files"] += 1 continue try: - header_cols = pd.read_csv(local_path, nrows=0).columns.tolist() - raw_inst_col = next( - ( - c - for c in header_cols - if COLUMN_RENAMES.get( - convert_to_snake_case(c), convert_to_snake_case(c) - ) - == inst_col - ), - None, - ) - dtype = {raw_inst_col: str} if raw_inst_col else None - df_full = pd.read_csv(local_path, on_bad_lines="warn", dtype=dtype) - df_full = df_full.rename( - columns={c: convert_to_snake_case(c) for c in df_full.columns} - ) - df_full = df_full.rename(columns=COLUMN_RENAMES) - - file_student_count = None - try: - student_col = next( - ( - c - for c in ("student_id", "study_id", "student_guid") - if c in df_full.columns - ), - None, - ) - if student_col: - file_student_count = int(df_full[student_col].nunique(dropna=True)) - except Exception: - file_student_count = None - - file_cohort = None - try: - if "cohort" in df_full.columns: - vals = ( - df_full["cohort"] - .dropna() - .astype(str) - .map(lambda x: x.strip()) - .tolist() - ) - vals = [ - v for v in vals if v and v.lower() not in {"nan", "none", "null"} - ] - file_cohort = sorted(set(vals)) or None - except Exception: - file_cohort = None - - file_cohort_term_pairs = None - try: - if {"cohort", "cohort_term"}.issubset(df_full.columns): - tmp = df_full[["cohort", "cohort_term"]].dropna() - tmp = tmp.assign( - cohort=tmp["cohort"].astype(str).map(lambda x: x.strip()), - cohort_term=tmp["cohort_term"] - .astype(str) - .map(lambda x: x.strip().upper()), - ) - tmp = tmp[ - (tmp["cohort"] != "") - & (tmp["cohort_term"] != "") - & (~tmp["cohort"].str.lower().isin({"nan", "none", "null"})) - & (~tmp["cohort_term"].str.lower().isin({"nan", "none", "null"})) - ] - tmp = tmp.drop_duplicates().sort_values(by=["cohort", "cohort_term"]) - pairs = [ - {"cohort": r.cohort, "cohort_term": r.cohort_term} - for r in tmp.itertuples(index=False) - ] - file_cohort_term_pairs = pairs or None - except Exception: - file_cohort_term_pairs = None - + df_full = load_staged_csv(local_path, renames=COLUMN_RENAMES, inst_col=inst_col) + student_count, file_cohort, cohort_term_pairs = summarize_file_metrics(df_full) logger.info( - "file=%s fp=%s: student_count=%s cohort_count=%s", - sftp_file_name, + "file=%s fp=%s students=%s cohorts=%s institutions=%s", + file_name, fp, - file_student_count, - (len(file_cohort) if file_cohort else 0), + student_count, + len(file_cohort or []), + len(inst_ids), ) if inst_col not in df_full.columns: - err = f"Expected institution column '{inst_col}' not found after normalization/renames for file={sftp_file_name} fp={fp}" - logger.error(err) update_manifest( spark, MANIFEST_TABLE_PATH, fp, status="FAILED", - error_message=err[:8000], - run_id=workflow_run_id, + error_message=f"Missing institution column '{inst_col}'"[:8000], + run_id=run_id, cohort=file_cohort, - cohort_term_pairs=file_cohort_term_pairs, - student_count=file_student_count, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, ) - failed_files += 1 + counts["failed_files"] += 1 continue - inst_ids = ( - plan_new_df.where(F.col("file_fingerprint") == fp) - .select("institution_id") - .distinct() - .collect() - ) - inst_ids = [r["institution_id"] for r in inst_ids] - if not inst_ids: - logger.info( - f"No institution_ids in plan for file={sftp_file_name} fp={fp}. Marking BRONZE_WRITTEN (no-op)." - ) update_manifest( spark, MANIFEST_TABLE_PATH, fp, status="BRONZE_WRITTEN", error_message=None, - run_id=workflow_run_id, + run_id=run_id, cohort=file_cohort, - cohort_term_pairs=file_cohort_term_pairs, - student_count=file_student_count, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, ) - skipped_files += 1 + counts["skipped_files"] += 1 continue - preview_inst_ids = inst_ids[:10] - logger.info( - f"file={sftp_file_name} fp={fp}: ingesting {len(inst_ids)} institution(s) " - f"using inst_col='{inst_col}'. Preview first 10 IDs={preview_inst_ids}" - ) - - file_errors = [] + # One pass over the frame instead of N equality filters. + grouped = { + str(k): g.reset_index(drop=True) + for k, g in df_full.groupby(inst_col, sort=False) + if str(k) in set(map(str, inst_ids)) + } + file_errors: list[str] = [] + out_name = output_file_name_from_sftp(file_name) for inst_id in inst_ids: try: - target_inst_id = str(inst_id) - filtered_df = df_full[df_full[inst_col] == target_inst_id].reset_index( - drop=True - ) - - if filtered_df.empty: - institutions_empty += 1 - logger.info( - f"file={sftp_file_name} fp={fp}: institution {inst_id} has 0 rows; skipping." - ) + filtered = grouped.get(str(inst_id)) + if filtered is None or filtered.empty: + counts["institutions_empty"] += 1 continue - # Parity with interactive PIPELINE_pdp_to_databricks checks. - if {"cohort", "cohort_term"}.issubset(filtered_df.columns): - latest_cohort = filtered_df["cohort"].max() - latest_cohort_terms = ( - filtered_df.loc[ - filtered_df["cohort"] == latest_cohort, "cohort_term" - ] - .dropna() - .astype(str) - .unique() - .tolist() - ) - logger.info( - "School check file=%s inst=%s rows=%s latest_cohort=%s " - "latest_cohort_terms=%s", - sftp_file_name, - inst_id, - len(filtered_df), - latest_cohort, - latest_cohort_terms, - ) - else: - logger.info( - "School check file=%s inst=%s rows=%s " - "(no cohort/cohort_term columns)", - sftp_file_name, - inst_id, - len(filtered_df), - ) + _school_check_log(file_name, inst_id, filtered) try: - inst_info = fetch_institution_by_pdp_id(api_client, inst_id) + info = fetch_institution_by_pdp_id(api_client, inst_id) except Exception as api_err: - institutions_unresolved += 1 - raise ValueError( - f"SST API lookup failed for pdp_id={inst_id}: {api_err}" - ) from api_err + counts["institutions_unresolved"] += 1 + raise ValueError(f"SST API lookup failed: {api_err}") from api_err - inst_name = inst_info.get("name") + inst_name = info.get("name") if not inst_name: - institutions_unresolved += 1 - raise ValueError( - f"SST API returned no 'name' for pdp_id={inst_id}. " - f"Response={inst_info}" - ) - - inst_prefix = databricksify_inst_name(inst_name) - logger.info( - "Resolved school file=%s pdp_id=%s name=%r prefix=%s", - sftp_file_name, - inst_id, - inst_name, - inst_prefix, - ) - - try: - bronze_schema = find_bronze_schema(spark, CATALOG, inst_prefix) - bronze_volume_name = find_bronze_volume_name( - spark, CATALOG, bronze_schema - ) - except ValueError as bronze_err: - institutions_no_bronze += 1 - raise ValueError( - f"Bronze not provisioned for pdp_id={inst_id} " - f"name={inst_name!r} prefix={inst_prefix}: {bronze_err}" - ) from bronze_err - - volume_dir = f"/Volumes/{CATALOG}/{bronze_schema}/{bronze_volume_name}" - - out_file_name = output_file_name_from_sftp(sftp_file_name) - full_path = os.path.join(volume_dir, out_file_name) - + counts["institutions_unresolved"] += 1 + raise ValueError(f"SST API returned no name for pdp_id={inst_id}") + + prefix = databricksify_inst_name(inst_name) + if prefix not in bronze_dir_cache: + try: + bronze_dir_cache[prefix] = resolve_bronze_volume_dir( + spark, CATALOG, prefix + ) + except ValueError as bronze_err: + counts["institutions_no_bronze"] += 1 + raise ValueError( + f"Bronze missing for {inst_name!r} ({prefix}): {bronze_err}" + ) from bronze_err + + volume_dir = bronze_dir_cache[prefix] + full_path = os.path.join(volume_dir, out_name) if os.path.exists(full_path): - institutions_skipped_existing += 1 + counts["institutions_skipped_existing"] += 1 logger.info( - f"file={sftp_file_name} inst={inst_id}: already exists in " - f"{volume_dir}; skipping write." + "Skip existing file=%s inst=%s path=%s", + file_name, + inst_id, + full_path, ) continue logger.info( - f"file={sftp_file_name} inst={inst_id}: writing to {volume_dir} " - f"as {out_file_name}" + "Write file=%s inst=%s name=%r -> %s/%s", + file_name, + inst_id, + inst_name, + volume_dir, + out_name, ) process_and_save_file( - volume_dir=volume_dir, file_name=out_file_name, df=filtered_df + volume_dir=volume_dir, file_name=out_name, df=filtered ) - institutions_written += 1 - logger.info(f"file={sftp_file_name} inst={inst_id}: write complete.") - - except Exception as e: + counts["institutions_written"] += 1 + except Exception as exc: msg = ( - f"inst_ingest_failed file={sftp_file_name} fp={fp} " - f"inst={inst_id}: {e}" + f"inst_ingest_failed file={file_name} fp={fp} inst={inst_id}: {exc}" ) logger.exception(msg) file_errors.append(msg) if file_errors: - err = " | ".join(file_errors)[:8000] update_manifest( spark, MANIFEST_TABLE_PATH, fp, status="FAILED", - error_message=err, - run_id=workflow_run_id, + error_message=" | ".join(file_errors)[:8000], + run_id=run_id, cohort=file_cohort, - cohort_term_pairs=file_cohort_term_pairs, - student_count=file_student_count, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, ) - failed_files += 1 + counts["failed_files"] += 1 else: update_manifest( spark, @@ -452,46 +268,30 @@ def _get_workflow_run_id(): fp, status="BRONZE_WRITTEN", error_message=None, - run_id=workflow_run_id, + run_id=run_id, cohort=file_cohort, - cohort_term_pairs=file_cohort_term_pairs, - student_count=file_student_count, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, ) - processed_files += 1 + counts["processed_files"] += 1 - except Exception as e: - msg = f"fatal_file_error file={sftp_file_name} fp={fp}: {e}" - logger.exception(msg) + except Exception as exc: + logger.exception("fatal_file_error file=%s fp=%s: %s", file_name, fp, exc) update_manifest( spark, MANIFEST_TABLE_PATH, fp, status="FAILED", - error_message=msg[:8000], - run_id=workflow_run_id, + error_message=f"fatal_file_error file={file_name} fp={fp}: {exc}"[:8000], + run_id=run_id, ) - failed_files += 1 - -logger.info( - "Done. processed_files=%s failed_files=%s skipped_files=%s " - "institutions_written=%s institutions_skipped_existing=%s " - "institutions_unresolved=%s institutions_no_bronze=%s institutions_empty=%s", - processed_files, - failed_files, - skipped_files, - institutions_written, - institutions_skipped_existing, - institutions_unresolved, - institutions_no_bronze, - institutions_empty, -) -if institutions_unresolved or institutions_no_bronze: - logger.warning( - "Some institutions were not fully ingestible (API unresolved or missing bronze). " - "See per-institution errors above; file-level manifest status reflects failures." - ) + counts["failed_files"] += 1 + +logger.info("Done counts=%s", dict(counts)) dbutils.notebook.exit( - f"PROCESSED={processed_files};FAILED={failed_files};SKIPPED={skipped_files};" - f"WRITTEN={institutions_written};EXISTING={institutions_skipped_existing};" - f"UNRESOLVED={institutions_unresolved};NO_BRONZE={institutions_no_bronze}" + "PROCESSED={processed_files};FAILED={failed_files};SKIPPED={skipped_files};" + "WRITTEN={institutions_written};EXISTING={institutions_skipped_existing};" + "UNRESOLVED={institutions_unresolved};NO_BRONZE={institutions_no_bronze}".format_map( + defaultdict(int, counts) + ) ) diff --git a/tests/ingestion/test_file_selection.py b/tests/ingestion/test_file_selection.py index a5fd7f8bd..a656720e1 100644 --- a/tests/ingestion/test_file_selection.py +++ b/tests/ingestion/test_file_selection.py @@ -74,21 +74,13 @@ def test_select_file_pair_uningested_skips_bronze_written(): _row("B_Cohort_20240201101010.csv"), _row("B_Course_20240201101010.csv"), ] - fingerprint_by_name = { - "B_Cohort_20240201101010.csv": "fp_b_cohort", - "B_Course_20240201101010.csv": "fp_b_course", - "A_Cohort_20240115123045.csv": "fp_a_cohort", - "A_Course_20240115123045.csv": "fp_a_course", - } - status_by_fingerprint = { - "fp_b_cohort": "BRONZE_WRITTEN", - "fp_b_course": "BRONZE_WRITTEN", - } c, o, mode = select_file_pair( rows, mode="uningested", - fingerprint_by_name=fingerprint_by_name, - status_by_fingerprint=status_by_fingerprint, + ingested_file_names={ + "B_Cohort_20240201101010.csv", + "B_Course_20240201101010.csv", + }, ) assert mode == "uningested" assert c == "A_Cohort_20240115123045.csv" @@ -100,18 +92,12 @@ def test_select_file_pair_uningested_all_done_raises(): _row("A_Cohort_20240115123045.csv"), _row("A_Course_20240115123045.csv"), ] - fingerprint_by_name = { - "A_Cohort_20240115123045.csv": "fp_a_cohort", - "A_Course_20240115123045.csv": "fp_a_course", - } - status_by_fingerprint = { - "fp_a_cohort": "BRONZE_WRITTEN", - "fp_a_course": "BRONZE_WRITTEN", - } with pytest.raises(FileNotFoundError, match="already BRONZE_WRITTEN"): select_file_pair( rows, mode="uningested", - fingerprint_by_name=fingerprint_by_name, - status_by_fingerprint=status_by_fingerprint, + ingested_file_names={ + "A_Cohort_20240115123045.csv", + "A_Course_20240115123045.csv", + }, ) From fc84acca4cc1b61761d639ac368eb1494fc9c8ee Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Fri, 7 Aug 2026 11:46:21 -0700 Subject: [PATCH 04/10] chore: remove NSC SFTP notebooks superseded by DAB scripts Ingestion now runs from pipelines/ingestion/pdp spark_python_tasks. Co-authored-by: Cursor --- .gitignore | 4 +- .../01_sftp_receive_scan.ipynb | 377 ------------ .../02_file_institution_expand.ipynb | 386 ------------ .../03_per_institution_bronze_ingest.ipynb | 582 ------------------ 4 files changed, 1 insertion(+), 1348 deletions(-) delete mode 100644 notebooks/nsc_sftp_automated_data_ingestion/01_sftp_receive_scan.ipynb delete mode 100644 notebooks/nsc_sftp_automated_data_ingestion/02_file_institution_expand.ipynb delete mode 100644 notebooks/nsc_sftp_automated_data_ingestion/03_per_institution_bronze_ingest.ipynb diff --git a/.gitignore b/.gitignore index e44ccec27..69e97a31f 100644 --- a/.gitignore +++ b/.gitignore @@ -216,6 +216,4 @@ marimo/_lsp/ __marimo__/ # Claude -.claude/ -*notebooks/nsc_sftp_automated_data_ingestion/tmp/ -*notebooks/nsc_sftp_automated_data_ingestion/gcp_config.yaml \ No newline at end of file +.claude/ \ No newline at end of file diff --git a/notebooks/nsc_sftp_automated_data_ingestion/01_sftp_receive_scan.ipynb b/notebooks/nsc_sftp_automated_data_ingestion/01_sftp_receive_scan.ipynb deleted file mode 100644 index 3093c2306..000000000 --- a/notebooks/nsc_sftp_automated_data_ingestion/01_sftp_receive_scan.ipynb +++ /dev/null @@ -1,377 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "7dc0a9a7-1db8-42b9-b0c4-07946f392d5e", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "# 1. Connect to SFTP and scan the receive folder for files.\n", - "# 2. Upsert unseen files into `ingestion_manifest` with status=NEW.\n", - "# 3. Download and stage NEW + unqueued files locally and upsert them into `pending_ingest_queue`.\n", - "\n", - "# Recent refactor:\n", - "# - SFTP helpers moved to `helper.py` (`connect_sftp`, `list_receive_files`, `download_sftp_atomic`).\n", - "# - `list_receive_files` now takes `source_system` explicitly (no hidden notebook globals).\n", - "\n", - "# Constraints:\n", - "# - SFTP connection required\n", - "# - NO API calls\n", - "# - Stages files to UC volume (CATALOG.default.tmp) + writes to Delta tables only\n", - "\n", - "# Inputs:\n", - "# - SFTP folder: `./receive`\n", - "# - Required workflow parameters (exact SFTP file names):\n", - "# - `cohort_file_name`\n", - "# - `course_file_name`\n", - "# - Both file names must end with the same 14-digit file stamp (e.g. `..._YYYYMMDDHHMMSS.csv`).\n", - "\n", - "# Outputs:\n", - "# - `CATALOG.default.ingestion_manifest`\n", - "# - `CATALOG.default.pending_ingest_queue`\n", - "# - Staged files written to UC Volume: `CATALOG.default.tmp` (path `/Volumes//default/tmp`)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "cbd7694b-4b30-41bf-9371-259479726010", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "%pip install paramiko python-box pyyaml\n", - "%pip install git+https://github.com/datakind/edvise.git@Automated_Ingestion_Workflow" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "b9ae88af-ade1-4df0-86a0-34d6d492383a", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "%restart_python" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "5888f9b8-bda7-4586-9f9f-ed1243d878de", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "import logging\n", - "import os\n", - "import re\n", - "from databricks.connect import DatabricksSession\n", - "from pyspark.sql import functions as F\n", - "\n", - "from edvise.utils.sftp import connect_sftp, list_receive_files\n", - "from edvise.ingestion.nsc_sftp.constants import (\n", - " MANIFEST_TABLE_PATH,\n", - " QUEUE_TABLE_PATH,\n", - " SFTP_REMOTE_FOLDER,\n", - " SFTP_SOURCE_SYSTEM,\n", - " SFTP_TMP_DIR,\n", - ")\n", - "from edvise.ingestion.nsc_sftp.helpers import (\n", - " build_listing_df,\n", - " download_new_files_and_queue,\n", - " ensure_manifest_and_queue_tables,\n", - " get_files_to_queue,\n", - " upsert_new_to_manifest,\n", - ")\n", - "from edvise import utils\n", - "\n", - "try:\n", - " dbutils # noqa: F821\n", - "except NameError:\n", - " from unittest.mock import MagicMock\n", - "\n", - " dbutils = MagicMock()\n", - "spark = DatabricksSession.builder.getOrCreate()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "61b348b8-aa62-4b5a-9442-d48d52e1a862", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "logging.basicConfig(\n", - " level=logging.INFO,\n", - " format=\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\",\n", - ")\n", - "logger = logging.getLogger(__name__)\n", - "\n", - "asset_scope = \"nsc-sftp-asset\"\n", - "\n", - "host = dbutils.secrets.get(scope=asset_scope, key=\"nsc-sftp-host\")\n", - "user = dbutils.secrets.get(scope=asset_scope, key=\"nsc-sftp-user\")\n", - "password = dbutils.secrets.get(scope=asset_scope, key=\"nsc-sftp-password\")\n", - "\n", - "cohort_file_name = utils.databricks.get_db_widget_param(\"cohort_file_name\", default=\"\")\n", - "course_file_name = utils.databricks.get_db_widget_param(\"course_file_name\", default=\"\")\n", - "cohort_file_name = str(cohort_file_name).strip()\n", - "course_file_name = str(course_file_name).strip()\n", - "if not cohort_file_name or not course_file_name:\n", - " raise ValueError(\n", - " \"Missing required workflow parameters: cohort_file_name and course_file_name. \"\n", - " \"Pass them as Databricks job base parameters.\"\n", - " )\n", - "\n", - "\n", - "def _extract_file_stamp(file_name: str) -> str:\n", - " base = os.path.basename(file_name)\n", - " m = re.search(r\"_(\\d{14})(?:\\.[^.]+)?$\", base)\n", - " if not m:\n", - " raise ValueError(\n", - " \"Expected file name to end with a 14-digit file stamp, e.g. \"\n", - " \"'..._YYYYMMDDHHMMSS.csv'. Got: \"\n", - " f\"{file_name}\"\n", - " )\n", - " return m.group(1)\n", - "\n", - "\n", - "cohort_stamp = _extract_file_stamp(cohort_file_name)\n", - "course_stamp = _extract_file_stamp(course_file_name)\n", - "if cohort_stamp != course_stamp:\n", - " raise ValueError(\n", - " \"cohort_file_name and course_file_name must end with the same file stamp. \"\n", - " f\"Got cohort stamp={cohort_stamp}, course stamp={course_stamp}.\"\n", - " )\n", - "logger.info(f\"Validated file stamp: {cohort_stamp}\")\n", - "logger.info(f\"Staging to UC volume path: {SFTP_TMP_DIR}\")\n", - "logger.info(\n", - " \"Manual file selection enabled: \"\n", - " f\"cohort_file_name={cohort_file_name}, course_file_name={course_file_name}\"\n", - ")\n", - "\n", - "logger.info(\"SFTP secured assets loaded successfully.\")" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "80968f66-5082-49ca-b03f-b3a1ef0bb908", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "transport = None\n", - "sftp = None\n", - "\n", - "try:\n", - " ensure_manifest_and_queue_tables(spark)\n", - "\n", - " transport, sftp = connect_sftp(host, user, password)\n", - " logger.info(\n", - " f\"Connected to SFTP host={host} and scanning folder={SFTP_REMOTE_FOLDER}\"\n", - " )\n", - "\n", - " file_rows_all = list_receive_files(sftp, SFTP_REMOTE_FOLDER, SFTP_SOURCE_SYSTEM)\n", - " if not file_rows_all:\n", - " logger.info(\n", - " f\"No files found in SFTP folder: {SFTP_REMOTE_FOLDER}. Exiting (no-op).\"\n", - " )\n", - " dbutils.notebook.exit(\"NO_FILES\")\n", - "\n", - " requested_names = {cohort_file_name, course_file_name}\n", - " logger.info(\n", - " f\"Found {len(file_rows_all)} file(s) on SFTP in folder={SFTP_REMOTE_FOLDER}; \"\n", - " f\"requested={sorted(requested_names)}\"\n", - " )\n", - " file_rows = [r for r in file_rows_all if r.get(\"file_name\") in requested_names]\n", - "\n", - " found_names = {r.get(\"file_name\") for r in file_rows}\n", - " missing_names = sorted(requested_names - found_names)\n", - " if missing_names:\n", - " available = sorted({r.get(\"file_name\") for r in file_rows_all})\n", - " preview = available[:25]\n", - " raise FileNotFoundError(\n", - " f\"Requested file(s) not found on SFTP in folder '{SFTP_REMOTE_FOLDER}': {missing_names}. \"\n", - " f\"Available file count={len(available)}; first 25={preview}\"\n", - " )\n", - "\n", - " for r in file_rows:\n", - " logger.info(\n", - " f\"Selected SFTP file: name={r.get('file_name')} size={r.get('file_size')} \"\n", - " f\"modified={r.get('file_modified_time')}\"\n", - " )\n", - "\n", - " df_listing = build_listing_df(spark, file_rows)\n", - " fingerprints = [\n", - " r[\"file_fingerprint\"] for r in df_listing.select(\"file_fingerprint\").collect()\n", - " ]\n", - "\n", - " logger.info(\"SFTP listing (selected files):\")\n", - " df_listing.select(\n", - " \"file_name\", \"file_size\", \"file_modified_time\", \"file_fingerprint\"\n", - " ).show(truncate=False)\n", - "\n", - " # 1) Ensure everything on SFTP is at least represented in manifest as NEW\n", - " upsert_new_to_manifest(spark, df_listing)\n", - "\n", - " logger.info(\"Manifest rows (selected files):\")\n", - " spark.table(MANIFEST_TABLE_PATH).where(\n", - " F.col(\"file_fingerprint\").isin(fingerprints)\n", - " ).select(\n", - " \"file_name\",\n", - " \"file_fingerprint\",\n", - " \"status\",\n", - " \"processed_at\",\n", - " \"error_message\",\n", - " ).show(truncate=False)\n", - "\n", - " # 2) Queue anything that is still NEW and not already queued\n", - " df_to_queue = get_files_to_queue(spark, df_listing)\n", - "\n", - " to_queue_count = df_to_queue.count()\n", - " if to_queue_count == 0:\n", - " logger.info(\n", - " \"No files to queue: either nothing is NEW, or NEW files are already queued. Exiting (no-op).\"\n", - " )\n", - " dbutils.notebook.exit(\"QUEUED_FILES=0\")\n", - "\n", - " logger.info(\"Files eligible to queue:\")\n", - " df_to_queue.select(\n", - " \"file_name\", \"file_size\", \"file_modified_time\", \"file_fingerprint\"\n", - " ).show(truncate=False)\n", - "\n", - " logger.info(\n", - " f\"Queuing {to_queue_count} NEW-unqueued file(s) to {QUEUE_TABLE_PATH} and staging to UC volume.\"\n", - " )\n", - " queued_count = download_new_files_and_queue(spark, sftp, df_to_queue, logger)\n", - "\n", - " logger.info(\"Queue rows (selected files):\")\n", - " spark.table(QUEUE_TABLE_PATH).where(\n", - " F.col(\"file_fingerprint\").isin(fingerprints)\n", - " ).select(\"file_name\", \"file_fingerprint\", \"local_tmp_path\", \"queued_at\").show(\n", - " truncate=False\n", - " )\n", - "\n", - " logger.info(\n", - " f\"Queued {queued_count} file(s) for downstream processing in {QUEUE_TABLE_PATH}.\"\n", - " )\n", - " dbutils.notebook.exit(f\"QUEUED_FILES={queued_count}\")\n", - "\n", - "finally:\n", - " try:\n", - " if sftp is not None:\n", - " sftp.close()\n", - " except Exception:\n", - " pass\n", - " try:\n", - " if transport is not None:\n", - " transport.close()\n", - " except Exception:\n", - " pass" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": {}, - "inputWidgets": {}, - "nuid": "edff98e1-0862-4e41-8c35-bd5fb6647136", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [] - } - ], - "metadata": { - "application/vnd.databricks.v1+notebook": { - "computePreferences": null, - "dashboards": [], - "environmentMetadata": { - "base_environment": "", - "environment_version": "4" - }, - "inputWidgetPreferences": null, - "language": "python", - "notebookMetadata": { - "pythonIndentUnit": 4 - }, - "notebookName": "01_sftp_receive_scan", - "widgets": {} - }, - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} diff --git a/notebooks/nsc_sftp_automated_data_ingestion/02_file_institution_expand.ipynb b/notebooks/nsc_sftp_automated_data_ingestion/02_file_institution_expand.ipynb deleted file mode 100644 index 5cd78beb4..000000000 --- a/notebooks/nsc_sftp_automated_data_ingestion/02_file_institution_expand.ipynb +++ /dev/null @@ -1,386 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# 1. Read each *staged* local file (from `pending_ingest_queue`), detect the institution id column,\n", - "# 2. extract unique institution IDs, and emit per-institution work items.\n", - "\n", - "# Constraints:\n", - "# - NO SFTP connection\n", - "# - NO API calls\n", - "# - NO volume writes\n", - "\n", - "# Input table:\n", - "# - `staging_sst_01.default.pending_ingest_queue`\n", - "\n", - "# Output table:\n", - "# - `staging_sst_01.default.institution_ingest_plan`\n", - "# - Columns: `file_fingerprint`, `file_name`, `local_path`, `institution_id`, `inst_col`, `file_size`, `file_modified_time`, `planned_at`\n" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "679b2064-2a15-4d89-abda-5e9c0148ff61", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "%pip install pandas python-box pyyaml paramiko\n", - "%pip install git+https://github.com/datakind/edvise.git@Automated_Ingestion_Workflow" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%restart_python" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "62608829-5027-4075-a4fc-1e4afc36ef3a", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "import logging\n", - "import os\n", - "import re\n", - "from datetime import datetime, timezone\n", - "\n", - "from pyspark.sql import functions as F\n", - "from pyspark.sql import types as T\n", - "from databricks.connect import DatabricksSession\n", - "\n", - "from edvise.ingestion.nsc_sftp.helpers import ensure_plan_table, extract_institution_ids\n", - "from edvise.ingestion.nsc_sftp.constants import (\n", - " QUEUE_TABLE_PATH,\n", - " PLAN_TABLE_PATH,\n", - " COLUMN_RENAMES,\n", - " INSTITUTION_COLUMN_PATTERN,\n", - ")\n", - "\n", - "try:\n", - " dbutils # noqa: F821\n", - "except NameError:\n", - " from unittest.mock import MagicMock\n", - "\n", - " dbutils = MagicMock()\n", - "spark = DatabricksSession.builder.getOrCreate()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "64156fce-07a6-4eb6-8612-6b29bc06edfe", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "logging.basicConfig(\n", - " level=logging.INFO,\n", - " format=\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\",\n", - ")\n", - "logger = logging.getLogger(__name__)\n", - "\n", - "INST_COL_PATTERN = re.compile(INSTITUTION_COLUMN_PATTERN, re.IGNORECASE)" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "87047914-fec0-4f35-b33f-d1b927605d11", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "ensure_plan_table(spark, PLAN_TABLE_PATH)\n", - "\n", - "# Pull queued staged files (Script 1 output)\n", - "if not spark.catalog.tableExists(QUEUE_TABLE_PATH):\n", - " logger.info(f\"Queue table {QUEUE_TABLE_PATH} not found. Exiting (no-op).\")\n", - " dbutils.notebook.exit(\"NO_QUEUE_TABLE\")\n", - "\n", - "queue_df = spark.read.table(QUEUE_TABLE_PATH)\n", - "\n", - "if queue_df.limit(1).count() == 0:\n", - " logger.info(\"pending_ingest_queue is empty. Exiting (no-op).\")\n", - " dbutils.notebook.exit(\"NO_QUEUED_FILES\")" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "21683394-0bec-42b8-82dd-1a4590519de5", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "# Avoid regenerating plans for files already expanded\n", - "existing_fp = (\n", - " spark.table(PLAN_TABLE_PATH).select(\"file_fingerprint\").distinct()\n", - " if spark.catalog.tableExists(PLAN_TABLE_PATH)\n", - " else None\n", - ")\n", - "if existing_fp is not None:\n", - " queue_df = queue_df.join(existing_fp, on=\"file_fingerprint\", how=\"left_anti\")\n", - "\n", - "if queue_df.limit(1).count() == 0:\n", - " logger.info(\n", - " \"All queued files have already been expanded into institution work items. Exiting (no-op).\"\n", - " )\n", - " dbutils.notebook.exit(\"NO_NEW_EXPANSION_WORK\")\n", - "\n", - "logger.info(\"Queued files to expand preview (after excluding already-expanded):\")\n", - "queue_df.select(\"file_fingerprint\", \"file_name\", \"local_tmp_path\", \"queued_at\").show(\n", - " 25, truncate=False\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "540c7880-f14a-4607-979a-856f17066c50", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "queued_files = queue_df.select(\n", - " \"file_fingerprint\",\n", - " \"file_name\",\n", - " F.col(\"local_tmp_path\").alias(\"local_path\"),\n", - " \"file_size\",\n", - " \"file_modified_time\",\n", - ").collect()\n", - "\n", - "logger.info(\n", - " f\"Expanding {len(queued_files)} staged file(s) into per-institution work items...\"\n", - ")\n", - "\n", - "work_items = []\n", - "missing_files = []\n", - "\n", - "for r in queued_files:\n", - " fp = r[\"file_fingerprint\"]\n", - " file_name = r[\"file_name\"]\n", - " local_path = r[\"local_path\"]\n", - "\n", - " if not local_path or not os.path.exists(local_path):\n", - " missing_files.append((fp, file_name, local_path))\n", - " continue\n", - "\n", - " try:\n", - " inst_col, inst_ids = extract_institution_ids(\n", - " local_path, renames=COLUMN_RENAMES, inst_col_pattern=INST_COL_PATTERN\n", - " )\n", - " if inst_col is None:\n", - " logger.warning(\n", - " f\"No institution id column found for file={file_name} fp={fp}. Skipping this file.\"\n", - " )\n", - " continue\n", - "\n", - " if not inst_ids:\n", - " logger.warning(\n", - " f\"Institution column found but no IDs present for file={file_name} fp={fp}. Skipping.\"\n", - " )\n", - " continue\n", - "\n", - " now_ts = datetime.now(timezone.utc)\n", - " for inst_id in inst_ids:\n", - " work_items.append(\n", - " {\n", - " \"file_fingerprint\": fp,\n", - " \"file_name\": file_name,\n", - " \"local_path\": local_path,\n", - " \"institution_id\": inst_id,\n", - " \"inst_col\": inst_col,\n", - " \"file_size\": r[\"file_size\"],\n", - " \"file_modified_time\": r[\"file_modified_time\"],\n", - " \"planned_at\": now_ts,\n", - " }\n", - " )\n", - "\n", - " preview_ids = inst_ids[:10]\n", - " logger.info(\n", - " f\"file={file_name} fp={fp}: found {len(inst_ids)} institution id(s) using column '{inst_col}'. \"\n", - " f\"Preview first 10 IDs={preview_ids}\"\n", - " )\n", - "\n", - " except Exception as e:\n", - " logger.exception(f\"Failed expanding file={file_name} fp={fp}: {e}\")\n", - " # We don't write manifests here per your division; fail fast so workflow can surface issue.\n", - " raise" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "32d5bc9c-16a1-42b4-adef-f1a442e5d447", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "if missing_files:\n", - " # This usually indicates the staged files were cleaned up or the staging path\n", - " # is not accessible from this cluster.\n", - " # Fail fast so the workflow stops (downstream cannot proceed without the staged files).\n", - " msg = (\n", - " \"Some staged files are missing on disk (staging path missing/inaccessible). \"\n", - " + \"; \".join([f\"fp={fp} file={fn} path={lp}\" for fp, fn, lp in missing_files])\n", - " )\n", - " logger.error(msg)\n", - " raise FileNotFoundError(msg)\n", - "\n", - "if not work_items:\n", - " logger.info(\"No work items generated from staged files. Exiting (no-op).\")\n", - " dbutils.notebook.exit(\"NO_WORK_ITEMS\")\n", - "\n", - "schema = T.StructType(\n", - " [\n", - " T.StructField(\"file_fingerprint\", T.StringType(), False),\n", - " T.StructField(\"file_name\", T.StringType(), False),\n", - " T.StructField(\"local_path\", T.StringType(), False),\n", - " T.StructField(\"institution_id\", T.StringType(), False),\n", - " T.StructField(\"inst_col\", T.StringType(), False),\n", - " T.StructField(\"file_size\", T.LongType(), True),\n", - " T.StructField(\"file_modified_time\", T.TimestampType(), True),\n", - " T.StructField(\"planned_at\", T.TimestampType(), False),\n", - " ]\n", - ")\n", - "\n", - "df_plan = spark.createDataFrame(work_items, schema=schema)\n", - "\n", - "logger.info(\"Work items summary by file (distinct institutions):\")\n", - "df_plan.groupBy(\"file_name\").agg(\n", - " F.countDistinct(\"institution_id\").alias(\"institution_count\")\n", - ").orderBy(\"file_name\").show(truncate=False)\n", - "\n", - "df_plan.createOrReplaceTempView(\"incoming_plan_rows\")\n", - "\n", - "# Idempotent upsert: unique per (file_fingerprint, institution_id)\n", - "spark.sql(\n", - " f\"\"\"\n", - " MERGE INTO {PLAN_TABLE_PATH} AS t\n", - " USING incoming_plan_rows AS s\n", - " ON t.file_fingerprint = s.file_fingerprint\n", - " AND t.institution_id = s.institution_id\n", - " WHEN MATCHED THEN UPDATE SET\n", - " t.file_name = s.file_name,\n", - " t.local_path = s.local_path,\n", - " t.inst_col = s.inst_col,\n", - " t.file_size = s.file_size,\n", - " t.file_modified_time = s.file_modified_time,\n", - " t.planned_at = s.planned_at\n", - " WHEN NOT MATCHED THEN INSERT *\n", - " \"\"\"\n", - ")\n", - "\n", - "count_out = df_plan.count()\n", - "logger.info(\n", - " f\"Wrote/updated {count_out} institution work item(s) into {PLAN_TABLE_PATH}.\"\n", - ")\n", - "dbutils.notebook.exit(f\"WORK_ITEMS={count_out}\")" - ] - } - ], - "metadata": { - "application/vnd.databricks.v1+notebook": { - "computePreferences": null, - "dashboards": [], - "environmentMetadata": { - "base_environment": "", - "environment_version": "4" - }, - "inputWidgetPreferences": null, - "language": "python", - "notebookMetadata": { - "pythonIndentUnit": 4 - }, - "notebookName": "02_file_institution_expand", - "widgets": {} - }, - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} diff --git a/notebooks/nsc_sftp_automated_data_ingestion/03_per_institution_bronze_ingest.ipynb b/notebooks/nsc_sftp_automated_data_ingestion/03_per_institution_bronze_ingest.ipynb deleted file mode 100644 index 898b46e00..000000000 --- a/notebooks/nsc_sftp_automated_data_ingestion/03_per_institution_bronze_ingest.ipynb +++ /dev/null @@ -1,582 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "0ed056e5-420d-4b47-8812-cf63f1f895c3", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "# Databricks notebook source\n", - "# Script 4 \u2014 04_per_institution_bronze_ingest\n", - "#\n", - "# Purpose:\n", - "# Consume institution_ingest_plan (created by Script 3), and for each (file \u00d7 institution):\n", - "# - get bearer token from SST staging using X-API-KEY (from Databricks secrets)\n", - "# - call /api/v1/institutions/pdp-id/{pdp_id} to resolve institution name\n", - "# - map name -> schema prefix via databricksify_inst_name()\n", - "# - locate _bronze schema in staging_sst_01\n", - "# - choose a volume in that schema containing \"bronze\"\n", - "# - filter rows by institution id (exactly like current script)\n", - "# - write to bronze volume using helper.process_and_save_file (exact same ingestion method)\n", - "# After all institutions for a file are processed, update ingestion_manifest:\n", - "# - BRONZE_WRITTEN if all institution ingests succeeded (or were already present)\n", - "# - FAILED if any error occurred for that file (store error_message)\n", - "#\n", - "# Constraints:\n", - "# - NO SFTP connection (uses staged local files from Script 1/3)\n", - "# - Uses existing ingestion function + behavior from current script\n" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "de7936c9-a18c-4a87-858a-2c15045481d0", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "%pip install pandas python-box pyyaml requests paramiko\n", - "%pip install git+https://github.com/datakind/edvise.git@Automated_Ingestion_Workflow" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%restart_python" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "83538ecc-3986-46a8-a755-fb037fee8039", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "import logging\n", - "import os\n", - "\n", - "import pandas as pd\n", - "from databricks.connect import DatabricksSession\n", - "\n", - "from pyspark.sql import functions as F\n", - "\n", - "from edvise.utils.api_requests import (\n", - " EdviseAPIClient,\n", - " fetch_institution_by_pdp_id,\n", - ")\n", - "from edvise.utils.data_cleaning import convert_to_snake_case\n", - "from edvise.utils.databricks import (\n", - " find_bronze_schema,\n", - " find_bronze_volume_name,\n", - ")\n", - "from edvise.utils.institution_naming import databricksify_inst_name\n", - "from edvise.utils.sftp import output_file_name_from_sftp\n", - "from edvise.ingestion.nsc_sftp.helpers import (\n", - " process_and_save_file,\n", - " update_manifest,\n", - ")\n", - "from edvise.ingestion.nsc_sftp.constants import (\n", - " CATALOG,\n", - " PLAN_TABLE_PATH,\n", - " MANIFEST_TABLE_PATH,\n", - " SST_BASE_URL,\n", - " SST_TOKEN_ENDPOINT,\n", - " INSTITUTION_LOOKUP_PATH,\n", - " SST_API_KEY_SECRET_KEY,\n", - " COLUMN_RENAMES,\n", - ")\n", - "\n", - "try:\n", - " dbutils # noqa: F821\n", - "except NameError:\n", - " from unittest.mock import MagicMock\n", - "\n", - " dbutils = MagicMock()\n", - "\n", - "try:\n", - " display # noqa: F821\n", - "except NameError:\n", - "\n", - " def display(x):\n", - " return x\n", - "\n", - "\n", - "spark = DatabricksSession.builder.getOrCreate()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "7aea7d3e-2734-40ed-ae5c-a32e67ce3541", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "logging.basicConfig(\n", - " level=logging.INFO,\n", - " format=\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\",\n", - ")\n", - "logger = logging.getLogger(__name__)\n", - "\n", - "asset_scope = \"nsc-sftp-asset\"\n", - "SST_API_KEY = dbutils.secrets.get(scope=asset_scope, key=SST_API_KEY_SECRET_KEY).strip()\n", - "if not SST_API_KEY:\n", - " raise RuntimeError(\n", - " f\"Empty SST API key from secrets: scope={asset_scope} key={SST_API_KEY_SECRET_KEY}\"\n", - " )\n", - "\n", - "api_client = EdviseAPIClient(\n", - " api_key=SST_API_KEY,\n", - " base_url=SST_BASE_URL,\n", - " token_endpoint=SST_TOKEN_ENDPOINT,\n", - " institution_lookup_path=INSTITUTION_LOOKUP_PATH,\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "1a0c7f38-ab8f-4a54-a778-6c2e79b5044d", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "if not spark.catalog.tableExists(PLAN_TABLE_PATH):\n", - " logger.info(f\"Plan table not found: {PLAN_TABLE_PATH}. Exiting (no-op).\")\n", - " dbutils.notebook.exit(\"NO_PLAN_TABLE\")\n", - "\n", - "if not spark.catalog.tableExists(MANIFEST_TABLE_PATH):\n", - " raise RuntimeError(f\"Manifest table missing: {MANIFEST_TABLE_PATH}\")\n", - "\n", - "plan_df = spark.table(PLAN_TABLE_PATH)\n", - "if plan_df.limit(1).count() == 0:\n", - " logger.info(\"institution_ingest_plan is empty. Exiting (no-op).\")\n", - " dbutils.notebook.exit(\"NO_WORK_ITEMS\")\n", - "\n", - "manifest_df = spark.table(MANIFEST_TABLE_PATH).select(\"file_fingerprint\", \"status\")\n", - "plan_new_df = plan_df.join(manifest_df, on=\"file_fingerprint\", how=\"inner\").where(\n", - " F.col(\"status\") == F.lit(\"NEW\")\n", - ")\n", - "if plan_new_df.limit(1).count() == 0:\n", - " logger.info(\"No planned work items where manifest status=NEW. Exiting (no-op).\")\n", - " dbutils.notebook.exit(\"NO_NEW_TO_INGEST\")\n", - "\n", - "plan_summary_df = (\n", - " plan_new_df.groupBy(\"file_name\", \"inst_col\", \"local_path\")\n", - " .agg(F.countDistinct(\"institution_id\").alias(\"institution_count\"))\n", - " .orderBy(\"file_name\")\n", - ")\n", - "logger.info(\"Planned work summary (manifest status=NEW):\")\n", - "display(plan_summary_df)\n", - "\n", - "# Collect file groups\n", - "file_groups = (\n", - " plan_new_df.select(\n", - " \"file_fingerprint\",\n", - " \"file_name\",\n", - " \"local_path\",\n", - " \"inst_col\",\n", - " \"file_size\",\n", - " \"file_modified_time\",\n", - " )\n", - " .distinct()\n", - " .collect()\n", - ")\n", - "\n", - "logger.info(f\"Preparing to ingest {len(file_groups)} NEW file(s).\")" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "application/vnd.databricks.v1+cell": { - "cellMetadata": { - "byteLimit": 2048000, - "rowLimit": 10000 - }, - "inputWidgets": {}, - "nuid": "cf0729e1-7a4f-402a-85b6-1bca3696f878", - "showTitle": false, - "tableResultSettingsMap": {}, - "title": "" - } - }, - "outputs": [], - "source": [ - "# ---------------------------\n", - "# Main per-file ingest loop\n", - "# ---------------------------\n", - "\n", - "# Best-effort: capture Databricks job/workflow run id for manifest logging.\n", - "def _get_workflow_run_id():\n", - " try:\n", - " ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()\n", - " tags = ctx.tags()\n", - " for k in (\"jobRunId\", \"runId\"):\n", - " try:\n", - " v = tags.apply(k)\n", - " if v:\n", - " return str(v)\n", - " except Exception:\n", - " pass\n", - " try:\n", - " v = ctx.currentRunId().get()\n", - " if v:\n", - " return str(v)\n", - " except Exception:\n", - " pass\n", - " except Exception:\n", - " pass\n", - " return None\n", - "\n", - "\n", - "workflow_run_id = _get_workflow_run_id()\n", - "logger.info(f\"Workflow run_id: {workflow_run_id}\")\n", - "\n", - "processed_files = 0\n", - "failed_files = 0\n", - "skipped_files = 0\n", - "\n", - "for fg in file_groups:\n", - " fp = fg[\"file_fingerprint\"]\n", - " sftp_file_name = fg[\"file_name\"]\n", - " local_path = fg[\"local_path\"]\n", - " inst_col = fg[\"inst_col\"]\n", - "\n", - " if not local_path or not os.path.exists(local_path):\n", - " err = f\"Staged local file missing for fp={fp}: {local_path}\"\n", - " logger.error(err)\n", - " update_manifest(\n", - " spark,\n", - " MANIFEST_TABLE_PATH,\n", - " fp,\n", - " status=\"FAILED\",\n", - " error_message=err[:8000],\n", - " run_id=workflow_run_id,\n", - " )\n", - " failed_files += 1\n", - " continue\n", - "\n", - " try:\n", - " # Read only the institution-id column as string at load time to avoid float promotion\n", - " header_cols = pd.read_csv(local_path, nrows=0).columns.tolist()\n", - " raw_inst_col = next(\n", - " (\n", - " c\n", - " for c in header_cols\n", - " if COLUMN_RENAMES.get(\n", - " convert_to_snake_case(c), convert_to_snake_case(c)\n", - " )\n", - " == inst_col\n", - " ),\n", - " None,\n", - " )\n", - " dtype = {raw_inst_col: str} if raw_inst_col else None\n", - " df_full = pd.read_csv(local_path, on_bad_lines=\"warn\", dtype=dtype)\n", - " df_full = df_full.rename(\n", - " columns={c: convert_to_snake_case(c) for c in df_full.columns}\n", - " )\n", - " df_full = df_full.rename(columns=COLUMN_RENAMES)\n", - "\n", - " # File-level metrics (best-effort) for downstream dashboarding\n", - " file_student_count = None\n", - " try:\n", - " student_col = next(\n", - " (\n", - " c\n", - " for c in (\"student_id\", \"study_id\", \"student_guid\")\n", - " if c in df_full.columns\n", - " ),\n", - " None,\n", - " )\n", - " if student_col:\n", - " file_student_count = int(df_full[student_col].nunique(dropna=True))\n", - " except Exception:\n", - " file_student_count = None\n", - "\n", - " file_cohort = None\n", - " try:\n", - " if \"cohort\" in df_full.columns:\n", - " vals = (\n", - " df_full[\"cohort\"]\n", - " .dropna()\n", - " .astype(str)\n", - " .map(lambda x: x.strip())\n", - " .tolist()\n", - " )\n", - " vals = [\n", - " v for v in vals if v and v.lower() not in {\"nan\", \"none\", \"null\"}\n", - " ]\n", - " file_cohort = sorted(set(vals)) or None\n", - " except Exception:\n", - " file_cohort = None\n", - "\n", - " file_cohort_term_pairs = None\n", - " try:\n", - " if {\"cohort\", \"cohort_term\"}.issubset(df_full.columns):\n", - " tmp = df_full[[\"cohort\", \"cohort_term\"]].dropna()\n", - " tmp = tmp.assign(\n", - " cohort=tmp[\"cohort\"].astype(str).map(lambda x: x.strip()),\n", - " cohort_term=tmp[\"cohort_term\"]\n", - " .astype(str)\n", - " .map(lambda x: x.strip().upper()),\n", - " )\n", - " tmp = tmp[\n", - " (tmp[\"cohort\"] != \"\")\n", - " & (tmp[\"cohort_term\"] != \"\")\n", - " & (~tmp[\"cohort\"].str.lower().isin({\"nan\", \"none\", \"null\"}))\n", - " & (~tmp[\"cohort_term\"].str.lower().isin({\"nan\", \"none\", \"null\"}))\n", - " ]\n", - " tmp = tmp.drop_duplicates().sort_values(by=[\"cohort\", \"cohort_term\"])\n", - " pairs = [\n", - " {\"cohort\": r.cohort, \"cohort_term\": r.cohort_term}\n", - " for r in tmp.itertuples(index=False)\n", - " ]\n", - " file_cohort_term_pairs = pairs or None\n", - " except Exception:\n", - " file_cohort_term_pairs = None\n", - "\n", - " logger.info(\n", - " \"file=%s fp=%s: student_count=%s cohort_count=%s\",\n", - " sftp_file_name,\n", - " fp,\n", - " file_student_count,\n", - " (len(file_cohort) if file_cohort else 0),\n", - " )\n", - "\n", - " if inst_col not in df_full.columns:\n", - " err = f\"Expected institution column '{inst_col}' not found after normalization/renames for file={sftp_file_name} fp={fp}\"\n", - " logger.error(err)\n", - " update_manifest(\n", - " spark,\n", - " MANIFEST_TABLE_PATH,\n", - " fp,\n", - " status=\"FAILED\",\n", - " error_message=err[:8000],\n", - " run_id=workflow_run_id,\n", - " cohort=file_cohort,\n", - " cohort_term_pairs=file_cohort_term_pairs,\n", - " student_count=file_student_count,\n", - " )\n", - " failed_files += 1\n", - " continue\n", - "\n", - " inst_ids = (\n", - " plan_new_df.where(F.col(\"file_fingerprint\") == fp)\n", - " .select(\"institution_id\")\n", - " .distinct()\n", - " .collect()\n", - " )\n", - " inst_ids = [r[\"institution_id\"] for r in inst_ids]\n", - "\n", - " if not inst_ids:\n", - " logger.info(\n", - " f\"No institution_ids in plan for file={sftp_file_name} fp={fp}. Marking BRONZE_WRITTEN (no-op).\"\n", - " )\n", - " update_manifest(\n", - " spark,\n", - " MANIFEST_TABLE_PATH,\n", - " fp,\n", - " status=\"BRONZE_WRITTEN\",\n", - " error_message=None,\n", - " run_id=workflow_run_id,\n", - " cohort=file_cohort,\n", - " cohort_term_pairs=file_cohort_term_pairs,\n", - " student_count=file_student_count,\n", - " )\n", - " skipped_files += 1\n", - " continue\n", - "\n", - " preview_inst_ids = inst_ids[:10]\n", - " logger.info(\n", - " f\"file={sftp_file_name} fp={fp}: ingesting {len(inst_ids)} institution(s) \"\n", - " f\"using inst_col='{inst_col}'. Preview first 10 IDs={preview_inst_ids}\"\n", - " )\n", - "\n", - " # Aggregate errors at file-level\n", - " file_errors = []\n", - "\n", - " for inst_id in inst_ids:\n", - " try:\n", - " target_inst_id = str(inst_id)\n", - " filtered_df = df_full[df_full[inst_col] == target_inst_id].reset_index(\n", - " drop=True\n", - " )\n", - "\n", - " if filtered_df.empty:\n", - " logger.info(\n", - " f\"file={sftp_file_name} fp={fp}: institution {inst_id} has 0 rows; skipping.\"\n", - " )\n", - " continue\n", - "\n", - " # Resolve institution -> name\n", - " inst_info = fetch_institution_by_pdp_id(api_client, inst_id)\n", - " inst_name = inst_info.get(\"name\")\n", - " if not inst_name:\n", - " raise ValueError(\n", - " f\"SST API returned no 'name' for pdp_id={inst_id}. Response={inst_info}\"\n", - " )\n", - "\n", - " inst_prefix = databricksify_inst_name(inst_name)\n", - "\n", - " # Find bronze schema + volume\n", - " bronze_schema = find_bronze_schema(spark, CATALOG, inst_prefix)\n", - " bronze_volume_name = find_bronze_volume_name(\n", - " spark, CATALOG, bronze_schema\n", - " )\n", - " volume_dir = f\"/Volumes/{CATALOG}/{bronze_schema}/{bronze_volume_name}\"\n", - "\n", - " # Output naming rule (same as current script)\n", - " out_file_name = output_file_name_from_sftp(sftp_file_name)\n", - " full_path = os.path.join(volume_dir, out_file_name)\n", - "\n", - " # Idempotency check\n", - " if os.path.exists(full_path):\n", - " logger.info(\n", - " f\"file={sftp_file_name} inst={inst_id}: already exists in {volume_dir}; skipping write.\"\n", - " )\n", - " continue\n", - "\n", - " logger.info(\n", - " f\"file={sftp_file_name} inst={inst_id}: writing to {volume_dir} as {out_file_name}\"\n", - " )\n", - " process_and_save_file(\n", - " volume_dir=volume_dir, file_name=out_file_name, df=filtered_df\n", - " )\n", - " logger.info(f\"file={sftp_file_name} inst={inst_id}: write complete.\")\n", - "\n", - " except Exception as e:\n", - " msg = f\"inst_ingest_failed file={sftp_file_name} fp={fp} inst={inst_id}: {e}\"\n", - " logger.exception(msg)\n", - " file_errors.append(msg)\n", - "\n", - " if file_errors:\n", - " err = \" | \".join(file_errors)[:8000]\n", - " update_manifest(\n", - " spark,\n", - " MANIFEST_TABLE_PATH,\n", - " fp,\n", - " status=\"FAILED\",\n", - " error_message=err,\n", - " run_id=workflow_run_id,\n", - " cohort=file_cohort,\n", - " cohort_term_pairs=file_cohort_term_pairs,\n", - " student_count=file_student_count,\n", - " )\n", - " failed_files += 1\n", - " else:\n", - " update_manifest(\n", - " spark,\n", - " MANIFEST_TABLE_PATH,\n", - " fp,\n", - " status=\"BRONZE_WRITTEN\",\n", - " error_message=None,\n", - " run_id=workflow_run_id,\n", - " cohort=file_cohort,\n", - " cohort_term_pairs=file_cohort_term_pairs,\n", - " student_count=file_student_count,\n", - " )\n", - " processed_files += 1\n", - "\n", - " except Exception as e:\n", - " msg = f\"fatal_file_error file={sftp_file_name} fp={fp}: {e}\"\n", - " logger.exception(msg)\n", - " update_manifest(\n", - " spark,\n", - " MANIFEST_TABLE_PATH,\n", - " fp,\n", - " status=\"FAILED\",\n", - " error_message=msg[:8000],\n", - " run_id=workflow_run_id,\n", - " )\n", - " failed_files += 1\n", - "\n", - "logger.info(\n", - " f\"Done. processed_files={processed_files}, failed_files={failed_files}, skipped_files={skipped_files}\"\n", - ")\n", - "dbutils.notebook.exit(\n", - " f\"PROCESSED={processed_files};FAILED={failed_files};SKIPPED={skipped_files}\"\n", - ")" - ] - } - ], - "metadata": { - "application/vnd.databricks.v1+notebook": { - "computePreferences": null, - "dashboards": [], - "environmentMetadata": { - "base_environment": "", - "environment_version": "4" - }, - "inputWidgetPreferences": null, - "language": "python", - "notebookMetadata": { - "pythonIndentUnit": 4 - }, - "notebookName": "03_per_institution_bronze_ingest", - "widgets": {} - }, - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} From fcb73635fc7e2bdc23232d467cdb018f5dc31b7c Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Fri, 7 Aug 2026 11:48:48 -0700 Subject: [PATCH 05/10] refactor: rename file_selection_mode uningested to skip_ingested Clearer default for scheduled runs; keep uningested as a short alias. Co-authored-by: Cursor --- .../pdp/resources/nsc_sftp_ingestion.yml | 4 ++-- .../ingestion/nsc_sftp/file_selection.py | 19 ++++++++++------ .../nsc_sftp/scripts/01_sftp_receive_scan.py | 4 ++-- tests/ingestion/test_file_selection.py | 22 ++++++++++++++----- 4 files changed, 32 insertions(+), 17 deletions(-) diff --git a/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml b/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml index f89859f57..029e43f0a 100644 --- a/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml +++ b/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml @@ -4,7 +4,7 @@ # # File selection: # - Set cohort_file_name + course_file_name for a manual run, OR -# - Leave them empty and use file_selection_mode=uningested|latest (default: uningested). +# - Leave them empty and use file_selection_mode=skip_ingested|latest (default: skip_ingested). resources: jobs: @@ -21,7 +21,7 @@ resources: - name: DB_workspace default: ${var.DB_workspace} - name: file_selection_mode - default: uningested + default: skip_ingested - name: cohort_file_name default: "" - name: course_file_name diff --git a/src/edvise/ingestion/nsc_sftp/file_selection.py b/src/edvise/ingestion/nsc_sftp/file_selection.py index fbbf74bbb..c21ecf2e0 100644 --- a/src/edvise/ingestion/nsc_sftp/file_selection.py +++ b/src/edvise/ingestion/nsc_sftp/file_selection.py @@ -13,7 +13,10 @@ from typing import Any, Iterable, Literal, Mapping, Optional FILE_STAMP_RE = re.compile(r"_(\d{14})(?:\.[^.]+)?$", re.IGNORECASE) -FileSelectionMode = Literal["manual", "latest", "uningested"] +FileSelectionMode = Literal["manual", "latest", "skip_ingested"] + +# Older job runs may still pass this; treat as skip_ingested. +_MODE_ALIASES = {"uningested": "skip_ingested"} @dataclass(frozen=True) @@ -87,13 +90,15 @@ def select_file_pair( """ Resolve cohort/course file names for an ingestion run. - ``uningested`` skips pairs whose cohort and course names are both present in - ``ingested_file_names`` (typically BRONZE_WRITTEN file_name values). Stamp-based - NSC names make file_name a stable version key without Spark fingerprinting. + ``skip_ingested`` picks the newest stamp pair that is not already fully + present in ``ingested_file_names`` (typically BRONZE_WRITTEN file_name values). + Stamp-based NSC names make file_name a stable version key without Spark + fingerprinting. """ cohort_file_name = (cohort_file_name or "").strip() course_file_name = (course_file_name or "").strip() - mode_norm = (mode or "uningested").strip().lower() + mode_norm = (mode or "skip_ingested").strip().lower() + mode_norm = _MODE_ALIASES.get(mode_norm, mode_norm) if cohort_file_name and course_file_name: cohort_stamp = extract_file_stamp(cohort_file_name) @@ -110,10 +115,10 @@ def select_file_pair( "file_selection_mode=manual requires both cohort_file_name and " "course_file_name job parameters." ) - if mode_norm not in {"latest", "uningested"}: + if mode_norm not in {"latest", "skip_ingested"}: raise ValueError( f"Unsupported file_selection_mode={mode!r}. " - "Use 'manual', 'latest', or 'uningested'." + "Use 'manual', 'latest', or 'skip_ingested'." ) pairs = discover_file_pairs(file_rows) diff --git a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py index 12c06d008..774106b51 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py @@ -39,7 +39,7 @@ cohort_file_name = runtime.job_param("cohort_file_name") course_file_name = runtime.job_param("course_file_name") file_selection_mode = ( - runtime.job_param("file_selection_mode", "uningested").lower() or "uningested" + runtime.job_param("file_selection_mode", "skip_ingested").lower() or "skip_ingested" ) logger.info( @@ -63,7 +63,7 @@ available = sorted({r["file_name"] for r in file_rows_all if r.get("file_name")}) logger.info("SFTP files=%s preview=%s", len(available), available[:25]) - # Cheap uningested check: BRONZE_WRITTEN file names only (no full Spark fingerprint pass). + # skip_ingested: BRONZE_WRITTEN file names only (no full Spark fingerprint pass). ingested_names: set[str] = set() if spark.catalog.tableExists(MANIFEST_TABLE_PATH): ingested_names = { diff --git a/tests/ingestion/test_file_selection.py b/tests/ingestion/test_file_selection.py index a656720e1..45315d0d7 100644 --- a/tests/ingestion/test_file_selection.py +++ b/tests/ingestion/test_file_selection.py @@ -47,7 +47,7 @@ def test_select_file_pair_manual(): course = "A_Course_20240115123045.csv" c, o, mode = select_file_pair( [], - mode="uningested", + mode="skip_ingested", cohort_file_name=cohort, course_file_name=course, ) @@ -67,7 +67,7 @@ def test_select_file_pair_latest(): assert o == "B_Course_20240201101010.csv" -def test_select_file_pair_uningested_skips_bronze_written(): +def test_select_file_pair_skip_ingested(): rows = [ _row("A_Cohort_20240115123045.csv"), _row("A_Course_20240115123045.csv"), @@ -76,18 +76,28 @@ def test_select_file_pair_uningested_skips_bronze_written(): ] c, o, mode = select_file_pair( rows, - mode="uningested", + mode="skip_ingested", ingested_file_names={ "B_Cohort_20240201101010.csv", "B_Course_20240201101010.csv", }, ) - assert mode == "uningested" + assert mode == "skip_ingested" assert c == "A_Cohort_20240115123045.csv" assert o == "A_Course_20240115123045.csv" -def test_select_file_pair_uningested_all_done_raises(): +def test_select_file_pair_skip_ingested_alias_uningested(): + rows = [ + _row("A_Cohort_20240115123045.csv"), + _row("A_Course_20240115123045.csv"), + ] + c, o, mode = select_file_pair(rows, mode="uningested") + assert mode == "skip_ingested" + assert c == "A_Cohort_20240115123045.csv" + + +def test_select_file_pair_skip_ingested_all_done_raises(): rows = [ _row("A_Cohort_20240115123045.csv"), _row("A_Course_20240115123045.csv"), @@ -95,7 +105,7 @@ def test_select_file_pair_uningested_all_done_raises(): with pytest.raises(FileNotFoundError, match="already BRONZE_WRITTEN"): select_file_pair( rows, - mode="uningested", + mode="skip_ingested", ingested_file_names={ "A_Cohort_20240115123045.csv", "A_Course_20240115123045.csv", From 3a75f5bedc27f0e03e537b451e043189c22f5f5b Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Fri, 7 Aug 2026 11:51:05 -0700 Subject: [PATCH 06/10] fix: removing unnecessary alias --- src/edvise/ingestion/nsc_sftp/file_selection.py | 4 ---- tests/ingestion/test_file_selection.py | 10 ---------- 2 files changed, 14 deletions(-) diff --git a/src/edvise/ingestion/nsc_sftp/file_selection.py b/src/edvise/ingestion/nsc_sftp/file_selection.py index c21ecf2e0..34d446d5e 100644 --- a/src/edvise/ingestion/nsc_sftp/file_selection.py +++ b/src/edvise/ingestion/nsc_sftp/file_selection.py @@ -15,9 +15,6 @@ FILE_STAMP_RE = re.compile(r"_(\d{14})(?:\.[^.]+)?$", re.IGNORECASE) FileSelectionMode = Literal["manual", "latest", "skip_ingested"] -# Older job runs may still pass this; treat as skip_ingested. -_MODE_ALIASES = {"uningested": "skip_ingested"} - @dataclass(frozen=True) class FilePair: @@ -98,7 +95,6 @@ def select_file_pair( cohort_file_name = (cohort_file_name or "").strip() course_file_name = (course_file_name or "").strip() mode_norm = (mode or "skip_ingested").strip().lower() - mode_norm = _MODE_ALIASES.get(mode_norm, mode_norm) if cohort_file_name and course_file_name: cohort_stamp = extract_file_stamp(cohort_file_name) diff --git a/tests/ingestion/test_file_selection.py b/tests/ingestion/test_file_selection.py index 45315d0d7..84f89026c 100644 --- a/tests/ingestion/test_file_selection.py +++ b/tests/ingestion/test_file_selection.py @@ -87,16 +87,6 @@ def test_select_file_pair_skip_ingested(): assert o == "A_Course_20240115123045.csv" -def test_select_file_pair_skip_ingested_alias_uningested(): - rows = [ - _row("A_Cohort_20240115123045.csv"), - _row("A_Course_20240115123045.csv"), - ] - c, o, mode = select_file_pair(rows, mode="uningested") - assert mode == "skip_ingested" - assert c == "A_Cohort_20240115123045.csv" - - def test_select_file_pair_skip_ingested_all_done_raises(): rows = [ _row("A_Cohort_20240115123045.csv"), From 08d84f16fce19181ed12e9da5e15efcc16b27282 Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Fri, 7 Aug 2026 11:55:07 -0700 Subject: [PATCH 07/10] refactor: tightening code and using existing shared helpers when possible ensuring everything is also parameter based and not hard coded --- .github/workflows/deploy.yml | 44 +++++++++ pipelines/ingestion/pdp/databricks.yml | 4 + .../pdp/resources/nsc_sftp_ingestion.yml | 13 ++- src/edvise/ingestion/nsc_sftp/constants.py | 65 ++----------- .../ingestion/nsc_sftp/file_selection.py | 4 +- src/edvise/ingestion/nsc_sftp/helpers.py | 38 ++++++-- src/edvise/ingestion/nsc_sftp/runtime.py | 86 +++++++++++++---- .../nsc_sftp/scripts/01_sftp_receive_scan.py | 45 +++------ .../scripts/02_file_institution_expand.py | 19 ++-- .../03_per_institution_bronze_ingest.py | 95 +++++++------------ 10 files changed, 230 insertions(+), 183 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index cd4afb1b1..2cae7cbaf 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -132,6 +132,28 @@ jobs: --var="datakind_group_to_manage_workflow=$GROUP_TO_MANAGE" \ --var="git_tag=${{ github.ref_name }}" + - name: Deploy NSC SFTP ingestion bundle (prod target -> staging_sst_01) + working-directory: pipelines/ingestion/pdp + env: + DATABRICKS_HOST: ${{ secrets.DATABRICKS_STAGING_HOST }} + DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_STAGING_CLIENT_ID }} + DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_STAGING_CLIENT_SECRET }} + SA_EXECUTER: ${{ secrets.STAGING_SERVICE_ACCOUNT_EXECUTER }} + DS_RUN_AS: ${{ secrets.STAGING_DS_RUN_AS }} + GROUP_TO_MANAGE: ${{ secrets.GROUP_TO_MANAGE }} + NSC_SFTP_SECRET_SCOPE: ${{ secrets.NSC_SFTP_SECRET_SCOPE }} + SST_API_KEY_SECRET_KEY: ${{ secrets.STAGING_SST_API_KEY_SECRET_KEY }} + run: | + databricks bundle deploy \ + --target=prod \ + --force-lock \ + --var="service_account_executer=$SA_EXECUTER" \ + --var="ds_run_as=$DS_RUN_AS" \ + --var="datakind_group_to_manage_workflow=$GROUP_TO_MANAGE" \ + --var="nsc_sftp_secret_scope=$NSC_SFTP_SECRET_SCOPE" \ + --var="sst_api_key_secret_key=$SST_API_KEY_SECRET_KEY" \ + --var="git_tag=${{ github.ref_name }}" + # DEV (dev-sst-02) deploy-dev: if: ${{ startsWith(github.ref, 'refs/tags/v') }} @@ -250,4 +272,26 @@ jobs: --var="ds_run_as=$DS_RUN_AS" \ --var="databricks_institution_name=synthetic" \ --var="datakind_group_to_manage_workflow=$GROUP_TO_MANAGE" \ + --var="git_commit=${{ github.sha }}" + + - name: Deploy NSC SFTP ingestion bundle (dev target -> dev_sst_02) + working-directory: pipelines/ingestion/pdp + env: + DATABRICKS_HOST: ${{ secrets.DATABRICKS_DEV_HOST }} + DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_DEV_CLIENT_ID }} + DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_DEV_CLIENT_SECRET }} + SA_EXECUTER: ${{ secrets.DEV_SERVICE_ACCOUNT_EXECUTER }} + DS_RUN_AS: ${{ secrets.DEV_DS_RUN_AS }} + GROUP_TO_MANAGE: ${{ secrets.GROUP_TO_MANAGE }} + NSC_SFTP_SECRET_SCOPE: ${{ secrets.NSC_SFTP_SECRET_SCOPE }} + SST_API_KEY_SECRET_KEY: ${{ secrets.DEV_SST_API_KEY_SECRET_KEY }} + run: | + databricks bundle deploy \ + --target=dev \ + --force-lock \ + --var="service_account_executer=$SA_EXECUTER" \ + --var="ds_run_as=$DS_RUN_AS" \ + --var="datakind_group_to_manage_workflow=$GROUP_TO_MANAGE" \ + --var="nsc_sftp_secret_scope=$NSC_SFTP_SECRET_SCOPE" \ + --var="sst_api_key_secret_key=$SST_API_KEY_SECRET_KEY" \ --var="git_commit=${{ github.sha }}" \ No newline at end of file diff --git a/pipelines/ingestion/pdp/databricks.yml b/pipelines/ingestion/pdp/databricks.yml index c74121196..d7a7dd50c 100644 --- a/pipelines/ingestion/pdp/databricks.yml +++ b/pipelines/ingestion/pdp/databricks.yml @@ -24,6 +24,10 @@ variables: description: "Workspace group with CAN_MANAGE on the job" ingestion_slack_webhook_id: description: "Slack webhook ID for ingestion pipeline failure notifications" + nsc_sftp_secret_scope: + description: "Databricks secret scope holding NSC SFTP + SST API credentials (from GitHub Secrets at deploy)" + sst_api_key_secret_key: + description: "Key name within nsc_sftp_secret_scope for the SST API key (from GitHub Secrets at deploy)" run_as: service_principal_name: ${var.ds_run_as} diff --git a/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml b/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml index 029e43f0a..5d228d99e 100644 --- a/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml +++ b/pipelines/ingestion/pdp/resources/nsc_sftp_ingestion.yml @@ -1,6 +1,7 @@ # NSC/PDP SFTP automated ingestion: Git-sourced spark_python_task chain (01→02→03). # Prerequisites: Job parameter DB_workspace (Unity Catalog), matching UC tables/volumes, -# secret scope nsc-sftp-asset (SFTP + SST API key), and cluster egress for SFTP/APIs. +# Databricks secret scope/key (passed via DAB vars from GitHub Secrets at deploy), +# and cluster egress for SFTP/APIs. # # File selection: # - Set cohort_file_name + course_file_name for a manual run, OR @@ -20,6 +21,10 @@ resources: parameters: - name: DB_workspace default: ${var.DB_workspace} + - name: nsc_sftp_secret_scope + default: ${var.nsc_sftp_secret_scope} + - name: sst_api_key_secret_key + default: ${var.sst_api_key_secret_key} - name: file_selection_mode default: skip_ingested - name: cohort_file_name @@ -35,6 +40,8 @@ resources: parameters: - --DB_workspace - "{{job.parameters.DB_workspace}}" + - --nsc_sftp_secret_scope + - "{{job.parameters.nsc_sftp_secret_scope}}" - --file_selection_mode - "{{job.parameters.file_selection_mode}}" - --cohort_file_name @@ -91,6 +98,10 @@ resources: parameters: - --DB_workspace - "{{job.parameters.DB_workspace}}" + - --nsc_sftp_secret_scope + - "{{job.parameters.nsc_sftp_secret_scope}}" + - --sst_api_key_secret_key + - "{{job.parameters.sst_api_key_secret_key}}" job_cluster_key: nsc-sftp-ingestion-cluster libraries: - pypi: diff --git a/src/edvise/ingestion/nsc_sftp/constants.py b/src/edvise/ingestion/nsc_sftp/constants.py index b1f6d3b95..98432b8d9 100644 --- a/src/edvise/ingestion/nsc_sftp/constants.py +++ b/src/edvise/ingestion/nsc_sftp/constants.py @@ -1,25 +1,19 @@ """ Constants for NSC SFTP ingestion pipeline. -Unity Catalog name must match the job's DB_workspace parameter (see -configure_nsc_catalog / resolve_nsc_catalog). Other values here are fixed or -scoped to default schema. +Unity Catalog name must match the job's DB_workspace parameter +(see runtime.bootstrap_catalog). Secret scope/key names are job params. """ from __future__ import annotations -import os -import sys - -# Unity Catalog name — set by configure_nsc_catalog (usually from job parameter DB_workspace). +# Unity Catalog name — set by configure_nsc_catalog. DEFAULT_CATALOG_FOR_LOCAL = "dev_sst_02" DEFAULT_SCHEMA = "default" -# Table names (without catalog.schema prefix) MANIFEST_TABLE = "ingestion_manifest" QUEUE_TABLE = "pending_ingest_queue" PLAN_TABLE = "institution_ingest_plan" - SFTP_TMP_VOLUME_NAME = "tmp" CATALOG: str @@ -48,66 +42,25 @@ def configure_nsc_catalog(catalog: str) -> None: SFTP_TMP_DIR = f"/Volumes/{CATALOG}/{DEFAULT_SCHEMA}/{SFTP_TMP_VOLUME_NAME}" -def parse_spark_python_task_params(argv: list[str] | None = None) -> dict[str, str]: - """Parse ``--key value`` pairs from ``spark_python_task.parameters``.""" - if argv is None: - argv = sys.argv - out: dict[str, str] = {} - i = 1 - while i < len(argv): - a = argv[i] - if a.startswith("--") and i + 1 < len(argv): - out[a[2:].replace("-", "_")] = argv[i + 1] - i += 2 - else: - i += 1 - return out - - -def resolve_nsc_catalog(argv: list[str] | None = None) -> str: - """ - Resolve Unity Catalog name in order: task argv ``--DB_workspace``, notebook widget - ``DB_workspace``, env ``NSC_DB_WORKSPACE``, else DEFAULT_CATALOG_FOR_LOCAL. - """ - argv = sys.argv if argv is None else argv - pairs = parse_spark_python_task_params(argv) - raw = pairs.get("DB_workspace", "").strip() - if raw: - return raw - try: - from edvise.utils.databricks import get_db_widget_param - - w = get_db_widget_param("DB_workspace", default="") - if str(w).strip(): - return str(w).strip() - except Exception: - pass - env = os.environ.get("NSC_DB_WORKSPACE", "").strip() - if env: - return env - return DEFAULT_CATALOG_FOR_LOCAL - - # SFTP settings SFTP_REMOTE_FOLDER = "./receive" SFTP_SOURCE_SYSTEM = "NSC" SFTP_PORT = 22 SFTP_DOWNLOAD_CHUNK_MB = 150 SFTP_VERIFY_DOWNLOAD = "size" # Options: "size", "sha256", "md5", "none" +SFTP_SECRET_KEY_HOST = "nsc-sftp-host" +SFTP_SECRET_KEY_USER = "nsc-sftp-user" +SFTP_SECRET_KEY_PASSWORD = "nsc-sftp-password" -# Edvise API settings -SST_BASE_URL = "https://staging-sst.datakind.org" -SST_TOKEN_ENDPOINT = f"{SST_BASE_URL}/api/v1/token-from-api-key" +# Edvise API path templates (base URL + secret scope/key come from job params). +SST_TOKEN_PATH = "/api/v1/token-from-api-key" INSTITUTION_LOOKUP_PATH = "/api/v1/institutions/pdp-id/{pdp_id}" -SST_API_KEY_SECRET_KEY = "sst_staging_api_key" # Key name in Databricks secrets -# File processing settings INSTITUTION_COLUMN_PATTERN = r"(?=.*institution)(?=.*id)" -# Column name mappings (mangled -> normalized) # Applied after snake_case conversion COLUMN_RENAMES = { - # NOTE: convert_to_snake_case splits trailing digit groups with an underscore, + # convert_to_snake_case splits trailing digit groups with an underscore, # e.g. "attemptedgatewaymathyear1" -> "attemptedgatewaymathyear_1". "attemptedgatewaymathyear_1": "attempted_gateway_math_year_1", "attemptedgatewayenglishyear_1": "attempted_gateway_english_year_1", diff --git a/src/edvise/ingestion/nsc_sftp/file_selection.py b/src/edvise/ingestion/nsc_sftp/file_selection.py index 34d446d5e..e685cb936 100644 --- a/src/edvise/ingestion/nsc_sftp/file_selection.py +++ b/src/edvise/ingestion/nsc_sftp/file_selection.py @@ -10,7 +10,7 @@ import os import re from dataclasses import dataclass -from typing import Any, Iterable, Literal, Mapping, Optional +from typing import Any, Iterable, Literal, Mapping, Optional, Sequence FILE_STAMP_RE = re.compile(r"_(\d{14})(?:\.[^.]+)?$", re.IGNORECASE) FileSelectionMode = Literal["manual", "latest", "skip_ingested"] @@ -77,7 +77,7 @@ def discover_file_pairs(file_rows: Iterable[Mapping[str, Any]]) -> list[FilePair def select_file_pair( - file_rows: list[Mapping[str, Any]], + file_rows: Sequence[Mapping[str, Any]], *, mode: str, cohort_file_name: str = "", diff --git a/src/edvise/ingestion/nsc_sftp/helpers.py b/src/edvise/ingestion/nsc_sftp/helpers.py index 1ff492707..356656b77 100644 --- a/src/edvise/ingestion/nsc_sftp/helpers.py +++ b/src/edvise/ingestion/nsc_sftp/helpers.py @@ -428,12 +428,7 @@ def normalize_staged_frame( df: pd.DataFrame, *, renames: dict[str, str] ) -> pd.DataFrame: """Apply snake_case + COLUMN_RENAMES to a staged PDP frame.""" - return df.rename( - columns={ - c: renames.get(convert_to_snake_case(c), convert_to_snake_case(c)) - for c in df.columns - } - ) + return df.rename(columns=_normalize_header_map(list(df.columns), renames)) def load_staged_csv( @@ -568,6 +563,37 @@ def resolve_bronze_volume_dir( return f"/Volumes/{catalog}/{bronze_schema}/{bronze_volume_name}" +def bronze_written_file_names(spark: pyspark.sql.SparkSession) -> set[str]: + """file_name values already marked BRONZE_WRITTEN in ingestion_manifest.""" + if not spark.catalog.tableExists(MANIFEST_TABLE_PATH): + return set() + rows = ( + spark.table(MANIFEST_TABLE_PATH) + .where(F.col("status") == F.lit("BRONZE_WRITTEN")) + .select("file_name") + .collect() + ) + return {r["file_name"] for r in rows if r["file_name"]} + + +def build_edvise_api_client( + *, + api_key: str, + db_workspace: str, + token_path: str, + institution_lookup_path: str, +): + """Construct EdviseAPIClient with workspace-derived base URL.""" + from edvise.utils.api_requests import EdviseAPIClient, get_base_url + + return EdviseAPIClient( + api_key=api_key, + base_url=get_base_url(db_workspace), + token_endpoint=token_path, + institution_lookup_path=institution_lookup_path, + ) + + def update_manifest( spark: pyspark.sql.SparkSession, manifest_table: str, diff --git a/src/edvise/ingestion/nsc_sftp/runtime.py b/src/edvise/ingestion/nsc_sftp/runtime.py index 978be365e..577015e05 100644 --- a/src/edvise/ingestion/nsc_sftp/runtime.py +++ b/src/edvise/ingestion/nsc_sftp/runtime.py @@ -3,33 +3,67 @@ from __future__ import annotations import logging +import os import sys from typing import Any from unittest.mock import MagicMock -from edvise.ingestion.nsc_sftp.constants import ( - configure_nsc_catalog, - parse_spark_python_task_params, - resolve_nsc_catalog, +from edvise.ingestion.nsc_sftp.constants import configure_nsc_catalog +from edvise.utils.databricks import ( + get_db_widget_param, + get_dbutils_or_none, + get_spark_session, ) +def parse_spark_python_task_params(argv: list[str] | None = None) -> dict[str, str]: + """Parse ``--key value`` pairs from ``spark_python_task.parameters``.""" + if argv is None: + argv = sys.argv + out: dict[str, str] = {} + i = 1 + while i < len(argv): + a = argv[i] + if a.startswith("--") and i + 1 < len(argv): + out[a[2:].replace("-", "_")] = argv[i + 1] + i += 2 + else: + i += 1 + return out + + +def resolve_nsc_catalog(argv: list[str] | None = None) -> str: + """ + Resolve Unity Catalog name: ``--DB_workspace`` → widget → ``NSC_DB_WORKSPACE`` + → default catalog for local imports. + """ + from edvise.ingestion.nsc_sftp.constants import DEFAULT_CATALOG_FOR_LOCAL + + pairs = parse_spark_python_task_params(argv) + raw = pairs.get("DB_workspace", "").strip() + if raw: + return raw + try: + w = get_db_widget_param("DB_workspace", default="") + if str(w).strip(): + return str(w).strip() + except Exception: + pass + return os.environ.get("NSC_DB_WORKSPACE", "").strip() or DEFAULT_CATALOG_FOR_LOCAL + + def bootstrap_catalog(argv: list[str] | None = None) -> None: """Configure UC catalog paths from job argv / widgets / env.""" configure_nsc_catalog(resolve_nsc_catalog(sys.argv if argv is None else argv)) def get_dbutils() -> Any: - try: - return dbutils # type: ignore[name-defined] # noqa: F821 - except NameError: - return MagicMock() + """Return real dbutils on Databricks; MagicMock locally so scripts import cleanly.""" + return get_dbutils_or_none() or MagicMock() def get_spark(): - from databricks.connect import DatabricksSession - - return DatabricksSession.builder.getOrCreate() + return get_spark_session() def get_logger(name: str) -> logging.Logger: @@ -41,13 +75,31 @@ def get_logger(name: str) -> logging.Logger: def job_param(name: str, default: str = "", *, argv: list[str] | None = None) -> str: - """Resolve a job/widget parameter as a stripped string.""" - from edvise import utils - + """Resolve a job/widget parameter as a stripped string (argv wins as default).""" pairs = parse_spark_python_task_params(sys.argv if argv is None else argv) - return str( - utils.databricks.get_db_widget_param(name, default=pairs.get(name, default)) - ).strip() + fallback = pairs.get(name, default) + try: + return str(get_db_widget_param(name, default=fallback)).strip() + except Exception: + return str(fallback).strip() + + +def require_job_param(name: str, *, argv: list[str] | None = None) -> str: + value = job_param(name, "", argv=argv) + if not value: + raise ValueError( + f"Missing required job parameter {name}. " + "Pass it via DAB var / job parameter at deploy or run time." + ) + return value + + +def notebook_exit(dbutils_obj: Any, message: str) -> None: + """Exit a Databricks task; raise SystemExit locally for testability.""" + try: + dbutils_obj.notebook.exit(message) + except Exception: + raise SystemExit(message) from None def workflow_run_id(dbutils_obj: Any) -> str | None: diff --git a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py index 774106b51..064553735 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py @@ -8,17 +8,18 @@ runtime.bootstrap_catalog() -from pyspark.sql import functions as F - from edvise.ingestion.nsc_sftp.constants import ( - MANIFEST_TABLE_PATH, QUEUE_TABLE_PATH, SFTP_REMOTE_FOLDER, + SFTP_SECRET_KEY_HOST, + SFTP_SECRET_KEY_PASSWORD, + SFTP_SECRET_KEY_USER, SFTP_SOURCE_SYSTEM, SFTP_TMP_DIR, ) from edvise.ingestion.nsc_sftp.file_selection import select_file_pair from edvise.ingestion.nsc_sftp.helpers import ( + bronze_written_file_names, build_listing_df, download_new_files_and_queue, ensure_manifest_and_queue_tables, @@ -31,10 +32,10 @@ spark = runtime.get_spark() logger = runtime.get_logger(__name__) -asset_scope = "nsc-sftp-asset" -host = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-host") -user = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-user") -password = dbutils.secrets.get(scope=asset_scope, key="nsc-sftp-password") +secret_scope = runtime.require_job_param("nsc_sftp_secret_scope") +host = dbutils.secrets.get(scope=secret_scope, key=SFTP_SECRET_KEY_HOST) +user = dbutils.secrets.get(scope=secret_scope, key=SFTP_SECRET_KEY_USER) +password = dbutils.secrets.get(scope=secret_scope, key=SFTP_SECRET_KEY_PASSWORD) cohort_file_name = runtime.job_param("cohort_file_name") course_file_name = runtime.job_param("course_file_name") @@ -58,29 +59,17 @@ file_rows_all = list_receive_files(sftp, SFTP_REMOTE_FOLDER, SFTP_SOURCE_SYSTEM) if not file_rows_all: logger.info("No files in %s; exiting.", SFTP_REMOTE_FOLDER) - dbutils.notebook.exit("NO_FILES") + runtime.notebook_exit(dbutils, "NO_FILES") available = sorted({r["file_name"] for r in file_rows_all if r.get("file_name")}) logger.info("SFTP files=%s preview=%s", len(available), available[:25]) - # skip_ingested: BRONZE_WRITTEN file names only (no full Spark fingerprint pass). - ingested_names: set[str] = set() - if spark.catalog.tableExists(MANIFEST_TABLE_PATH): - ingested_names = { - r["file_name"] - for r in spark.table(MANIFEST_TABLE_PATH) - .where(F.col("status") == F.lit("BRONZE_WRITTEN")) - .select("file_name") - .collect() - if r["file_name"] - } - cohort_file_name, course_file_name, mode_used = select_file_pair( file_rows_all, mode=file_selection_mode, cohort_file_name=cohort_file_name, course_file_name=course_file_name, - ingested_file_names=ingested_names, + ingested_file_names=bronze_written_file_names(spark), ) logger.info( "Selected via %s: cohort=%s course=%s", @@ -99,24 +88,16 @@ ) df_listing = build_listing_df(spark, file_rows) - fingerprints = [ - r.file_fingerprint for r in df_listing.select("file_fingerprint").collect() - ] upsert_new_to_manifest(spark, df_listing) df_to_queue = get_files_to_queue(spark, df_listing) if df_to_queue.limit(1).count() == 0: logger.info("Nothing NEW to queue; exiting.") - dbutils.notebook.exit("QUEUED_FILES=0") + runtime.notebook_exit(dbutils, "QUEUED_FILES=0") queued_count = download_new_files_and_queue(spark, sftp, df_to_queue, logger) - logger.info( - "Queued %s file(s). fingerprints=%s table=%s", - queued_count, - fingerprints, - QUEUE_TABLE_PATH, - ) - dbutils.notebook.exit(f"QUEUED_FILES={queued_count}") + logger.info("Queued %s file(s) into %s", queued_count, QUEUE_TABLE_PATH) + runtime.notebook_exit(dbutils, f"QUEUED_FILES={queued_count}") finally: for closer in (sftp, transport): try: diff --git a/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py b/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py index 8b2573ae4..d84000494 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py @@ -30,20 +30,19 @@ ensure_plan_table(spark, PLAN_TABLE_PATH) if not spark.catalog.tableExists(QUEUE_TABLE_PATH): - dbutils.notebook.exit("NO_QUEUE_TABLE") + runtime.notebook_exit(dbutils, "NO_QUEUE_TABLE") queue_df = spark.table(QUEUE_TABLE_PATH) if queue_df.limit(1).count() == 0: - dbutils.notebook.exit("NO_QUEUED_FILES") + runtime.notebook_exit(dbutils, "NO_QUEUED_FILES") -# Skip fingerprints already expanded. queue_df = queue_df.join( spark.table(PLAN_TABLE_PATH).select("file_fingerprint").distinct(), on="file_fingerprint", how="left_anti", ) if queue_df.limit(1).count() == 0: - dbutils.notebook.exit("NO_NEW_EXPANSION_WORK") + runtime.notebook_exit(dbutils, "NO_NEW_EXPANSION_WORK") queued_files = queue_df.select( "file_fingerprint", @@ -98,7 +97,7 @@ if missing: raise FileNotFoundError("Missing staged files: " + "; ".join(missing)) if not work_items: - dbutils.notebook.exit("NO_WORK_ITEMS") + runtime.notebook_exit(dbutils, "NO_WORK_ITEMS") schema = T.StructType( [ @@ -112,8 +111,9 @@ T.StructField("planned_at", T.TimestampType(), False), ] ) -df_plan = spark.createDataFrame(work_items, schema=schema) -df_plan.createOrReplaceTempView("incoming_plan_rows") +spark.createDataFrame(work_items, schema=schema).createOrReplaceTempView( + "incoming_plan_rows" +) spark.sql( f""" MERGE INTO {PLAN_TABLE_PATH} AS t @@ -129,6 +129,5 @@ WHEN NOT MATCHED THEN INSERT * """ ) -count_out = len(work_items) -logger.info("Wrote/updated %s plan row(s) into %s", count_out, PLAN_TABLE_PATH) -dbutils.notebook.exit(f"WORK_ITEMS={count_out}") +logger.info("Wrote/updated %s plan row(s) into %s", len(work_items), PLAN_TABLE_PATH) +runtime.notebook_exit(dbutils, f"WORK_ITEMS={len(work_items)}") diff --git a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py index 5ed36095d..2179b09c1 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py @@ -19,18 +19,17 @@ INSTITUTION_LOOKUP_PATH, MANIFEST_TABLE_PATH, PLAN_TABLE_PATH, - SST_API_KEY_SECRET_KEY, - SST_BASE_URL, - SST_TOKEN_ENDPOINT, + SST_TOKEN_PATH, ) from edvise.ingestion.nsc_sftp.helpers import ( + build_edvise_api_client, load_staged_csv, process_and_save_file, resolve_bronze_volume_dir, summarize_file_metrics, update_manifest, ) -from edvise.utils.api_requests import EdviseAPIClient, fetch_institution_by_pdp_id +from edvise.utils.api_requests import fetch_institution_by_pdp_id from edvise.utils.institution_naming import databricksify_inst_name from edvise.utils.sftp import output_file_name_from_sftp @@ -38,17 +37,20 @@ spark = runtime.get_spark() logger = runtime.get_logger(__name__) -asset_scope = "nsc-sftp-asset" -api_key = dbutils.secrets.get(scope=asset_scope, key=SST_API_KEY_SECRET_KEY).strip() +db_workspace = runtime.require_job_param("DB_workspace") +secret_scope = runtime.require_job_param("nsc_sftp_secret_scope") +sst_api_key_secret_key = runtime.require_job_param("sst_api_key_secret_key") + +api_key = dbutils.secrets.get(scope=secret_scope, key=sst_api_key_secret_key).strip() if not api_key: raise RuntimeError( - f"Empty SST API key: scope={asset_scope} key={SST_API_KEY_SECRET_KEY}" + f"Empty SST API key: scope={secret_scope} key={sst_api_key_secret_key}" ) -api_client = EdviseAPIClient( +api_client = build_edvise_api_client( api_key=api_key, - base_url=SST_BASE_URL, - token_endpoint=SST_TOKEN_ENDPOINT, + db_workspace=db_workspace, + token_path=SST_TOKEN_PATH, institution_lookup_path=INSTITUTION_LOOKUP_PATH, ) @@ -78,7 +80,7 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: if not spark.catalog.tableExists(PLAN_TABLE_PATH): - dbutils.notebook.exit("NO_PLAN_TABLE") + runtime.notebook_exit(dbutils, "NO_PLAN_TABLE") if not spark.catalog.tableExists(MANIFEST_TABLE_PATH): raise RuntimeError(f"Manifest table missing: {MANIFEST_TABLE_PATH}") @@ -92,9 +94,8 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: .where(F.col("status") == F.lit("NEW")) ) if plan_new_df.limit(1).count() == 0: - dbutils.notebook.exit("NO_NEW_TO_INGEST") + runtime.notebook_exit(dbutils, "NO_NEW_TO_INGEST") -# One collect: file metadata + institution ids grouped in Python. plan_rows = plan_new_df.select( "file_fingerprint", "file_name", "local_path", "inst_col", "institution_id" ).collect() @@ -113,7 +114,7 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: ) run_id = runtime.workflow_run_id(dbutils) -counts = defaultdict(int) +counts: dict[str, int] = defaultdict(int) bronze_dir_cache: dict[str, str] = {} for fp, meta in by_file.items(): @@ -122,21 +123,30 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: inst_col = meta["inst_col"] inst_ids = sorted(set(inst_ids_by_fp[fp])) - if not local_path or not os.path.exists(local_path): + def _fail(msg: str, **manifest_kwargs) -> None: update_manifest( spark, MANIFEST_TABLE_PATH, fp, status="FAILED", - error_message=f"Staged local file missing: {local_path}"[:8000], + error_message=msg[:8000], run_id=run_id, + **manifest_kwargs, ) counts["failed_files"] += 1 + + if not local_path or not os.path.exists(local_path): + _fail(f"Staged local file missing: {local_path}") continue try: df_full = load_staged_csv(local_path, renames=COLUMN_RENAMES, inst_col=inst_col) student_count, file_cohort, cohort_term_pairs = summarize_file_metrics(df_full) + metrics = dict( + cohort=file_cohort, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, + ) logger.info( "file=%s fp=%s students=%s cohorts=%s institutions=%s", file_name, @@ -147,18 +157,7 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: ) if inst_col not in df_full.columns: - update_manifest( - spark, - MANIFEST_TABLE_PATH, - fp, - status="FAILED", - error_message=f"Missing institution column '{inst_col}'"[:8000], - run_id=run_id, - cohort=file_cohort, - cohort_term_pairs=cohort_term_pairs, - student_count=student_count, - ) - counts["failed_files"] += 1 + _fail(f"Missing institution column '{inst_col}'", **metrics) continue if not inst_ids: @@ -169,18 +168,16 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: status="BRONZE_WRITTEN", error_message=None, run_id=run_id, - cohort=file_cohort, - cohort_term_pairs=cohort_term_pairs, - student_count=student_count, + **metrics, ) counts["skipped_files"] += 1 continue - # One pass over the frame instead of N equality filters. + wanted = set(map(str, inst_ids)) grouped = { str(k): g.reset_index(drop=True) for k, g in df_full.groupby(inst_col, sort=False) - if str(k) in set(map(str, inst_ids)) + if str(k) in wanted } file_errors: list[str] = [] out_name = output_file_name_from_sftp(file_name) @@ -249,18 +246,7 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: file_errors.append(msg) if file_errors: - update_manifest( - spark, - MANIFEST_TABLE_PATH, - fp, - status="FAILED", - error_message=" | ".join(file_errors)[:8000], - run_id=run_id, - cohort=file_cohort, - cohort_term_pairs=cohort_term_pairs, - student_count=student_count, - ) - counts["failed_files"] += 1 + _fail(" | ".join(file_errors), **metrics) else: update_manifest( spark, @@ -269,29 +255,20 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: status="BRONZE_WRITTEN", error_message=None, run_id=run_id, - cohort=file_cohort, - cohort_term_pairs=cohort_term_pairs, - student_count=student_count, + **metrics, ) counts["processed_files"] += 1 except Exception as exc: logger.exception("fatal_file_error file=%s fp=%s: %s", file_name, fp, exc) - update_manifest( - spark, - MANIFEST_TABLE_PATH, - fp, - status="FAILED", - error_message=f"fatal_file_error file={file_name} fp={fp}: {exc}"[:8000], - run_id=run_id, - ) - counts["failed_files"] += 1 + _fail(f"fatal_file_error file={file_name} fp={fp}: {exc}") logger.info("Done counts=%s", dict(counts)) -dbutils.notebook.exit( +runtime.notebook_exit( + dbutils, "PROCESSED={processed_files};FAILED={failed_files};SKIPPED={skipped_files};" "WRITTEN={institutions_written};EXISTING={institutions_skipped_existing};" "UNRESOLVED={institutions_unresolved};NO_BRONZE={institutions_no_bronze}".format_map( defaultdict(int, counts) - ) + ), ) From 05705a3ff38e04f41110e920099efd5ee9546c48 Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Fri, 7 Aug 2026 12:02:01 -0700 Subject: [PATCH 08/10] fix: type check --- src/edvise/ingestion/nsc_sftp/helpers.py | 3 +- .../03_per_institution_bronze_ingest.py | 45 +++++++++++++------ 2 files changed, 34 insertions(+), 14 deletions(-) diff --git a/src/edvise/ingestion/nsc_sftp/helpers.py b/src/edvise/ingestion/nsc_sftp/helpers.py index 356656b77..05ccf3026 100644 --- a/src/edvise/ingestion/nsc_sftp/helpers.py +++ b/src/edvise/ingestion/nsc_sftp/helpers.py @@ -16,6 +16,7 @@ if TYPE_CHECKING: import paramiko + from edvise.utils.api_requests import EdviseAPIClient import pandas as pd import pyspark.sql @@ -582,7 +583,7 @@ def build_edvise_api_client( db_workspace: str, token_path: str, institution_lookup_path: str, -): +) -> EdviseAPIClient: """Construct EdviseAPIClient with workspace-derived base URL.""" from edvise.utils.api_requests import EdviseAPIClient, get_base_url diff --git a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py index 2179b09c1..ab640cbb5 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py @@ -6,7 +6,9 @@ import os from collections import defaultdict +from typing import Optional +import pandas as pd from edvise.ingestion.nsc_sftp import runtime runtime.bootstrap_catalog() @@ -55,7 +57,7 @@ ) -def _school_check_log(file_name: str, inst_id: str, filtered) -> None: +def _school_check_log(file_name: str, inst_id: str, filtered: pd.DataFrame) -> None: if {"cohort", "cohort_term"}.issubset(filtered.columns): latest = filtered["cohort"].max() terms = ( @@ -99,7 +101,7 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: plan_rows = plan_new_df.select( "file_fingerprint", "file_name", "local_path", "inst_col", "institution_id" ).collect() -by_file: dict[str, dict] = {} +by_file: dict[str, dict[str, str]] = {} inst_ids_by_fp: dict[str, list[str]] = defaultdict(list) for row in plan_rows: fp = row["file_fingerprint"] @@ -123,7 +125,13 @@ def _school_check_log(file_name: str, inst_id: str, filtered) -> None: inst_col = meta["inst_col"] inst_ids = sorted(set(inst_ids_by_fp[fp])) - def _fail(msg: str, **manifest_kwargs) -> None: + def _fail( + msg: str, + *, + cohort: Optional[list[str]] = None, + cohort_term_pairs: Optional[list[dict[str, str]]] = None, + student_count: Optional[int] = None, + ) -> None: update_manifest( spark, MANIFEST_TABLE_PATH, @@ -131,7 +139,9 @@ def _fail(msg: str, **manifest_kwargs) -> None: status="FAILED", error_message=msg[:8000], run_id=run_id, - **manifest_kwargs, + cohort=cohort, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, ) counts["failed_files"] += 1 @@ -142,11 +152,6 @@ def _fail(msg: str, **manifest_kwargs) -> None: try: df_full = load_staged_csv(local_path, renames=COLUMN_RENAMES, inst_col=inst_col) student_count, file_cohort, cohort_term_pairs = summarize_file_metrics(df_full) - metrics = dict( - cohort=file_cohort, - cohort_term_pairs=cohort_term_pairs, - student_count=student_count, - ) logger.info( "file=%s fp=%s students=%s cohorts=%s institutions=%s", file_name, @@ -157,7 +162,12 @@ def _fail(msg: str, **manifest_kwargs) -> None: ) if inst_col not in df_full.columns: - _fail(f"Missing institution column '{inst_col}'", **metrics) + _fail( + f"Missing institution column '{inst_col}'", + cohort=file_cohort, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, + ) continue if not inst_ids: @@ -168,7 +178,9 @@ def _fail(msg: str, **manifest_kwargs) -> None: status="BRONZE_WRITTEN", error_message=None, run_id=run_id, - **metrics, + cohort=file_cohort, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, ) counts["skipped_files"] += 1 continue @@ -246,7 +258,12 @@ def _fail(msg: str, **manifest_kwargs) -> None: file_errors.append(msg) if file_errors: - _fail(" | ".join(file_errors), **metrics) + _fail( + " | ".join(file_errors), + cohort=file_cohort, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, + ) else: update_manifest( spark, @@ -255,7 +272,9 @@ def _fail(msg: str, **manifest_kwargs) -> None: status="BRONZE_WRITTEN", error_message=None, run_id=run_id, - **metrics, + cohort=file_cohort, + cohort_term_pairs=cohort_term_pairs, + student_count=student_count, ) counts["processed_files"] += 1 From 1ba72b77cb239b114c740b618f112a1dd4276901 Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Fri, 7 Aug 2026 12:21:57 -0700 Subject: [PATCH 09/10] fix: put src on sys.path for NSC SFTP job scripts Databricks GIT-sourced spark_python_task does not install the package, so entrypoints need the same path bootstrap as other pipeline scripts. Co-authored-by: Cursor --- .../nsc_sftp/scripts/01_sftp_receive_scan.py | 25 +++++++++++++++++++ .../scripts/02_file_institution_expand.py | 23 +++++++++++++++++ .../03_per_institution_bronze_ingest.py | 23 +++++++++++++++++ 3 files changed, 71 insertions(+) diff --git a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py index 064553735..ed2d06103 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/01_sftp_receive_scan.py @@ -4,6 +4,31 @@ from __future__ import annotations +import os +import sys + +# Ensure repo src/ is on sys.path so `import edvise.*` works in Databricks Jobs. +# Layout: /src/edvise/ingestion/nsc_sftp/scripts/ +_here = globals().get("__file__") +if _here: + _script_dir = os.path.dirname(os.path.abspath(_here)) +else: + _argv0 = os.path.abspath(sys.argv[0]) if sys.argv else "" + if _argv0.endswith(".py") and os.path.isfile(_argv0): + _script_dir = os.path.dirname(_argv0) + else: + _script_dir = os.path.abspath(os.getcwd()) +_current = _script_dir +for _ in range(8): + if os.path.isdir(os.path.join(_current, "edvise")): + if _current not in sys.path: + sys.path.insert(0, _current) + break + _parent = os.path.dirname(_current) + if _parent == _current: + break + _current = _parent + from edvise.ingestion.nsc_sftp import runtime runtime.bootstrap_catalog() diff --git a/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py b/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py index d84000494..1dc2f34a6 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/02_file_institution_expand.py @@ -6,8 +6,31 @@ import os import re +import sys from datetime import datetime, timezone +# Ensure repo src/ is on sys.path so `import edvise.*` works in Databricks Jobs. +# Layout: /src/edvise/ingestion/nsc_sftp/scripts/ +_here = globals().get("__file__") +if _here: + _script_dir = os.path.dirname(os.path.abspath(_here)) +else: + _argv0 = os.path.abspath(sys.argv[0]) if sys.argv else "" + if _argv0.endswith(".py") and os.path.isfile(_argv0): + _script_dir = os.path.dirname(_argv0) + else: + _script_dir = os.path.abspath(os.getcwd()) +_current = _script_dir +for _ in range(8): + if os.path.isdir(os.path.join(_current, "edvise")): + if _current not in sys.path: + sys.path.insert(0, _current) + break + _parent = os.path.dirname(_current) + if _parent == _current: + break + _current = _parent + from edvise.ingestion.nsc_sftp import runtime runtime.bootstrap_catalog() diff --git a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py index ab640cbb5..959faa37d 100644 --- a/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py +++ b/src/edvise/ingestion/nsc_sftp/scripts/03_per_institution_bronze_ingest.py @@ -5,9 +5,32 @@ from __future__ import annotations import os +import sys from collections import defaultdict from typing import Optional +# Ensure repo src/ is on sys.path so `import edvise.*` works in Databricks Jobs. +# Layout: /src/edvise/ingestion/nsc_sftp/scripts/ +_here = globals().get("__file__") +if _here: + _script_dir = os.path.dirname(os.path.abspath(_here)) +else: + _argv0 = os.path.abspath(sys.argv[0]) if sys.argv else "" + if _argv0.endswith(".py") and os.path.isfile(_argv0): + _script_dir = os.path.dirname(_argv0) + else: + _script_dir = os.path.abspath(os.getcwd()) +_current = _script_dir +for _ in range(8): + if os.path.isdir(os.path.join(_current, "edvise")): + if _current not in sys.path: + sys.path.insert(0, _current) + break + _parent = os.path.dirname(_current) + if _parent == _current: + break + _current = _parent + import pandas as pd from edvise.ingestion.nsc_sftp import runtime From 50f6401ee51c0d9061c39f281b631115d3a8a98c Mon Sep 17 00:00:00 2001 From: Noreen Mayat Date: Fri, 7 Aug 2026 17:06:31 -0700 Subject: [PATCH 10/10] fix: changing substring match --- .../ingestion/nsc_sftp/file_selection.py | 19 +++-- tests/ingestion/test_file_selection.py | 80 ++++++++----------- 2 files changed, 44 insertions(+), 55 deletions(-) diff --git a/src/edvise/ingestion/nsc_sftp/file_selection.py b/src/edvise/ingestion/nsc_sftp/file_selection.py index e685cb936..4217d717c 100644 --- a/src/edvise/ingestion/nsc_sftp/file_selection.py +++ b/src/edvise/ingestion/nsc_sftp/file_selection.py @@ -1,8 +1,11 @@ """ Select cohort/course SFTP file pairs for NSC PDP ingestion. -Files are expected to end with a shared 14-digit stamp ``_YYYYMMDDHHMMSS`` and -to contain ``cohort`` or ``course`` in the basename (case-insensitive). +Files share a trailing 14-digit stamp ``_YYYYMMDDHHMMSS``. Roles match basename +markers (course checked first — it contains the cohort marker as a suffix): + +- course: ``COURSE_LEVEL_AR_DEIDENTIFIED_STUDYID`` +- cohort: ``AR_DEIDENTIFIED_STUDYID`` """ from __future__ import annotations @@ -13,6 +16,8 @@ from typing import Any, Iterable, Literal, Mapping, Optional, Sequence FILE_STAMP_RE = re.compile(r"_(\d{14})(?:\.[^.]+)?$", re.IGNORECASE) +COURSE_MARKER = "COURSE_LEVEL_AR_DEIDENTIFIED_STUDYID" +COHORT_MARKER = "AR_DEIDENTIFIED_STUDYID" FileSelectionMode = Literal["manual", "latest", "skip_ingested"] @@ -42,13 +47,11 @@ def try_extract_file_stamp(file_name: str) -> Optional[str]: def classify_pdp_file_role(file_name: str) -> Optional[Literal["cohort", "course"]]: - base = os.path.basename(file_name).lower() - has_cohort = "cohort" in base - has_course = "course" in base - if has_cohort and not has_course: - return "cohort" - if has_course and not has_cohort: + base = os.path.basename(file_name).upper() + if COURSE_MARKER in base: return "course" + if COHORT_MARKER in base: + return "cohort" return None diff --git a/tests/ingestion/test_file_selection.py b/tests/ingestion/test_file_selection.py index 84f89026c..081614d58 100644 --- a/tests/ingestion/test_file_selection.py +++ b/tests/ingestion/test_file_selection.py @@ -7,6 +7,15 @@ select_file_pair, ) +COHORT_A = "AO1600pdp_AO1600_AR_DEIDENTIFIED_STUDYID_20240115123045.csv" +COURSE_A = "AO1600pdp_AO1600_COURSE_LEVEL_AR_DEIDENTIFIED_STUDYID_20240115123045.csv" +COHORT_B = "AO1600pdp_AO1600_AR_DEIDENTIFIED_STUDYID_20240201101010.csv" +COURSE_B = "AO1600pdp_AO1600_COURSE_LEVEL_AR_DEIDENTIFIED_STUDYID_20240201101010.csv" +COHORT_C = "AO1600pdp_AO1600_AR_DEIDENTIFIED_STUDYID_20260724030759.csv" +COURSE_C = "AO1600pdp_AO1600_COURSE_LEVEL_AR_DEIDENTIFIED_STUDYID_20260724030759.csv" +COHORT_D = "AO1600pdp_AO1600_AR_DEIDENTIFIED_STUDYID_20260724040738.csv" +COURSE_D = "AO1600pdp_AO1600_COURSE_LEVEL_AR_DEIDENTIFIED_STUDYID_20260724040738.csv" + def _row(name: str, size: int = 10) -> dict: return { @@ -19,85 +28,62 @@ def _row(name: str, size: int = 10) -> dict: def test_extract_file_stamp(): - assert extract_file_stamp("PDP_Cohort_File_20240115123045.csv") == "20240115123045" + assert extract_file_stamp(COHORT_A) == "20240115123045" def test_classify_pdp_file_role(): - assert classify_pdp_file_role("School_Cohort_20240115123045.csv") == "cohort" - assert classify_pdp_file_role("School_Course_20240115123045.csv") == "course" + assert classify_pdp_file_role(COHORT_C) == "cohort" + assert classify_pdp_file_role(COURSE_C) == "course" assert classify_pdp_file_role("readme.txt") is None def test_discover_file_pairs_requires_both_roles(): rows = [ - _row("A_Cohort_20240115123045.csv"), - _row("A_Course_20240115123045.csv"), - _row("B_Cohort_20240201101010.csv"), # incomplete pair + _row(COHORT_A), + _row(COURSE_A), + _row(COHORT_B), # incomplete pair _row("noise_20240301111111.csv"), ] pairs = discover_file_pairs(rows) assert len(pairs) == 1 assert pairs[0].stamp == "20240115123045" - assert pairs[0].cohort_file_name.endswith("Cohort_20240115123045.csv") - assert pairs[0].course_file_name.endswith("Course_20240115123045.csv") + assert pairs[0].cohort_file_name == COHORT_A + assert pairs[0].course_file_name == COURSE_A + + +def test_discover_file_pairs_and_latest(): + rows = [_row(COHORT_C), _row(COURSE_C), _row(COHORT_D), _row(COURSE_D)] + pairs = discover_file_pairs(rows) + assert [p.stamp for p in pairs] == ["20260724030759", "20260724040738"] + c, o, mode = select_file_pair(rows, mode="latest") + assert (c, o, mode) == (COHORT_D, COURSE_D, "latest") def test_select_file_pair_manual(): - cohort = "A_Cohort_20240115123045.csv" - course = "A_Course_20240115123045.csv" c, o, mode = select_file_pair( [], mode="skip_ingested", - cohort_file_name=cohort, - course_file_name=course, + cohort_file_name=COHORT_A, + course_file_name=COURSE_A, ) - assert (c, o, mode) == (cohort, course, "manual") - - -def test_select_file_pair_latest(): - rows = [ - _row("A_Cohort_20240115123045.csv"), - _row("A_Course_20240115123045.csv"), - _row("B_Cohort_20240201101010.csv"), - _row("B_Course_20240201101010.csv"), - ] - c, o, mode = select_file_pair(rows, mode="latest") - assert mode == "latest" - assert c == "B_Cohort_20240201101010.csv" - assert o == "B_Course_20240201101010.csv" + assert (c, o, mode) == (COHORT_A, COURSE_A, "manual") def test_select_file_pair_skip_ingested(): - rows = [ - _row("A_Cohort_20240115123045.csv"), - _row("A_Course_20240115123045.csv"), - _row("B_Cohort_20240201101010.csv"), - _row("B_Course_20240201101010.csv"), - ] + rows = [_row(COHORT_A), _row(COURSE_A), _row(COHORT_B), _row(COURSE_B)] c, o, mode = select_file_pair( rows, mode="skip_ingested", - ingested_file_names={ - "B_Cohort_20240201101010.csv", - "B_Course_20240201101010.csv", - }, + ingested_file_names={COHORT_B, COURSE_B}, ) - assert mode == "skip_ingested" - assert c == "A_Cohort_20240115123045.csv" - assert o == "A_Course_20240115123045.csv" + assert (c, o, mode) == (COHORT_A, COURSE_A, "skip_ingested") def test_select_file_pair_skip_ingested_all_done_raises(): - rows = [ - _row("A_Cohort_20240115123045.csv"), - _row("A_Course_20240115123045.csv"), - ] + rows = [_row(COHORT_A), _row(COURSE_A)] with pytest.raises(FileNotFoundError, match="already BRONZE_WRITTEN"): select_file_pair( rows, mode="skip_ingested", - ingested_file_names={ - "A_Cohort_20240115123045.csv", - "A_Course_20240115123045.csv", - }, + ingested_file_names={COHORT_A, COURSE_A}, )