CRDC CDE alignment, DataCatalog/SCDM integration, and kg-pipeline buildout

.gitattributes

@@ -0,0 +1,3 @@
+* text=auto eol=lf
+
+*.csv text eol=lf

.gitignore

@@ -78,4 +78,5 @@ kg-pipeline/data/mc2_assay/rdf/*
 kg-pipeline/__pycache__/
 kg-pipeline/**/__pycache__/
 kg-pipeline/.venv/
-.pytest_cache/
+.pytest_cache/
+/.vscode

CLAUDE.md

@@ -0,0 +1,64 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this repo is
+
+Data models and controlled vocabularies for the [Cancer Complexity Knowledge Portal](https://cancercomplexity.synapse.org/) (CCKP). The model is maintained as CSV files in domain-specific `modules/`, collated into `mc2.model.csv`, and converted to JSON-LD/JSON Schema via `synapseclient.extensions.curator` (the actively-maintained successor to [schematicpy](https://pypi.org/project/schematicpy/), which Sage Bionetworks has announced will be retired by end of 2026) for use by the [Data Curator App](https://dca.app.sagebionetworks.org/).
+
+## Commands
+
+```bash
+# Install dependencies (Python 3.10+)
+pip install -r requirements.txt
+
+# Full build: update valid values in all modules → collate → generate JSON Schemas
+make all
+
+# Steps individually:
+python update_valid_values.py   # reads modules/mapping.yaml, rewrites annotationProperty.csv Valid Values columns
+make collate                    # concatenates all modules/*/annotationProperty.csv → mc2.model.csv
+make convert                    # python convert_model_to_jsonld.py → mc2.model.jsonld
+make generate-json              # python create_json_from_model.py <data types> → json_schemas/
+
+# Generate JSON schemas for specific data types only
+python create_json_from_model.py Biospecimen Study Dataset
+
+# QC model build (NOTE: qc_convert still calls the old, non-functional
+# `schematic schema convert` against qc_model/mc2_qc.model.csv, a separately
+# hand-formatted copy of the model with schematicpy's older required columns
+# -- unlike `make convert`, this was not migrated; needs a follow-up decision)
+make qc
+
+# Docs dev server
+mkdocs serve   # http://localhost:8000
+```
+
+## Architecture
+
+### The module + collation pattern
+
+Each domain lives in `modules/<domain>/`:
+- `annotationProperty.csv` — attribute definitions for that domain (type, description, valid values, validation rules)
+- One CSV per controlled vocabulary (e.g., `specimenType.csv`, `fixative.csv`) — the actual enumerated terms
+
+`modules/mapping.yaml` is the central registry: it maps each attribute name to the CV CSV that provides its valid values. `update_valid_values.py` reads this file and rewrites the `Valid Values` column in each `annotationProperty.csv`.
+
+`make collate` then concatenates all `annotationProperty.csv` files into `mc2.model.csv` (the header comes from the consortium module, body from all others via `tail -n +2`).
+
+### CSV conventions
+- All columns read/written as `dtype=str` — `TRUE`/`FALSE` must not become `True`/`False`
+- No NaN — use empty strings (`keep_default_na=False`)
+- Index on `Attribute` column when updating via pandas
+
+### PR requirements
+PRs to main must have exactly one semantic label: `major`, `minor`, `patch`, or `non-release`. The `pr-check.yml` workflow enforces this.
+
+### CI workflows (`.github/workflows/`)
+| Workflow | Trigger | What it does |
+|----------|---------|--------------|
+| `build-jsonld.yml` | PR to main (module changes) | `make all` to validate collation + schema conversion |
+| `build-docs.yml` | Push to main | Builds MkDocs site → GitHub Pages |
+| `pr-check.yml` | PR events | Validates semantic label |
+| `google-sheet-sync.yml` | Scheduled/manual | Syncs RFC Google Sheets to a CSV branch |
+| `create-release.yml` | Manual trigger | Creates GitHub release with version bump |

Makefile

@@ -1,10 +1,7 @@
 CSV := mc2.model.csv
-QC := ./qc_model/mc2_qc.model.csv
 DATA := DataDSP Study FileView PublicationView GrantView ToolView EducationalResource DatasetView DataCatalog Biospecimen Model Individual SequencingLevel1 SequencingLevel2 SequencingLevel3 SequencingRNALevel1 ImagingLevel1 ImagingLevel2 ImagingLevel3Image ImagingLevel3Segments ImagingLevel4 NanoStringGeoMxAuxiliaryFiles NanoStringGeoMxDSPImaging NanoStringGeoMxDSPLevel1 NanoStringGeoMxDSPLevel2 NanoStringGeoMxDSPLevel3 NanoStringGeoMXROISegmentAnnotation 10xVisiumAuxiliaryFiles 10xVisiumRNALevel1 10xVisiumRNALevel2 10xVisiumRNALevel3 10xVisiumRNALevel4
 
-all: collate generate-json
-
-qc: collate qc_convert
+all: collate convert generate-json
 
 collate:
 	@echo "Collating module components..."
@@ -13,10 +10,7 @@ collate:
 	tail -n +2 -q modules/*/annotationProperty.csv >> ${CSV}
 
 convert:
-	schematic schema convert ${CSV}
-
-qc_convert:
-	schematic schema convert ${QC}
+	python convert_model_to_jsonld.py
 
 generate-json:
 	python create_json_from_model.py ${DATA}

README.md

@@ -26,16 +26,14 @@
 
 🔎 **Data Models Explorer**: https://mc2-center.github.io/data-models/
 
-📊 **Data Curator App**: https://dca.app.sagebionetworks.org/
-
 ---
 
 ## Overview
 
 This project contains the released versions of the JSON-LD schemas for the
 [Cancer Complexity Knowledge Portal] (CCKP), and more broadly, MC2 Center.
 You can learn more about the schemas/data models and other aspects of this
-project in our portal documentation - coming soon! The MC2 Center data model
+project in our Data Models Explorer. The MC2 Center data model
 is in both CSV and JSON-LD format, and individual entity schemas are also
 exported as standalone JSON Schemas in `./json_schemas`.
 
@@ -46,21 +44,47 @@ assay-level metadata for imaging (multiplexed/single-channel imaging),
 NanoString GeoMx Digital Spatial Profiler (DSP) spatial transcriptomics,
 bulk/single-cell sequencing, and 10x Genomics Visium spatial transcriptomics.
 
+## Usage
+
+Requires Python 3.10+.
+
+```bash
+pip install -r requirements.txt
+
+# Full build: update valid values in all modules -> collate -> generate JSON Schemas
+make all
+```
+
+Steps individually:
+
+```bash
+python update_valid_values.py   # reads modules/mapping.yaml, rewrites annotationProperty.csv Valid Values columns
+make collate                    # concatenates all modules/*/annotationProperty.csv -> mc2.model.csv
+make convert                    # converts mc2.model.csv -> mc2.model.jsonld
+make generate-json              # python create_json_from_model.py <data types> -> json_schemas/
+
+# Generate JSON schemas for specific data types only
+python create_json_from_model.py Biospecimen Study Dataset
+```
+
+To build and preview the docs site locally:
+
+```bash
+mkdocs serve   # http://localhost:8000
+```
+
+See [contributing guidelines] for the full development and release process.
+
 ## Folder Structure
 
 ```
 .
-├── dca_config/
 ├── docs/
 ├── modules/
 ├── scripts/
 └── templates/
 ```
 
-### DCA Configuration
-
-MC2 Center's configurations for the DCA is located in `./dca_config`.
-
 ### Documentation
 
 All docs are located in the `./docs` directory and are written in Markdown