CRDC CDE alignment, DataCatalog/SCDM integration, and kg-pipeline buildout - #262

Draft

Bankso wants to merge 227 commits into

mainfrom

cde-model-revisions

Commits on Apr 2, 2026

Move cv files to shared
Bankso
committed
bc2021d
Update duo from governance model
Bankso
committed
745fe28
Add governance-focused models
Bankso
committed
46218e4
Add updated governance attributes
Bankso
committed
affe79d
Update mapping.yaml
Bankso
committed
9eb952f
Update valid values
Bankso
committed
51255dd
Update all_valid_values.csv
Bankso
committed
fa7e429
Build model CSV
Bankso
committed
3eea5f5
Build JSON-LD
Bankso
committed
8522b0d
Add script to build JSON with curator
Bankso
committed
1cd13bc
Use curator-based JSON build tool
Bankso
committed
13ca6f2
Remove unused schemas
Bankso
committed
229f765
Remove outdated schemas
Bankso
committed
0a30990
Delete cckp_datacatalog.jsonld
Bankso
committed
b8a4128
Add json schemas from curator
Bankso
committed
b0ba90b

Commits on Apr 7, 2026

Rebuild JSON schemas with synapseclient 4.12
Bankso
committed
0334e96

Commits on Apr 24, 2026

Add base NGS Level 1 attributes to RNA Level 1
Bankso
committed
b4ce174
Build schemas from CSV, not JSON-LD
Bankso
committed
617fbfe
Remove unused functions
Bankso
committed
2bd316f
Make model
Bankso
committed
f3858c3

Commits on Apr 25, 2026

Integrate FileView attributes for curator adoption
Bankso
committed
c9f3de3
Update mc2.model.csv
Bankso
committed
dbcea81

Commits on Apr 27, 2026

Use file-specific data use codes attribute
Bankso
committed
ec6da3b
Use File Data Use Codes attribute
Bankso
committed
faaffa3
Update mc2.model.csv
Bankso
committed
a42905d
Create SequencingRNALevel1.json
Bankso
committed
b5a8b6c
Use catalog-specific DUO attribute
Bankso
committed
09cee5b
Use DatasetView-specific DUO attribute
Bankso
committed
8483e25
Update docs link
Bankso
committed
7ca2f30
Use Study specific DUO attribute
Bankso
committed
9857dda
Use catalog-specific DUO attribute
Bankso
committed
06dfe44
Update annotationProperty.csv
Bankso
committed
b41f9e7
Only use DUO valid values for Study
Bankso
committed
8813490
Collate model CSV
Bankso
committed
55653da
Don't use conditional DUO elements in DSP
Bankso
committed
d8da15b
Pass JSON schema types via makefile
Bankso
committed
a03bcda
Pull data types from CLI instead
Bankso
committed
484b71b
Update all_valid_values.csv
Bankso
committed
2d45873
Update annotationProperty.csv
Bankso
committed
ff5eff7
Update mc2.model.csv
Bankso
committed
2de8787
Fix link
Bankso
committed
1ecde30
rebuild model
Bankso
committed
6a947ed
Rebuild JSON schemas
Bankso
committed
8ddb580
Update link
Bankso
committed
e4970ba
Re-separate fileview attributes
Bankso
committed
8073f5a

Commits on Apr 28, 2026

Remove specimen info from file attributes
Bankso
committed
357cc38
Remove specimen info from file schema
Bankso
committed
79fb283
Combine current file attributes with RNA seq assay
Bankso
committed
6df50a7
Collate model
Bankso
committed
d2b3044
Update SequencingRNALevel1.json
Bankso
committed
93825b1
Update FileView.json
Bankso
committed
7b46e5e

Commits on Apr 29, 2026

Update DataDSP.json
Bankso
committed
8cc3124

Commits on May 12, 2026

Update primary key descriptions
Bankso
committed
33987ad

Commits on May 13, 2026

Update to current curator CSV format
Bankso
committed
e2c5908
Update to current curator CSV format
Bankso
committed
d5dee2c
Update to current curator CSV format
Bankso
committed
8253958
Update to current curator CSV format
Bankso
committed
f2d531f
Update to current curator CSV format
Bankso
committed
9d93ecb
Update to current curator CSV format
Bankso
committed
f86b0c7
Update to current curator CSV format
Bankso
committed
dcf70b1
Pattern rule is uri, not url
Bankso
committed
c7b9485

Commits on May 28, 2026

Reformatting for curator model compliance
Bankso
committed
eb10f71

Commits on Jun 10, 2026

Reformatting for curator model compliance (part 2)
Bankso
committed
6b8820e
Update collection attributes
Bankso
committed
7476142

Commits on Jun 11, 2026

Add primary_key property label and regex patterns
Bankso
committed
dfd813e
Add common file attributes to file-based models
Bankso
committed
e59f143
Collate CSVs
Bankso
committed
bb04519
Fix copy/paste error
Bankso
committed
1cf5460
Remove old schema
Bankso
committed
378c28c
Add update curator JSON schemas
Bankso
committed
2d744bc

Commits on Jun 22, 2026

Update id pattern
Bankso
committed
cc8c7ca
Update id pattern
Bankso
committed
001b0d4
Update id pattern
Bankso
committed
0c91cdd
Update key patterns to align with ids
Bankso
committed
70b3059
Collate model
Bankso
committed
645fb33
Add updated JSON
Bankso
committed
5cd2589

Commits on Jun 24, 2026

Update File Design description
Bankso
committed
1c49d0f

Commits on Jun 25, 2026

Add CLAUDE.md

Bankso
and
claude
committed
c293346

Commits on Jun 27, 2026

Use OLS MCP claude skill to prototype ontology mapping tools
Bankso
committed
2042e02

Commits on Jul 9, 2026

Add results of CDE matching by claude skill
Bankso
committed
984fef4

Commits on Jul 21, 2026

Update annotationProperty CSVs and add review docs
Bankso
committed
8a3af11

Commits on Jul 22, 2026

Remove CC1.0 license

vpchung
authored and
Bankso
committed
33b5f94
Add Apache 2.0 License (#254)

vpchung
authored and
Bankso
committed
5840772
Governance model integration and curator updates (#247)
Bankso
committed
f423f07
Fix duplicate DependsOn key in VisiumRNALevel2 and resync DUO codes

Bankso
and
claude
committed
feecc80
Document previously-undocumented modules and rebuild Full Field Reference

Bankso
and
claude
committed
4ec7a52
Add/fix CDE mappings surfaced by mapping-description-update's CDE review

Bankso
and
claude
committed
d4b467b
Regenerate mc2.model.csv after CDE mapping fixes

Bankso
and
claude
committed
384910b
Move mapping artifacts to worklists folder

Bankso
and
claude
committed
e27a240
Surface key/CDE/ontology mappings in the docs site

Bankso
and
claude
committed
1e01fb7
Backfill descriptions/ontology terms for 15 small controlled-vocab files

Bankso
and
claude
committed
aac37bc
Fill in DUO ontology identifiers/URLs mechanically in shared/duo.csv

Bankso
and
claude
committed
eca8195
Backfill descriptions/ontology terms for sequencing small CV files

Bankso
and
claude
committed
8aa9d16
Backfill descriptions/ontology terms for remaining Wave 1 CV files

Bankso
and
claude
committed
90e83c8

Commits on Jul 23, 2026

Normalize all controlled-vocabulary CSVs to LF line endings

Bankso
and
claude
committed
8c5a746
Backfill descriptions/ontology terms for Wave 2 medium CV files

Bankso
and
claude
committed
5212bd2
Add descriptions and ontology mappings for toolFormat

Bankso
and
claude
committed
dcd8856
Don't list non-preferred terms in enum tables

Bankso
and
claude
committed
3b09b27

Commits on Jul 24, 2026

Add ontology identifiers and clean CSV headers

Bankso
and
claude
committed
f17a42b

Commits on Jul 31, 2026

Add ontology IDs and metadata to shared CSVs

Bankso
and
claude
committed
6fcb053

Commits on Aug 14, 2026

Add CCKP knowledge-graph pipeline

Bankso
and
claude
committed
cecd8da

Commits on Aug 17, 2026

Normalize mapping URIs and update CSVs

Bankso
and
claude
committed
960db95
Fill in ontology mappings for modules/tool/ and lymphStage N-stage codes

Bankso
and
claude
committed
4aa31ab

Commits on Aug 18, 2026

Vendor csv-to-linkml converter for reproducibility

Bankso
and
claude
committed
5625633
Normalize Study deidentification property names
Bankso
committed
f58b0fb
Add suggest-mappings and update-baseline targets

Bankso
and
claude
committed
30dd0a2
Add suggest-mappings workflow and SSSOM files

Bankso
and
claude
committed
0408a05
Add ROR and ontology IDs to CSVs

Bankso
and
claude
committed
970176d
Align kg-pipeline with sagebrain-model conventions (Part A)

Bankso
and
claude
committed
7c2da66
Add MC2 assay-metadata KG, linked into sagebrain-model (Part B)

Bankso
and
claude
committed
e9d8196
Add .gitkeep placeholders for mc2_assay data dirs
Bankso
committed
3923961

Commits on Aug 19, 2026

Add Synapse secure-publish target for both kg-pipeline outputs

Bankso
and
claude
committed
1c5fd6f
Merge branch 'main' into database-model-kg
Bankso
committed
c0dddfb

Commits on Sep 1, 2026

Correct stale malformed_cv_terms.csv claim in kg-pipeline README

Bankso
and
claude
committed
c91d04c
Attach File Tissue/File Tumor Type to every File-attribute class

Bankso
and
claude
committed
c8f64c9
Promote grantNumber to a shared top-level slot in cckp_portal schema

Bankso
and
claude
committed
a904146
Remove redundant Tissue/Tumor Type harmonization in link_sagebrain.py

Bankso
and
claude
committed
7ab7232
Add plans/ for SCDM alignment and kg-pipeline README-fix proposals

Bankso
and
claude
committed
fe877e9
Vendor SageCommonDataModel + add Makefile targets for SCDM interop

Bankso
and
claude
committed
890520c
Add institution/consortium crosswalks to SCDM entities

Bankso
and
claude
committed
e379d01
Link the CCKP graph to SCDM Organization/Program/Person entities

Bankso
and
claude
committed
879c822
Document SCDM interop in kg-pipeline README

Bankso
and
claude
committed
5717fef
Record SCDM alignment steps 1-6 Implementation Report

Bankso
and
claude
committed
e2f7caf
Add Institution_id/Consortium_id + shared Institution Key attribute

Bankso
and
claude
committed
cb2346a
Wire Institution/Consortium/PersonView Key into Grant/Project/Person Views

Bankso
and
claude
committed
6a0edeb
Regenerate mc2.model.csv + kg-pipeline schema for Institution/Consortium Key

Bankso
and
claude
committed
ba90b9f
Record SCDM alignment step 7 Implementation Report

Bankso
and
claude
committed
6e86cac

Commits on Sep 2, 2026

Align modules/dataCatalog with real Synapse Dataset entity annotations

Bankso
and
claude
committed
5734464
Extend existing CVs for dataCatalog fields, preferring reuse over new files

Bankso
and
claude
committed
c651408
Update modules/mapping.yaml for dataCatalog's realigned CVs

Bankso
and
claude
committed
8e2d7f4
Regenerate mc2.model.csv + kg-pipeline schema for dataCatalog changes

Bankso
and
claude
committed
b3b68fa
Add Data Catalog pipeline stage: extract, triples, merge + tests

Bankso
and
claude
committed
09058e6
Document the Data Catalog stage in kg-pipeline README

Bankso
and
claude
committed
158f49c
Record Data Catalog integration Implementation Report

Bankso
and
claude
committed
f840848

Commits on Sep 3, 2026

Add CRDC CDE reference data, mapping report, and integration plan

Bankso
and
claude
committed
cd942ec
CRDC CDE integration for Individual/Model/Biospecimen clinical modules

Bankso
and
claude
committed
7fb45a6
CRDC CDE integration + attribute consolidation for remaining modules

Bankso
and
claude
committed
a57a593
Update modules/mapping.yaml for the CRDC integration + consolidation

Bankso
and
claude
committed
a3b0c8b
Regenerate mc2.model.csv + all_valid_values.csv

Bankso
and
claude
committed
f0bcf2b
Retire redundant FK-shadowing attributes + naming fixes

Bankso
and
claude
committed
6552e4d
Consolidate Grant/Project/Study Investigator into shared Investigator

Bankso
and
claude
committed
1755c05
Legacy CDE cleanup, description/columnType fixes

Bankso
and
claude
committed
eaf3ddd
Add Study Number of Samples, resolving the long-standing dangling reference

Bankso
and
claude
committed
58fed0e
Document round 7 in the Implementation Report

Bankso
and
claude
committed
dcb6982
Update modules/mapping.yaml for round 7's attribute changes

Bankso
and
claude
committed
872023a
Regenerate mc2.model.csv + all_valid_values.csv

Bankso
and
claude
committed
8db1ad2
Backfill real Valid Values from live caDSR data

Bankso
and
claude
committed
07918f9
Add live CDE reference data, update mapping report with real CV findings

Bankso
and
claude
committed
8b53811
Document Round 8 in the Implementation Report

Bankso
and
claude
committed
55bacdd
Regenerate mc2.model.csv + all_valid_values.csv

Bankso
and
claude
committed
6c8ea2c
Re-scope Biospecimen Type Category from CDE 11253427 to CDE 12445832

Bankso
and
claude
committed
3f77119
Un-map Tool Entity Role from CDE 2201713

Bankso
and
claude
committed
9780fdb
Reformat Tumor Grade to match CDE 11325685's real serialization

Bankso
and
claude
committed
7eb2172
Update crdc_cde_mapping_report.csv for the three Round 9 decisions

Bankso
and
claude
committed
77ca6e0
Document Round 9 in the Implementation Report

Bankso
and
claude
committed
51e62c0
Regenerate mc2.model.csv + all_valid_values.csv

Bankso
and
claude
committed
dba153e
Remove Tool Entity Role's legacy CDE:2201713 tag entirely

Bankso
and
claude
committed
4110108
Remove redundant Biospecimen Analyte Type; document 2 final decisions

Bankso
and
claude
committed
9d42c41
Fix Biospecimen Taxonomy ID: columnType number -> string

Bankso
and
claude
committed
5b6f54f
Switch make convert to synapseclient's curator extension

Bankso
and
claude
committed
a378f83
Remove schematicpy from requirements.txt; bump synapseclient to 4.13.0

Bankso
and
claude
committed
12fa7f6

Commits on Sep 4, 2026

Regenerate kg-pipeline mc2_model LinkML schema from this session's model changes

Bankso
and
claude
committed
14a36af
Fix kg-pipeline scripts for renamed/consolidated MC2 attributes

Bankso
and
claude
committed
debfa7e
Update kg-pipeline tests for consolidated attributes and the retired Consortium CV

Bankso
and
claude
committed
4b6946e
Refresh license/species SSSOM crosswalks after enum consolidation

Bankso
and
claude
committed
caa4941
Consolidate per-entity SSSOM crosswalks into per-enum files

Bankso
and
claude
committed
e3348ad
Fix stale mc2_enum references in cckp_portal.linkml.yaml, drop the retired Consortium CV

Bankso
and
claude
committed
9f5ff05
Update kg-pipeline coverage baseline after consolidation

Bankso
and
claude
committed
bf7ee2b
Document the kg-pipeline propagation round in plans/crdc_cde_integration.md

Bankso
and
claude
committed
ac514f2
Register a real CRDC_CDE: prefix, resolving the unregistered-prefix warning

Bankso
and
claude
committed
52f0396
Verify and document: Grant.consortium repoint rejected, link_sagebrain.py not redundant

Bankso
and
claude
committed
01b1a3c
Document Round 11: kg-pipeline follow-ups verified and resolved

Bankso
and
claude
committed
a6beddb
Complete the Grant.consortium human review: 11 SCDM Programs sourced

Bankso
and
claude
committed
dc5ad59
Add combined-kg/full-kg targets so DataCatalog/SCDM merges can't go stale

Bankso
and
claude
committed
4901a69
Fix DataCatalog license/dataUseModifiers field-name drift in kg-pipeline

Bankso
and
claude
committed
707fe7e
Surface caDSR CDE mappings in the docs site's field reference tables

Bankso
and
claude
committed
ade1326
Add DataCatalog to the docs site

Bankso
and
claude
committed
25c3e73
Remove QC model
Bankso
committed
f1e03cb

Commits on Sep 8, 2026

Add a Usage section to the root README

Bankso
and
claude
committed
cd901f2
Separate kg-pipeline's README into usage docs and decision rationale

Bankso
and
claude
committed
40dabb9

Commits on Sep 9, 2026

Fix malformed rows in shared/assay.csv breaking strict CSV parsing

Bankso
and
claude
committed
6130f03
Finish Biospecimen Acquisition Method's CRDC CDE alignment; remove orphaned legacy CVs

Bankso
and
claude
committed
1d07453
Backfill CV descriptions from existing ontology mappings (OLS + SPDX)

Bankso
and
claude
committed
063338a
Synthesize descriptions for CV terms with no real ontology equivalent

Bankso
and
claude
committed
62e074a
Docs: render "No description provided" instead of nan; link CDE tags

Bankso
and
claude
committed
6577de7
Add cv_quality_pass plan + implementation report; regenerate model

Bankso
and
claude
committed
cefd07f
File module: free up Longitudinal timepoint fields, add Batch Identifier

Bankso
and
claude
committed
2dbf9a7
Regenerate mc2.model.csv/jsonld + JSON schemas for File module changes

Bankso
and
claude
committed
019798d
Move File Batch Identifier's ontology ref to Properties; propagate to all file-based templates

Bankso
and
claude
committed
e70e569
Regenerate model + JSON schemas for File Batch Identifier propagation

Bankso
and
claude
committed
3ef833e
kg-pipeline: register the MIxS prefix for File Batch Identifier

Bankso
and
claude
committed
0b1a2b5
Propagate model changes to kg-pipeline schema artifacts

Bankso
and
claude
committed
4f948a6
harmonize.py/build_triples.py: split comma-crammed scalar CV values

Bankso
and
claude
committed
aae3cba
Update kg-pipeline provenance/coverage baseline for live-data re-run

Bankso
and
claude
committed
2c43af4
suggest_mappings.py: pluggable registry backends (add SPDX)

Bankso
and
claude
committed
e203812
suggest_mappings.py: make registry selection prefix-driven, not path-guessed

Bankso
and
claude
committed
729beca
suggest_mappings.py: skip confirmed-unmappable values, fix short-code queries

Bankso
and
claude
committed
fe7433f

Commits on Sep 10, 2026

cckp_portal.linkml.yaml: align 5 CCKP classes to schema.org classes

Bankso
and
claude
committed
fa9d63c
Document schema.org class alignment in kg-pipeline README + decisions log

Bankso
and
claude
committed
fda0bd7
plans: add schema class alignment plan (implemented) + MONDO/UBERON federation promotion plan

Bankso
and
claude
committed
2449408
crosswalk_ontology.py: gate MONDO/UBERON crosswalk rows behind a reviewed flag

Bankso
and
claude
committed
5a2eb91
Add link_ontology_crosswalk.py: promote reviewed MONDO/UBERON edges

Bankso
and
claude
committed
4e1d8a5
Document MONDO/UBERON crosswalk promotion in README + decisions log

Bankso
and
claude
committed
1038223
plans: append Implementation Report to mondo_uberon_federation_promotion.md

Bankso
and
claude
committed
d3fec49
Update .gitignore
Bankso
committed
9fd071e
Update README to remove DCA config and link
Bankso
committed
ec48c14
crosswalk_scdm.py: fix broken consortium crosswalk, point at consortium_id.csv

Bankso
and
claude
committed
191c8f1
Register Biospecimen Type Category CV, fix CDE-drift Analyte row

Bankso
and
claude
committed
5585deb
Model-wide cleanup: remove process/historical narration from Description/Notes

Bankso
and
claude
committed
9da0482
Regenerate derived artifacts for Biospecimen Type Category fix + description cleanup

Bankso
and
claude
committed
30ec13c
Biospecimen Type Category: achieve full CDE 12445832 parity (19/19)

Bankso
and
claude
committed
ceb1093
Model-wide CDE alignment audit: fix 4 confirmed missing-value gaps

Bankso
and
claude
committed
9a83080
Regenerate derived artifacts for full-parity + CDE-alignment fixes

Bankso
and
claude
committed
53376ec

Commits on Sep 11, 2026

Split File Assay Category out of Assay/DSP Dataset Assay for CRDC_CDE 12373576 parity

Bankso
and
claude
committed
e187a64
Resolve Image Assay Type / GeoMx DSP Assay Type CDE 7789196 conflict

Bankso
and
claude
committed
03d6437
Remove wrong CDE:8037927 tag from Biospecimen Embedding Medium

Bankso
and
claude
committed
6dfef5a
Biospecimen Preservation Method: achieve full CDE/CRDC_CDE 8028962 parity (14/14)

Bankso
and
claude
committed
2597356
Biospecimen Composition: trim CV to exact CDE/CRDC_CDE 12922545 parity (11/11)

Bankso
and
claude
committed
a6c5bf0
Split File Format / Dataset File Formats CVs for CDE/CRDC_CDE 11416926 parity

Bankso
and
claude
committed
26a5205
NGS Library Strategy: add 4 missing CDE/CRDC_CDE 6273393 values

Bankso
and
claude
committed
73e102d
Sex: trim CV to exact CDE/CRDC_CDE 7572817 parity (3/3, approved)

Bankso
and
claude
committed
325e449
Remove unresolvable CDE tags from Study_id and DSP Data Use Codes

Bankso
and
claude
committed
7baf373
plans/cde_alignment_audit.md: finalize Implementation Report

Bankso
and
claude
committed
ac37599
Regenerate derived artifacts for full CDE/CRDC_CDE alignment pass

Bankso
and
claude
committed
81ca56c
kg-pipeline: skip rows with a blank declared identifier instead of crashing

Bankso
and
claude
committed
b9a1c39
kg-pipeline: regenerate schema + re-extract/harmonize provenance for this session's model changes

Bankso
and
claude
committed
d487af8
qc_model/qc_attribute_mapping.csv: align with current Publication/Dataset/Tool/EducationalResource models

Bankso
and
claude
committed
f7d63ec