CRDC CDE alignment, DataCatalog/SCDM integration, and kg-pipeline buildout - #262
Draft
Bankso wants to merge 227 commits into
Draft
CRDC CDE alignment, DataCatalog/SCDM integration, and kg-pipeline buildout#262Bankso wants to merge 227 commits into
Bankso wants to merge 227 commits into
Conversation
It is no longer necessary to modify the columnType in this way
Avoid needing to collect study information in file templates
Valid values included for conditional info
… log Notes the new class-level mappings in the schema/ directory-layout table, and records the full CURIE-vs-full-IRI generator quirk plus the pre-existing (unrelated) linkml generate owl blank-node ordering nondeterminism found while regenerating, so a future cckp_portal.ttl diff isn't mistaken for unexpected schema drift. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ederation promotion plan Two initiatives from reviewing the SageBrain RDF Knowledge Graph Construction draft doc's ontology-alignment recommendations against this repo's kg-pipeline. cckp_schema_class_alignment.md is implemented (see its Implementation Report); mondo_uberon_federation_promotion.md is planned, not yet implemented. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ewed flag Adds a `reviewed` column (default "false") to the 4 committed NCIT->MONDO/ UBERON crosswalk files, mirroring crosswalk_scdm.py's exact convention for consortium_to_scdm_program.tsv. load_existing_reviewed() preserves a human's prior true flips across a re-run, keyed on the stable source CURIE rather than row order, so re-running `make crosswalk-ontology` after a CV edit can't silently reset review work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New stage (make link-ontology-crosswalk) reads the harmonized CCKP CSVs (same pattern link_scdm.py already uses, not the built cckp_kg.ttl) and mints cckp:tumorTypeMondoTerm/tissueUberonTerm edges for every reviewed crosswalk row - additive alongside the existing NCIT-anchored tumorTypeTerm/tissueTerm edges build_triples.py emits, never replacing them. Output stays its own file, data/rdf/ontology_crosswalk_links.ttl, folded into `make full-kg` alongside scdm_links.ttl. diseaseType/diseaseStatus crosswalks are deliberately not wired up here - those CVs back MC2 individual/biospecimen-level attributes with no corresponding field on any of the 5 CCKP portal classes, so there's no existing Term edge anywhere to attach a MONDO edge to yet. Surfaced via this script's own "Not yet consumable" run output, not silently dropped. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the new stage to the Running/Consuming-the-graph/Additional-pipeline- stages/Directory-layout sections, and records the scope-narrowing (only tumorType/tissue, not diseaseType/diseaseStatus) and the CL-prefix crosswalk row gap found during implementation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ion.md Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Remove references to the Data Curator App and the DCA configuration from the README. Update wording to point users to the Data Models Explorer instead of the portal documentation, and remove the dca_config entry and the 'DCA Configuration' section from the project tree to clean up outdated links and notes.
…um_id.csv modules/consortium/consortium_name.csv (bare-acronym CV, Attribute=CCBIR) was retired in 6552e4d ("Retire redundant FK-shadowing attributes + naming fixes") in favor of Consortium_id/Consortium Key, but crosswalk_scdm.py's default --consortium-cv path and Makefile invocation were never updated - `make crosswalk-scdm` has crashed with FileNotFoundError ever since. Fixed to read modules/consortium/consortium_id.csv directly: Attribute is already the program.<slug> id (Consortium_id is primary_key), and the bare acronym real CCKP `consortium` values store lives in that CV's own Notes column (its NCIt Code column is blank for every row - verified against the live file, not assumed from the column name). Also added load_existing_program_rows()/existing_rows merge-preserve logic (same pattern as crosswalk_ontology.py's reviewed-column preservation added earlier this session) - the committed consortium_to_scdm_program.tsv turned out to already carry real human curation (11/11 rows reviewed=true with real descriptions), which the original unconditional overwrite would have silently destroyed on any future re-run. Verified end-to-end: `make crosswalk-scdm` now regenerates the file byte-identical to its curated committed state, and `make link-scdm` resolves 6673 consortiumRef edges against real harmonized data (previously untestable since the crosswalk could never be regenerated to confirm it still matched consortium_id.csv). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
modules/biospecimen/biospecimenCategory.csv existed but was never wired into modules/mapping.yaml, so Biospecimen Type Category's Valid Values stayed a hardcoded list. Fetched CDE 12445832 (Specimen Material Category) directly from caDSR to get ground truth before wiring it up: confirmed Analyte is not one of its 19 real permissible values (the CV's own Analyte row cited the wrong CDS field as its source) and that the "Biospecimen Analyte Type" sibling attribute referenced in the old description doesn't exist anywhere in this model. Per user direction: added Not Reported (NCIT:C43234) and Unknown (NCIT:C17998) to the CV - both already this model's standing convention for these two sentinels - removed the Analyte row, and registered the CV in mapping.yaml. The other CDE-real values not yet in the CV (Body Fluid or Substance, Xenograft, Skin, Saliva, RNA, DNA, Central Nervous System, Cell Line) are deliberately left for separate handling, not lost - see plans/biospecimen_type_category_cde_fix.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ion/Notes
Standing directive: model content (Description/Notes/etc.) must stay
user-facing - help someone understand, navigate, and consume the model.
Process narration, curation history, and internal-review commentary
belong in plans/reports instead.
- Cleared 623 pure "Added by <name> <date>"-style curator-attribution
stamps from the Notes column across 9 CV files (tool_data/tool_operation/
tool_topic/tool_format/tool_language/tool_license/assay/tumorType/tissue)
- zero navigational value; git blame is the right tool for that, not a
per-row model field.
- 13 assay.csv rows individually reviewed: pure "added while aligning
modules/dataCatalog... (2026-09-01)" narration cleared (already
documented in plans/datacatalog_kg_integration.md); a handful shortened
to just their genuinely useful disambiguation fragment.
- 7 dataCatalog_data_type.csv rows cleared (crosswalk-decision narration,
already documented in the same plan).
- biospecimen/pathology.csv: 9 rows repeating an identical, version-tied
"Not in use as of MC2 data model v8.x.x" Notes cleared; the underlying
fact restated once, version-independently, on the attribute's own
Description in annotationProperty.csv.
- Biospecimen Acquisition Method and Individual Year of Birth Descriptions
shortened - the follow-up/PHI-consideration context they carried is
already fully tracked in plans/crdc_cde_integration.md.
Full findings, rationale, and what was deliberately left alone documented
in plans/model_description_cleanup.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ription cleanup mc2.model.csv, all_valid_values.csv (make collate), json_schemas/Biospecimen, FileView, Individual.json (create_json_from_model.py - the only 3 data types whose schema actually references a changed attribute, confirmed via a full make generate-json run), mc2.model.jsonld (convert_model_to_jsonld.py), and kg-pipeline's own schema/mc2_model.linkml.yaml + mc2_model.ttl + mc2_model_prefixes_report.md (make mc2-model-linkml + make schema, per kg-pipeline/README.md's documented propagation step for upstream modules/ changes). All regenerated via documented make targets, no manual edits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New standing rule: a CDE/CRDC_CDE-mapped attribute's CV must fully match the CDE's real permissible-value list - no partial coverage. The prior commit left 8 of 19 real values unadded; this closes that gap. Added Cell Line (NCIT:C16403) and Central Nervous System (NCIT:C12438) via fresh OLS lookups (exact label matches); reused already-curated terms for Xenograft (NCIT:C156443, from specimenComp.csv/pathology.csv), Skin (NCIT:C12470, from tissue.csv), and Saliva/RNA/DNA (from specimenType.csv's existing curation of the same terms at the more granular Biospecimen Type level) - same concepts, consistent codes, not re-derived from scratch. Also caught and fixed a wrong-sense code on an existing row: Fluids (NCIT:C204466, "Body Fluid Specimen" - the specimen-instance sense) did not match the CDE's real permissible value "Body Fluid or Substance" (NCIT:C13236, the material-category sense) - only visible by reading the OLS definition, not the label. Renamed the row to the CDE's exact literal value with the corrected code. Verified 19/19 exact parity programmatically. See plans/biospecimen_type_category_cde_fix.md's Follow-up/Addendum sections. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran three parallel audit forks across all 115 CDE/CRDC_CDE-referencing attribute rows (18 files), fetching real caDSR permissible-value lists and diffing against actual current model content, per the standing full-parity rule. Most CRDC_CDE-tagged fields were already exact matches. Of the real findings, this commit closes the 4 that were genuine, safe, additive gaps: - Treatment Type (CRDC_CDE:14737565): added 7 missing values with fresh NCIT lookups (38/38: 37 real + legitimate extra "Control"). - NGS Sequencing Platform (CRDC_CDE:6352164): added 10 missing instruments/ arrays (77/77: 76 real + legitimate extra "Complete Genomics"); 2 values have no defensible ontology match (documented, matching this file's own pre-existing precedent for such rows). - File Data Checksum Type (CRDC_CDE:11475057): corrected SHA-1/SHA-256/ SHA-512 (wrong casing, one non-existent value) to the CDE's real literal values md5sum/sha1/sha256. - Biospecimen Stain (CDE:8120269): was completely unenumerated despite being Required=True. New CV (biospecimen/stainType.csv, 6 rows, real NCIT codes) registered in mapping.yaml. 5 structural-mismatch findings (where literal alignment would mean reassigning to a different CDE or deleting real, currently-valid values) are deliberately NOT fixed here - flagged for explicit user direction, see plans/cde_alignment_audit.md. 3 more categories need careful per-value reconciliation (File Format, NGS Library Strategy, License) or a CDE-fetch retry (3 Study-module ids) - also documented there, not resolved. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers both the Biospecimen Type Category 19/19 parity fix and the 4 CDE-alignment gaps (Treatment Type, NGS Sequencing Platform, File Data Checksum Type, Biospecimen Stain). mc2.model.csv/all_valid_values.csv (make collate), 10 affected json_schemas/*.json (full make generate-json run), mc2.model.jsonld (convert_model_to_jsonld.py), and kg-pipeline's own schema/mc2_model.linkml.yaml/mc2_model.ttl (make mc2-model-linkml && make schema) - all regenerated via documented make targets. kg-pipeline's 97-test suite passes unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… 12373576 parity Assay's 391-value CV never matched CDE 12373576's 9-value permissible set; carve out a dedicated File Assay Category attribute (new CV shared/fileAssayCategory.csv, 9/9 exact) and propagate it into DependsOn for all 19 file-level modules, matching how File Format is already handled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both attributes carried the same CDE:7789196 tag but only one CV could be correct. GeoMx DSP Assay Type's 4-value CV (RNA/Protein nCounter/NGS) is not this CDE's vocabulary, so the tag is dropped there. Image Assay Type is the real match: retagged and its CV rewritten to the CDE's 23 exact values (renames H&E->H and E, MxIF->mIF; drops SABER; adds 10 missing values such as ELISA, CyTOF, ATAC-Seq - NULISA has no defensible ontology code, documented in the plan). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CDE 8037927 does not correspond to this attribute's CV; no replacement CDE was found via cdeMatch search, so the tag is dropped and the existing CV (Paraffin wax, Carbowax, etc.) is left untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rity (14/14) CV rewritten from 19 loosely-related terms to the CDE's exact 14 values (9 reused existing codes after renaming, 5 fresh OLS lookups). Biospecimen Preservation Medium reconfirmed already exact (17/17), left untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…y (11/11) Dropped 13 values with no home in the real CDE (approved by user); 2 case-only fixes (Not applicable/Not reported -> Not Applicable/Not Reported). All retained codes reused from the pre-existing file, no new ontology lookups needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…6 parity File Format now backed by new modules/shared/fileFormat.csv (97 values, 97/97 exact parity: 50 reused from the old shared CV, 47 fresh OLS lookups). Dataset File Formats' overlap with the real CDE measured at only 48% - not a good-faith fit - so its legacy CDE/CRDC_CDE tag is dropped instead and its 81-value CV left as-is. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Added mRNA-Seq, Bulk RNA-Seq, scRNA-Seq, scDNA-Seq via fresh OLS lookups. Kept the pre-existing RNA-Seq/DNA-Seq entries as legitimate extras rather than removing them, since they're still valid, commonly-used terms not in the CDE's list. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dropped 6 inclusive/demographic values not in the CDE's permissible set (Decline to answer, Don't know, Intersex, None of these describe me, Prefer not to answer, X), per explicit user approval. All 3 retained codes reused, no new lookups. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Study_id's CDE:12960571 had no matching replacement (cdeMatch search on 4 name variants returned zero candidates - the API matches on permissible values, and Study_id is a free-form primary key with none); tag dropped, primary_key kept. DSP Data Use Codes' CDE:0002001 is likewise unresolvable; dropped, DUO values left untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fill in concrete verification numbers (json_schemas diff scope, LinkML/ TTL regeneration stats, kg-pipeline test results) now that all 10 sections are implemented and verified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
update_valid_values.py + make collate + make generate-json + convert_model_to_jsonld.py + kg-pipeline's make mc2-model-linkml && make schema, reflecting all 10 sections of the CDE alignment plan (File Assay Category split, Image Assay Type, Biospecimen Embedding Medium/Preservation Method/Composition, File Format split, NGS Library Strategy, Sex, Study_id/DSP Data Use Codes). kg-pipeline's 97-test suite passes and both .ttl files parse cleanly against the regenerated schema. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ashing The live Dataset table now carries 40 fully-blank placeholder rows (every field empty except an incidental Synapse row version) that build_triples.py's mint_id() correctly refused to synthesize an id for. Skip such rows in build_triples.py's build_class_graph() and in link_scdm.py's three identifier-minting loops, logging a count, rather than hard-failing the whole build. A row missing its identifier but carrying other real data still raises, preserving the existing never-silently-drop guarantee. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…this session's model changes schema/*.ttl regenerated from the CDE-alignment model changes; data_sources.yaml and mappings/sssom/*.tsv reflect today's live extract+harmonize run (Dataset grew 1141->1181 rows, including the 40 blank rows the previous commit's fix now handles; new datacataloglicense.sssom.tsv). Published to Synapse (syn76958235, "portal-ttl-builds") via make publish-portal-kg. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…aset/Tool/EducationalResource models Publication Assay/Tumor Type/Tissue, Dataset Assay/Species/Tumor Type/Tissue, Tool License, and Resource License were renamed to their shared, unprefixed attribute names (Assay, Tumor Type, Tissue, Species, License) in a prior consolidation pass; this mapping file still referenced the old prefixed names. All 4 components' attribute counts already matched their current View's DependsOn list exactly - these 9 renames were the only drift. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch accumulates the model and knowledge-graph pipeline work since main was last updated: full CRDC CDE/CDE alignment across the model, DataCatalog and SCDM (Sage Common Data Model) integration, a new kg-pipeline knowledge-graph build system, and a curator-format migration away from schematicpy.
Model + curator migration
make convertfrom schematicpy to synapseclient's curator extension; drop schematicpy from requirements.txtCRDC CDE / CDE alignment (rounds 1-11 + this session's full-parity pass)
File Assay Categoryout ofAssay/DSP Dataset Assay(CRDC_CDE 12373576), resolved theImage Assay Type/GeoMx DSP Assay TypeCDE 7789196 conflict, brought Biospecimen Preservation Method/Composition/Embedding Medium, File Format, NGS Library Strategy, and Sex to exact or corrected CDE parity, and removed unresolvable legacy CDE tags (Study_id, DSP Data Use Codes)DataCatalog integration
modules/dataCatalogwith real Synapse Dataset entity annotations; extend CVs preferring reuse over new filesSCDM (Sage Common Data Model) alignment
consortium_id.csv(was pointing at a deleted file)kg-pipeline architecture
suggest_mappings.py's registry backend pluggable (add SPDX), prefix-driven rather than path-guessedTest plan
make all(update_valid_values -> collate -> convert -> generate-json) runs clean;mc2.model.csv,mc2.model.jsonld, and alljson_schemas/*.jsonregenerated and spot-checkedmake mc2-model-linkml && make schema && make test- 568 slots / 142 enums, both.ttlfiles parse cleanly, 97/97 tests passfetch_cde_record()set-diff against the live caDSR record