An ADAPT curation pipeline from public downloadable sources to a PharmacoSet RDS. The source-tag-corrected Anchor PSet, scientific QC and exact source-to-object checks completed; the original artifact is preserved; validation status distinguishes complete checks from pending work. Processing decisions explain the reference-pipeline correspondence and source boundaries.
pixi install --locked
pixi run setup
pixi run pipeline
pixi run qc
pixi run zipupReview config/pipeline.yaml first. packages records fixed GitHub revisions; sources records download URL, ID and an expected SHA-256 for molecular inputs. Those hashes reject silent release changes, including changes behind CMP metadata URLs; a changed upstream file requires an intentional source update. Setup installs into the dedicated Pixi R library and records resolved revisions. It refuses a global R library. Contact: michaelcao-anh.tran@uhn.ca.
workflow/Snakefile orchestrates source downloads → safe extraction → response/molecular processing → annotation → fitting and synergy → PharmacoSet construction and TSV exports. pixi run qc separately renders the report. Fit checkpoints allow restart. Default fitting uses four workers with one concurrent pipeline job; reduce workers if memory is constrained.
Full end-to-end runtime has not yet been measured. Downloads and curve fitting dominate; reserve space for compressed inputs, extracted tables, fitting intermediates and final outputs.
Public original and validation raw wells are normalized by scan as (intensity - mean B positive control) / (mean NC-1 negative control - mean B positive control), clipped to [0,1]. Repeated tag records become one physical well. Anchor-only controls provide observed anchor viability. Mixed-treatment component doses remain in the response keys; curve fitting uses the first component dose as the reference script does. Day-1 and fitted/portal tables are comparison inputs, not substitutes for raw wells. A 600-record peer-normalized sample had 596 public matches, agreeing within 5.6e-16; the four unmatched records relate to known public-release coverage differences.
Expression uses the original public CMP rnaseq_all_20220624.zip processed FPKM, TPM and read-count matrices. The workflow downloads and imports this archive directly. Microarray and CEL processing are excluded; no local processed microarray file or archived PSet is required.
Treatment IDs use readable source names. Identical names share one treatment entry, while treatmentid_dataset and the response fields treatment1id_source/treatment2id_source retain every original identity. Grouping names does not aggregate measurements or refit curves.
Full response import checked: 3,634,666 drugged physical wells; 126 source sample IDs; 68 role/mix treatment IDs. Source counts are not annotation match counts. Original inspection records and links remain in data sources; config/pipeline.yaml is the executable source list. SHA-256 receipts bind a run to actual downloaded bytes, including mutable URLs.
data/results/GDSC_ANCHOR_PSet.rds: validated serialized PharmacoSet.data/results/tables/*.tsv: sample, treatment, sensitivity, curation, fit, combination and molecular-summary tables.data/procdata/metadata/: package revisions, session info, source receipts, annotation misses, feature exclusions and object validation.docs/pset_qc.html: rendered QC after a successful pipeline run.
The canonical PSet uses saveRDS(). Data and generated reports are ignored by Git. zipup creates a maximum-gzip TSV archive.
The single full-data QC report explores response distributions, fit support, representative curves, molecular coverage and small dataset-appropriate analyses. Published comparisons use matched metrics, contexts and units, with missingness, Pearson/Spearman correlations, agreement/bias and outlier diagnostics. See QC standard, implementation, validation, and source history. Lightweight integration fixtures remain internal software tests.
Maintain Conda dependencies with pixi add and retain pixi.lock. R dependencies installed from GitHub/Bioconductor are recorded separately. Keep processing in workflow/scripts/, QC in workflow/qc/, and generated files under data/{rawdata,procdata,results}. Never replace public input provenance with an opaque peer PSet.
Run pixi install --locked, pixi run setup, then pixi run pipeline. pixi.lock fixes the system/R environment; renv.lock fixes the tested R dependency versions, Bioconductor release and CoreGx/PharmacoGx commits. Setup modifies only this project’s Pixi library. Input data, API responses, fitted intermediates and final RDS/HTML files are generated and ignored by Git. No existing PSet is required.
Original scientific downloads are checksum-pinned in config/pipeline.yaml; a changed remote file stops the build. AnnotationDB defaults to mode: live, reusing local responses when available. Set refresh: true for an explicit live refresh. API annotations can change independently of the measured data. mode: replay with refresh: false requires every response to match config/annotationdb-checksums.json; it never silently accepts changed bytes. Those checksums describe the reviewed cache, not a publicly archived API snapshot. A fresh replay can fail if the service changes, in which case the original response cache is needed; a live build remains possible. Generated response hashes are recorded in data/procdata/metadata/annotationdb_responses.tsv.
Verification covered a clean macOS ARM64 installation of the identical shared dependency lock, each dataset’s existing setup checks, all four isolated workflow graphs, access to every configured source URL, and selected fresh download/extraction stages. The fresh HMCL response stage reproduced all 904,376 observations and its sample, treatment and published-profile tables exactly. Full fitting and end-to-end regeneration of all four PSets were not repeated; current final objects retain the previously verified fitted measurements.