Icca data source

.claude/skills/generate-database-options/SKILL.md

@@ -85,6 +85,7 @@ Incorporate answers into the dict before writing.
 ```python
 import json
 from pathlib import Path
+
 Path(output_path).write_text(json.dumps(db_options, indent=4, ensure_ascii=False), encoding="utf-8")
 ```
 

.claude/skills/new-datasource/SKILL.md

@@ -5,7 +5,7 @@ description: Add a brand-new data source module to the ClinicalScope project. Us
 
 # New Datasource Skill
 
-Add a complete, production-ready datasource module by **mirroring the closest existing source**. The 11 existing datasources already encode every pattern this skill needs to cover — the skill's job is to route correctly and remind you of the cross-cutting concerns (registration, tests, snapshots, docs).
+Add a complete, production-ready datasource module by **mirroring the closest existing source**. The existing datasources already encode every pattern this skill needs to cover — the skill's job is to route correctly and remind you of the cross-cutting concerns (registration, tests, snapshots, docs).
 
 ## Step 0 — Gather materials and identity
 
@@ -57,7 +57,7 @@ Internal — **do not show this table to the user**. Use it to choose which exis
 
 | Raw format / complexity | Primary |
 |---|---|
-| Plain CSV/parquet, datetime column present, one signal per column | `philips_waves` |
+| Plain CSV/parquet, datetime column present, one signal per column | **Stop — write no module.** Tell the user to drop the file into `other/` and configure it under an `other::<stem>` key; a datasource is only justified by format-specific parsing. |
 | Long-format, needs pivot to wide | `mindray_respi_numerics` |
 | Custom text parser (header blocks, metadata sections) | `servo_u` |
 | Custom binary/XML/less-structured | `eit` |
@@ -97,9 +97,13 @@ Beyond the three new files in `src/clinical_scope/datasource/sources/<name>/`, e
 - **`tests/datasource/conftest.py`** — add a session-scoped `<datasource_name>_cls` fixture.
 - **`docs/user_guide/tutorial.md`** → *Patient Data & Supported Data Sources* canonical table — add a row.
 - **`CLAUDE.md`** → *Supported Data Sources* bullet list — add a bullet, list order aligned with `AVAILABLE`.
+- **`example/template_patient_data_structure/<EXPECTED_FOLDER_NAME>/.gitkeep`** — the empty scaffold that ships in the release bundle.
+- **`example/demo_database/database_options.xlsx`** — add a section for the new source (a `*` sentinel row plus a curated handful of signals), then **regenerate `database_options.json` from it**; the demo must plot every source it ships.
+
+`tests/unit/test_example_assets.py` guards both of those last two and prints the regeneration one-liner.
 
 Conditional — update only if the file currently enumerates all datasources:
-- `README.md`, `example/option_files/*`.
+- `README.md`.
 
 ## Step 6 — Example data
 
@@ -158,10 +162,12 @@ Once everything is in place, mention the primary for transparency:
 - [ ] `src/clinical_scope/datasource/sources/<name>/find_load_format.py`
 - [ ] `src/clinical_scope/datasource/registry.py` (import, inner class, `AVAILABLE`)
 - [ ] `example/demo_database/demo_patient/<folder>/` — example data
+- [ ] `example/template_patient_data_structure/<folder>/.gitkeep` — guard test asserts this set matches the registry
 - [ ] `tests/datasource/conftest.py` — fixture added
 - [ ] `tests/datasource/test_<name>.py` — copied from primary and adapted
 - [ ] `tests/expected_results/<name>/` — snapshots generated
 - [ ] `docs/user_guide/tutorial.md` — table row added
 - [ ] `CLAUDE.md` — Supported Data Sources bullet updated
-- [ ] `README.md`, `example/option_files/*` — updated only if they enumerate sources
+- [ ] `example/demo_database/database_options.{xlsx,json}` — section added to the xlsx, json regenerated from it
+- [ ] `README.md` — updated only if it enumerates sources
 - [ ] All tests pass, `ruff check` clean, smoke test sees the new datasource

.claude/skills/organize-patient-folder/SKILL.md

@@ -10,7 +10,7 @@ Reorganize files in a user-provided folder into the correct `ClinicalScope` pati
 The expected structure is:
 ```
 Patient01/
-├── philips_waves/          ← one subfolder per active datasource
+├── eit/                    ← one subfolder per active datasource
 ├── fluxmed_signals/
 ├── servo_u/
 ├── other/                  ← catch-all for unclassifiable files
@@ -100,12 +100,12 @@ Collect all files marked `confidence: "ask"` and present them together (not one-
 Ambiguous files — please assign each one:
 
   1. raw_data_20240101.csv  (200 KB)
-     Candidates: philips_waves · philips_numerics · fluxmed_signals · other · skip
+     Candidates: mindray_respi_numerics · fluxmed_signals · other · skip
 
   2. unrecognised_format.bin  (4 KB)
      Candidates: other · skip
 
-Enter: 1=philips_waves, 2=other  (or use numbers from the list above)
+Enter: 1=other, 2=other  (or use numbers from the list above)
 ```
 
 Use `AskUserQuestion` for this — present all ambiguous files in a single question when feasible.
@@ -119,8 +119,8 @@ Before touching anything, print a full summary grouped by target subfolder:
 Organization plan for: /path/to/Patient01
 Operation: COPY
 
-  philips_waves/
-    ├── patient_data_wave.parquet      [auto: unique extension match]
+  other/
+    ├── patient_data_wave.parquet      [auto: generic tabular export]
   servo_u/
     ├── recording.sta                  [auto: unique extension match]
   fluxmed_signals/
@@ -170,17 +170,14 @@ The classification logic in Steps 2–4 must always be driven by the live output
 
 | Datasource | Unique extension? | Reliable file keywords (≥5 chars) | Known ambiguities |
 |---|---|---|---|
-| `philips_waves` | — | `waveform`, `timeseries`, `philips` | Generic names like `data_waves_*.parquet` → ask user |
-| `philips_numerics` | — | `philips_numeric`, `philips` | `*numerics.parquet` ties with mindray_respi_numerics → ask user |
 | `eit` | `.asc` ✓ | — | Unique extension — always auto |
 | `fluxmed_signals` | `.txt` (shared) | `signals`, `signal`, `fluxmed` | `Parameters.txt` disambiguates from fluxmed_signals |
 | `fluxmed_parameters` | `.txt` (shared) | `parameters` | `Signals.txt` disambiguates from fluxmed_parameters |
 | `servo_u` | `.sta` ✓ | — | Unique extension — always auto |
 | `mindray_scope` | `.xml` ✓ for xml; `.csv` ambiguous | — | `.csv` mindray files (e.g., `Art-*.csv`) → always ask user |
 | `mindray_respi_waves` | — | `respi_wave`, `resp_wave`, `mndry_wave`, `mndry` | `mndry_waveform*.parquet` → auto |
 | `mindray_respi_numerics` | — | `respi_numeric`, `resp_numeric`, `mndry_numeric`, `mndry` | `mndry_numerics*.parquet` → auto |
-| `syringe` | — | `syringe`, `seringues` | Reliable if filename contains "syringe" |
-| `other` | catch-all | — | — |
+| `other` | catch-all | — | Plain CSV/parquet with a datetime column belongs here — monitor exports, syringe logs, anything tabular |
 
 ## Edge cases
 

.claude/skills/organize-patient-folder/evals/evals.json

@@ -3,12 +3,12 @@
   "evals": [
     {
       "id": 1,
-      "prompt": "I have a patient data folder at example/example_patients/Patient_full/ — all the datasource subfolders are there. Can you do a dry-run audit and tell me which folders are correctly named for ClinicalScope and which are not? Don't move anything.",
-      "expected_output": "Correctly identifies standard folders (eit, philips_waves, philips_numerics, fluxmed_signals, fluxmed_parameters, servo_u, mindray_scope, syringe) as already in place, and flags Mindray_resp_waves and Mindray_resp_numerics as non-standard (wrong case/spelling vs expected mindray_respi_waves and mindray_respi_numerics). No files moved.",
+      "prompt": "I have a patient data folder at example/demo_database/demo_patient/ — all the datasource subfolders are there. Can you do a dry-run audit and tell me which folders are correctly named for ClinicalScope and which are not? Don't move anything.",
+      "expected_output": "Correctly identifies standard folders (eit, edf, fluxmed_signals, fluxmed_parameters, servo_u, mindray_scope, other) as already in place, and flags Mindray_resp_waves and Mindray_resp_numerics as non-standard (wrong case/spelling vs expected mindray_respi_waves and mindray_respi_numerics). No files moved.",
       "files": [],
       "assertions": [
-        "Unique-extension files classified correctly: .asc→eit/, .sta→servo_u/",
-        "Keyword files: Signals.txt→fluxmed_signals/, Parameters.txt→fluxmed_parameters/, mndry_waveform*.parquet→mindray_respi_waves/, mndry_numerics*.parquet→mindray_respi_numerics/, syringe*.csv→syringe/",
+        "Unique-extension files classified correctly: .asc→eit/, .sta→servo_u/, .edf→edf/",
+        "Keyword files: Signals.txt→fluxmed_signals/, Parameters.txt→fluxmed_parameters/, mndry_waveform*.parquet→mindray_respi_waves/, mndry_numerics*.parquet→mindray_respi_numerics/, syringe*.csv→other/",
         "Flags Mindray_resp_waves and Mindray_resp_numerics as non-standard folder names",
         "Standard folders reported as already in place",
         "No files moved or copied (dry run)",
@@ -17,26 +17,28 @@
     },
     {
       "id": 2,
-      "prompt": "I have a folder at example/example_patients/ with multiple patients. Check if all patient subfolders are correctly organized for ClinicalScope. Dry run only — no moves.",
-      "expected_output": "Detects batch mode across Patient_full, Patient_difficult_format, etc. For each patient, identifies correctly-named vs non-standard datasource subfolders. Mindray_resp_* flagged as non-standard in patients where they appear.",
+      "prompt": "I have a folder at example/demo_database/ holding patient subfolders. Check if every patient subfolder is correctly organized for ClinicalScope. Dry run only — no moves.",
+      "expected_output": "Detects batch mode and reports on each patient subfolder it finds under demo_database/ (demo_patient). For each, identifies correctly-named vs non-standard datasource subfolders, flagging Mindray_resp_*. Ignores loose files such as database_options.xlsx rather than treating them as patients.",
       "files": [],
       "assertions": [
-        "Batch mode detected: reports on multiple patient subfolders",
+        "Batch mode detected: iterates the patient subfolders found under the given root",
+        "Loose files at the root (database_options.xlsx) not mistaken for patient folders",
         "Flags Mindray_resp_numerics and Mindray_resp_waves as non-standard (expected: mindray_respi_*)",
-        "Standard folders (eit, philips_waves, fluxmed_*, servo_u, syringe, mindray_scope) reported as already correct",
+        "Standard folders (eit, edf, fluxmed_*, servo_u, other, mindray_scope) reported as already correct",
         "No files moved or copied",
         "Provides per-patient summary"
       ]
     },
     {
       "id": 3,
-      "prompt": "I have a folder at example/example_patients/Patient_full/ — can you organize it? The Mindray_resp_waves and Mindray_resp_numerics folders have wrong names. Copy the files into correctly named folders (mindray_respi_waves and mindray_respi_numerics). Leave everything else alone.",
-      "expected_output": "Correctly identifies that Mindray_resp_waves→mindray_respi_waves and Mindray_resp_numerics→mindray_respi_numerics need renaming. Creates correctly-named subfolders and copies the files. All other folders left untouched.",
+      "prompt": "Copy example/demo_database/demo_patient/ to a scratch directory outside the repo first, then organize the copy. The Mindray_resp_waves and Mindray_resp_numerics folders have wrong names — copy the files into correctly named folders (mindray_respi_waves and mindray_respi_numerics). Leave everything else alone, and don't touch the original under example/.",
+      "expected_output": "Copies the patient folder to a scratch location, then identifies that Mindray_resp_waves→mindray_respi_waves and Mindray_resp_numerics→mindray_respi_numerics need renaming. Creates correctly-named subfolders and copies the files inside the scratch copy. All other folders left untouched; the shipped demo under example/ is unmodified.",
       "files": [],
       "assertions": [
+        "Works on a scratch copy — example/demo_database/demo_patient/ is byte-identical afterwards",
         "Creates mindray_respi_waves/ and copies files from Mindray_resp_waves/",
         "Creates mindray_respi_numerics/ and copies files from Mindray_resp_numerics/",
-        "Other datasource folders (eit, philips_waves, etc.) reported as already correct and not touched",
+        "Other datasource folders (eit, edf, servo_u, etc.) reported as already correct and not touched",
         "Original Mindray_resp_* folders preserved (copy not move)",
         "Skill uses library folder_keywords to verify expected folder names"
       ]

.gitignore

@@ -38,11 +38,12 @@ logs/
 LOCAL_SCRIPT/*
 *_from_xlsx.json
 
+/tmp
+
 # IA
 PLANS/*
 # QWEN.md — now a symlink to CLAUDE.md
-.qwen/settings.json.orig
-.qwen/settings.json
+.qwen/
 
 # mattpocock skills
 docs/agents/

CHANGELOG.md

@@ -4,6 +4,62 @@ All notable changes to this project will be documented in this file.
 
 ---
 
+## [Unreleased]
+
+_Nothing yet._
+
+---
+
+## [1.1.0] — 2026-08-12
+
+> **Note:** The first release since v1.0.0 went public. It carries one breaking change (three datasources removed, with a migration path below), two new plot types, a new per-person settings tier, and a substantial reduction in load time and memory for large recordings.
+
+### Removed — **breaking**
+- Drop the `philips_waves`, `philips_numerics` and `syringe` datasources. They performed no format-specific parsing — a plain `read_csv` / `read_parquet` — so they were the generic `other` source with extra machinery. Now that each file inside `other/` carries its own configuration and its own `time_shift`, they no longer earn a module.
+
+  **To migrate:** move the files into the patient's `other/` subfolder, then rename their `database_options` and `patient_options` sections from `philips_waves` / `philips_numerics` / `syringe` to `other::<filename-without-extension>`. A folder still named after a removed source is now reported with a warning instead of being loaded. Three details to check while migrating:
+  - **Syringe timestamps.** The old source defaulted to `Europe/Paris`; `other` defaults to UTC. Add `"additional_informations": {"timezone": "Europe/Paris"}` to the section, or timestamps shift by an hour or two.
+  - **Marker traces.** Syringe and Philips numerics drew `lines+markers`. Restore it with the new per-file `trace_options` block (see the user guide).
+  - **Signal names and plot order.** Signals inside `other/` are named `<stem>::<column>`, so cross-source `grouped_fields` and `loop` entries need the qualified form (`other::waves::art`). `other` also renders last by default; use per-signal `priority` to restore a specific order.
+
+### Added
+- **Spectrogram plot type.** A time–frequency view for high-rate signals (EEG, pressure waveforms), configured per signal with a window length, overlap and dB range. It renders the same samples a different way — it does not label or score them ([ADR-0006](docs/adr/0006-no-clinical-analysis.md)).
+- **PSD plot type.** Power spectral density over the visible window; several PSDs share one subplot so bands can be compared side by side.
+- **`.edf` datasource.** Reads European Data Format recordings, the usual container for EEG and polysomnography exports.
+- **`user_options`, a third configuration tier.** Per-person app behaviour and display fallbacks, edited in the Settings modal and stored at `~/.clinical_scope/user_options.json`. It only ever supplies values `database_options` left unset — it never overrides them ([ADR-0005](docs/adr/0005-user-options-are-fallbacks.md)).
+- **Per-file configuration for `other/`.** Each file gets its own `other::<stem>` section carrying its own `time_shift`, timezone, grouping and trace style, so unrelated CSVs in one folder stop sharing a single configuration.
+- **Output redirection.** `save_path` / `save_folder` write results outside the patient folder, so a read-only input directory no longer blocks a run ([ADR-0003](docs/adr/0003-output-root-redirection.md)).
+- Per-file `trace_options` in an `other::<stem>` section (`mode`, `line_width`, `line_dash`, `opacity`, `marker_symbol`, `marker_size`), so one file can be drawn with markers while its neighbours stay plain lines.
+- Files read from `other/` are now symlinked into `clinical_scope_output/`, extending the traceability guarantee previously limited to `philips_waves`.
+
+### Changed
+- **`datetime_start` / `datetime_end` are stored as timezone-aware instants** rather than wall-clock text, so a window means the same moment regardless of the machine reading it.
+- **Display timezone moved into `user_options`.** It is now a per-person setting in the Settings modal, and the patient-options datetime fields re-render in the chosen zone as soon as it changes.
+- **Datetime columns are auto-detected by both name and content.** A column is only accepted as the time axis once its values parse as timestamps, so a `date`-ish name over unparseable data no longer hijacks the axis ([ADR-0004](docs/adr/0004-validate-datetime-column-candidates.md)).
+- Plot types are ordered with time series first, so the familiar view leads.
+- Reloading patient options now falls back to the default for any entry the saved file omits, instead of leaving it blank.
+- A patient folder that yields nothing now says which folders were searched and why each was skipped.
+- CLI scripts that produce no output explain what was missing rather than exiting quietly.
+- `example/` is split by audience, with the demo configuration promoted to the canonical reference; every demo datasource now starts at a common time, so the default view opens with all sources overlapping.
+
+### Fixed
+- **Spectrogram and PSD magnitudes are now a true one-sided power spectral density** (`unit²/Hz`), following `scipy.signal.welch`'s windowing, normalisation and detrending. Previously the magnitude tracked window length — a noise floor climbing 3 dB per doubling — so two windows of the same signal could not be compared and an absolute dB range slid out from under the data.
+- Applying a timezone no longer mutates the caller's DataFrame index in place.
+- A folder holding both `data.csv` and `data.parquet` no longer loads the same data twice under colliding names — one file per stem is kept, preferring parquet.
+- Global groups authored in the Excel format now emit qualified references for `other::<stem>` signals, which previously could not resolve.
+- The bundled EEG example no longer ships channels that were pure artefact.
+
+### Performance
+- **Columns are selected and pruned before loading.** Only the columns a run actually needs are read from disk, cutting load time and peak memory on wide recordings; `inspect` prunes at read too. Redundant deep copies were removed and the deduplicate-then-sort step unified across every path that used both.
+- A second read of the same file is skipped when no time window narrows it.
+- Parquet datetime-column detection reads metadata instead of the column body.
+
+### Documentation
+- Seven new ADRs: output-root redirection (0003), datetime-column validation (0004), user-options-as-fallbacks (0005), the no-clinical-analysis scope boundary (0006), read-time pruning as an optimization rather than a filter (0007), the format-specific-parsing criterion for datasource modules (0008), and `other::<stem>` as a configuration scope (0009).
+- `CONTRIBUTING.md` now states what belongs in the library — ClinicalScope derives for display, it does not interpret — and when a new datasource needs a module of its own rather than a slot in `other/`.
+
+---
+
 ## [1.0.0] — 2026-06-24 *(First public release)*
 
 > **Note:** First public, open-source release of ClinicalScope — installable from PyPI

CITATION.cff

@@ -8,9 +8,9 @@ authors:
     given-names: Alexis
     email: alexis.janin@inria.fr
     affiliation: "Assistance Publique – Hôpitaux de Paris"
-version: 1.0.0
+version: 1.1.0
 doi: "10.5281/zenodo.20830140"
-date-released: "2026-06-24"
+date-released: "2026-08-12"
 license: Apache-2.0
 repository-code: "https://github.com/larib-data/clinical-scope"
 url: "https://github.com/larib-data/clinical-scope"