Skip to content
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
69c6b65
renamed the parameter for clarity
peterlavalle-nhs Sep 28, 2026
ba6742d
always check sheets we're checking (for hidden dates et al)
peterlavalle-nhs Sep 28, 2026
ff2f4a2
detects the problem
peterlavalle-nhs Sep 28, 2026
1789a14
more detail for the error message
peterlavalle-nhs Sep 28, 2026
920b301
validates what it should
peterlavalle-nhs Sep 28, 2026
0547569
fixes ignores to skip temp xlsx files
peterlavalle-nhs Sep 28, 2026
34469d5
passes all tests again
peterlavalle-nhs Sep 28, 2026
45ab14d
checks for padded cell values
peterlavalle-nhs Sep 29, 2026
5250972
updates it to handle ignored values with spaces around them
peterlavalle-nhs Sep 29, 2026
d727465
Update log message and comments for clarity
g-pechorin Sep 29, 2026
592c130
might do the dates
peterlavalle-nhs Sep 29, 2026
32bc10c
Merge remote-tracking branch 'origin/shift_ignore' into shift_ignore
peterlavalle-nhs Sep 29, 2026
4b4479a
more corrections
peterlavalle-nhs Sep 29, 2026
19ff976
raises a specific error for missing patient columns
peterlavalle-nhs Oct 1, 2026
af53890
(WIP) need to digest/adapt the patient_id loading
peterlavalle-nhs Sep 30, 2026
3fa2186
updates the rest of the date shift
peterlavalle-nhs Oct 1, 2026
38ece50
improved function and comments
peterlavalle-nhs Oct 1, 2026
7247526
uses the new exception
peterlavalle-nhs Oct 1, 2026
db26a0c
changes to allow pip to access book_page
peterlavalle-nhs Oct 2, 2026
65de773
small logging messages
peterlavalle-nhs Oct 2, 2026
f470909
Merge remote-tracking branch 'origin/book_page' into book_page
peterlavalle-nhs Oct 2, 2026
77245b8
more logging
peterlavalle-nhs Oct 2, 2026
9e98b1e
log during id generation
peterlavalle-nhs Oct 2, 2026
dc1b873
remove the pd call
peterlavalle-nhs Oct 2, 2026
651c457
restored it - it wasn't the problem
peterlavalle-nhs Oct 2, 2026
57e1bd2
ignore the coverage database
peterlavalle-nhs Oct 5, 2026
117a386
removes duplicate parameters from _inplace
peterlavalle-nhs Oct 5, 2026
3fc3d9d
update docs (oops)
peterlavalle-nhs Oct 5, 2026
68309d8
removes duplicate parameters from _inplace
peterlavalle-nhs Oct 5, 2026
700a568
update docs (oops)
peterlavalle-nhs Oct 5, 2026
f3dcf66
Merge branch 'shift_ignore' into shift_ignore_yaml
peterlavalle-nhs Oct 5, 2026
83c87ef
shift ignore reading from YAML
peterlavalle-nhs Oct 5, 2026
5091e8e
detects old (bad) configs
peterlavalle-nhs Oct 5, 2026
9441cf4
don't count tests/ when checking coverage
peterlavalle-nhs Oct 5, 2026
be338b1
debug logging for the ignores
peterlavalle-nhs Oct 5, 2026
4215f13
tests all pass with the new yaml file approach
peterlavalle-nhs Oct 5, 2026
6e70a83
Merge branch 'shift_ignore' into book_page
peterlavalle-nhs Oct 5, 2026
de2be87
just-enough to see if it helps
peterlavalle-nhs Oct 5, 2026
1451435
test/uv updates
peterlavalle-nhs Oct 8, 2026
184230e
Merge branch 'book_page' into book_stream
peterlavalle-nhs Oct 8, 2026
77b5476
pointed to the live copy
peterlavalle-nhs Oct 8, 2026
cdc970d
WIP 93/111 tests pass
peterlavalle-nhs Oct 8, 2026
817acb3
implements "row is blank" test and skipping
peterlavalle-nhs Oct 9, 2026
e15626d
"cheats" and switches the ambiguous date to iso8061
peterlavalle-nhs Oct 9, 2026
9266d43
corrected overlap detection and testing
peterlavalle-nhs Oct 9, 2026
8e04437
fixes missing patient id column detection
peterlavalle-nhs Oct 9, 2026
1206f2d
added comments and checks for "skip holes"
peterlavalle-nhs Oct 12, 2026
357067c
Merge remote-tracking branch 'origin/main' into book_page
peterlavalle-nhs Oct 12, 2026
7801176
corrected the result of a rename
peterlavalle-nhs Oct 12, 2026
bc74743
fix again
peterlavalle-nhs Oct 12, 2026
a17e990
allow header default to 0 in patient page, and, allow date values in …
peterlavalle-nhs Oct 12, 2026
17de1f4
refactored the existing functions (as is) into separate files
peterlavalle-nhs Oct 12, 2026
4a5e407
brings book_page (and tests) into the project
peterlavalle-nhs Oct 12, 2026
8dd5540
raise different exception
peterlavalle-nhs Oct 12, 2026
cde02e7
refactors the shift_delta generation to be simpler and move checking …
peterlavalle-nhs Oct 12, 2026
7266555
try to separate the sheet/row checks to be a standalone thing
peterlavalle-nhs Oct 12, 2026
bca28c5
refactored the loop control and parameter lookups into a scanner class
peterlavalle-nhs Oct 12, 2026
1fc05bf
extracts the ignore rules to a subroutine
peterlavalle-nhs Oct 12, 2026
7f75ba7
idea for how to fixit
peterlavalle-nhs Oct 12, 2026
0f06412
more cleanup
peterlavalle-nhs Oct 13, 2026
93da71d
comments and removed a redundant check
peterlavalle-nhs Oct 13, 2026
fe70510
"orchestration" changes
peterlavalle-nhs Oct 14, 2026
e15b3c5
csv and excel workbooks now both try to parse dates to be consistent …
peterlavalle-nhs Oct 14, 2026
54d8c3e
adds the hidden-date bypass to book_page
peterlavalle-nhs Oct 16, 2026
09e2e8a
always allow "NULL" in date columns
peterlavalle-nhs Oct 16, 2026
a4eb73b
option to skip dating date
peterlavalle-nhs Oct 16, 2026
c7d8776
add "skip" to sheet/page types
peterlavalle-nhs Oct 16, 2026
c17e023
cut out normalise in a different way
peterlavalle-nhs Oct 16, 2026
a17deb2
renamed and documented the bypasses
peterlavalle-nhs Oct 16, 2026
c623097
raise exceptions on abstract invocation
peterlavalle-nhs Oct 16, 2026
24bd638
Potential fix for pull request finding 'CodeQL / Clear-text logging o…
g-pechorin Oct 16, 2026
780682a
renamed self to self
peterlavalle-nhs Oct 16, 2026
10c96df
Merge remote-tracking branch 'origin/book_page' into book_page
peterlavalle-nhs Oct 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,4 @@ tests/output/
*.jsonl.tar.gz

.DS_Store

# ignore temp excel files
~*
.coverage
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ sheet_configs = {
],
"header_row": 1,
"skip_rows_after_header": [],
"pass_as_is": { # the parameter is here
"shift_ignore": { # the parameter is here
"dob": [ # any column can have "as is" values added
"missing", # the values are each listed here
]
Expand Down Expand Up @@ -180,7 +180,7 @@ While this does require repeated manual intervention ...
- `header_row`: (Optional) Zero-based row index for the row that contains column names
- `skip_rows_after_header`: (Optional) List of zero-based row indices to exclude from data (e.g. a data-type row immediately below the header)
- `shift_exceptions`: (Optional) Dict mapping column names to lists of date strings that should never be shifted (e.g. a fixed end-of-study date). Dates are parsed using the same flexible parser as regular date values.
- `pass_as_is`: (Optional) Dict mapping column names to lists of "non dates" that are passed through without being changed
- `shift_ignore`: (Optional) Dict mapping `page:{column:[values]}` to lists of values that `shift_excel_dates_inplace()` passes as-is with no manipulation or checking.
- `patient_header_row`: (Optional) Zero-based header row for the patient sheet (default: 0). If the patient sheet is in `sheet_configs`, that sheet’s `header_row` is used instead.
- `patient_skip_rows`: (Optional) Zero-based row indices to exclude from patient data (e.g. a data-type row). If the patient sheet is in `sheet_configs`, that sheet’s `skip_rows_after_header` is used instead.
- `min_shift_days` / `max_shift_days`: Range of days to shift (default: -15 to 15)
Expand Down
Loading
Loading