Hotfix/additional invalid issues - #854

Open

alex-silverman wants to merge 170 commits into

devfrom

hotfix/additional-invalid-issues

Commits on Sep 22, 2025

replace client-invalid demographic values check with fuller check based on FY26 specs. And smal updates to this fuller check to handle special columns and beginnings of nullunless check
Alex Silverman
committed
123b472

Commits on Oct 27, 2025

handle NULL Export.csv fields during logToConsole
Alex Silverman
committed
c223483
add initially invalid check for empty files
Alex Silverman
committed
292c30d

Commits on Nov 4, 2025

Merge branch 'feat/monthly-inflow-outflows' into hotfix/additional-invalid-issues
alex-silverman
authored
9850d1a
making sure to correctly grab file sizes without needing to unzip and displaying more consistent messaging
Alex Silverman
committed
8b1fc9c

Commits on Nov 11, 2025

Commits on Jan 23, 2026

Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman
authored
2b997c1

Commits on Jan 25, 2026

reference local version of machine-readable specs
Alex Silverman
committed
2fd479f

Commits on Feb 3, 2026

for some reason, zip_list wasn't getting found, so explicitly prefixing the zip package
Alex Silverman
committed
5db3b70
refacor to consolidate existing checks, better initial helper data, and more automatically parse the specs (after updating it)
Alex Silverman
committed
45ed7ec
more updateable lsit of files to ignore.
Alex Silverman
committed
d1e82c6
better handling of which Export.csv fields we want to store for logging session data. Whether we actually store them will depend on whether they're in the Export.csv file
Alex Silverman
committed
86966b8
better check naming and consolidation
Alex Silverman
committed
e6a1991
updating/consolidating checks in EvaChecks.csv
Alex Silverman
committed
56031f4
only use tryCatch when not IN_DEV_MODE
Alex Silverman
committed
cec86fd
changing cols_and_Data_types to be the machine-readable specs. and the old cols_and_data-types (now called column_priorities) now just provides Eva's DataTypeHighPriority indicator.
Alex Silverman
committed
e365c93
moving Brackets, Incorrect columns, and Unexpected Data Types checks to the detect_invalid_values script, since they're all based on the specs doc now. Also removing the ExportID inconsistency, sin…
Alex Silverman
committed
73fb9ef

Commits on Feb 4, 2026

doing the merge with check_info dt at the end, just once.
Alex Silverman
committed
63abda4

Commits on Feb 6, 2026

fixing the datatype mapping on import
Alex Silverman
committed
6be1dfc
adding file/csv name as a variable to facilitate dynamic Detail text
Alex Silverman
committed
e8c676b
making a standardized DEtail column just once, but adding a few special cases
Alex Silverman
committed
ceeea8a
fix bug in grabbing foreign_key_checks data
Alex Silverman
committed
754d6a7

Commits on Feb 8, 2026

we need to get all data files in order to run the FSA checks on all of them
Alex Silverman
committed
a6ea040
make sure we're checking all foreign keys, not just the first
Alex Silverman
committed
e0d61bc
remove files not used by the rest of Eva at the end of FSA
Alex Silverman
committed
ece36e4

Commits on Feb 9, 2026

Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman
authored
b792fc3

Commits on Feb 10, 2026

most things are now handled in the new machine_readable_specs_validation.R script. =
Alex Silverman
committed
3b5a220
updating columns.csv file to only include File, Column, and DataTypeHighPriority
Alex Silverman
committed
7a2ab64
Merge branch 'hotfix/additional-invalid-issues' of github.com:abtassociates/eva into hotfix/additional-invalid-issues
Alex Silverman
committed
9ca492f

Commits on Feb 11, 2026

we've renamed the machine-readable-specs-validation script. reflect that in 03
Alex Silverman
committed
b455d87
simplify and speed up check for duplicate unique IDs
Alex Silverman
committed
f64dc1a
now that we're doing systematic FSA checks for all files, we don't want to remove them from the reduced datasets or the test "upload" will fail.
Alex Silverman
committed
6443b56
let's not load readxl package. Instead, we'll prefix it to the only 2 places we call read_excel (machine readbale specs tabs)
Alex Silverman
committed
34c3879
making sure all necessary packages are loaded in order to run create_test_datasets.R without first running the app (which would load all necessary libraries).
Alex Silverman
committed
6fb5bc8

Commits on Feb 12, 2026

clean up the validation specs
Alex Silverman
committed
355a0e1
fixes to machine-readable validation:
Alex Silverman
committed
381411f
since we're now using all files for fsa check, need to stop checking for nrows = 1 in creating test datasets
Alex Silverman
committed
d7eeb2e
need to force to integers or fcase won't work when created updated data
Alex Silverman
committed
50ff224
fixed fsa test zip (dropped -test) suffix
Alex Silverman
committed
e532f79

Commits on Feb 13, 2026

removed old duplicate ID checks
Alex Silverman
committed
67bcf85

Commits on Feb 18, 2026

Commits on Mar 5, 2026

moving more specs-prep stuff into its own script to be loaded up front when app loads (rather than for each user session). In theory, this has everything needed to do the processing in the FSA, DQ,…
Alex Silverman
committed
081c491
carrying through other updates:
Alex Silverman
committed
7a903de
handling null_unless and foreign key checks in DQ and PDDE
Alex Silverman
committed
80fdbdf
updating EvaChecks to reflect latest checks
Alex Silverman
committed
b8c00c4

Commits on Mar 10, 2026

major refactor:
Alex Silverman
committed
189c9d1
the new specs checks turned up a problem with the demo file (invalid SubTypeProvided). Fix
Alex Silverman
committed
9aeeaaa
In testing:
Alex Silverman
committed
3da22a2
forgot to save the RDS files!
Alex Silverman
committed
77b2817
fixing the warning about NA values of daterange inputs. We don't need NA. We can just not specify, which makes them Null, which is still empty to begin with.
Alex Silverman
committed
1fc6c6c

Commits on Mar 13, 2026

change the "Type" column in EvaChecks to a more intuitive "Priority". This requires changes throughout the code. Similarly, "check_type" to "issue_type" (same as "Issue", i.e. the name of the issue).
Alex Silverman
committed
f8d274c
make sure to rename AnchorValue to EnrollmentID/ProjectID so it will merge correctly with the 05/06 data.
Alex Silverman
committed
7ee2ead
remove old code
Alex Silverman
committed
cefd3b7
to ensure sure all Detail Texts evaluate correctly, need to include validation_notes and handle NAs
Alex Silverman
committed
47d4fa3
small bug fixes to incorrect-column code.
Alex Silverman
committed
6268995
include valid_values and csv_join_prereqs map as mirai dependencies
Alex Silverman
committed
70f249f
lingering update of "check_type" to "issue_type"
Alex Silverman
committed
0a4135e
updated specs_prepped rds
Alex Silverman
committed
15e8f50

Commits on Mar 17, 2026

lingering references to the old "Type" instead of "priority"
Alex Silverman
committed
0e80a0e
make sure to account for the fact that only the specs in DQ have Details.
Alex Silverman
committed
313267c
For DQ Null Unless, there's no CSV or Name field, so need to add the Name field to the detail
Alex Silverman
committed
6d75eb0
bring back accidentally-deleted valid_list_lookup
Alex Silverman
committed
3c7b7ac
Rename columns to more readable versions for FSA output
Alex Silverman
committed
f434af2
make sure to include AnchorValue in the FSA output
Alex Silverman
committed
e8e211b
set session-valid to 0 correctly now that we're using DataTypeHighPriority
Alex Silverman
committed
5be75c7
adding DQ and PDDE versions of Null Unless checks to Evachecks
Alex Silverman
committed
2ce8d60
updating specs_prepped withthe latest specs reflecting that we DO want to include the Anchor ID/Value for Null unless in DQ and PDDE.
Alex Silverman
committed
008d480

Commits on Mar 19, 2026

joining column_priorities in specs_helpers, and updating Priority column there. Basically, for any specs, if the *column* is HP, then the issue is HP.
Alex Silverman
committed
cd3a247
another forgotten place we were referencing old Type variable name instead of new Priority. Also switching to collapse in a couple places for consistency
Alex Silverman
committed
77410ab
use fill=TRUE in pdde rowbind
Alex Silverman
committed
8f33418
instead of converting Priority to factor in lots of places, do it once in data prep.
Alex Silverman
committed
33e29b8
code cleanups
Alex Silverman
committed
6be1b69
funique as needed
Alex Silverman
committed
8f96a50
modify the Detail Text for Null Unless ot specify the Name and CSV. This way, we don't have these extra floating CSV and Column columns that make less sense in DQ and PDDE outputs.
Alex Silverman
committed
2fb5109
add sync=TRUE to let mirai pause at browsers
Alex Silverman
committed
0b74324
updated specs_prepped
Alex Silverman
committed
57cc745
deleting specS_prepped for now, as it's not yet public
Alex Silverman
committed
591b87b

Commits on Apr 22, 2026

include cols_and_data_types in the saved machine-specs file! It's used in helper_functions.R
Alex Silverman
committed
334390c
fix Incorrect Data Types check. We were returning the underlying csv data, without the flagged RClass and such used in the Detail
Alex Silverman
committed
a46f514
rename Name to Column for the 3 specs issues so that it aligns with the other FSA checks that use "Column"
Alex Silverman
committed
6fd6b12
remove empty Key Info phrase inthe Detail if there are no key fields. Also switch to stringi::stri_replace_all_*; it's faster than gsub
Alex Silverman
committed
8ee51a9
adding logging statements
Alex Silverman
committed
7871d47
we don't need the _fixed version of the original data anymore because of how we're reading things in.
Alex Silverman
committed
3a5d4d3

Commits on Apr 29, 2026

adding code to create readable versions of the Null Unless rules, which will be piped to the "Detail" text, e.g. "[COLUMN] is [HUMAN RULE]".
Alex Silverman
committed
7742991
fixing this
Alex Silverman
committed
0269162
timeout-overlay formatting
Alex Silverman
committed
b88f823

Commits on May 1, 2026

custom Detail display for 2 specific Null Unless rules
Alex Silverman
committed
b81682e

Commits on May 4, 2026

Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman
authored
f047a50
adding eva's version of specs
Alex Silverman
committed
a5dcf36

Commits on May 6, 2026

include the more detailed NullUnless - Additional Requirements, which specify additional requirements for null-unless variables being non-null, and otherwise null.
Alex Silverman
committed
ec6a159
fix column name from null unless additional reqs: it's Funder, not FundingSource
Alex Silverman
committed
9b55160
go back to using NA, instead of NULL for date inputs. otherwise defaults to current date which will keep changing/failing for tests.
Alex Silverman
committed
8f3238e
Merge branch 'hotfix/additional-invalid-issues' of github.com:abtassociates/eva into hotfix/additional-invalid-issues
Alex Silverman
committed
e131d4d

Commits on May 11, 2026

fix update of new Null Unless logic. Using data.table so we can subset update vars at the same time.
Alex Silverman
committed
3c1473f
getting rid of DataTypeHighPriority. Not helpful since we already have Priority
Alex Silverman
committed
7c5371e
updating specs_prepped
Alex Silverman
committed
973908b

Commits on May 14, 2026

Add "List" element to be populated into the Invalid Non-Null
Alex Silverman
committed
b30ca5b
Instead of hardcoding string limit to 32, use what's in the specs.
Alex Silverman
committed
5adb5f6
if a Detail Template contains a variable that doesn't exist in the dataset, it will crash. So instead we wrap in a `try`.
Alex Silverman
committed
2d4ac75
if we subset to variables in the dataset, then we'll lose variables if they're misspelled.
Alex Silverman
committed
124e28c
instead of just flagging extra and missing (and misordered) columns, we also want to flag misspelled and miscapitalized.
Alex Silverman
committed
0bcacaf
another place where we need to include the List column
Alex Silverman
committed
4653a51
only do the work of joining the reporting info if there are any issues in the first place
Alex Silverman
committed
8e36808
update EvaChecks guidance for invalid non-null values
Alex Silverman
committed
f138857
handle variables that have a dynamic valid list
Alex Silverman
committed
aea9aa3

Commits on May 15, 2026

Commits on May 18, 2026

consolidate manual edits
Alex Silverman
committed
937d839
add SubTypeProvided back to special_validation_rules.
Alex Silverman
committed
7799555
move invalid_non__null_dynamic_lists code to more sensible location and make it a dt
Alex Silverman
committed
bcd04b2
order csv_join_prereqs alphabetically and add any missing that are needed for the Funder/ProjectType condition for null_unless
Alex Silverman
committed
58de674
only define additional_reqs (for null_unless) if there's even a funder and/or projecttype
Alex Silverman
committed
d6d1879
instead of joining for the rule, since we need to actually evaluate the underlying data (e.g. check if the actual Funder is in the null unless rule list), we add it later when doing the invalid check.
Alex Silverman
committed
369d787
fix setting of List element in detail template
Alex Silverman
committed
94674f6
some code cleanup
Alex Silverman
committed
df1d375
code cleanup
Alex Silverman
committed
f681732
cleanup
Alex Silverman
committed
800cb98
Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman
authored
a865052
updating specs_prepped
Alex Silverman
committed
1628378
Merge branch 'hotfix/additional-invalid-issues' of github.com:abtassociates/eva into hotfix/additional-invalid-issues
Alex Silverman
committed
ee5d8cc

Commits on May 19, 2026

handle empty special_validation_issues and empty FSA
Alex Silverman
committed
b8c13c4
fix bug when Funder and ProjectType are empty in Null Unless Additional Reqs dataset
Alex Silverman
committed
82a97d4
Fix column capitalization in columns.csv
Alex Silverman
committed
e802baa
update test good and demo. Demo needs to pass new FSA checks that are now HP so it can also do DQ.
Alex Silverman
committed
4003977
add special non-null checks for CLS ,LivingSituation, and Destination. The specs had nothing, but we want to exclude some values.
Alex Silverman
committed
83019a8
create an invalid non-null value for EmploymentType, as this is not an HP column. This will allow demo mode to proceed
Alex Silverman
committed
fe6c83a
remove RaceNone injected problem from demo mode FSA
Alex Silverman
committed
3b059be
need to make sure we run the templatable validation checks
Alex Silverman
committed
2945232
make sure to load cols_and_Data_types if specs_prepped exists.
Alex Silverman
committed
9693639
fixing typo in cols_and_data_types assignment
Alex Silverman
committed
e81fcb6
instead of creating a new dataset claled validation_info (which was referenced in other places), just update cols_and_data_types, and reference that instead, as it's already saved with specs_prepped.
Alex Silverman
committed
554bb0f
flipped sign on invalid non-null check of CLS/LivingSituation/Destination
Alex Silverman
committed
788a81b
update specs_prepped
Alex Silverman
committed
ee2bcd2

Commits on Jun 29, 2026

split Null Unless check into 2: "*should be* null unless..." vs. "*should not be* null when..."
Alex Silverman
committed
74a1abd
code cleanup
Alex Silverman
committed
51f907a
Test File updates:
Alex Silverman
committed
fdd943c
add necessary intermediate files for specs.
Alex Silverman
committed
9f7a6b0
add null_unless_additional_reqs to saved dataset
Alex Silverman
committed
7f21409
fix typo in Impermissible Characters reference
Alex Silverman
committed
8bab1da
make sure machine readable specs script is loaded for creating test datasets.
Alex Silverman
committed
432480f
make sure downloadImpermissibleButton is visibile before doing download in shinytest
Alex Silverman
committed
5bbef54
updated shinytests
Alex Silverman
committed
9dde2f8
merge in latest dev
Alex Silverman
committed
60f683b

Commits on Jul 10, 2026

merging in latest dev with timeliness updates
Alex Silverman
committed
ebf6063
update snapshots
Alex Silverman
committed
0960a1d

Commits on Jul 15, 2026

Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman
authored
aee9ae7

Commits on Jul 28, 2026

handle key fields missing from underlying dataset
Alex Silverman
committed
0d1c0b5
adding library calls in load_machine_readbale_Specs to make sure it can be run with no prior code run.
Alex Silverman
committed
8e03e4d
Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman
authored
77f0149

Commits on Jul 31, 2026

include necessary helper function scripts in case load specs script is run before anything else
Alex Silverman
committed
275198c
updating specs path to point to spreadsheet on ACE and updating specs based on latest version
Alex Silverman
committed
e137e3f
updating specs sheet for Foreign Key Missing checks
Alex Silverman
committed
056a1e1

Commits on Aug 3, 2026

allow for no key fields (at leats used by Foreign Key Missing check for EnrollmentCoC, which is now a pdde check
Alex Silverman
committed
0c95122
update tests snapshots
Alex Silverman
committed
d679a32
updating snapshots
Alex Silverman
committed
27f3c8d

Commits on Aug 12, 2026

correctly select invalid records. `fsubset` might not have been working
Alex Silverman
committed
cd16e0e
update detail_template with "Not Null When" detail text.
Alex Silverman
committed
b124662
logging memory usage
Alex Silverman
committed
81c5644
cleanup
Alex Silverman
committed
b4ccaeb
update specs
Alex Silverman
committed
99f61d1
add missing session to `log_memory`
Alex Silverman
committed
0eed3cd
update specs rds
Alex Silverman
committed
d52d31d
update snapshots, including new `demo-test`
Alex Silverman
committed
1f4c924
Merge branch 'dev' into hotfix/additional-invalid-issues
Alex Silverman
committed
33d231c
updating snapshot after merge with dev
Alex Silverman
committed
926e6e7
update snapshots for new demo test
Alex Silverman
committed
3b69880