Hotfix/additional invalid issues - #854
Conversation
…ed on FY26 specs. And smal updates to this fuller check to handle special columns and beginnings of nullunless check
… displaying more consistent messaging
replace client-invalid demographic values check with fuller check bas…
…ng the zip package
…nd more automatically parse the specs (after updating it)
…ng session data. Whether we actually store them will depend on whether they're in the Export.csv file
…e old cols_and_data-types (now called column_priorities) now just provides Eva's DataTypeHighPriority indicator. Switching to the specs doc, though, means we have to update references to the File and Column columns, which are now named CSV and Name, respectively. also using an updated data_type_mapping to map the data types referenced in the specs with the R Classes of the columns of the read-in csvs.
…to the detect_invalid_values script, since they're all based on the specs doc now. Also removing the ExportID inconsistency, since that's captured by the Relational Integrity Violation check
Merge branch 'dev' into hotfix/additional-invalid-issues # Conflicts: # 03_file_structure_analysis.R # global.R # helper_functions.R # public-resources/EvaChecks.csv
Merge branch 'dev' into hotfix/additional-invalid-issues # Conflicts: # tests/helper_data/main-valid/dq_main.csv # tests/testthat/_snaps/linux-4.5/main-valid/test-main-valid-exportTestValues.json
i think this is resolved? I set |
I think this is no longer a problem:
|
|
Overall the changes are looking much improved and working for most scenarios. Found one modified dataset that still broke the upload, in demo dataset mods dir: |
|
In reviewing, noticed that DQ and PDDE checks for "Nulls not allowed" and "Invalid Non-Null Value" were appearing as mostly empty rows in the PDDE And DQ reports. 
|
|
@alex-silverman I did some digging based on the above comment, and it seems like these two types of checks, along with others, do not have AnchorIDs in What is the reasoning for setting up some checks with no AnchorID? |
Nice catch. Just pushed a fix (exiting out early from part of the code if any Key Fields are not in the dataset). |
@vlopez0603 - which dataset did you use and which tab were you looking at? I just tried DEMO mode and didn't see a blank org in any of the HP, Error, or Warning tabs. |
…an be run with no prior code run.
@vlopez0603 - I think the specs dataset hadn't been updated from my dev version to the one on eva-dev. Can you try again? |
|
Most of the blank rows have been resolved though there's still one on the PDDE side (see screenshot below). This case is actually flagging a missing Geocode for a project. 
I think it may be on me to update the specs sheet on ACE for at least one field. I may have missed indicating to include the AnchorID for that data element/column. |
|
@alex-silverman / @mlfurman3 - I made minor updates to the specs spreadsheet. I noticed the Incorrect Data Type check had three columns incorrectly set as PDDE instead of FSA. The Non-Null Invalid check also had one column incorrectly set as PDDE instead of FSA. I'm still in the process of reviewing the other checks in the specs spreadsheet. |
…s run before anything else
… based on latest version
…or EnrollmentCoC, which is now a pdde check
3 participants
Using machine specs to detect invalid values across all columns in all files.