Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
159 commits
Select commit Hold shift + click to select a range
123b472
replace client-invalid demographic values check with fuller check bas…
Sep 22, 2025
c223483
handle NULL Export.csv fields during logToConsole
Oct 27, 2025
292c30d
add initially invalid check for empty files
Oct 27, 2025
9850d1a
Merge branch 'feat/monthly-inflow-outflows' into hotfix/additional-in…
alex-silverman Nov 4, 2025
8b1fc9c
making sure to correctly grab file sizes without needing to unzip and…
Nov 4, 2025
0c7e15b
Merge branch 'hotfix/additional-invalid-issues' into invalid_vals_check
alex-silverman Nov 11, 2025
e407a1b
Merge pull request #874 from abtassociates/invalid_vals_check
alex-silverman Nov 11, 2025
2b997c1
Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman Jan 23, 2026
2fd479f
reference local version of machine-readable specs
Jan 25, 2026
5db3b70
for some reason, zip_list wasn't getting found, so explicitly prefixi…
Feb 3, 2026
45ed7ec
refacor to consolidate existing checks, better initial helper data, a…
Feb 3, 2026
d1e82c6
more updateable lsit of files to ignore.
Feb 3, 2026
86966b8
better handling of which Export.csv fields we want to store for loggi…
Feb 3, 2026
e6a1991
better check naming and consolidation
Feb 3, 2026
56031f4
updating/consolidating checks in EvaChecks.csv
Feb 3, 2026
cec86fd
only use tryCatch when not IN_DEV_MODE
Feb 3, 2026
e365c93
changing cols_and_Data_types to be the machine-readable specs. and th…
Feb 3, 2026
73fb9ef
moving Brackets, Incorrect columns, and Unexpected Data Types checks …
Feb 3, 2026
63abda4
doing the merge with check_info dt at the end, just once.
Feb 4, 2026
6be1dfc
fixing the datatype mapping on import
Feb 6, 2026
e8c676b
adding file/csv name as a variable to facilitate dynamic Detail text
Feb 6, 2026
ceeea8a
making a standardized DEtail column just once, but adding a few speci…
Feb 6, 2026
754d6a7
fix bug in grabbing foreign_key_checks data
Feb 6, 2026
a6ea040
we need to get all data files in order to run the FSA checks on all o…
Feb 8, 2026
e0d61bc
make sure we're checking all foreign keys, not just the first
Feb 8, 2026
ece36e4
remove files not used by the rest of Eva at the end of FSA
Feb 8, 2026
b792fc3
Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman Feb 9, 2026
3b5a220
most things are now handled in the new machine_readable_specs_validat…
Feb 10, 2026
7a2ab64
updating columns.csv file to only include File, Column, and DataTypeH…
Feb 10, 2026
9ca492f
Merge branch 'hotfix/additional-invalid-issues' of github.com:abtasso…
Feb 10, 2026
b455d87
we've renamed the machine-readable-specs-validation script. reflect t…
Feb 11, 2026
f64dc1a
simplify and speed up check for duplicate unique IDs
Feb 11, 2026
6443b56
now that we're doing systematic FSA checks for all files, we don't wa…
Feb 11, 2026
34c3879
let's not load readxl package. Instead, we'll prefix it to the only 2…
Feb 11, 2026
6fb5bc8
making sure all necessary packages are loaded in order to run create_…
Feb 11, 2026
355a0e1
clean up the validation specs
Feb 12, 2026
381411f
fixes to machine-readable validation:
Feb 12, 2026
d7eeb2e
since we're now using all files for fsa check, need to stop checking …
Feb 12, 2026
50ff224
need to force to integers or fcase won't work when created updated data
Feb 12, 2026
e532f79
fixed fsa test zip (dropped -test) suffix
Feb 12, 2026
67bcf85
removed old duplicate ID checks
Feb 13, 2026
f75cbdd
we will continue doing the null_unless and foreign_key_issue checks i…
Feb 18, 2026
ef39c03
using the more precise and accurate "integer" RClass, instead of "num…
Feb 18, 2026
081c491
moving more specs-prep stuff into its own script to be loaded up fron…
Mar 5, 2026
7a903de
carrying through other updates:
Mar 5, 2026
80fdbdf
handling null_unless and foreign key checks in DQ and PDDE
Mar 5, 2026
b8c00c4
updating EvaChecks to reflect latest checks
Mar 5, 2026
189c9d1
major refactor:
Mar 10, 2026
9aeeaaa
the new specs checks turned up a problem with the demo file (invalid …
Mar 10, 2026
3da22a2
In testing:
Mar 10, 2026
77b2817
forgot to save the RDS files!
Mar 10, 2026
1fc6c6c
fixing the warning about NA values of daterange inputs. We don't need…
Mar 10, 2026
f8d274c
change the "Type" column in EvaChecks to a more intuitive "Priority".…
Mar 13, 2026
7ee2ead
make sure to rename AnchorValue to EnrollmentID/ProjectID so it will …
Mar 13, 2026
cefd3b7
remove old code
Mar 13, 2026
47d4fa3
to ensure sure all Detail Texts evaluate correctly, need to include v…
Mar 13, 2026
6268995
small bug fixes to incorrect-column code.
Mar 13, 2026
70f249f
include valid_values and csv_join_prereqs map as mirai dependencies
Mar 13, 2026
0a4135e
lingering update of "check_type" to "issue_type"
Mar 13, 2026
15e8f50
updated specs_prepped rds
Mar 13, 2026
0e80a0e
lingering references to the old "Type" instead of "priority"
Mar 17, 2026
313267c
make sure to account for the fact that only the specs in DQ have Deta…
Mar 17, 2026
6d75eb0
For DQ Null Unless, there's no CSV or Name field, so need to add the …
Mar 17, 2026
3c7b7ac
bring back accidentally-deleted valid_list_lookup
Mar 17, 2026
f434af2
Rename columns to more readable versions for FSA output
Mar 17, 2026
e8e211b
make sure to include AnchorValue in the FSA output
Mar 17, 2026
5be75c7
set session-valid to 0 correctly now that we're using DataTypeHighPri…
Mar 17, 2026
2ce8d60
adding DQ and PDDE versions of Null Unless checks to Evachecks
Mar 17, 2026
008d480
updating specs_prepped withthe latest specs reflecting that we DO wan…
Mar 17, 2026
cd3a247
joining column_priorities in specs_helpers, and updating Priority col…
Mar 19, 2026
77410ab
another forgotten place we were referencing old Type variable name in…
Mar 19, 2026
8f33418
use fill=TRUE in pdde rowbind
Mar 19, 2026
33e29b8
instead of converting Priority to factor in lots of places, do it onc…
Mar 19, 2026
6be1b69
code cleanups
Mar 19, 2026
8f96a50
funique as needed
Mar 19, 2026
2fb5109
modify the Detail Text for Null Unless ot specify the Name and CSV. T…
Mar 19, 2026
0b74324
add sync=TRUE to let mirai pause at browsers
Mar 19, 2026
57cc745
updated specs_prepped
Mar 19, 2026
591b87b
deleting specS_prepped for now, as it's not yet public
Mar 19, 2026
334390c
include cols_and_data_types in the saved machine-specs file! It's use…
Apr 22, 2026
a46f514
fix Incorrect Data Types check. We were returning the underlying csv …
Apr 22, 2026
6fd6b12
rename Name to Column for the 3 specs issues so that it aligns with t…
Apr 22, 2026
8ee51a9
remove empty Key Info phrase inthe Detail if there are no key fields.…
Apr 22, 2026
7871d47
adding logging statements
Apr 22, 2026
3a5d4d3
we don't need the _fixed version of the original data anymore because…
Apr 22, 2026
7742991
adding code to create readable versions of the Null Unless rules, whi…
Apr 29, 2026
0269162
fixing this
Apr 29, 2026
b88f823
timeout-overlay formatting
Apr 29, 2026
b81682e
custom Detail display for 2 specific Null Unless rules
May 1, 2026
f047a50
Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman May 4, 2026
a5dcf36
adding eva's version of specs
May 4, 2026
ec6a159
include the more detailed NullUnless - Additional Requirements, which…
May 6, 2026
9b55160
fix column name from null unless additional reqs: it's Funder, not Fu…
May 6, 2026
8f3238e
go back to using NA, instead of NULL for date inputs. otherwise defau…
May 6, 2026
e131d4d
Merge branch 'hotfix/additional-invalid-issues' of github.com:abtasso…
May 6, 2026
3c1473f
fix update of new Null Unless logic. Using data.table so we can subse…
May 11, 2026
7c5371e
getting rid of DataTypeHighPriority. Not helpful since we already hav…
May 11, 2026
973908b
updating specs_prepped
May 11, 2026
b30ca5b
Add "List" element to be populated into the Invalid Non-Null
May 14, 2026
5adb5f6
Instead of hardcoding string limit to 32, use what's in the specs.
May 14, 2026
2d4ac75
if a Detail Template contains a variable that doesn't exist in the da…
May 14, 2026
124e28c
if we subset to variables in the dataset, then we'll lose variables i…
May 14, 2026
0bcacaf
instead of just flagging extra and missing (and misordered) columns, …
May 14, 2026
4653a51
another place where we need to include the List column
May 14, 2026
8e36808
only do the work of joining the reporting info if there are any issue…
May 14, 2026
f138857
update EvaChecks guidance for invalid non-null values
May 14, 2026
aea9aa3
handle variables that have a dynamic valid list
May 14, 2026
91ece3c
pull in the list names for the elements whose valid list is dynamic. …
May 15, 2026
cbed250
add some manual edits to the specs. These are situations where the sp…
May 15, 2026
97d907e
Add a new "Conditonally Invalid Value" check. Unlike the Invalid Non-…
May 15, 2026
4aa761a
now that we've better separated the Invalid Non-Null check, including…
May 15, 2026
937d839
consolidate manual edits
May 18, 2026
7799555
add SubTypeProvided back to special_validation_rules.
May 18, 2026
bcd04b2
move invalid_non__null_dynamic_lists code to more sensible location a…
May 18, 2026
58de674
order csv_join_prereqs alphabetically and add any missing that are ne…
May 18, 2026
d6d1879
only define additional_reqs (for null_unless) if there's even a funde…
May 18, 2026
369d787
instead of joining for the rule, since we need to actually evaluate t…
May 18, 2026
94674f6
fix setting of List element in detail template
May 18, 2026
df1d375
some code cleanup
May 18, 2026
f681732
code cleanup
May 18, 2026
800cb98
cleanup
May 18, 2026
a865052
Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman May 18, 2026
1628378
updating specs_prepped
May 18, 2026
ee5d8cc
Merge branch 'hotfix/additional-invalid-issues' of github.com:abtasso…
May 18, 2026
b8c13c4
handle empty special_validation_issues and empty FSA
May 19, 2026
82a97d4
fix bug when Funder and ProjectType are empty in Null Unless Addition…
May 19, 2026
e802baa
Fix column capitalization in columns.csv
May 19, 2026
4003977
update test good and demo. Demo needs to pass new FSA checks that are…
May 19, 2026
83019a8
add special non-null checks for CLS ,LivingSituation, and Destination…
May 19, 2026
fe6c83a
create an invalid non-null value for EmploymentType, as this is not a…
May 19, 2026
3b059be
remove RaceNone injected problem from demo mode FSA
May 19, 2026
2945232
need to make sure we run the templatable validation checks
May 19, 2026
9693639
make sure to load cols_and_Data_types if specs_prepped exists.
May 19, 2026
e81fcb6
fixing typo in cols_and_data_types assignment
May 19, 2026
554bb0f
instead of creating a new dataset claled validation_info (which was r…
May 19, 2026
788a81b
flipped sign on invalid non-null check of CLS/LivingSituation/Destina…
May 19, 2026
ee2bcd2
update specs_prepped
May 19, 2026
74a1abd
split Null Unless check into 2: "*should be* null unless..." vs. "*sh…
Jun 29, 2026
51f907a
code cleanup
Jun 29, 2026
fdd943c
Test File updates:
Jun 29, 2026
9f7a6b0
add necessary intermediate files for specs.
Jun 29, 2026
7f21409
add null_unless_additional_reqs to saved dataset
Jun 29, 2026
8bab1da
fix typo in Impermissible Characters reference
Jun 29, 2026
432480f
make sure machine readable specs script is loaded for creating test d…
Jun 29, 2026
5bbef54
make sure downloadImpermissibleButton is visibile before doing downlo…
Jun 29, 2026
9dde2f8
updated shinytests
Jun 29, 2026
60f683b
merge in latest dev
Jun 29, 2026
ebf6063
merging in latest dev with timeliness updates
Jul 10, 2026
0960a1d
update snapshots
Jul 10, 2026
aee9ae7
Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman Jul 15, 2026
0d1c0b5
handle key fields missing from underlying dataset
Jul 28, 2026
8e03e4d
adding library calls in load_machine_readbale_Specs to make sure it c…
Jul 28, 2026
77f0149
Merge branch 'dev' into hotfix/additional-invalid-issues
alex-silverman Jul 28, 2026
275198c
include necessary helper function scripts in case load specs script i…
Jul 31, 2026
e137e3f
updating specs path to point to spreadsheet on ACE and updating specs…
Jul 31, 2026
056a1e1
updating specs sheet for Foreign Key Missing checks
Jul 31, 2026
0c95122
allow for no key fields (at leats used by Foreign Key Missing check f…
Aug 3, 2026
d679a32
update tests snapshots
Aug 3, 2026
27f3c8d
updating snapshots
Aug 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,5 @@ vscDebugger
tests/testthat/_snaps/**/*.png
www/client-level-export-data-dictionary.csv
.Renviron
sandbox/update-FY24-ICF-good-to-FY26.R
sandbox/update-FY24-ICF-good-to-FY26.R
public-resources/FY26 HMIS-CSV-Machine-Readable-Specifications.xlsx
14 changes: 13 additions & 1 deletion 00_initially_valid_import.R
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,15 @@ if(tolower(tools::file_ext(upload_filepath)) != "zip") {
zipFiles <- zipContents$Name %>% str_replace(".csv", "")

# expected files
expected_files <- unique(cols_and_data_types$File)
expected_files <- unique(cols_and_data_types$CSV)

# get missing files by comparing what we expect with what we got
missing_files <- expected_files[!(expected_files %in% zipFiles)]

# empty files
empty_files <- zip::zip_list(upload_filepath) %>%
fsubset(gsub(".csv", "", filename) %in% expected_files & uncompressed_size <= 1)

### Now check whether the file is hashed, has the expected structure, and contains
# the expected csv files
if(grepl("/", zipContents$Name[1])) {
Expand Down Expand Up @@ -151,6 +155,14 @@ if(tolower(tools::file_ext(upload_filepath)) != "zip") {
If you are not sure how to run a SHA-256 hashed HMIS CSV Export in your HMIS, please contact your HMIS vendor."
)
logMetadata(session, "Unsuccessful upload - not hashed")
} else if(nrow(empty_files) > 0) {
show_invalid_popup(
issueID = 150,
title = "Unsuccessful Upload - Empty file(s)",
popupText = glue::glue("In your HMIS CSV Export, the following files are empty:
{paste(empty_files$filename, collapse=', ')}. All files should at least contain headers.")
)
logMetadata(session, "Unsuccessful upload - not hashed")
} else {
session$userData$initially_valid_import(1)
}
Expand Down
2 changes: 1 addition & 1 deletion 01_get_Export.R
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ logToConsoleFull(session, "Running get export")
list_of_problems <- list()

# for each file in the csv, loop through the file names in the csv
for (file in unique(cols_and_data_types$File)) {
for (file in unique(cols_and_data_types$CSV)) {
#import the csv and save it as a data frame
logToConsole(session, paste0("importing ", file))
assign(file, importFile(upload_filepath=NULL, csvFile=file))
Expand Down
Loading
Loading