feat: Enhance config validation (Resolves #33) - #42
Conversation
Resolves draeger-lab#33. Replaces verbose YAML traversal and USER/_USER_ keyword logic with a recursive dictionary schema validator. Adds support for type enforcement, range checking, path validation, and sensible defaults.
There was a problem hiding this comment.
Pull request overview
This PR adds a schema-driven, dictionary-based configuration validation utility intended to centralize and simplify config validation logic, in line with Issue #33.
Changes:
- Added
src/specimen/util/config_validator.pyintroducingSchemaField,validate_config, andConfigValidationError. - Updated
src/specimen/util/__init__.pyto expose the new validation utilities at thespecimen.utilpackage level.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| src/specimen/util/config_validator.py | New schema-based config validation implementation with type/constraint/path checks and legacy keyword handling. |
| src/specimen/util/init.py | Re-exports new validator symbols from the util package namespace. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Fixes nested USER missing value check logic, cleans up type error formatting, removes unused Callable import, and resolves export ambiguity in __init__.py.
|
Thanks for the thorough review! I've pushed updates to address all the feedback:
Let me know if anything else is needed! |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (4)
src/specimen/util/config_validator.py:97
- This “Clean up legacy 'USER' keyword” block is unreachable because the earlier
if value is None or value == "_USER_" or value == "USER": ... continuealready handles allvalue == "USER"cases. Removing this dead code will simplify control flow and prevent confusion when modifying the placeholder logic later.
# Clean up legacy 'USER' keyword
if value == "USER":
logger.warning(
f"Deprecated 'USER' keyword found at '{current_path}'. Using default: {rule.default}"
)
value = rule.default
src/specimen/util/config_validator.py:105
boolis a subclass ofintin Python, so this numeric bounds check will also run for boolean values (andexpected_type=intwould accept booleans). Excludingboolhere avoids surprising validation behavior when the schema contains integer fields and the YAML providestrue/false.
if isinstance(value, (int, float)):
src/specimen/util/config_validator.py:90
- The legacy placeholder for “required user input” in this repo is
__USER__(seespecimen/util/set_up.py:309and the example YAMLs), but this validator only checks for_USER_. That means__USER__would pass through as a normal string and likely reach downstream code. Also, whenvalue == "_USER_"and the field is required, the code raises before the deprecation log/specific message, so users won’t see the intended guidance. Consider normalizing both__USER__and_USER_up front and always raising a consistent error for them.
This issue also appears in the following locations of the same file:
- line 91
- line 105
# 1. Check Required fields (Replaces _USER_ and USER logic)
if value is None or value == "_USER_" or value == "USER":
if rule.required:
raise ConfigValidationError(
f"Missing required configuration parameter: '{current_path}'"
src/specimen/util/init.py:3
- The PR description states that this PR “replac[es] the previously verbose YAML traversal” for config validation, but the current entrypoints still call
specimen.util.set_up.validate_config(e.g.specimen/hqtb/workflow.py:49,specimen/cmpb/workflow.py:48-49). Exportingconfig_validatorfromspecimen.utildoesn’t switch any existing code over, so Issue #33 isn’t fully resolved by these changes alone.
__all__ = ["util", "set_up", "config_validator"]
from . import util, set_up, config_validator
Description
This PR introduces a robust, dictionary-based configuration validation system to resolve Issue #33. It centralizes the validation logic into a new
config_validator.pyutility, replacing the previously verbose YAML traversal[cite: 2].Key Changes
SchemaFieldandvalidate_configto allow developers to define expected configuration structures cleanly.USER->Nonefallback with a system that warns the user and automatically applies sensible developer-defined defaults[cite: 2]._USER_keyword now throws an explicitConfigValidationErrorfor missing required parameters[cite: 2].min_val/max_val) and if string parameters match an expected list (allowed_values)[cite: 2].is_pathandpath_must_existflags to strictly verify that provided file/directory paths actually exist on the system[cite: 2].Testing
Tested locally to ensure accurate error throwing for missing parameters and out-of-bound variables, as well as successful fallbacks for deprecated keywords.