feat(rails): declare and validate rail dependencies - #2195
Conversation
github-actions
Bot
added
status: needs triage
Pouyanpi
removed
the
status: needs triage
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Pouyanpi
added
the
status: triaged
Greptile SummaryThis PR makes rail runtime dependencies explicit by introducing a typed
|
| Filename | Overview |
|---|---|
| nemoguardrails/manifests/manifest.py | Adds PythonPackage model with PEP 440/508 validators and integrates it into RailRequirements; uniqueness normalization misses dot-separated package names. |
| nemoguardrails/manifests/validation.py | New validation module; static/runtime package checks, env-var checks, and report aggregation are correctly implemented with appropriate status distinctions. |
| nemoguardrails/cli/rails.py | New CLI sub-command wiring validated rail results to stdout with correct exit-code semantics. |
| nemoguardrails/manifests/catalog.py | Adds config-owner index built during catalog construction; owner_for_config_key lookup is consistent with existing flow-owner pattern. |
| nemoguardrails/library/sensitive_data_detection/actions.py | Migrates presidio/spacy lazy imports to require_python_package; _load_presidio caching is correct and None-guard removals are clean. |
| nemoguardrails/library/injection_detection/actions.py | Replaces module-level yara sentinel with lru_cache'd _load_yara(); stale _check_yara_available() fully removed. |
| nemoguardrails/library/jailbreak_detection/actions.py | Converts silent ImportError suppression into actionable RailDependencyError; removes jailbreak_result=False fallback for missing deps. |
| nemoguardrails/library/hf_classifier/backends.py | Switches pipeline loading to require_python_package + raise_for_missing_package for torch. |
| tests/rails/llm/test_rail_requirements.py | Comprehensive test coverage for static/runtime validation, env-var checks, transitive import failures, and configured-manifest selection. |
| pyproject.toml | Adds packaging>=23.0 as core dependency; removes rail-specific extras and cleans them from the all aggregate. |
Reviews (7): Last reviewed commit: "test(rails): isolate unknown validator e..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
29e08df to
08d9c5c
Compare
b233b7d to
7eed276
Compare
fe1b46c to
7e708b4
Compare
7eed276 to
02e7a2a
Compare
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
📝 WalkthroughWalkthroughThe PR adds declarative rail dependency modeling and validation, migrates catalog rails to structured package requirements, introduces ChangesRail requirements and manifest integration
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant RailsConfig
participant RailCatalog
participant validate_rail_requirements
participant RailValidationReport
CLI->>RailsConfig: load --config
CLI->>RailCatalog: resolve configured rail manifests
CLI->>validate_rail_requirements: validate static or runtime requirements
validate_rail_requirements->>RailValidationReport: aggregate checks and install guidance
RailValidationReport-->>CLI: print statuses and exit code
sequenceDiagram
participant RuntimeV2_x
participant ActionDispatcher
participant ActionFinished
participant co_v2
RuntimeV2_x->>ActionDispatcher: resolve and execute manifest action
ActionDispatcher-->>RuntimeV2_x: action result and ownership status
RuntimeV2_x->>ActionFinished: add is_rail_action marker
ActionFinished->>co_v2: process event
co_v2-->>RuntimeV2_x: omit rail-action output from history
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
nemoguardrails/manifests/manifest.py (1)
333-339: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winUse
packaging.utils.canonicalize_namefor the duplicate check.The manual normalization (
lower().replace("_", "-")) doesn't fully implement PEP 503: it misses dots, so distributions likeruamel.yamlvsruamel-yamlwould not be flagged as duplicates.packagingis already a dependency here.♻️ Proposed fix
+from packaging.utils import canonicalize_name + `@model_validator`(mode="after") def _python_packages_must_be_unique(self) -> "RailRequirements": - normalized = [package.distribution.lower().replace("_", "-") for package in self.python_packages] + normalized = [canonicalize_name(package.distribution) for package in self.python_packages] duplicates = sorted({name for name in normalized if normalized.count(name) > 1})🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nemoguardrails/manifests/manifest.py` around lines 333 - 339, Update RailRequirements._python_packages_must_be_unique to normalize each distribution with packaging.utils.canonicalize_name instead of the manual lower().replace("_", "-") logic, preserving the existing duplicate detection and error behavior.nemoguardrails/manifests/validation.py (1)
90-104: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
RailDependencyErrormessage omits the environment marker; duplicates_package_requirement.
RailDependencyError.__init__re-implements the distribution+version string without the marker suffix that_package_requirement(defined just below) already produces. This can suggest an install command that ignores a package's actual applicability constraint (e.g.python_version < '3.13'), and duplicates logic that should be shared.♻️ Proposed fix
class RailDependencyError(ImportError): def __init__(self, rail_name: str, requirement: PythonPackage) -> None: self.rail_name = rail_name self.requirement = requirement - package = requirement.distribution + (requirement.version or "") + package = _package_requirement(requirement) super().__init__( f"Rail {rail_name!r} requires Python package {package!r}. Install it with: pip install '{package}'" ) + + +def _package_requirement(requirement: PythonPackage) -> str: + package = requirement.distribution + (requirement.version or "") + if requirement.marker is not None: + package = f"{package}; {Marker(requirement.marker)}" + return package(Move
_package_requirementaboveRailDependencyError, or forward-declare, to call it from__init__.)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nemoguardrails/manifests/validation.py` around lines 90 - 104, Update RailDependencyError.__init__ to build the package string via _package_requirement(requirement), ensuring the exception message and install command include any environment marker. Move or otherwise make _package_requirement available before RailDependencyError uses it, and remove the duplicated distribution/version formatting.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/getting-started/installation-guide.mdx`:
- Line 117: Update the catalog rails installation note in the installation guide
to replace “NeMo Guardrails extras” with the canonical wording “extras for the
NVIDIA NeMo Guardrails library,” while preserving the existing guidance about
installing rail-specific Python packages directly.
In `@tests/test_guardrails_ai_e2e_actions.py`:
- Around line 253-255: Configure the patched _load_guard_class in the
validate_guardrails_ai test to raise the expected lookup error via side_effect,
ensuring the unknown validator path still raises GuardrailsAIValidationError.
Keep the existing pytest.raises assertion and validation arguments unchanged.
---
Nitpick comments:
In `@nemoguardrails/manifests/manifest.py`:
- Around line 333-339: Update RailRequirements._python_packages_must_be_unique
to normalize each distribution with packaging.utils.canonicalize_name instead of
the manual lower().replace("_", "-") logic, preserving the existing duplicate
detection and error behavior.
In `@nemoguardrails/manifests/validation.py`:
- Around line 90-104: Update RailDependencyError.__init__ to build the package
string via _package_requirement(requirement), ensuring the exception message and
install command include any environment marker. Move or otherwise make
_package_requirement available before RailDependencyError uses it, and remove
the duplicated distribution/version formatting.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 24a61993-561b-4731-827f-e7a6fd1eb5b2
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (48)
CONTRIBUTING.mddocs/configure-rails/guardrail-catalog/agentic-security.mdxdocs/configure-rails/guardrail-catalog/community/cleanlab.mdxdocs/configure-rails/guardrail-catalog/community/gcp-text-moderations.mdxdocs/configure-rails/guardrail-catalog/community/presidio.mdxdocs/configure-rails/guardrail-catalog/content-safety.mdxdocs/configure-rails/guardrail-catalog/jailbreak-protection.mdxdocs/getting-started/installation-guide.mdxdocs/getting-started/tutorials/jailbreak-detection-heuristics.mdxdocs/reference/cli/index.mdxnemoguardrails/actions/action_dispatcher.pynemoguardrails/cli/__init__.pynemoguardrails/cli/rails.pynemoguardrails/colang/runtime.pynemoguardrails/colang/v2_x/runtime/runtime.pynemoguardrails/library/cleanlab/actions.pynemoguardrails/library/cleanlab/rail.pynemoguardrails/library/content_safety/rail.pynemoguardrails/library/gcp_moderate_text/actions.pynemoguardrails/library/gcp_moderate_text/rail.pynemoguardrails/library/guardrails_ai/actions.pynemoguardrails/library/guardrails_ai/rail.pynemoguardrails/library/hf_classifier/backends.pynemoguardrails/library/hf_classifier/rail.pynemoguardrails/library/injection_detection/actions.pynemoguardrails/library/injection_detection/rail.pynemoguardrails/library/jailbreak_detection/actions.pynemoguardrails/library/jailbreak_detection/rail.pynemoguardrails/library/sensitive_data_detection/actions.pynemoguardrails/library/sensitive_data_detection/rail.pynemoguardrails/llm/filters.pynemoguardrails/manifests/__init__.pynemoguardrails/manifests/catalog.pynemoguardrails/manifests/manifest.pynemoguardrails/manifests/validation.pypyproject.tomltests/cli/test_cli_main.pytests/manifests/test_manifest.pytests/rails/llm/test_rail_requirements.pytests/test_action_dispatcher.pytests/test_filters.pytests/test_guardrails_ai_actions.pytests/test_guardrails_ai_e2e_actions.pytests/test_injection_detection.pytests/test_jailbreak_actions.pytests/test_jailbreak_nim.pytests/test_rail_packaging.pytests/v2_x/test_run_actions.py
|
|
||
| ### Rail Dependencies | ||
|
|
||
| Catalog rails declare their own Python package, model, service, and environment-variable requirements. Rail-specific Python packages are not included in the NeMo Guardrails extras. Install them directly by following the setup instructions in the [Guardrail Catalog](/configure-guardrails/guardrail-catalog). |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use the canonical package name.
Replace “NeMo Guardrails extras” with wording that refers to “the NVIDIA NeMo Guardrails library,” for example: “Rail-specific Python packages are not included in extras for the NVIDIA NeMo Guardrails library.”
As per coding guidelines, “Refer to this package as ‘the NVIDIA NeMo Guardrails library’.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/getting-started/installation-guide.mdx` at line 117, Update the catalog
rails installation note in the installation guide to replace “NeMo Guardrails
extras” with the canonical wording “extras for the NVIDIA NeMo Guardrails
library,” while preserving the existing guidance about installing rail-specific
Python packages directly.
Source: Coding guidelines
| with patch("nemoguardrails.library.guardrails_ai.actions._load_guard_class"): | ||
| with pytest.raises(GuardrailsAIValidationError) as exc_info: | ||
| validate_guardrails_ai(validator_name="completely_unknown_validator", text="Test text") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Make the patched loader fail.
Line 253 replaces the unknown-validator failure with a callable MagicMock, so validate_guardrails_ai() can complete instead of raising. Configure side_effect with a lookup error.
Proposed fix
- with patch("nemoguardrails.library.guardrails_ai.actions._load_guard_class"):
+ with patch(
+ "nemoguardrails.library.guardrails_ai.actions._load_guard_class",
+ side_effect=LookupError("unknown validator"),
+ ):📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| with patch("nemoguardrails.library.guardrails_ai.actions._load_guard_class"): | |
| with pytest.raises(GuardrailsAIValidationError) as exc_info: | |
| validate_guardrails_ai(validator_name="completely_unknown_validator", text="Test text") | |
| with patch( | |
| "nemoguardrails.library.guardrails_ai.actions._load_guard_class", | |
| side_effect=LookupError("unknown validator"), | |
| ): | |
| with pytest.raises(GuardrailsAIValidationError) as exc_info: | |
| validate_guardrails_ai(validator_name="completely_unknown_validator", text="Test text") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_guardrails_ai_e2e_actions.py` around lines 253 - 255, Configure
the patched _load_guard_class in the validate_guardrails_ai test to raise the
expected lookup error via side_effect, ensuring the unknown validator path still
raises GuardrailsAIValidationError. Keep the existing pytest.raises assertion
and validation arguments unchanged.
02e7a2a to
f545de8
Compare
|
Staged Fern docs preview: https://nvidia-preview-pr-2195.docs.buildwithfern.com/nemo/guardrails |
tgasser-nv
left a comment
tgasser-nv
left a comment
There was a problem hiding this comment.
This approach doesn't feel user-friendly, and is a regression on the current experience. I think we need to rethink how this works.
The only advantage is we're reducing the size of the install, but it's a weak argument since removing half a dozen packages from all whose total size if only around 3MB isn't much of a saving. Users would reasonably expect to use Nvidia-vended rails like multilingual Content-safety (using fast-langdetect) or yara regex rules after installing Guardrails.
The disadvantage is that it requires learning a whole new flow around nemoguardrails validate where you have to validate every possible rail you'd use in a Guardrails install/environment. Then copy-paste pip commands for each of these in sequence. These pip commands are less likely to work than a single uv install, since it can't jointly solve all package versions and can silently corrupt the uv core installation. Every time you do a uv sync these will be lost. The uv add command would be better than pip install.
The tension is that the Manifest approach does discovery dynamically by walking the library directories, whereas the package installation has to be from a static file. We can iterate over the rail manifests to create a dependency list for each. Then use tomlkit to write a new section for each rail, and an all-rails which combines all of them into the pyproject.toml:
[project.optional-dependencies]
tracing = [...] # hand-written, untouched
server = [...]
# >>> generated rail extras — run `make gen-rail-extras`; do not edit >>>
content-safety-rail = ["fast-langdetect>=1"]
injection-detection-rail = ["yara-python>=4.5.1,<5"]
sensitive-data-detection-rail = ["presidio-analyzer>=2.2; python_version < '3.13'", "presidio-anonymizer>=2.2; python_version < '3.13'", "spacy>=3.4.4,<4,!=3.7.0; python_version < '3.13'"]
# ... one per rail ...
all-rails = [ ...union of every rail's packages... ]
# <<< generated rail extras <<<
Then if any rail isn't installed the error message would be to run uv add nemoguardrails[<rail-name>]
| @@ -144,12 +160,6 @@ async def jailbreak_detection_model( | |||
| except RuntimeError as e: | |||
| log.error(f"Jailbreak detection model not available: {e}") | |||
| jailbreak_result = False | |||
There was a problem hiding this comment.
Without an except clause here, it'll just raise a ModuleNotFoundError without helping the user solve the problem. We need to give them guidance on what to do to remedy the failure, using the manifest to give the missing modules
2 participants
Description
Make rail runtime dependencies explicit in manifests and validate the requirements
of configured rails without importing optional packages or contacting services.
Add typed Python package declarations, structured validation results, dependency
errors, and
nemoguardrails rails validate. Static validation checks installeddistribution versions, environment markers, and required environment-variable
names;
--runtimeadditionally verifies declared imports. Service and modelrequirements remain declared but unverified.
Migrate built-in rail manifests and lazy runtime imports to the new contract.
Remove rail-specific installation extras so users install each rail's packages
directly. Add the lightweight
packagingcore dependency for standards-compliantPEP 440 version and PEP 508 marker validation.
Adds
uv run nemoguardrails rails validate --config=./examples/configs/gliner gliner: [OK] environment_variable: NVIDIA_API_KEY (set) [INFO] service: GLiNER endpoint (declared; not verified)Impact
provider, model, or network side effects.
RailDependencyErrormessages.allextra.Verification
AI Assistance
Checklist
Summary by CodeRabbit
New Features
rails validateCLI command to check configured rail dependencies, environment variables, and runtime imports.Documentation
Bug Fixes