docs(skills): add library rail contribution and testing guidance - #2223
Draft
Pouyanpi wants to merge 62 commits into
Draft
docs(skills): add library rail contribution and testing guidance#2223Pouyanpi wants to merge 62 commits into
Pouyanpi wants to merge 62 commits into
Conversation
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Decides when a change requires a recorded/VCR test and whether a test belongs in tests/recorded/ or tests/. Core rule: no provider request crossing the wire means nothing to record, so the test is a unit test. Covers the when-required triggers (wire path, public output contract, new provider, model-calling rail, provider-shaped bug), the negative-path ladder, the pure_runtime exception, and dialect-single recorded coverage. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Step-by-step guide for adding a rail under nemoguardrails/library/: archetype exemplars, the rail.py manifest contract, RailOutcome actions, manifest-declared Python package requirements, HTTP rules for vendor backends, the three test layers including both-dialect coverage, and docs and examples. Rules are enumerated concretely because evaluation showed agents comply with what a skill names and miss what it omits; the eval's misses are baked in as named rules (data_retention, dialect-consistent flows, RetryingHTTPClient semantics, docs_url extension, flow-level raise test). Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Reviewer counterpart for rail-contribution PRs: scope and duplicate checks, mechanical gates first (and no re-reviewing what a green gate proves), nine judgment dimensions tests cannot check (privacy honesty, retry semantics verified against retry.py, fail-closed tested not asserted, dialect coverage, docs accuracy, and more), a security pass, and a verdict format that reports clean dimensions as well as findings. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Records why the rail and testing skills are checklists rather than principles (eval finding: agents comply with what is named), the division of labor between skills and mechanical gates, and which test layer owns Colang dialect coverage (gate: structure; unit tests: both-dialect behavior; recorded tests: dialect-single with two named exceptions). Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Gates verify what exists but cannot flag what is absent. Reviewing a live inbound rail PR showed the two most load-bearing protections (a working Colang 2 path and mandatory recorded coverage) were only implied via the author skill, not named as review checks. Name them: manifest presence (an unmanifested rail bypasses every catalog-keyed gate), both-dialect unit tests, the flow-level raise test, the recorded outcome triad, and docs plus example config. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Vendor rails sometimes offer fail-open behavior. Name the review rules: fail-closed by default, the switch belongs in the rail config model rather than an env var because it changes the rail's security posture, fail-open results must be distinguishable from genuine vendor clears (no synthesized vendor payloads) and logged, and the fail-open path needs its own test. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
tests/recorded/AGENTS.md is a thin tripwire for agents editing files in the suite: the placement rule, the marker requirement, the dialect-single rule, and pointers to README.md for mechanics and the recorded-tests skill for the decision procedure, without restating either. The README now states the placement rule in its opening paragraph instead of only inside the negative-paths section, since misplacement is the most common mistake and the rule was easy to skim past. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
The snapshot workflow was documented but its semantics were not, and agents route around snapshot() when unsure. Verified empirically and documented: create/fix rewrites the test file in place (the diff is the expected outcome, with a normal teardown notice on the same run); an unfilled snapshot never passes silently (teardown error serially, failed comparison under xdist); always snapshot the normalized form. The contribution skill's recorded-suite step now states the same rules in brief. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Cassettes are claims by their submitters; an agent or contributor without service access can fabricate one that looks recorded. Name the review rule: rely on the sanitizer fixed-point gate for naive fabrication, ask how external cassettes were recorded, and when unverifiable either re-record as a maintainer or demote to the explicit fake_cassette regime rather than merging them as recorded truth. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
The skill predated the Stack 4c conformance layer. Name test_builtin_rail_conformance.py in the catalog-gates step and the final verification loop, explain the schema snapshot it enforces and how to regenerate it, and state the rule that a contributor fixes the manifest rather than editing a generic gate or extending an exception list. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
The skill was greenfield-only. Add a change-impact matrix for the common case of changing a rail that already exists (action rename, parameter or return changes, config schema updates that need snapshot regeneration, dependency and privacy changes), so a contributor knows which mirrored artifacts must move together before the conformance gates flag the drift. The same matrix is duplicated in nemoguardrails/library/AGENTS.md as the edit-time tripwire; both are kept self-contained because they merge on independent branches, and the contract they encode is frozen. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
clavata factors its client into request.py for a multi-endpoint API, which reads as required layering. Point at nemoguardrails/library/f5/ as the smallest complete vendor rail on the managed HTTP client: a single inline http_call in actions.py returning RailOutcome, no request layer. Copy F5 for a one-endpoint vendor, clavata for a multi-endpoint one. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
The conformance gates pass if a contributor made them pass the wrong way, so add a review dimension that inspects the generic infrastructure, not just the rail: reject edits to the generic gates, challenge new exception-list entries, and treat any rails_config.snapshot.json change as a reviewed public-schema change rather than a mechanical refresh. Also add the conformance gate to the Step 1 mechanical-gate command. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
The skills named only a single raise test and one recorded provider-error, so a contributor could ship a rail that swallows most vendor failures and still be compliant. Require exhaustive error-branch coverage in the UNIT layer (timeout, connection error, status classes, 429 retried and exhausted, malformed payload, missing credential, missing dependency, each crossed with fail-open/closed), enumerated as a checklist because a principle would be skipped. Keep the recorded suite to the recordable real error only, since a synthetic error forced into tests/recorded/ violates the unit-vs-recorded placement rule. Adds review dimension 13 to enforce it. Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
github-actions
Bot
added
status: needs triage
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issue(s)
Verification
AI Assistance
Checklist