Skip to content

feat(server): add output rail checking mode to check_async() - #2205

Open
m-misiura wants to merge 2 commits into
NVIDIA-NeMo:developfrom
m-misiura:output_rail
Open

feat(server): add output rail checking mode to check_async()#2205
m-misiura wants to merge 2 commits into
NVIDIA-NeMo:developfrom
m-misiura:output_rail

Conversation

@m-misiura

Copy link
Copy Markdown
Contributor

Description

This PR intends to expose the existing rail_types parameter through the /v1/checks server
endpoint, closing the gap between the Python API (LLMRails.check_async())
and the HTTP API via:

  • adding rail_types field to GuardrailCheckDataInput schema
  • pass it through to checks_async() in the endpoint handler

Related Issue(s)

This PR addresses GH Issue #2063

Verification

  1. Added unit tests, which pass:
pytest tests/server/test_guardrail_checks.py

tests/server/test_guardrail_checks.py ...... [ 26%]
.................                            [100%]

================ 23 passed in 2.51s ================
  1. Tested against a local live server with a following configuration
colang_version: "1.0"

models: []

rails:
  config:
    hf_classifier:
      hap:
        engine: local
        model: ibm-granite/granite-guardian-hap-125m
        threshold: 0.5
        blocked_labels:
          - LABEL_1

    sensitive_data_detection:
      input:
        entities:
          - EMAIL_ADDRESS
          - PHONE_NUMBER
          - CREDIT_CARD
      output:
        entities:
          - EMAIL_ADDRESS
          - PHONE_NUMBER
          - CREDIT_CARD

  input:
    flows:
      - hf classifier check input $classifier=hap
      - mask sensitive data on input

  output:
    flows:
      - hf classifier check output $classifier=hap
      - mask sensitive data on output

and the following requests:

curl -s http://localhost:8000/v1/checks \
  -H "Content-Type: application/json" \
  -d '{
    "model": "test",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "guardrails": {"config_id": "checks_rail_types", "rail_types": ["input"]}
  }'
  

which returns

{"status":"passed","content":"What is the capital of France?"}
curl -s http://localhost:8000/v1/checks \
  -H "Content-Type: application/json" \
  -d '{
    "model": "test",
    "messages": [{"role": "user", "content": "You are stupid and I hate you"}],
    "guardrails": {"config_id": "checks_rail_types", "rail_types": ["input"]}
  }'

which returns

{"status":"blocked","content":"I'm sorry, I can't respond to that.","rail":"hf classifier check input $classifier=hap"}
curl -s http://localhost:8000/v1/checks \
  -H "Content-Type: application/json" \
  -d '{
    "model": "test",
    "messages": [{"role": "user", "content": "Contact me at john@example.com or 555-123-4567"}],
    "guardrails": {"config_id": "checks_rail_types", "rail_types": ["input"]}
  }'

which returns

{"status":"modified","content":"Contact me at <EMAIL_ADDRESS> or <PHONE_NUMBER>"}
curl -s http://localhost:8000/v1/checks \
  -H "Content-Type: application/json" \
  -d '{
    "model": "test",
    "messages": [
      {"role": "user", "content": "What is your email?"},
      {"role": "assistant", "content": "You can reach us at support@company.com"}
    ],
    "guardrails": {"config_id": "checks_rail_types", "rail_types": ["output"]}
  }'

which returns

{"status":"modified","content":"You can reach us at <EMAIL_ADDRESS>"}
curl -s http://localhost:8000/v1/checks \
  -H "Content-Type: application/json" \
  -d '{
    "model": "test",
    "messages": [
      {"role": "user", "content": "Tell me what you think"},
      {"role": "assistant", "content": "You are stupid and I hate you"}
    ],
    "guardrails": {"config_id": "checks_rail_types", "rail_types": ["output"]}
  }'

which returns

{"status":"blocked","content":"I'm sorry, I can't respond to that.","rail":"hf classifier check output $classifier=hap"}
curl -s http://localhost:8000/v1/checks \
  -H "Content-Type: application/json" \
  -d '{
    "model": "test",
    "messages": [
      {"role": "user", "content": "Email me at user@test.com"},
      {"role": "assistant", "content": "Sure, I see your email is user@test.com"}
    ],
    "guardrails": {"config_id": "checks_rail_types", "rail_types": ["input", "output"]}
  }'

which returns

{"status":"modified","content":"Sure, I see your email is <EMAIL_ADDRESS>"}

AI Assistance

  • No AI tools were used.
  • AI tools were used; a human reviewed and can explain every change (tool: Claude Code).

Checklist

  • I've read the CONTRIBUTING guidelines.
  • This PR links to a triaged issue assigned to me.
  • My PR title follows the project commit convention.
  • I've updated the documentation if applicable.
  • [] I've added tests if applicable.
  • I've noted any verification beyond CI and any checks I couldn't run.
  • I did not update generated changelog files manually.
  • I addressed all CodeRabbit, Greptile, and other review comments, or replied with why no change is needed.
  • @mentions of the person or team responsible for reviewing proposed changes.

cc @Pouyanpi @tgasser-nv

@github-actions github-actions Bot added status: needs triage New issues that have not yet been reviewed or categorized. size: M labels Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size: S status: needs triage New issues that have not yet been reviewed or categorized.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant