diff --git a/docs/configure-rails/guardrail-catalog/tool-calling.mdx b/docs/configure-rails/guardrail-catalog/tool-calling.mdx index e527a55116..7f3241fe15 100644 --- a/docs/configure-rails/guardrail-catalog/tool-calling.mdx +++ b/docs/configure-rails/guardrail-catalog/tool-calling.mdx @@ -348,7 +348,7 @@ IORails streams the text response as it arrives, accumulates the tool-call fragm The NVIDIA NeMo Guardrails library includes other tool-related capabilities that are distinct from the IORails tool-calling rails described here: -- [Tools Integration](/integration-with-third-party-libraries/tools-integration) covers LangChain tool passthrough and output-rail validation on the `LLMRails` engine. +- [Tools Integration](/integration-with-third-party-libraries/tools-integration) covers LangChain tool calling and output-rail validation on the `LLMRails` engine. - [Rail types](/about-nemo-guardrails-library/rail-types) describes Colang execution rails, which run actions before and after execution within the `LLMRails` event-driven pipeline. The rails on this page operate at the request and response boundary of the IORails engine and do not execute tools. diff --git a/docs/integration/langchain/langgraph-integration.mdx b/docs/integration/langchain/langgraph-integration.mdx index 24d701de34..c77ee09812 100644 --- a/docs/integration/langchain/langgraph-integration.mdx +++ b/docs/integration/langchain/langgraph-integration.mdx @@ -401,7 +401,7 @@ result = graph.invoke({ ### 1. Passthrough Mode Configuration -For tool calling and complex flows, use `passthrough=True` to maintain the original prompt structure: +Use `passthrough=True` to maintain the original prompt structure for complex flows. ```python guardrails = RunnableRails(config=config, passthrough=True) @@ -427,7 +427,7 @@ guardrails = RunnableRails(config=config, passthrough=True, verbose=True) ### Common Issues -1. **Empty Content with Tool Calls**: When using tools, ensure `passthrough=True` is set. +1. **Empty Content with Tool Calls**: Check that the guardrails configuration does not define dialog flows. 2. **Authorization Errors**: Verify API keys for both main model and safety models. 3. **Configuration Not Found**: Ensure guardrail config paths are correct. diff --git a/docs/integration/langchain/runnable-rails.mdx b/docs/integration/langchain/runnable-rails.mdx index 7cb534340f..9fc08dd23a 100644 --- a/docs/integration/langchain/runnable-rails.mdx +++ b/docs/integration/langchain/runnable-rails.mdx @@ -199,14 +199,14 @@ The `passthrough` parameter controls this behavior. - **`passthrough=False`**: Allows guardrails to modify prompts for enhanced protection. ```python -# Minimal intervention (required for tool calling) +# Minimal intervention guardrails = RunnableRails(config, passthrough=True) # Enhanced guardrails (modifies prompts as needed) guardrails = RunnableRails(config, passthrough=False) ``` -**Tool Calling Requirement**: Set `passthrough=True` for proper tool call handling. +**Tool Calling**: Tool calls work with either `passthrough` setting, as long as the guardrails configuration does not define dialog flows (`user_messages` or `rails.dialog.single_call.enabled`). ### Custom Input/Output Keys @@ -238,13 +238,13 @@ When a guardrail is triggered and predefined messages must be returned instead o ## Tool Calling -`RunnableRails` supports LangChain tool calling with full metadata preservation and streaming. Tool calling requires `passthrough=True` to work properly. +`RunnableRails` supports LangChain tool calling with full metadata preservation and streaming. Tool calling works with or without `passthrough`, as long as the guardrails configuration does not define dialog flows (`user_messages` or `rails.dialog.single_call.enabled`). Configurations that define dialog flows drop tool calls and return only the generated text. -The following steps are required to use tool calling with `RunnableRails`: +To use tool calling with `RunnableRails`: -- Set `passthrough=True` when creating `RunnableRails` instance. - Use `bind_tools()` to attach tools to your model. - Handle tool execution in your application logic. +- Avoid guardrail configurations that define dialog flows if you need tool calls to surface in the response. ### Basic Tool Setup diff --git a/docs/integration/tools-integration.mdx b/docs/integration/tools-integration.mdx index 05d2baea17..795262a273 100644 --- a/docs/integration/tools-integration.mdx +++ b/docs/integration/tools-integration.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "Tools Integration with the NeMo Guardrails Library" sidebar-title: "Tools Integration" -description: "Integrate LangChain tools with guardrails using passthrough mode and output rail validation." +description: "Integrate LangChain tools with guardrails and output rail validation." content: type: "how_to" --- @@ -52,9 +52,11 @@ For detailed information on creating custom tools, refer to the [LangChain Tools ## Configuration Settings -### Passthrough Mode +### Passthrough Mode and Dialog Flows -When using tools with the NeMo Guardrails library, it's recommended to use **passthrough mode**. This mode is essential because: +Tool calls are supported in any guardrails configuration, with or without `passthrough`, as long as the configuration does not define dialog flows. A configuration defines dialog flows when it declares `user_messages` (canonical form) or when `rails.dialog.single_call.enabled` is `true`. Those dialog-flow paths drop tool calls and return only the generated text. + +If your configuration does not use dialog flows, using **passthrough mode** is still recommended because: - Internal NeMo Guardrails library tasks do not require tool use and might provide erroneous results if tools are enabled - It ensures that the LLM can properly handle tool calls and responses diff --git a/docs/reference/engine-feature-support.mdx b/docs/reference/engine-feature-support.mdx index ba9795c954..a63bb0db29 100644 --- a/docs/reference/engine-feature-support.mdx +++ b/docs/reference/engine-feature-support.mdx @@ -135,7 +135,7 @@ A topic-control or jailbreak flow on the output rail is a fallback condition. | Tool-result validation rail | ✓ | ✓ | IORails flow: `tool result validation` | Both engines support passing model tool calls through to the caller and validating tool calls and tool results. -`LLMRails` handles these through the Colang runtime and tool rails. +`LLMRails` handles these through the Colang runtime and tool rails, and surfaces tool calls regardless of the `passthrough` setting, as long as the configuration does not define dialog flows (`user_messages` or `rails.dialog.single_call.enabled`); those flows drop tool calls and return only the generated text. `IORails` validates tool calls and tool results through directional flows: `tool call validation` on the tool-output rail and `tool result validation` on the tool-input rail. Tool calls are returned in the OpenAI-style `tool_calls` field of the response message. diff --git a/nemoguardrails/actions/llm/generation.py b/nemoguardrails/actions/llm/generation.py index 58a7dcaa2b..03245b2f8c 100644 --- a/nemoguardrails/actions/llm/generation.py +++ b/nemoguardrails/actions/llm/generation.py @@ -32,6 +32,7 @@ from nemoguardrails.actions.llm.utils import ( flow_to_colang, get_and_clear_reasoning_trace_contextvar, + get_and_clear_tool_calls_contextvar, get_first_nonempty_line, get_last_bot_intent_event, get_last_user_intent_event, @@ -653,17 +654,10 @@ async def _emit_general_bot_turn( context_updates["bot_thinking"] = reasoning_trace output_events.append(new_event_dict("BotThinking", content=reasoning_trace)) - if self.config.passthrough: - from nemoguardrails.actions.llm.utils import ( - get_and_clear_tool_calls_contextvar, - ) + tool_calls = get_and_clear_tool_calls_contextvar() - tool_calls = get_and_clear_tool_calls_contextvar() - - if tool_calls: - output_events.append(new_event_dict("BotToolCalls", tool_calls=tool_calls)) - else: - output_events.append(new_event_dict("BotMessage", text=text)) + if tool_calls: + output_events.append(new_event_dict("BotToolCalls", tool_calls=tool_calls)) else: output_events.append(new_event_dict("BotMessage", text=text)) diff --git a/nemoguardrails/server/api.py b/nemoguardrails/server/api.py index e585a594f9..93505e5cb4 100644 --- a/nemoguardrails/server/api.py +++ b/nemoguardrails/server/api.py @@ -542,17 +542,29 @@ async def chat_completion(body: GuardrailsChatCompletionRequest, request: Reques detail="thread_id message-history replay is not supported for Colang 2.0.", ) - if (body.tools or body.tool_choice is not None or body.parallel_tool_calls is not None) and ( - llm_rails.config.passthrough is not True or body.stream - ): - raise HTTPException( - status_code=422, - detail=( - "The 'tools', 'tool_choice', and 'parallel_tool_calls' parameters are only " - "supported for non-streaming requests when the guardrails configuration has 'passthrough: true'." - ), - ) + tools_requested = body.tools or body.tool_choice is not None or body.parallel_tool_calls is not None + if tools_requested: + if body.stream: + raise HTTPException( + status_code=422, + detail=( + "The 'tools', 'tool_choice', and 'parallel_tool_calls' parameters are only " + "supported for non-streaming requests." + ), + ) + # Dialog-flow configs (canonical-form user_messages or single_call dialog + # rails) never surface tool calls to the request, so reject explicitly + has_dialog_flows = bool(llm_rails.config.user_messages) or llm_rails.config.rails.dialog.single_call.enabled + if has_dialog_flows: + raise HTTPException( + status_code=422, + detail=( + "The 'tools', 'tool_choice', and 'parallel_tool_calls' parameters are not " + "supported when the guardrails configuration defines dialog flows " + "(user_messages or single_call dialog rails)." + ), + ) try: messages = body.messages or [] if body.guardrails.context: diff --git a/tests/integrations/langchain/test_tool_calling_passthrough_only.py b/tests/integrations/langchain/test_tool_calling_passthrough.py similarity index 52% rename from tests/integrations/langchain/test_tool_calling_passthrough_only.py rename to tests/integrations/langchain/test_tool_calling_passthrough.py index 92c64ba021..d6fca6623f 100644 --- a/tests/integrations/langchain/test_tool_calling_passthrough_only.py +++ b/tests/integrations/langchain/test_tool_calling_passthrough.py @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Test that tool calling ONLY works in passthrough mode.""" +"""Test tool-calling behavior across passthrough and non-dialog-flow general modes.""" from unittest.mock import MagicMock @@ -94,16 +94,12 @@ def config_no_passthrough(): class TestToolCallingPassthroughOnly: - """Test that tool calling only works in passthrough mode.""" + """Test that tool calling works in passthrough mode.""" def test_config_passthrough_true(self, config_passthrough): """Test that passthrough config is correctly set.""" assert config_passthrough.passthrough is True - def test_config_passthrough_false(self, config_no_passthrough): - """Test that non-passthrough config is correctly set.""" - assert config_no_passthrough.passthrough is False - @pytest.mark.asyncio async def test_tool_calls_work_in_passthrough_mode(self, config_passthrough, mock_llm_with_tool_calls): """Test that tool calls create BotToolCalls events in passthrough mode.""" @@ -136,13 +132,54 @@ async def test_tool_calls_work_in_passthrough_mode(self, config_passthrough, moc assert len(result.events) == 1 assert result.events[0]["type"] == "BotToolCalls" stored = result.events[0]["tool_calls"] + assert len(stored) == 1 assert stored[0]["function"]["name"] == "test_tool" assert stored[0]["function"]["arguments"] == {"param": "value"} assert stored[0]["id"] == "call_123" @pytest.mark.asyncio - async def test_tool_calls_ignored_in_non_passthrough_mode(self, config_no_passthrough, mock_llm_with_tool_calls): + async def test_no_tool_calls_creates_bot_message_in_passthrough(self, config_passthrough, mock_llm_with_tool_calls): + """Test that no tool calls creates BotMessage event even in passthrough mode.""" + tool_calls_var.set(None) + + mock_response_no_tools = AIMessage(content="Regular text response") + mock_llm_with_tool_calls.ainvoke.return_value = mock_response_no_tools + mock_llm_with_tool_calls.invoke.return_value = mock_response_no_tools + + generation_actions = LLMGenerationActions( + config=config_passthrough, + llm=LangChainLLMAdapter(mock_llm_with_tool_calls), + llm_task_manager=MagicMock(), + get_embedding_search_provider_instance=MagicMock(return_value=None), + ) + + events = [{"type": "UserMessage", "text": "test"}] + context = {} + + result = await generation_actions.generate_user_intent( + events=events, context=context, config=config_passthrough + ) + + assert len(result.events) == 1 + assert result.events[0]["type"] == "BotMessage" + + def test_llm_rails_integration_passthrough_mode(self, config_passthrough, mock_llm_with_tool_calls): + """Test LLMRails with passthrough mode allows tool calls.""" + rails = LLMRails(config=config_passthrough, llm=LangChainLLMAdapter(mock_llm_with_tool_calls)) + + assert rails.config.passthrough is True + + +class TestToolCallingNonPassthrough: + """Test that tool calling works in non-passthrough mode.""" + + def test_config_no_passthrough_true(self, config_no_passthrough): + """Test that non-passthrough config is correctly set.""" + assert config_no_passthrough.passthrough is False + + @pytest.mark.asyncio + async def test_tool_calls_non_passthrough_mode(self, config_no_passthrough, mock_llm_with_tool_calls): """Test that tool calls are ignored when not in passthrough mode.""" tool_calls = [ { @@ -169,12 +206,19 @@ async def test_tool_calls_ignored_in_non_passthrough_mode(self, config_no_passth ) assert len(result.events) == 1 - assert result.events[0]["type"] == "BotMessage" - assert "tool_calls" not in result.events[0] + assert result.events[0]["type"] == "BotToolCalls" + stored = result.events[0]["tool_calls"] + + assert len(stored) == 1 + assert stored[0]["function"]["name"] == "test_tool" + assert stored[0]["function"]["arguments"] == {"param": "value"} + assert stored[0]["id"] == "call_123" @pytest.mark.asyncio - async def test_no_tool_calls_creates_bot_message_in_passthrough(self, config_passthrough, mock_llm_with_tool_calls): - """Test that no tool calls creates BotMessage event even in passthrough mode.""" + async def test_no_tool_calls_creates_bot_message_in_non_passthrough( + self, config_no_passthrough, mock_llm_with_tool_calls + ): + """Test that no tool calls creates BotMessage event even in non-passthrough mode.""" tool_calls_var.set(None) mock_response_no_tools = AIMessage(content="Regular text response") @@ -182,7 +226,7 @@ async def test_no_tool_calls_creates_bot_message_in_passthrough(self, config_pas mock_llm_with_tool_calls.invoke.return_value = mock_response_no_tools generation_actions = LLMGenerationActions( - config=config_passthrough, + config=config_no_passthrough, llm=LangChainLLMAdapter(mock_llm_with_tool_calls), llm_task_manager=MagicMock(), get_embedding_search_provider_instance=MagicMock(return_value=None), @@ -192,20 +236,106 @@ async def test_no_tool_calls_creates_bot_message_in_passthrough(self, config_pas context = {} result = await generation_actions.generate_user_intent( - events=events, context=context, config=config_passthrough + events=events, context=context, config=config_no_passthrough ) assert len(result.events) == 1 - assert result.events[0]["type"] == "BotMessage" + assert result.events[0]["type"] == "BotToolCalls" + stored = result.events[0]["tool_calls"] + assert len(stored) == 1 + assert stored[0]["function"]["name"] == "test_tool" + assert stored[0]["function"]["arguments"] == {"param": "value"} + assert stored[0]["id"] == "call_123" - def test_llm_rails_integration_passthrough_mode(self, config_passthrough, mock_llm_with_tool_calls): - """Test LLMRails with passthrough mode allows tool calls.""" - rails = LLMRails(config=config_passthrough, llm=LangChainLLMAdapter(mock_llm_with_tool_calls)) + @pytest.mark.asyncio + async def test_tool_calls_with_prompt_mode_non_passthrough(self, config_no_passthrough, mock_llm_with_tool_calls): + """Test that tool calls are ignored when not in passthrough mode.""" + tool_calls = [ + { + "id": "call_123", + "type": "tool_call", + "name": "test_tool", + "args": {"param": "value"}, + } + ] + tool_calls_var.set(tool_calls) - assert rails.config.passthrough is True + generation_actions = LLMGenerationActions( + config=config_no_passthrough, + llm=LangChainLLMAdapter(mock_llm_with_tool_calls), + llm_task_manager=MagicMock(), + get_embedding_search_provider_instance=MagicMock(return_value=None), + ) + + events = [{"type": "UserMessage", "text": "test"}] + context = {} + + result = await generation_actions.generate_user_intent( + events=events, context=context, config=config_no_passthrough + ) + assert len(result.events) == 1 + assert result.events[0]["type"] == "BotToolCalls" + stored = result.events[0]["tool_calls"] + + assert len(stored) == 1 + assert stored[0]["function"]["name"] == "test_tool" + assert stored[0]["function"]["arguments"] == {"param": "value"} + assert stored[0]["id"] == "call_123" + + @pytest.mark.asyncio + async def test_complex_tool_calls_non_passthrough(self, config_no_passthrough, mock_llm_with_tool_calls): + """Test that complex tool calls are ignored when not in passthrough mode.""" + tool_calls = [ + { + "id": "call_123", + "type": "tool_call", + "name": "test_tool", + "args": {"param": "value"}, + } + ] + tool_calls_var.set(tool_calls) + generation_actions = LLMGenerationActions( + config=config_no_passthrough, + llm=LangChainLLMAdapter(mock_llm_with_tool_calls), + llm_task_manager=MagicMock(), + get_embedding_search_provider_instance=MagicMock(return_value=None), + ) - def test_llm_rails_integration_non_passthrough_mode(self, config_no_passthrough, mock_llm_with_tool_calls): - """Test LLMRails without passthrough mode.""" - rails = LLMRails(config=config_no_passthrough, llm=LangChainLLMAdapter(mock_llm_with_tool_calls)) + events = [{"type": "UserMessage", "text": "test"}] + context = {} - assert rails.config.passthrough is False + result = await generation_actions.generate_user_intent( + events=events, context=context, config=config_no_passthrough + ) + assert len(result.events) == 1 + assert result.events[0]["type"] == "BotToolCalls" + stored = result.events[0]["tool_calls"] + assert len(stored) == 1 + assert stored[0]["function"]["name"] == "test_tool" + assert stored[0]["function"]["arguments"] == {"param": "value"} + assert stored[0]["id"] == "call_123" + + @pytest.mark.asyncio + async def test_get_and_clear_tool_calls_called_correctly_non_passthrough( + self, config_no_passthrough, mock_llm_with_tool_calls + ): + generation_actions = LLMGenerationActions( + config=config_no_passthrough, + llm=LangChainLLMAdapter(mock_llm_with_tool_calls), + llm_task_manager=MagicMock(), + get_embedding_search_provider_instance=MagicMock(return_value=None), + ) + + events = [{"type": "UserMessage", "text": "test"}] + context = {} + + result = await generation_actions.generate_user_intent( + events=events, context=context, config=config_no_passthrough + ) + assert len(result.events) == 1 + assert result.events[0]["type"] == "BotToolCalls" + stored = result.events[0]["tool_calls"] + assert len(stored) == 1 + assert stored[0]["function"]["name"] == "test_tool" + assert stored[0]["function"]["arguments"] == {"param": "value"} + assert stored[0]["id"] == "call_123" diff --git a/tests/integrations/langchain/test_tool_calls_event_extraction.py b/tests/integrations/langchain/test_tool_calls_event_extraction.py index f0c40552c5..a16353e727 100644 --- a/tests/integrations/langchain/test_tool_calls_event_extraction.py +++ b/tests/integrations/langchain/test_tool_calls_event_extraction.py @@ -126,7 +126,7 @@ def mock_get_and_clear(): return None with patch( - "nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar", + "nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar", side_effect=mock_get_and_clear, ): chat = TestChat(config, llm_completions=[""]) diff --git a/tests/integrations/langchain/test_tool_output_rails.py b/tests/integrations/langchain/test_tool_output_rails.py index 6233fb5bfb..79744fee1d 100644 --- a/tests/integrations/langchain/test_tool_output_rails.py +++ b/tests/integrations/langchain/test_tool_output_rails.py @@ -85,7 +85,7 @@ async def test_tool_output_rails_basic(): """, ) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat(config, llm_completions=[""]) @@ -204,7 +204,7 @@ async def test_multiple_tool_output_rails(): """, ) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat(config, llm_completions=[""]) diff --git a/tests/server/test_api.py b/tests/server/test_api.py index 9e469dae71..43e60446cd 100644 --- a/tests/server/test_api.py +++ b/tests/server/test_api.py @@ -296,6 +296,8 @@ async def mock_generate_async(*, messages, options, state): mock_rails.generate_async = mock_generate_async mock_rails.config.colang_version = "1.0" mock_rails.config.passthrough = True + mock_rails.config.user_messages = {} + mock_rails.config.rails.dialog.single_call.enabled = False tools = [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}] @@ -318,11 +320,59 @@ async def mock_generate_async(*, messages, options, state): assert captured_options["options"].llm_params["parallel_tool_calls"] is False -def test_chat_completion_rejects_tools_for_non_passthrough_config(): - """Test that tools/tool_choice/parallel_tool_calls are rejected unless passthrough is True.""" +def test_chat_completion_accepts_tools_for_non_passthrough_config(): + """Tools are allowed for a non-passthrough config without no dialog flows.""" mock_rails = AsyncMock() mock_rails.config.colang_version = "1.0" mock_rails.config.passthrough = False + mock_rails.config.user_messages = {} + mock_rails.config.rails.dialog.single_call.enabled = False + + tools = [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}] + + with patch("nemoguardrails.server.api._get_rails", new=AsyncMock(return_value=mock_rails)): + response = client.post( + "/v1/chat/completions", + json={ + "model": "gpt-4o", + "messages": [{"role": "user", "content": "Weather?"}], + "tools": tools, + "guardrails": {"config_id": "with_custom_llm"}, + }, + ) + + assert response.status_code == 200 + + +def test_chat_completion_accepts_tool_choice_non_passthrough(): + """Test that``tool_choice`` alone is allowed for a non-passthrough, no-dialog-flow config.""" + mock_rails = AsyncMock() + mock_rails.config.colang_version = "1.0" + mock_rails.config.passthrough = None + mock_rails.config.user_messages = {} + mock_rails.config.rails.dialog.single_call.enabled = False + + with patch("nemoguardrails.server.api._get_rails", new=AsyncMock(return_value=mock_rails)): + response = client.post( + "/v1/chat/completions", + json={ + "model": "gpt-4o", + "messages": [{"role": "user", "content": "Weather?"}], + "tool_choice": "auto", + "guardrails": {"config_id": "with_custom_llm"}, + }, + ) + + assert response.status_code == 200 + + +def test_chat_completion_rejects_tools_for_dialog_flow_config(): + """Test that tools are rejected when the config defines dialog flows.""" + mock_rails = AsyncMock() + mock_rails.config.colang_version = "1.0" + mock_rails.config.passthrough = False + mock_rails.config.user_messages = {"express greeting": ["hello"]} + mock_rails.config.rails.dialog.single_call.enabled = False tools = [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}] @@ -338,14 +388,16 @@ def test_chat_completion_rejects_tools_for_non_passthrough_config(): ) assert response.status_code == 422 - assert "passthrough" in response.json()["detail"].lower() + assert "dialog flows" in response.json()["detail"].lower() -def test_chat_completion_rejects_tool_choice_without_passthrough(): - """Test that tool_choice alone is rejected for non-passthrough configs.""" +def test_chat_completion_rejects_tool_choice_for_single_call_config(): + """Test that ``tool_choice`` alone is rejected when ``single_call`` dialog rails are enabled.""" mock_rails = AsyncMock() mock_rails.config.colang_version = "1.0" mock_rails.config.passthrough = None + mock_rails.config.user_messages = {} + mock_rails.config.rails.dialog.single_call.enabled = True with patch("nemoguardrails.server.api._get_rails", new=AsyncMock(return_value=mock_rails)): response = client.post( @@ -359,7 +411,7 @@ def test_chat_completion_rejects_tool_choice_without_passthrough(): ) assert response.status_code == 422 - assert "passthrough" in response.json()["detail"].lower() + assert "dialog flows" in response.json()["detail"].lower() def test_chat_completion_rejects_tools_with_streaming_even_in_passthrough(): @@ -367,6 +419,8 @@ def test_chat_completion_rejects_tools_with_streaming_even_in_passthrough(): mock_rails = AsyncMock() mock_rails.config.colang_version = "1.0" mock_rails.config.passthrough = True + mock_rails.config.user_messages = {} + mock_rails.config.rails.dialog.single_call.enabled = False tools = [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}] @@ -383,7 +437,7 @@ def test_chat_completion_rejects_tools_with_streaming_even_in_passthrough(): ) assert response.status_code == 422 - assert "passthrough" in response.json()["detail"].lower() + assert "non-streaming" in response.json()["detail"].lower() def test_chat_completion_returns_tool_calls(): diff --git a/tests/test_bot_tool_call_events.py b/tests/test_bot_tool_call_events.py index 32294c5c53..6de09b9d40 100644 --- a/tests/test_bot_tool_call_events.py +++ b/tests/test_bot_tool_call_events.py @@ -36,7 +36,7 @@ async def test_bot_tool_call_event_creation(): } ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls config = RailsConfig.from_content(config={"models": [], "passthrough": True}) @@ -55,7 +55,7 @@ async def test_bot_message_vs_bot_tool_call_event(): config = RailsConfig.from_content(config={"models": [], "passthrough": True}) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = None chat_text = TestChat(config, llm_completions=["Regular text response"]) @@ -73,7 +73,7 @@ async def test_bot_message_vs_bot_tool_call_event(): } ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat_tools = TestChat(config, llm_completions=[""]) @@ -113,7 +113,7 @@ async def test_tool_calls_bypass_output_rails(): """, ) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat(config, llm_completions=[""]) @@ -138,7 +138,7 @@ async def test_mixed_content_and_tool_calls(): config = RailsConfig.from_content(config={"models": [], "passthrough": True}) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat( @@ -172,7 +172,7 @@ async def test_multiple_tool_calls(): config = RailsConfig.from_content(config={"models": [], "passthrough": True}) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat(config, llm_completions=[""]) @@ -201,7 +201,7 @@ async def test_regular_text_still_goes_through_output_rails(): """, ) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = None chat = TestChat(config, llm_completions=["This is a regular response"]) @@ -230,7 +230,7 @@ async def test_empty_text_without_tool_calls_still_blocked(): """, ) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = None chat = TestChat(config, llm_completions=[""]) diff --git a/tests/test_generation_equivalence.py b/tests/test_generation_equivalence.py index 4e535d5754..e1da6e25b3 100644 --- a/tests/test_generation_equivalence.py +++ b/tests/test_generation_equivalence.py @@ -301,6 +301,70 @@ def test_multi_step_generation(self): "generate_bot_message", ] + def test_tools_not_supported_when_dialog_flows_are_enabled(self): + """Tool calls the model returns are dropped when dialog flows are enabled.""" + config = RailsConfig.from_content( + """ + define user express greeting + "hello" + define flow + user express greeting + bot express greeting + """ + ) + config.rails.dialog.single_call.enabled = True + + llm = RecordingFakeLLM( + llm_responses=[ + LLMResponse( + content=' express greeting\nbot express greeting\n "Hello, there!"', + tool_calls=[ + ToolCall( + id="call_1", + type="function", + function=ToolCallFunction(name="get_weather", arguments={"city": "Boston"}), + ) + ], + ) + ] + ) + chat = TestChat(config, llm=llm) + response = cast( + GenerationResponse, + chat.app.generate( + "hello there!", + options={ + **LOG_OPTS, + "llm_params": { + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the weather", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + }, + }, + } + ] + }, + }, + ), + ) + + assert response.response == "Hello, there!" + assert response.tool_calls is None + assert event_sequence(response) == [ + "UserIntent:express greeting|cache:bot_intent_event,bot_message_event", + "BotIntent:express greeting", + "BotMessage:Hello, there!", + ] + assert llm_tasks(response) == ["generate_intent_steps_message"] + assert llm.calls == [("generate", None)] + assert 'user "hello there!"' in call_prompts(response)[0] + _EMBEDDINGS_COLANG = """ define user express greeting @@ -819,6 +883,43 @@ def test_passthrough_tool_calls(self): {"id": "call_1", "type": "function", "function": {"name": "t", "arguments": {"p": "v"}}} ] + def test_non_passthrough_tool_calls(self): + """Tool calls in the non-passthrough general path also emit ``BotToolCalls``. + + ``_emit_general_bot_turn`` reads ``tool_calls_var`` after generating the + text regardless of which branch produced it, so the rendered-GENERAL + (non-passthrough) branch reaches the same ``BotToolCalls`` outcome as + ``test_passthrough_tool_calls`` -- the divergence is only in the prompt + and the ``stop=["User:"]`` call mode, both pinned here via ``llm.calls``. + """ + config = RailsConfig.from_content(yaml_content="models: []\n") + llm = RecordingFakeLLM( + llm_responses=[ + LLMResponse( + content="", + tool_calls=[ + ToolCall( + id="call_1", + type="function", + function=ToolCallFunction(name="t", arguments={"p": "v"}), + ) + ], + ) + ] + ) + chat = TestChat(config, llm=llm) + response = cast(GenerationResponse, chat.app.generate("call the tool", options=LOG_OPTS)) + + assert event_sequence(response) == ["BotToolCalls:1"] + assert llm_tasks(response) == ["general"] + assert llm.calls == [("generate", ("User:",))] + # The full tool-call payload (id, function name, arguments) is surfaced on + # the public response, not just the count. + assert response.response == "" + assert response.tool_calls == [ + {"id": "call_1", "type": "function", "function": {"name": "t", "arguments": {"p": "v"}}} + ] + def test_single_call_general_no_user_messages(self): """Single-call enabled but no user messages: the converged general bot turn. @@ -878,6 +979,45 @@ def test_dialog_with_passthrough_uses_bot_message_branch(self): assert llm_tasks(response) == ["generate_user_intent", "generate_bot_message"] assert event_sequence(response)[-1] == "BotMessage:passthrough bot answer" + def test_single_call_tools_not_supported(self): + """Test that tool calls are dropped in the single-call path when dialog flows exist.""" + config = RailsConfig.from_content( + """ + define user express greeting + "hello" + define flow + user express greeting + bot express greeting + """ + ) + config.rails.dialog.single_call.enabled = True + llm = RecordingFakeLLM( + llm_responses=[ + LLMResponse( + content=' express greeting\nbot express greeting\n "Hello, there!"', + tool_calls=[ + ToolCall( + id="call_1", + type="function", + function=ToolCallFunction(name="get_weather", arguments={"city": "Boston"}), + ) + ], + ) + ] + ) + chat = TestChat(config, llm=llm) + response = cast(GenerationResponse, chat.app.generate("call the tool", options=LOG_OPTS)) + + assert response.response == "Hello, there!" + assert response.tool_calls is None + assert event_sequence(response) == [ + "UserIntent:express greeting|cache:bot_intent_event,bot_message_event", + "BotIntent:express greeting", + "BotMessage:Hello, there!", + ] + assert llm_tasks(response) == ["generate_intent_steps_message"] + assert llm.calls == [("generate", None)] + class TestGenerateValue: """``generate_value`` action ($var = ...).""" diff --git a/tests/test_input_tool_rails.py b/tests/test_input_tool_rails.py index 405b5b153f..984e274567 100644 --- a/tests/test_input_tool_rails.py +++ b/tests/test_input_tool_rails.py @@ -684,7 +684,7 @@ async def test_bot_tool_calls_event_generated(self): } ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls config = RailsConfig.from_content(config={"models": [], "passthrough": True}) @@ -714,7 +714,7 @@ async def test_multiple_tool_calls_in_bot_tool_calls_event(self): }, ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls config = RailsConfig.from_content(config={"models": [], "passthrough": True}) diff --git a/tests/test_tool_calling_passthrough_integration.py b/tests/test_tool_calling_passthrough_integration.py index 0c7408fd88..d462031375 100644 --- a/tests/test_tool_calling_passthrough_integration.py +++ b/tests/test_tool_calling_passthrough_integration.py @@ -50,7 +50,7 @@ async def test_tool_calls_work_in_passthrough_mode_with_options(self): }, ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat( @@ -86,7 +86,7 @@ async def test_tool_calls_work_in_passthrough_mode_dict_response(self): } ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat( @@ -104,7 +104,7 @@ async def test_tool_calls_work_in_passthrough_mode_dict_response(self): @pytest.mark.asyncio async def test_no_tool_calls_in_passthrough_mode(self): - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = None chat = TestChat( @@ -123,7 +123,7 @@ async def test_no_tool_calls_in_passthrough_mode(self): @pytest.mark.asyncio async def test_empty_tool_calls_in_passthrough_mode(self): - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = [] chat = TestChat(self.passthrough_config, llm_completions=["I understand your request."]) @@ -145,7 +145,7 @@ async def test_tool_calls_with_prompt_mode_passthrough(self): } ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat( @@ -185,7 +185,7 @@ async def test_complex_tool_calls_passthrough_integration(self): }, ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = complex_tool_calls chat = TestChat( @@ -256,7 +256,7 @@ async def test_tool_calls_integration_preserves_other_response_data(self): } ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat( @@ -296,7 +296,7 @@ async def test_tool_calls_with_real_world_examples(self): }, ] - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = realistic_tool_calls chat = TestChat( @@ -332,3 +332,309 @@ async def test_tool_calls_with_real_world_examples(self): @pytest.mark.asyncio async def test_passthrough_config_requirement(self): assert self.passthrough_config.passthrough is True + + +class TestToolCallingNonPassthroughIntegration: + def setup_method(self): + self.non_passthrough_config = RailsConfig.from_content( + colang_content="", + yaml_content=""" + models: [] + passthrough: false + """, + ) + + @pytest.mark.asyncio + async def test_tool_calls_work_in_non_passthrough_mode_with_options(self): + test_tool_calls = [ + { + "name": "get_weather", + "args": {"location": "NYC"}, + "id": "call_123", + "type": "tool_call", + }, + ] + + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: + mock_get_clear.return_value = test_tool_calls + + chat = TestChat( + self.non_passthrough_config, + llm_completions=[""], + ) + + result = await chat.app.generate_async( + messages=[ + { + "role": "user", + "content": "What's the weather in NYC?", + } + ], + options=GenerationOptions(), + ) + + assert isinstance(result, GenerationResponse) + assert result.tool_calls == test_tool_calls + assert len(result.tool_calls) == 1 + assert isinstance(result.response, list) + assert result.response[0]["role"] == "assistant" + assert result.response[0]["content"] == "" + + @pytest.mark.asyncio + async def test_tool_calls_work_in_non_passthrough_mode_dict_response(self): + test_tool_calls = [ + { + "name": "get_weather", + "args": {"location": "London"}, + "id": "call_weather", + "type": "tool_call", + } + ] + + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: + mock_get_clear.return_value = test_tool_calls + + chat = TestChat( + self.non_passthrough_config, + llm_completions=["I'll check the weather for you."], + ) + + result = await chat.app.generate_async(messages=[{"role": "user", "content": "What's the weather like?"}]) + + assert isinstance(result, dict) + assert "tool_calls" in result + assert result["tool_calls"] == test_tool_calls + assert result["role"] == "assistant" + assert result["content"] == "" + + @pytest.mark.asyncio + async def test_no_tool_calls_in_non_passthrough_mode(self): + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: + mock_get_clear.return_value = None + + chat = TestChat( + self.non_passthrough_config, + llm_completions=["Hello! How can I help you today?"], + ) + + result = await chat.app.generate_async( + messages=[{"role": "user", "content": "Hello"}], + options=GenerationOptions(), + ) + + assert isinstance(result, GenerationResponse) + assert result.tool_calls is None + assert "Hello! How can I help" in result.response[0]["content"] + + @pytest.mark.asyncio + async def test_empty_tool_calls_in_non_passthrough_mode(self): + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: + mock_get_clear.return_value = [] + + chat = TestChat(self.non_passthrough_config, llm_completions=["I understand your request."]) + + result = await chat.app.generate_async(messages=[{"role": "user", "content": "Tell me a joke"}]) + + assert isinstance(result, dict) + assert "tool_calls" not in result + assert "understand your request" in result["content"] + + @pytest.mark.asyncio + async def test_tool_calls_with_prompt_mode_non_passthrough(self): + test_tool_calls = [ + { + "name": "search", + "args": {"query": "latest news"}, + "id": "call_prompt", + "type": "tool_call", + } + ] + + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: + mock_get_clear.return_value = test_tool_calls + + chat = TestChat( + self.non_passthrough_config, + llm_completions=["I'll search for that information."], + ) + + result = await chat.app.generate_async(prompt="Search for the latest news", options=GenerationOptions()) + + assert isinstance(result, GenerationResponse) + assert result.tool_calls == test_tool_calls + assert isinstance(result.response, str) + assert result.response == "" + + @pytest.mark.asyncio + async def test_complex_tool_calls_non_passthrough_integration(self): + complex_tool_calls = [ + { + "name": "get_current_weather", + "args": {"location": "San Francisco", "unit": "fahrenheit"}, + "id": "call_weather_001", + "type": "tool_call", + }, + { + "name": "calculate_tip", + "args": {"bill_amount": 85.50, "tip_percentage": 18}, + "id": "call_calc_002", + "type": "tool_call", + }, + { + "name": "web_search", + "args": {"query": "best restaurants near me", "limit": 5}, + "id": "call_search_003", + "type": "tool_call", + }, + ] + + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: + mock_get_clear.return_value = complex_tool_calls + + chat = TestChat( + self.non_passthrough_config, + llm_completions=["I'll help you with the weather, calculate the tip, and find restaurants."], + ) + + result = await chat.app.generate_async( + messages=[ + { + "role": "user", + "content": "I need weather, tip calculation, and restaurant search", + } + ], + options=GenerationOptions(), + ) + + assert isinstance(result, GenerationResponse) + assert result.tool_calls == complex_tool_calls + assert len(result.tool_calls) == 3 + + weather_call = result.tool_calls[0] + assert weather_call["name"] == "get_current_weather" + assert weather_call["args"]["location"] == "San Francisco" + assert weather_call["args"]["unit"] == "fahrenheit" + assert weather_call["id"] == "call_weather_001" + assert weather_call["type"] == "tool_call" + + tip_call = result.tool_calls[1] + assert tip_call["name"] == "calculate_tip" + assert tip_call["args"]["bill_amount"] == 85.50 + assert tip_call["args"]["tip_percentage"] == 18 + assert tip_call["id"] == "call_calc_002" + assert tip_call["type"] == "tool_call" + + search_call = result.tool_calls[2] + assert search_call["name"] == "web_search" + assert search_call["args"]["query"] == "best restaurants near me" + assert search_call["args"]["limit"] == 5 + assert search_call["id"] == "call_search_003" + assert search_call["type"] == "tool_call" + + @pytest.mark.asyncio + async def test_tool_calls_integration_preserves_other_response_data_non_passthrough(self): + test_tool_calls = [ + { + "name": "preserve_test", + "args": {"data": "preserved"}, + "id": "call_preserve", + "type": "tool_call", + } + ] + + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: + mock_get_clear.return_value = test_tool_calls + + chat = TestChat( + self.non_passthrough_config, + llm_completions=["Response with preserved data."], + ) + + result = await chat.app.generate_async( + messages=[{"role": "user", "content": "Test message"}], + options=GenerationOptions(), + ) + + assert isinstance(result, GenerationResponse) + assert result.tool_calls == test_tool_calls + assert result.response is not None + assert result.llm_output is None + assert result.state is None + assert isinstance(result.response, list) + assert len(result.response) == 1 + assert result.response[0]["role"] == "assistant" + assert result.response[0]["content"] == "" + + @pytest.mark.asyncio + async def test_tool_calls_with_real_world_examples_non_passthrough(self): + realistic_tool_calls = [ + { + "name": "get_weather", + "args": {"location": "London"}, + "id": "call_JMTxzsfy21izMf248MHZvj3G", + "type": "tool_call", + }, + { + "name": "add", + "args": {"a": 15, "b": 27}, + "id": "call_INoaqHesFOrZdjHynU78qjX4", + "type": "tool_call", + }, + ] + + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: + mock_get_clear.return_value = realistic_tool_calls + + chat = TestChat( + self.non_passthrough_config, + llm_completions=["I'll get the weather in London and add 15 + 27 for you."], + ) + + result = await chat.app.generate_async( + messages=[ + { + "role": "user", + "content": "What's the weather in London and what's 15 + 27?", + } + ], + options=GenerationOptions(), + ) + + assert isinstance(result, GenerationResponse) + assert result.tool_calls == realistic_tool_calls + + weather_call = result.tool_calls[0] + assert weather_call["name"] == "get_weather" + assert weather_call["args"] == {"location": "London"} + assert weather_call["id"] == "call_JMTxzsfy21izMf248MHZvj3G" + assert weather_call["type"] == "tool_call" + + add_call = result.tool_calls[1] + assert add_call["name"] == "add" + assert add_call["args"] == {"a": 15, "b": 27} + assert add_call["id"] == "call_INoaqHesFOrZdjHynU78qjX4" + assert add_call["type"] == "tool_call" + + @pytest.mark.asyncio + async def test_passthrough_config_requirement_non_passthrough(self): + assert self.non_passthrough_config.passthrough is False + + @pytest.mark.asyncio + async def test_tool_calls_fail_in_dialog_flows_non_passthrough(self): + config = RailsConfig.from_content( + colang_content="", + yaml_content=""" + models: [] + passthrough: false + dialog: + user_messages: true + single_call: + enabled: true + """, + ) + chat = TestChat(config, llm_completions=["I'll check the weather for you."]) + result = await chat.app.generate_async(messages=[{"role": "user", "content": "What's the weather like?"}]) + assert isinstance(result, dict) + assert "tool_calls" not in result + assert result["role"] == "assistant" + assert result["content"] == "I'll check the weather for you." diff --git a/tests/test_tool_calling_passthrough_only.py b/tests/test_tool_calling_passthrough_only.py index 76c458369c..09f4afa06b 100644 --- a/tests/test_tool_calling_passthrough_only.py +++ b/tests/test_tool_calling_passthrough_only.py @@ -13,6 +13,15 @@ # See the License for the specific language governing permissions and # limitations under the License. +"""Test tool-calling behavior across passthrough and non-dialog-flow general modes. + +Tool calls surface as ``BotToolCalls`` in any config that reaches +``_emit_general_bot_turn`` -- i.e. any config with no dialog flows (no +``user_messages`` and no ``single_call`` dialog rails) -- whether or not +``passthrough`` is set. Only dialog-flow configs (covered separately in +``tests/test_generation_equivalence.py``) drop tool calls. +""" + from unittest.mock import MagicMock import pytest @@ -92,6 +101,8 @@ def config_no_passthrough(): class TestToolCallingPassthroughOnly: + """Tool calls surface in any config without dialog flows, passthrough or not.""" + def test_config_passthrough_true(self, config_passthrough): assert config_passthrough.passthrough is True @@ -135,13 +146,24 @@ async def test_tool_calls_work_in_passthrough_mode(self, config_passthrough, moc assert stored[0]["id"] == "call_123" @pytest.mark.asyncio - async def test_tool_calls_ignored_in_non_passthrough_mode(self, config_no_passthrough, mock_llm_with_tool_calls): + async def test_tool_calls_work_in_non_passthrough_mode_without_dialog_flows( + self, config_no_passthrough, mock_llm_with_tool_calls + ): + """Tool calls still surface as ``BotToolCalls`` without passthrough. + + ``config_no_passthrough`` has no dialog flows either (no + ``user_messages``, ``single_call`` disabled), so ``generate_user_intent`` + still reaches ``_emit_general_bot_turn``, which reads ``tool_calls_var`` + regardless of ``passthrough``. + """ tool_calls = [ { "id": "call_123", - "type": "tool_call", - "name": "test_tool", - "args": {"param": "value"}, + "type": "function", + "function": { + "name": "test_tool", + "arguments": {"param": "value"}, + }, } ] tool_calls_var.set(tool_calls) @@ -161,8 +183,12 @@ async def test_tool_calls_ignored_in_non_passthrough_mode(self, config_no_passth ) assert len(result.events) == 1 - assert result.events[0]["type"] == "BotMessage" - assert "tool_calls" not in result.events[0] + assert result.events[0]["type"] == "BotToolCalls" + stored = result.events[0]["tool_calls"] + assert len(stored) == 1 + assert stored[0]["function"]["name"] == "test_tool" + assert stored[0]["function"]["arguments"] == {"param": "value"} + assert stored[0]["id"] == "call_123" @pytest.mark.asyncio async def test_no_tool_calls_creates_bot_message_in_passthrough(self, config_passthrough): diff --git a/tests/test_tool_calls_event_extraction.py b/tests/test_tool_calls_event_extraction.py index 5a30c29ab3..873cc5b92a 100644 --- a/tests/test_tool_calls_event_extraction.py +++ b/tests/test_tool_calls_event_extraction.py @@ -114,7 +114,7 @@ def mock_get_and_clear(): return None with patch( - "nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar", + "nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar", side_effect=mock_get_and_clear, ): chat = TestChat(config, llm_completions=[""]) diff --git a/tests/test_tool_output_rails.py b/tests/test_tool_output_rails.py index 464a9f32e8..f5a33b59e3 100644 --- a/tests/test_tool_output_rails.py +++ b/tests/test_tool_output_rails.py @@ -92,7 +92,7 @@ async def test_tool_output_rails_basic(): """, ) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat(config, llm_completions=[""]) @@ -191,7 +191,7 @@ async def test_multiple_tool_output_rails(): """, ) - with patch("nemoguardrails.actions.llm.utils.get_and_clear_tool_calls_contextvar") as mock_get_clear: + with patch("nemoguardrails.actions.llm.generation.get_and_clear_tool_calls_contextvar") as mock_get_clear: mock_get_clear.return_value = test_tool_calls chat = TestChat(config, llm_completions=[""])