feat: add NVIDIA NIM provider support - #635
Open
mmustafasenoglu wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
There are runtime-breaking issues in the NVIDIA provider wiring (notably an invalid import for NVIDIA_BASE_URL) and missing streaming event emission that will impair the frontend experience.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds NVIDIA NIM as a first-class LLM provider (OpenAI-compatible chat-completions) so the app can generate code using NVIDIA Nemotron 3.5 Flash models, including wiring the NVIDIA API key from frontend settings through the backend pipeline/provider factory.
Changes:
- Frontend: added
nvidiaApiKeyto settings, new NVIDIA key input in Settings, and added Nemotron 3.5 Flash variants to the model list/descriptions. - Backend: added NVIDIA models to
Llm/provider mapping, plumbednvidia_api_keythrough request extraction → runner, and added an NVIDIA provider session + factory branch usingAsyncOpenAI(base_url=https://integrate.api.nvidia.com/v1). - Ops/evals/cost: added
NVIDIA_API_KEYconfig, pricing entry, and eval-time key checks for NVIDIA models.
File summaries
| File | Description |
|---|---|
| frontend/src/types.ts | Extends Settings shape with nvidiaApiKey. |
| frontend/src/lib/models.ts | Adds Nemotron 3.5 Flash variants to the frontend model enum + descriptions. |
| frontend/src/components/settings/SettingsTab.tsx | Adds NVIDIA API key input to the Settings UI. |
| frontend/src/App.tsx | Initializes nvidiaApiKey in default settings state. |
| backend/routes/model_choice_sets.py | Introduces NVIDIA_ONLY_MODELS for variant selection fallbacks. |
| backend/routes/generate_code.py | Plumbs nvidia_api_key through extraction, model selection, and runner invocation; updates missing-key error message. |
| backend/llm.py | Adds NVIDIA models to Llm, MODEL_PROVIDER, and NVIDIA_MODELS membership set. |
| backend/evals/core.py | Adds NVIDIA key checks for eval runs and passes the key into the Agent runner. |
| backend/costs/pricing.py | Adds estimated pricing for Nemotron 3.5 Flash for cost tracking. |
| backend/config.py | Adds NVIDIA_API_KEY env var to config. |
| backend/agent/providers/nvidia/provider.py | New NVIDIA provider implementation using OpenAI-compatible chat.completions streaming + tool calls + cost tracking. |
| backend/agent/providers/nvidia/init.py | Exposes NVIDIA provider session + tool serializer. |
| backend/agent/providers/factory.py | Adds NVIDIA provider branch in the provider-session factory. |
| backend/agent/providers/init.py | Re-exports NVIDIA provider symbols from the providers package. |
| backend/agent/engine.py | Threads nvidia_api_key into provider session creation. |
Review details
Suppressed comments (1)
backend/routes/model_choice_sets.py:81
NVIDIA_ONLY_MODELSomits the base and medium Nemotron variants even though they exist inLlmand are exposed in the frontend model list. This means variant-cycling with only an NVIDIA key will never select the base/medium variants.
NVIDIA_ONLY_MODELS = (
Llm.NVIDIA_NEMOTRON_3_5_FLASH_HIGH,
Llm.NVIDIA_NEMOTRON_3_5_FLASH_LOW,
)
- Files reviewed: 15/15 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if not nvidia_api_key: | ||
| raise Exception("NVIDIA API key is missing.") | ||
|
|
||
| from agent.providers.nvidia import NVIDIA_BASE_URL |
Comment on lines
+195
to
+198
| state = _NvidiaParseState() | ||
| stream = await self._client.chat.completions.create(**params) # type: ignore | ||
| async for chunk in stream: # type: ignore | ||
| _parse_chunk(chunk, state, on_event) |
Comment on lines
+136
to
+139
| try: | ||
| args = json.loads(entry["arguments"]) | ||
| except (json.JSONDecodeError, TypeError): | ||
| args = {"INVALID_JSON": entry["arguments"]} |
Comment on lines
12
to
+14
| # Image generation (optional) | ||
| REPLICATE_API_KEY = os.environ.get("REPLICATE_API_KEY", None) | ||
| NVIDIA_API_KEY = os.environ.get("NVIDIA_API_KEY", None) |
mmustafasenoglu
force-pushed
the
feat/nvidia-nim-provider
branch
2 times, most recently
from
September 2, 2026 11:15
73d1cc9 to
1aab91e
Compare
- Add NvidiaProviderSession implementing ProviderSession protocol with chat completions streaming, tool calling, and cost tracking - Strip image content from messages since Nemotron 3.5 Flash is text-only - Add stream_options for proper usage tracking - Add nvidia_api_key parameter across the full pipeline: frontend settings, WebSocket extraction, model selection, and agent engine - Add NVIDIA model enums (NEMOTRON_3_5_FLASH with LOW/MEDIUM/HIGH thinking variants) to both backend LLM and frontend model lists - Add NVIDIA_API_KEY env var support in config and pricing data - Update evals runner to pass nvidia_api_key to Agent - Fix missing MODEL_PROVIDER import in evals/core.py by using the new NVIDIA_MODELS convenience set
mmustafasenoglu
force-pushed
the
feat/nvidia-nim-provider
branch
from
September 2, 2026 12:19
1aab91e to
3dc2498
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds full NVIDIA NIM (NVIDIA Inference Microservice) support as a first-class model provider, enabling users to run code generation with Nemotron 3.5 Flash models via NVIDIA's OpenAI-compatible API.
Changes
Backend
Frontend
Testing
ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL Command "tsc" not found
Did you mean "pnpm test"? passes cleanly
ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL Command "build" not found succeeds