Add e2e tests for the new Configure LLM Providers modal - #15109
Open
bricestacey wants to merge 9 commits into
Open
Add e2e tests for the new Configure LLM Providers modal#15109bricestacey wants to merge 9 commits into
bricestacey wants to merge 9 commits into
Conversation
|
E2E Tests 🚀 Why these tags?
More on automatic tags from changed files. |
bricestacey
force-pushed
the
new-modal-e2e
branch
from
July 24, 2026 20:36
e376167 to
5609f22
Compare
The testid sits on a zero-size layout wrapper (its dialog container is position:absolute), so Playwright reported it hidden. Scope the modal locator to the visible .positron-modal-dialog-box inside it. See #14818
Drive login/logout off the modal's observed provider section instead of
guessing auto-sign-in from process.env, and add a test.use({ extraEnv })
hook that unsets ANTHROPIC_API_KEY / OPENAI_API_KEY for the launched app
so the API-key providers start disconnected and the modal's connect flow
is genuinely exercised.
See #14818
Take the value setter from the input's own realm and dispatch a change event in addition to input, so React's controlled-input onChange fires reliably across the desktop and web builds. Candidate fix for the web e2e failure where the Connect button stayed disabled after entering an API key. See #14818
bricestacey
force-pushed
the
new-modal-e2e
branch
from
July 31, 2026 13:18
5609f22 to
4bc3245
Compare
The new-modal sign-in suite failed in CI for anthropic-api and openai-api: after filling the key and clicking Connect, the modal stayed on the Connect view showing "Could not validate <provider> API key within 5 seconds", so the Connected view never appeared. The reload in beforeAll was the trigger. Restarting the extension host makes the authentication extension re-probe the cloud credential-chain metadata endpoints (AWS/Azure IMDS at 169.254.169.254, metadata.google.internal). Those are unreachable in the CI container and hang, and the test clicks Connect ~3s later while they are still in flight. The extension host logs show the DNS/connect for the provider host taking 1.0s, 4.7s, and 7.6s in the three failing runs, against the fixed 5s AbortController budget in the key validation. The legacy suite passes the same providers in the same job because it never reloads: its app is idle by then and TLS starts in ~0.1s. The reload was never needed. assistant.newProviderModal is read live each time the Configure Providers command runs, so writing the setting is enough. See #14818
bricestacey
marked this pull request as ready for review
July 31, 2026 20:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft to exercise CI for the new-provider-modal e2e tests. This targets
main, so it includes the commits from #15093 (the modal itself); the new work in this PR is the e2e coverage on top.Summary
Adds e2e (Playwright) coverage for the new "Configure LLM Providers" modal (behind the hidden
assistant.newProviderModalsetting), mirroring the existing legacy-modal sign-in suite with minimal changes:data-testid/data-provider-sectionhooks to the modal's React components so the page object can target rows, views, and the dialog reliably.ModelProviderModalpage object exposing the sameloginModelProvider/logoutModelProvidersurface as the legacy one, driving the new list -> Connect -> Connected navigation.posit-assistant-signin-new-modal.test.ts, which enables the switch and reuses the legacy test body verbatim (sign in -> "Say hello" -> assert a real response -> sign out) across anthropic-api, openai-api, amazon-bedrock, posit-ai, and ms-foundry.Release Notes
New Features
Bug Fixes
Validation Steps
@:assistant
The new suite exercises the sign-in / send-hello / sign-out flow against the new modal for each provider. It needs provider credentials (CI-provided) and a headed browser for the Posit AI OAuth flow. Locally verified: type-check + lint clean, the affected positronAssistant vitest suites pass, and Playwright collects all 5 tests.
See #14818