Add APIUsage tracking to extChatEndpoint#5102
Conversation
fix(ext-endpoint): propagate usage from DataPart stream in ExtensionContributedChatEndpoint The extension-contributed chat endpoint previously returned a hardcoded zeroed-out usage object on every successful response, causing the Copilot Chat context-window meter to never update when routing through third-party language model providers. Add a `usageFromDataPart()` helper that checks incoming `LanguageModelDataPart` chunks for a native usage shape (`prompt_tokens`, `completion_tokens`, `total_tokens` — all numbers) without keying on any MIME type. This mirrors the shape-based check already present in the non-provider (internal) streaming path. In `makeChatRequest2()`: - Declare `usage: APIUsage | undefined` before the stream loop. - After handling known MIME types (StatefulMarker, ContextManagement), call `usageFromDataPart()` on every DataPart and capture the result. - Return `usage ?? <zeroed fallback>` so existing behaviour is preserved when no usage chunk arrives. No new MIME branch is introduced; any provider that emits a DataPart whose JSON payload matches the OpenAI usage shape will automatically populate the meter.
|
@microsoft-github-policy-service agree company="CLDMV" |
There was a problem hiding this comment.
Pull request overview
This PR updates the extension-contributed chat endpoint so usage data emitted by third-party language model providers can flow back into Copilot Chat’s normal response pipeline instead of always being replaced with zeros. It fits into the endpoint layer that adapts VS Code LanguageModelChat providers to the extension’s internal ChatResponse/usage model.
Changes:
- Adds a helper in
extChatEndpoint.tsto detect OpenAI-shaped usage payloads insideLanguageModelDataPartchunks. - Captures usage while consuming the provider response stream and returns it on successful responses.
- Keeps the previous zeroed fallback when no usage chunk is present.
…arsing - Extend usageFromDataPart to include completion_tokens_details when present in the payload, preventing reasoning/prediction token breakdown from being silently dropped for extension-contributed models. - Add extChatEndpoint.spec.ts with provider-specific regression tests covering shape-based usage chunk detection and completion_tokens_details passthrough, mirroring the coverage pattern of the regular SSE stream.
|
Thanks for the contribution! This repository has been archived because the project has moved into the main VS Code repository. Could you please reopen/recreate this PR against: We’ll continue reviewing contributions there. Thanks! |
Token usage reported by third-party language models is currently thrown out entirely — unceremoniously discarded before it can reach the context-window meter. This patch corrects that oversight by plumbing the usage data already present in the LLM response stream through to the appropriate locations within
vscode-copilot-chat, so the meter finally gets the numbers it deserves.fix(ext-endpoint): propagate usage from DataPart stream in ExtensionContributedChatEndpoint
The extension-contributed chat endpoint previously returned a hardcoded zeroed-out usage object on every successful response, causing the Copilot Chat context-window meter to never update when routing through third-party language model providers.
Add a
usageFromDataPart()helper that checks incomingLanguageModelDataPartchunks for a native usage shape (prompt_tokens,completion_tokens,total_tokens— all numbers) without keying on any MIME type. This mirrors the shape-based check already present in the non-provider (internal) streaming path.In
makeChatRequest2():usage: APIUsage | undefinedbefore the stream loop.usageFromDataPart()on every DataPart and capture the result.usage ?? <zeroed fallback>so existing behaviour is preserved when no usage chunk arrives.No new MIME branch is introduced; any provider that emits a DataPart whose JSON payload matches the OpenAI usage shape will automatically populate the meter.