feat: per-variant cost attribution via variantCost WebSocket message - #632
Open
AseemPrasad wants to merge 2 commits into
Open
feat: per-variant cost attribution via variantCost WebSocket message#632AseemPrasad wants to merge 2 commits into
AseemPrasad wants to merge 2 commits into
Conversation
…red output mode Adds three inter-related improvements to the agentic tool-call runtime, all gated behind the new AGENT_STRUCTURED_OUTPUT opt-in flag.
Author
|
@abi would love to get this contribution reviewed and incorporated.. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds per-variant cost attribution to the WebSocket pipeline. After each variant's agent run completes,
the backend now emits a
variantCostWebSocket message carrying the USD cost. The frontend stores it on theVariantobject, making it available for live display, the eval compare page, and any future cost UI.This is the minimal, highest-value slice of PR 2 — it works with every existing model today (including all
GPT-5, Claude Opus 4/5, and Gemini variants) and requires no changes to pricing tables.
New model enum variants (GPT-5.6, Claude Opus 4.10, etc.) and their pricing entries are tracked as TODOs
pending provider API confirmation.
What Changed
Backend
agent/engine.py_last_cost_usd: float | Nonefield andlast_cost_usdproperty toAgentEngine.run(), the cost is captured fromsession.total_cost_usd()before the sessionfinallyblockcloses it. Callers can always read
runner.last_cost_usdafterrun()returns.routes/generate_code.pyAdded
"variantCost"to theMessageTypeLiteral union.In
_run_variant(), afterrunner.run()completes:Note: variantCost is sent before variantComplete. If the variant failed with an error, no cost message is sent
(the provider session may not have a final cost in that case).
────────────────────────────────────────────────────────────────────────────────
Backward Compatibility
doesn't handle it.
requires both (a) a token-usage record from the provider and (b) a ModelPricing entry for the API model
name. All currently defined models have pricing entries.
────────────────────────────────────────────────────────────────────────────────
Testing Notes