Skip to content

Experiment: Integrate OpenAI’s Public Beta Decision Model for Chart Selection - #48

Open
YosiElias wants to merge 3 commits into
rh-ai-quickstart:mainfrom
YosiElias:feat/decision-api-chart-selection
Open

YosiElias wants to merge 3 commits into
rh-ai-quickstart:mainfrom
YosiElias:feat/decision-api-chart-selection

Conversation

@YosiElias

@YosiElias YosiElias commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR is the first experiment integrating OpenAI’s new Decision Model, currently in public beta, into the Quickstart Factory harness. It tests the Decision API as the primary chart-selection path in rh-qs-architect, replacing the LLM-based chart-selector subagent for normal runs. The subagent remains as a fallback if the API fails.

The goal is to evaluate this beta model in a real harness workflow and establish a pattern for applying it to other decisions across the quickstart flow.

Cost comparison

The table compares token usage and estimated cost for selecting charts with each approach:

Approach Measured usage Approx. cost
LLM subagent (Sonnet 4.5) 32.8K tokens ~$0.18
Decision Model 3,459 input tokens; no output tokens billed ~$0.00035

In this comparison, the Decision API approach cost about 1/500 as much, and got the same results!

@github-actions

Copy link
Copy Markdown

🔍 Skill Validation (Advisory — does not block merge)

❌ Errors

  • internal link escapes skill directory: ../../../docs/foundation/validation-skill-template.md
  • broken internal link: ./references/ai-architecture-charts.md (file not found)
  • internal link escapes skill directory: ../rh-qs-test-suite/references/workflow-catalog.md

⚠️ Warnings

  • unknown directory: subagents/ (contains 6 files) — agents using the standard skill structure won't discover these files; should this be assets/?

📋 Full report

Use parallel chart checks as the primary selection path to reduce cost and response time, keeping the subagent as an API-failure fallback. This is the first Decision API integration in the
quickstart factory flow.

Signed-off-by: Yossi Elias <yelias@redhat.com>
@YosiElias
YosiElias force-pushed the feat/decision-api-chart-selection branch from f5f5274 to 46ec369 Compare October 18, 2026 12:06
@github-actions

Copy link
Copy Markdown

🔍 Skill Validation (Advisory — does not block merge)

❌ Errors

  • internal link escapes skill directory: ../../../docs/foundation/validation-skill-template.md
  • broken internal link: ./references/ai-architecture-charts.md (file not found)
  • internal link escapes skill directory: ../rh-qs-test-suite/references/workflow-catalog.md

⚠️ Warnings

  • unknown directory: subagents/ (contains 6 files) — agents using the standard skill structure won't discover these files; should this be assets/?

📋 Full report

Add descriptions for all architecture charts to improve capability matching

Signed-off-by: Yossi Elias <yelias@redhat.com>
Signed-off-by: Yossi Elias <yelias@redhat.com>
@github-actions

Copy link
Copy Markdown

🔍 Skill Validation (Advisory — does not block merge)

❌ Errors

  • internal link escapes skill directory: ../../../docs/foundation/validation-skill-template.md
  • broken internal link: ./references/ai-architecture-charts.md (file not found)
  • internal link escapes skill directory: ../rh-qs-test-suite/references/workflow-catalog.md

⚠️ Warnings

  • unknown directory: subagents/ (contains 6 files) — agents using the standard skill structure won't discover these files; should this be assets/?

📋 Full report

@YosiElias
YosiElias requested a review from mhdawson November 16, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant