Skip to content

feat: move nightly testing generation to subagent - #42

Merged
mhdawson merged 3 commits into
rh-ai-quickstart:mainfrom
mhdawson:test-nightly-skill
Sep 29, 2026
Merged

mhdawson merged 3 commits into
rh-ai-quickstart:mainfrom
mhdawson:test-nightly-skill

Conversation

@mhdawson

Copy link
Copy Markdown
Member
  • move generation of nighly testing actions to subagent

    • move to subagent
    • change to generated workflows based on re-usable actions in common quickstart-ci-testing repository
    • add basic test framework/skill - $validate-skill
    • add tests for nightly testing generation subagent
  • developed/tested primarily with codex and luna, then lightly tested on claude

NOTE: be careful when running, my testing/fixes with with claude used over $25 of my budget. Only by using codex luna for most of the work was it possible to keep the budget impact acceptable.

- move generation of nighly testing actions to subagent
  - move to subagent
  - change to generated workflows based on re-usable actions in common
    quickstart-ci-testing repository
  - add basic test framework/skill - $validate-skill
  - add tests for nightly testing generation subagent

Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: Michael Dawson <midawson@redhat.com>
@mhdawson
mhdawson requested a review from YosiElias September 18, 2026 15:21
@github-actions

Copy link
Copy Markdown

🔍 Skill Validation (Advisory — does not block merge)

⚠️ Warnings

  • unknown directory: subagents/ (contains 2 files) — agents using the standard skill structure won't discover these files; should this be assets/?
  • unexpected file at root: output-templates.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unexpected file at root: reasoning-guardrails.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unknown directory: runners/ (contains 8 files) — agents using the standard skill structure won't discover these files; should this be assets/?
  • unexpected file at root: spec-template.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unknown directory: subagents/ (contains 4 files) — agents using the standard skill structure won't discover these files; should this be assets/?

📋 Full report

Signed-off-by: Michael Dawson <midawson@redhat.com>
@github-actions

Copy link
Copy Markdown

🔍 Skill Validation (Advisory — does not block merge)

⚠️ Warnings

  • unknown directory: subagents/ (contains 2 files) — agents using the standard skill structure won't discover these files; should this be assets/?
  • unexpected file at root: output-templates.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unexpected file at root: reasoning-guardrails.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unknown directory: runners/ (contains 8 files) — agents using the standard skill structure won't discover these files; should this be assets/?
  • unexpected file at root: spec-template.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unknown directory: subagents/ (contains 4 files) — agents using the standard skill structure won't discover these files; should this be assets/?

📋 Full report

@mhdawson

Copy link
Copy Markdown
Member Author

Exploring with codex it seems like the warning are inaccurate.

@YosiElias YosiElias left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @mhdawson! The two-stage design (deterministic diff, then the rater on top) looks great!

Curious to see how it holds up on a skill like rh-qs-implement, where the output
varies between executions and additional-instructions becomes more important (likely with targeted mode). Not blocking of course, just worth watching once we get to those skills.

*I've left 2 comments, not blockers, just cleanup.

"runner_output": str(runner_output.resolve()),
}
if args.runner_adapter:
runner_prompt = render_prompt(args.runner_prompt_file.read_text(), case_replacements)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

render_prompt never actually replaces anything here, the runner subagent
prompt contains no {...} token, so this returns the template unchanged. The
values reach the agent through the f-string block appended right after, which make it not a blocker, but worth cleaning up.

BTW, the runner_output key is not used.

"runner_output": str(runner_output.resolve()),
}
if args.rater_adapter:
rater_prompt = render_prompt(args.rater_prompt_file.read_text(), replacements)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, the rater subagent prompt contains no {...} token either, so render_prompt is a no-op and the values arrive via the appended block.

BTW, the rater_output key and runner_outputis not used

- address review comments
- tweak invocation scripts to improve reliability

Signed-off-by: Michael Dawson <midawson@redhat.com>
@mhdawson

Copy link
Copy Markdown
Member Author

Pushed commit to address comments

@github-actions

Copy link
Copy Markdown

🔍 Skill Validation (Advisory — does not block merge)

⚠️ Warnings

  • unknown directory: subagents/ (contains 2 files) — agents using the standard skill structure won't discover these files; should this be assets/?
  • unexpected file at root: output-templates.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unexpected file at root: reasoning-guardrails.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unknown directory: runners/ (contains 8 files) — agents using the standard skill structure won't discover these files; should this be assets/?
  • unexpected file at root: spec-template.md — if agents need this file, move it into references/ or assets/ as appropriate; otherwise remove it to avoid unnecessary context window usage
  • unknown directory: subagents/ (contains 4 files) — agents using the standard skill structure won't discover these files; should this be assets/?

📋 Full report

@mhdawson
mhdawson merged commit 39f2f4d into rh-ai-quickstart:main Sep 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants