Repository navigation
feat: move nightly testing generation to subagent - #42
Conversation
- move generation of nighly testing actions to subagent
- move to subagent
- change to generated workflows based on re-usable actions in common
quickstart-ci-testing repository
- add basic test framework/skill - $validate-skill
- add tests for nightly testing generation subagent
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Michael Dawson <midawson@redhat.com>
🔍 Skill Validation (Advisory — does not block merge)
|
Signed-off-by: Michael Dawson <midawson@redhat.com>
🔍 Skill Validation (Advisory — does not block merge)
|
|
Exploring with codex it seems like the warning are inaccurate. |
YosiElias
left a comment
There was a problem hiding this comment.
LGTM, thanks @mhdawson! The two-stage design (deterministic diff, then the rater on top) looks great!
Curious to see how it holds up on a skill like rh-qs-implement, where the output
varies between executions and additional-instructions becomes more important (likely with targeted mode). Not blocking of course, just worth watching once we get to those skills.
*I've left 2 comments, not blockers, just cleanup.
| "runner_output": str(runner_output.resolve()), | ||
| } | ||
| if args.runner_adapter: | ||
| runner_prompt = render_prompt(args.runner_prompt_file.read_text(), case_replacements) |
There was a problem hiding this comment.
render_prompt never actually replaces anything here, the runner subagent
prompt contains no {...} token, so this returns the template unchanged. The
values reach the agent through the f-string block appended right after, which make it not a blocker, but worth cleaning up.
BTW, the runner_output key is not used.
| "runner_output": str(runner_output.resolve()), | ||
| } | ||
| if args.rater_adapter: | ||
| rater_prompt = render_prompt(args.rater_prompt_file.read_text(), replacements) |
There was a problem hiding this comment.
Same as above, the rater subagent prompt contains no {...} token either, so render_prompt is a no-op and the values arrive via the appended block.
BTW, the rater_output key and runner_outputis not used
- address review comments - tweak invocation scripts to improve reliability Signed-off-by: Michael Dawson <midawson@redhat.com>
|
Pushed commit to address comments |
🔍 Skill Validation (Advisory — does not block merge)
|
move generation of nighly testing actions to subagent
developed/tested primarily with codex and luna, then lightly tested on claude
NOTE: be careful when running, my testing/fixes with with claude used over $25 of my budget. Only by using codex luna for most of the work was it possible to keep the budget impact acceptable.