[draft][agent_sys] task package init - #142
Draft
dorado269 wants to merge 19 commits into
Draft
Conversation
dorado269
force-pushed
the
dev.yihou.aiopt.task_package
branch
from
September 1, 2026 03:04
920ee3f to
3613481
Compare
`single_real_task` proved one deployment and names its model in the task, the brief and the validator. This package is that shape with the model taken out of it: `main.yaml` and `steps/deploy.yaml` name no model, no architecture and no port, and the plain instruction the series book asks for is the agent's actual input (`$E2E_INSTRUCTION`). `check_deploy_kit` is `check_packup_shape` plus four rules, each a fault observed in a real kit rather than a rule liked in the abstract: an `Expected output` section (it is the only criterion the reproduction check has), machine-readable `results/` (stage 2 consumes files, not paragraphs), `environment.md` naming GPU architecture / image / model, and a refusal of a kit whose evidence shows the model registered under a filesystem path. Both non-vacuity controls were run before any agent run, against kits nobody wrote for this validator: the human-written glm53flash kit fails with five specific faults, and the `b6` kit that *passed* `single_real_task`'s check fails on exactly one rule -- the served name, `/data/<user>/.../Qwen3.6-27B`, which that run had itself recorded as a trap. `agent-sys show` loads the package: 2 tasks, 1 root, 2 output validators. Signed-off-by: yihou <yihou@amd.com>
…ent-sys` `PATH` without `~/.local/bin`; `HOME=/opt/spur`, so `claude` finds no credentials and reports "Not logged in"; and `extensions.preciousObjects` landing in a worktree's SHARED common config, where it would make `git gc` refuse in the main checkout and every other worktree on a box other people are using. Each was found with `examples/demo` as a 90-second harness smoke test rather than with a 40-minute run of the real package. The third is answered with a standalone clone rather than by changing state outside this worktree. Signed-off-by: yihou <yihou@amd.com>
…e kit
Run `qwen-r1`: `check_deploy_kit` PASS, `check_deploy_reproduces` FAIL with no
`claude.log` and no `reproduction.json` -- the signature of `claude_command()`
returning None before the reproducer ever started. The validation zone gets
`env_mgr`'s policy-derived PATH and an empty `home/` of its own, so neither
`shutil.which("claude")` nor `$HOME/.local/bin/claude` resolves, and
`AGENT_SYS_CLAUDE_CLI` is a name `env_mgr` never published.
That is a false FAIL on a handoff the shape check had just passed and whose own
producer had cold-tested it -- the worst verdict this validator can emit. The
path now travels the same route as every other site fact, a package variable
(`--var claude_cli=`), with PATH and ~/.local/bin kept as fallbacks; and the
refusal message now says plainly that the fault is the environment's, not the
handoff's.
Signed-off-by: yihou <yihou@amd.com>
… asserts
Mission steps 3 and 5: a run that passes is how the next run gets a better
example. Both kits -- Qwen3.6-27B TP1 and GLM-5.3-Flash TP8, same closure, only
`--var` between them -- are sanitised into `examples/`, stripped of hostnames,
ports, run tags and user paths, keeping the structure, the evidence set and the
wording of the criteria.
What the exemplars demonstrated is then fixed as two checks rather than left as
prose:
require_mode_readback the deployment mode must be evidenced from two
independent components -- the worker's own log
line and the router's worker listing. The mode is
selected by OMITTING a flag, so the launch command
is not evidence, and one component agreeing with
itself is not either.
require_completion_evidence one file must carry `"finish_reason": "stop"` AND
a non-empty `"content"`. /health returning 200
shows a process is listening, not a model that
answers.
Controls, all run: both real kits PASS; a copy of the qwen kit with
`chat_completion.json`, `verification.json` and `worker_mode_line.txt` removed
FAILS on exactly those two rules and nothing else.
The brief gains the two traps the runs paid for: `--served-model-name` is absent
from infera's `--help` (parse_known_args forwards the rest to the engine), so an
agent that greps the help ships a kit publishing the container mount path; and
cold start is 274 s for 51 GB but 910 s for 328 GB over NFS, against a kit that
documented 200 s.
Signed-off-by: yihou <yihou@amd.com>
Signed-off-by: yihou <yihou@amd.com>
…result Stage 1 is green: two models, three consecutive clean runs each, package unchanged across both series (869cf21), verified by the driver rather than by hand. Evidence packed at /shared_nfs/yihou/packups/packups/e2e_deploy_standardized.packup_20260901. The bug note is the part worth keeping in the tree. A validator body that shells out to `claude` -- which is how both AI validators in this repository are written -- has no supported way to find the binary the run itself already located: a validation zone gets a policy-derived PATH and an empty home/, and `AGENT_SYS_CLAUDE_CLI` is a name env_mgr never published. The package works around it with a variable; the gap, and the fact that a body which fails gracefully is indistinguishable from a handoff that is bad, are `validator`'s. Signed-off-by: yihou <yihou@amd.com>
Signed-off-by: yihou <yihou@amd.com>
dorado269
force-pushed
the
dev.yihou.aiopt.task_package
branch
from
September 2, 2026 07:26
0a6a3dd to
1a01a0b
Compare
add profile/analyze/integration demos
Stage 4 of the six-stage `llm_e2e_performance_optimization` series, as its own self-contained demo folder beside `analyze-demo`, `profiling-demo` and `integration-demo` — the layout the other stages already use, one folder per stage, each loadable on its own. It ships `optimize_kernel` (ai, Opus, KernelForge wrapper) and `publish_workset` (program), the `workset` / `kernel_optimization` handoffs, and three validators: shape on the workset, shape plus re-measurement on the output. Stage 3 does not exist yet, so a real traced operator — the sglang sampler's vocabulary softmax, `[8, 151936]` fp32, 55.59 us/call — ships as package data under `assets/worksets/` and `publish_workset` puts it into the graph. `agent-sys show --package .../kernel-opt-demo` loads it: 3 tasks, 1 root and 2 subtasks, six validation phases wired. Signed-off-by: yihou <yihou@amd.com>
A one-leaf package that runs `publish_workset` and `check_workset_shape` and nothing else, so a candidate workset can be checked against the contract in seconds instead of after an hour of a KernelForge campaign measuring nothing. Its `--var real_package=` / `--var workset_dir=` defaults point at `llm_e2e_performance_optimization/kernel-opt-demo`, where stage 4 now lives. Signed-off-by: yihou <yihou@amd.com>
[draft]Dev.yihou.aiopt.kernel opt module
…comes a container Every other stage already lives in its own folder — `profiling-demo`, `analyze-demo`, `kernel-opt-demo`, `integration-demo`. Stage 1 was the exception: its `main.yaml`, `assets/` and `steps/deploy.yaml` sat at the package root, so the root was simultaneously a task package and the directory holding four other task packages. That was not only untidy, it was broken. `--package <root>` enumerated the nested `main.yaml` files and loaded them into one namespace, and the load failed with `closure 'main' names subtask closure 'seed_table', which is not declared` — a message about `analyze-demo` produced by pointing at stage 1. The root now carries no `main.yaml`, so the same command says what is actually true: *not a task package*. Pure moves plus the paths that had to follow them: `--package` in `README.md` and `steps/deploy.yaml`'s usage comment now name `deploy-demo`, and the two `temp/mission.md` references become `../temp/mission.md`. No package content changed. The root `README.md` is replaced by an index of the six stages, since the file it replaces was stage 1's own and moved with it. Checked with `agent-sys show`: `deploy-demo` loads 2 tasks and four validation phases; `kernel-opt-demo` loads 3; the other three still stop only at their missing site variables, as before. Signed-off-by: yihou <yihou@amd.com>
Each of the five stage packages was driven end to end on the spur cluster
until it produced a sealed handoff set, so that a later run of the stitched
flow can mock any stage rather than execute it. The deliveries live outside
the repo, under /shared_nfs/yihou/agent_sys/{debugging,cheat_for_mock}/.
Localisation, since these packages were written for a different cluster:
the transport seam (srun -> spur exec), the GPU target, the visible cards,
the container name and the context length all become package variables
instead of hard-coded values. profiling-demo gains a 'local' transport for
driving from the node itself.
Real defects fixed in the packages:
- analyze-demo's analyze_packup kind declared six items under
additionalProperties:false while packup.py writes ten, so the seal
refused every version and the run reported its terminal task as still
running.
- kernel-opt-demo inherited a gfx942 baseline into a gfx950 run, and put
TMPDIR on NFS, which segfaults every ROCm kernel launch.
- profiling-demo's mix_worker.sh hard-coded GLM-only flags, which produce
empty content on any other model.
- analyze-demo's locality helper merged two distinct findings into one.
Three framework gaps are recorded under temp/bugs/ and worked around rather
than fixed, agent_sys itself being outside this round's scope:
- spec_loader/variables.py accepts ${N} and ${N:-d} only; any other ${...}
is passed through verbatim with no diagnostic, so a typo arrives at a
subprocess as a malformed argument.
- a refused seal is filed under seal_refused, which has no reader anywhere
outside tests, so the one message naming the cause is discarded and the
operator is shown a stall timer instead.
- two functions named copy_out one level apart have opposite safety
properties, and the consuming path uses the one that does not verify the
digest -- while grants.py justifies its design by the other one's
guarantee.
Per-module debugging notes are at the repo root, one file per module, plus
the running checkpoint record.
Signed-off-by: yihou <yihou@amd.com>
Two pieces of work that turned out to be one: making a run legible from outside. AgentsView is never modified -- every knob is one it publishes. ## The panel An external Go binary reads the JSONL transcripts `claude` already writes, so the integration is only *where transcripts land* and *which directory the panel reads*. It installs into a prefix `~/.infera_agent_sys` (laid out like `~/.local`, owned by env_mgr), starts at deploy time on 18888, stays resident, and prints its URL. Nothing is written to /usr/local/bin or ~/.local/bin. **It shows only agent_sys's own sessions**, by five gates. The strongest was found by measuring rather than designing: `HOME` is redirected into the prefix for the binary's own subprocesses, so every provider's *default* root resolves inside it -- no list to go stale when upstream adds a provider we have never heard of. Verified live: of 122 session roots AgentsView would scan, 122 are inside the prefix and 0 outside. The user's own ~/.claude is never read. **o11y may never fail the thing it observes.** Binary missing, port taken, daemon wedged, health timed out, mapping refused: one `log.warning` and a skip, with a test per mode and a blanket catch at the single call site. Reuse of a daemon on the port needs two gates -- it answers /api/v1/agents with JSON, *and* a live `daemon.<pid>.json` in our own data directory names that port. A stranger's daemon writes that record into their data directory, so the isolation is the filesystem's rather than a convention we maintain. ## A browsable run tree, and one project per run Runtime directories now carry the name they already knew: `zones/task.main.<uuid>/task.solve_a.<uuid>.0.<hash>/`. Nothing resolves through the label, so old trees still resolve and a renamed closure does not move an artefact. That made the names readable; it cannot group them. AgentsView derives a project from the session's *deepest* path segment, and attempts nest, so one run arrived as a dozen unrelated projects. The only filesystem fix would be putting every attempt under one directory, which is what zone isolation exists to prevent. So one `explicit` mapping over the run root, posted at run start. **Measured before written, and three measurements changed the code**: `Origin` is mandatory on mutating calls (without it a plain-text 403 that reads like a missing endpoint); `machine` comes from the daemon's own `local_machine`, never assembled, because a wrong one matches nothing silently; names normalise `-` to `_`, so we normalise before posting. Scaling is O(rows x sessions) -- visible only by repeating the curve at 4x the sessions -- and the path the daemon actually runs is flat, +32 ms at 5000 rows. **Acceptance was the rendered dropdown**, not an exit code and not the API: one per-run project holding 9 sessions against 43 per-attempt projects holding 1 each, both behaviours in the same control. ## Also fixed, and not in the original scope - A plain `pytest` wrote **128 MB** into the operator's ~/.infera_agent_sys -- a real 45 MB download -- because three test files reached main(["run", ...]) without patching ensure_installed. Nothing said so, and it made the suite need the network. Fixed structurally in two conftests; a scratch prefix now takes 0. - The readiness probe ran in the caller's cwd, filing ten identical `Reply with exactly one word: ready` sessions under whichever repository you were standing in. It gets its own directory in the prefix. ## Left open, each with the command that would settle it `repo_dot_worktrees` matched nothing across thirteen prefix shapes; whether `enabled=false` rows still cost a scan; the zone symlink under an enforcing policy, which is **untested because currently untestable, never "safe"**; and TODO 4g, the backend's `claude` children not exiting when their task completes. 2287 passed, 3 skipped, 4 xfailed. Signed-off-by: yihou <yihou@amd.com>
feat(agent_sys): AgentsView as the o11y panel, one project per run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please include a brief summary of the changes, relevant motivation and context.
Fixes # (issue)
Type of change
Changes
Please list the changes introduced in this PR:
Checklist: