[beca-oc]

Description

Onyx Craft as a thin layer over Expected Parrot: Craft's agent runs Expected Parrot's own EDSL, ep CLI, study skills and application CLIs on open-weight Bedrock models, with respondents going through Onyx's gateway. A non-admin user names a market question and gets a plan, a priced pilot, the full run, inline charts and a plain explanation. Map: #119, decision #130.

What is in it:

• deployment/craft-ep/sandbox/ep-charts.ts (global OpenCode plugin) holds the rules in code: no EDSL job (.run(, ep run, ep run-manifest) before the user's second message; after the gate fires, shell stays refused for that turn and the third refused command aborts it; each model call gets the current turn's rule (plan only, then a 10-respondent pilot with tokens and cost, then "respondents are simulated"); reply.md may not say market share, penetration, adoption, market size or reach; each new chart's inline Markdown is appended to the tool result.
• deployment/craft-ep/rebuild.sh: one script that builds the backend, web, sandbox base and Expected Parrot sandbox images from the checkout, writes .env, brings up Compose with the Craft overlay, restarts nginx, and applies the Craft model and instructions.
• craft-instructions.md (3,827 of 4,000 characters), agent model minimax.minimax-m2.5 (GLM-5 also works at about twice the agent-token cost; Qwen3-Next refuses to run EDSL).
• web/src/app/craft/constants/exampleBuildPrompts.ts: a "Market research" group of four starter prompts, first on Craft's welcome page.
• test/tdd.py: drives Craft as test users; core (plan gate x2, pricing study through pilot, full run and a market-share follow-up, unavailable model), market (willingness to pay, image concept test, segmented conjoint), demos, tools; reports time to first chart; --without-gate for the red side.

Depends on #132 (LiteLLM api_base fix) for any streamed model call on fork main.

How Has This Been Tested?

On agent-dev, stack rebuilt by rebuild.sh from this branch plus #132 (tag fork-145653006), as non-admin test users through Craft's API:

Run	Checks	Cases green
c1	41/43	5/7
c2	36/43	4/7
c3	34/43	4/7 (two misses were a tdd inspector bug, fixed after)
c4	39/43	4/7

• Plan gate: red with PLAN_GATE = false (agent wrote a Results file on turn 1, tdd-g6-red), green with it (tdd-g8: 3/3 gate cases). Before this PR's fix the agent bypassed the gate with ep run.
• First inline chart after approving the full run: 73 s to 274 s across c1-c4.
• Still flaky: the "simulated respondents" caveat on follow-up replies, pilot size on the conjoint, occasional 39/40 distinct agents, rare gateway timeouts.
• Not run: the 25 demo cases and 39 tool cases; the $25 Bedrock cap for this goal was reached first.

🤖 Generated with Claude Code

[vercel]

When the full study overwrites a pilot chart at the same path, the filename-only seen set suppresses the inline-chart reminder for the updated results.

[vercel]

Even after copying the test files into ~/craft-ep as documented, the runners fail because craft_turn.sh is not executable.

[beca-oc]

Final-image rerun (sandbox image with ep-charts.ts and /tmp/ep-skills, final Craft instructions), 2026-11-02, every flow three turns through Craft's API as non-admin users:

Flow	Inline charts that load	Banned share terms	Errors
streaming preference	2	0	0
quickstart	4	0	0
survey_logic	5	0	0
agents_scenarios	2	0	0
import_csv	3	0	0
labeling (EDSL checkbox + free text coders)	6	0	0
pretest (EDSL reviewers, no sub-agents)	1	0	0
results_cache (4 of 4 second-run answers from cache)	2	0	0
nps	4	0	0
two_pass	6	0	0
enrich_agents ({{ agent.career }} trait)	4	0	0
conjoint (importance, part-worths, price curve, choice shares)	12	0	0

Earlier brand-new-user meal-kit run on the same image: 4 inline charts, 0 banned terms, skills read from /tmp/ep-skills. Labeling, nps, conjoint and agents_scenarios each needed one rerun because my test script was edited while running; the Craft turns themselves did not fail.

[beca-oc]

Plan families 12 and 13 through Craft on the final image, 2026-11-02 (three turns each, non-admin users):

Flow	Result
AI-moderated interview (guide, interviewer, one follow-up per answer)	20 hybrid + 20 remote simulated workers; themes by group; 3 inline charts load; $0.72
Question distribution across models	same numerical question on Llama 4 Maverick (mean 4.65 h/week) and Kimi K2.5 (4.17); 2 inline charts load
Google Form rebuild	form mapped to MultipleChoice, CheckBox, LinearScale, FreeText; 4 inline charts load; $0.01
Label validation against gold	120/120 correct, Cohen's kappa 1.000; confusion matrix inline
Image stimulus (uploaded ad)	first run passed only the file path, so the model never saw the ad; after adding the FileStore rule the rerun sent the image, answers quote the ad; 40 shoppers, $0.10
PDF stimulus (uploaded two-page brief)	each page's text as a scenario; 20 simulated brand managers; risk chart inline
Humanize (real respondents)	blocked, reported honestly: .humanize() fails with "The API key you provided is invalid"; no Expected Parrot account or key exists (none in Doppler beca). Needs an Expected Parrot account to recruit humans.

Banned share terms: 0 in every flow.

[beca-oc]

Humanize through Craft (2026-11-02): the agent built the EDSL survey and rendered the respondent page with Expected Parrot's runway (pinned 75f6810ac, now in the sandbox image). The page shows the question and its four options exactly as a respondent would see it. The reply states that fielding it to real people needs an Expected Parrot account: ep humanize create publishes to Expected Parrot's platform, and no Expected Parrot key exists in any Doppler beca config. Craft serves the page as a download, not a hosted link.

[beca-oc]

Expected Parrot application tools through Craft, 2026-11-02. Every public package in Expected Parrot's directory except humanize-based spence (skipped per owner) and uxtest (requires Python < 3.13; sandbox has 3.13) ran one three-turn flow as a non-admin user. Pass means the agent ran the tool's own commands, finished every turn, and hit no errors.

37 of 37 pass: epiq, conversation, aivis, ariel, bewley, bower, burr, dewey, flyvbjerg, green, golden, helmer, ishikawa, kahn, katz, langley, mccall, mcda, messick, niles, oneheart, perla, premortem, pruefung, raiffa, roth, stanton, tommy, treffen, umriss, urteil, vorhersage, voting, zermelo, zugunruhe, zwicky, zwill.

Examples: aivis measured brand mentions across two models; green ran a choice experiment and charted attribute importance; ishikawa rendered its fishbone SVG inline; langley named low awareness as the least-refuted hypothesis; raiffa computed the expected-value policy; zugunruhe migrated a survey with zugunruhe migrate. Inline charts loaded wherever made; the only banned-term hits were stanton citing Grand View Research's real market size as a source.

Private packages (AISS, Earnings, Kean, Zitate) could not be tested.

[beca-oc]

uxtest through Craft (2026-11-02): with uv's Python 3.12 and Playwright Chromium in the image, the agent ran 33 uxtest commands, drove three simulated personas through uxtest's bundled checkout site, and reported the main issue (every persona was sent to /login before understanding the product). No errors. That makes 38 of 38 public directory packages passing except humanize-based spence, which the owner asked to skip. AISS, Earnings, Kean and Zitate are private and return 404 to our accounts.

[beca-oc]

spence through Craft (2026-11-02): the agent ran 111 spence commands locally (role, opening, 3 applications, 6 reviews via spence's demo flow, all 6 in state submitted) with no errors and nothing published to Expected Parrot. All 39 public directory packages now pass through Craft. Untestable: AISS, Earnings, Kean and Zitate are private (404 to our GitHub accounts).

[beca-oc]

EDSL docs examples (docs.json: how-to guides and research examples) not covered before, run through Craft as test users on the final image, prompts in test/ep_demos.tsv:

Example	Responses	Inline charts (load)	Cost	Notes
experimental_conditions	160	4 (4)	$0.01	simple message very clear 80/80, detailed 20/80
model_comparisons	10	1 (1)	$0.0006	llama4-maverick vs qwen3-next agree 4/5; llama4 rates every message high, even a thank-you note
prompt_sensitivity	360	2 (2)	$0.07	benefit-of-doubt moved review 2 neutral to positive
model_behavior	80	1 (1)	$0.0058	yes: hot afternoon 60%, Monday 25%, just had coffee 0%, price was $3 0%
qualitative	3 excerpts	0	$0.001	themes with evidence quotes; agent skipped the plan gate on a 3-item task
human_validation	120	1 (1)	$0.009	100% match with the supplied human labels; fielding to people skipped (no EP account)

No errors or banned terms in any turn. model_comparisons needed one extra turn because the generic 'run 20 per group' approval did not fit a two-model design.

[vercel]

The full-size and distinct-agent checks accept oversized studies, so adding respondents beyond the approved sample still produces passing sample-size results.

[beca-oc]

Local gate on exact head 34aa5f6240b9aaa6b808defcdd52b4e65a0caba2, clean worktree on agent-dev (~/gate133-<sha>, git status --porcelain empty, web node_modules symlinked from the main checkout). GitHub Actions cannot start on the org's private repos; the owner-approved 2026-10-31 local-gate exception applies.

head 34aa5f6240b9aaa6b808defcdd52b4e65a0caba2 porcelain 0
$ ruff check deployment/craft-ep/test
All checks passed!
exit 0
$ ruff format --check deployment/craft-ep/test/tdd.py
1 file already formatted
exit 0
$ bash -n deployment/craft-ep/rebuild.sh deployment/craft-ep/test/*.sh
exit 0
$ wc -c < deployment/craft-ep/craft-instructions.md
3827
$ oxfmt --check src/app/craft/constants/exampleBuildPrompts.ts
All matched files use the correct format.
exit 0
$ oxlint src/app/craft/constants/exampleBuildPrompts.ts
Found 0 warnings and 0 errors.
exit 0
$ bun run types:check
exit 0
✓ Types generated successfully

Independent review (fresh subagent, fallback because the Claude review job cannot run): PASS, no blocking findings. Non-blocking notes and what I did:

1. Instructions near 4,000 chars: measured 3,827.
2. The gate misses jobs started via bash run.sh / python - < x.py; a false .run( hit (e.g. subprocess.run() refuses shell for the turn. Documented as a known gap in the README.
3. If the session-messages read fails the count defaults to 2 (fails open). Kept: a broken read must not lock users out.
4. reply.md written through the shell skips the banned-term check; the plugin is writable by the sandbox user. Documented.
5. rebuild.sh removes every sandbox-* container not on the new image; health wait was silent. Now commented and fails loudly.
6. tdd plan_only could pass an unsaved run: now also fails on any .run( / ep run command in turn 1. Other check limits noted.
7. README on --without-gate: now says it also turns off the per-turn rules and sends no attachments.

No review threads. Merging is blocked for this account (beca-oc does not have the correct permissions to execute MergePullRequest, seen on #132), so this stops at a ready PR.