feat(craft): run Expected Parrot studies in Onyx Craft

deployment/craft-ep/README.md

@@ -0,0 +1,72 @@
+# Expected Parrot studies in Onyx Craft
+
+Onyx Craft runs Expected Parrot's own tools: EDSL and its `ep` CLI (MIT) in the
+Craft sandbox, Expected Parrot's study skills (`expectedparrot/ep-skills`,
+pinned), and Craft's OpenCode agent on open-weight Bedrock models. A user asks a
+market question in Craft; the agent plans the study, waits for approval, runs a
+priced pilot, waits again, then runs the full study on simulated respondents and
+shows the charts inline.
+
+Running on agent-dev as Compose project `craft-fork`
+(`~/craft-ep/fork/deployment/docker_compose`), web on
+`http://agent-dev.taila3aadb.ts.net:3390`. Tracked on
+[map #119](https://github.com/Subconscious-ai/onyx/issues/119).
+
+## What is configuration and what is ours
+
+| Piece | Source |
+| --- | --- |
+| Craft, sandbox, model gateway, skills import, Craft instructions setting | Onyx (this fork) |
+| EDSL 1.0.9, `ep` CLI, study skills | Expected Parrot, installed or fetched at image build |
+| `sandbox/Dockerfile` | ours: fork sandbox image plus EDSL, `openai[aiohttp]`, pinned skills, concurrency cap |
+| `sandbox/ep-charts.ts` | ours: global OpenCode plugin, same mechanism as Onyx's `turn-budget.ts` |
+| `craft-instructions.md` | ours: the Craft organization instructions (admin setting, under Onyx's 4,000-character limit) |
+| `rebuild.sh` | ours: builds every image from this checkout, updates `.env`, restarts the stack, applies the admin settings |
+| `web/src/app/craft/constants/exampleBuildPrompts.ts` | fork change: a "Market research" group of starter prompts, first on Craft's welcome page |
+
+## Stack settings (`deployment/docker_compose/.env`)
+
+Start from `env.template`, then set (no secrets listed here):
+
+- `COMPOSE_PROJECT_NAME=craft-fork`, `HOST_PORT=3390`, `HOST_PORT_80=3391`, `WEB_DOMAIN=http://agent-dev.taila3aadb.ts.net:3390`, `AUTH_TYPE=basic`, `USER_AUTH_SECRET` generated (`openssl rand -hex 32`).
+- `ENABLE_CRAFT=true`, `SANDBOX_BACKEND=docker`, `SANDBOX_CONTAINER_IMAGE=craft-ep/sandbox:<fork tag>-edsl1.0.9`.
+- `ONYX_BACKEND_IMAGE` and `ONYX_WEB_SERVER_IMAGE` built from this fork (`backend/Dockerfile`, `web/Dockerfile`); `ONYX_MODEL_SERVER_IMAGE=onyxdotapp/onyx-model-server:v5.0.3`; `CODE_INTERPRETER_IMAGE_TAG=0.4.9`.
+- `LICENSE_ENFORCEMENT_ENABLED=false` and `ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=true`: Craft needs the Business tier; `backend/ee/LICENSE` permits development and testing without a subscription. Production needs an Onyx subscription.
+- `OPENCODE_PROMPT_INACTIVITY_TIMEOUT_SECONDS=1800`: the 200 s default ends a turn while a full EDSL run is still quiet.
+- `LLM_SOCKET_READ_TIMEOUT=600`: MiniMax M2.5 can go quiet for minutes while it writes a long tool call; the 60 s default (and 180 s) returned "The selected model did not respond in time."
+- The fork's Compose file names `minio/minio:RELEASE.2025-07-23T15-54-02Z-cpuv1`, which Docker Hub no longer serves; tag Onyx's mirror (`onyxdotapp/minio` at that release) under that name.
+
+`rebuild.sh` does the rest: it builds the backend, web, sandbox base and Expected Parrot sandbox images tagged `fork-<commit>`, creates the sandbox network and proxy volume, tags the MinIO mirror, writes the image tags and timeouts into `.env`, runs `docker compose up -d`, removes sandboxes on an older image (Craft re-creates them), and applies the default Craft model and `craft-instructions.md`. The steps by hand: before `docker compose up`, `docker network create onyx_craft_sandbox`, `docker volume create sandbox_proxy_ca`, and build the sandbox image from `sandbox/` on top of the fork's `backend/onyx/server/features/build/sandbox/image` (tagged `craft-ep/sandbox-base:<fork tag>`). Docker needs buildx for the fork images.
+
+The fork needs [fix(llm): pass api_base through the patched LiteLLM bridge check (#132)](https://github.com/Subconscious-ai/onyx/pull/132); without it every streamed model call fails.
+
+## After the stack is up (admin API)
+
+1. Bedrock provider with the open-weight models (Anthropic models on Bedrock are blocked for this account): `minimax.minimax-m2.5`, `moonshotai.kimi-k2.5`, `zai.glm-5`, `deepseek.v3.2`, `qwen.qwen3-next-80b-a3b`, `openai.gpt-oss-120b-1:0`, `us.meta.llama4-maverick-17b-instruct-v1:0`, `mistral.mistral-large-3-675b-instruct`, `google.gemma-3-27b-it`. `test/copy_bedrock.py` copies the dev chat stack's Bedrock key without printing it.
+2. `POST /api/admin/llm/default` and `/default-craft` with `minimax.minimax-m2.5` (`rebuild.sh` sets the Craft default). GLM-5 also passes but costs about twice as much: about $1.50 of agent tokens per full study against $0.70 to $1.00 for MiniMax (measured from `user_usage`). Kimi K2.5 and DeepSeek v3.2 replaced the study with web research; Qwen3-Next refused to run EDSL.
+3. `PATCH /api/admin/settings` with `{"craft_instructions": <craft-instructions.md>}`.
+
+Respondents run inside the sandbox through Onyx's own gateway: EDSL's `openai_compatible` service with base URL `$ONYX_SERVER_URL$ONYX_API_PREFIX/gateway/v1` and key `$ONYX_PAT`, model `1/<model id>`. No model key enters the sandbox.
+
+## Why each piece of glue exists
+
+- **Skills for every user without switching them on.** Onyx enables custom skills per user, and Craft's OpenCode rules let the agent read only the session folder, enabled skills and `/tmp`. The image fetches the pinned skills to `/opt/ep-skills`, and the sandbox entrypoint copies them into the fresh `/tmp/ep-skills` at start. (Craft starts sandboxes with `/workspace/entrypoint.sh`, not the image `ENTRYPOINT`.)
+- **Rules the plugin enforces, not just the instructions.** Instructions alone held in about 15 of 18 runs. `ep-charts.ts` (1) refuses any shell command that starts an EDSL job (`.run(` inline or in the Python file it runs, or `ep run` / `ep run-manifest`) before the user's second message, and once it fires refuses every shell command for the rest of that turn (the third refused command ends the turn), so the first reply is a plan; it also adds the current turn's rule to each model call (turn 1 plan only, turn 2 pilot only with tokens and cost, later turns say the respondents are simulated); (2) refuses a write of `outputs/reply.md` that says market share, penetration, adoption, market size or reach; (3) appends to each tool result the exact inline Markdown for every new PNG and warns when a chart lands outside `outputs/graphs/`; (4) flags the banned terms in anything else the agent writes.
+- **`EDSL_MAX_CONCURRENT_TASKS=8`.** EDSL's default of 50 exhausted the API's Redis connection pool ("Too many connections").
+
+## Test
+
+`test/tdd.py` drives Craft through its own API as the test users `ep-user2`..`ep-user8` (cookie jars in `~/craft-ep/`, copied with `test/craft_turn.sh`, `ep_demos.tsv`, `ep_tool_prompts.tsv` and `stimuli/`), then checks what ran: EDSL Results files in the session's sandbox, the commands the agent ran, the reply, and whether each inline chart loads.
+
+```bash
+python3 tdd.py <run> [--jobs 4] [--model ID] [--without-gate] [core | market | demos | tools | <case> ...]
+```
+
+- `core`: the plan gate on a small coding task and on a user who demands an immediate run (`gate_immediate_run`), a priced pricing study through pilot and full run with a market-share follow-up, and an unavailable respondent model.
+- `market`: three scripted investigations: willingness to pay (Van Westendorp plus a price ladder), an ad concept test with an attached image, and a conjoint with two segments.
+- `demos`: each Expected Parrot demo and EDSL docs example in `ep_demos.tsv`; `tools`: each application CLI in `ep_tool_prompts.tsv` (pass means the tool's own command ran and its result is in the reply).
+- `first_chart_time` reports the seconds from approving the full run to the reply with the first inline chart.
+- `--without-gate` sets `PLAN_GATE = false` in the plugin in the test user's sandbox (the gate and the per-turn rules) and restarts `opencode serve`, then restores it: `gate_immediate_run` goes red without it and green with it. Attachments are not sent in this mode.
+- Known gaps, from review: the gate does not see an EDSL job started from a script run some other way (`bash run.sh`, `python - < x.py`), and a reply written to `reply.md` through the shell skips the banned-term check.
+
+`welcome_screenshot.py` renders Craft's welcome page with the Market research starter prompts open. `flow.sh`, `audit_flow.sh`, `run_demos.sh` and `craft_screenshot.py` are the earlier hand tools; `craft_screenshot.py` renders a session in headless Chromium.

deployment/craft-ep/craft-instructions.md

@@ -0,0 +1,28 @@
+Expected Parrot. For any survey, market, pricing, message test, experiment, conjoint, interview, persona, pretest, or text coding, labeling or theme request: first read /tmp/ep-skills/create-study/SKILL.md, then the skills it points to. Run everything with Expected Parrot's EDSL and `ep` CLI from /workspace/.venv: simulated respondents answer, simulated coders label. No web research first. Never answer, code or classify yourself, and never build your own survey framework. Use exactly any EDSL mechanism the user names (trait, scenario, memory, piping, skip rule, cache).
+
+Model. Before any run:
+  export OPENAI_COMPATIBLE_BASE_URL="$ONYX_SERVER_URL$ONYX_API_PREFIX/gateway/v1"
+  export OPENAI_COMPATIBLE_API_KEY="$ONYX_PAT"
+Use Model("1/us.meta.llama4-maverick-17b-instruct-v1:0", service_name="openai_compatible") unless the user names another Bedrock model (1/moonshotai.kimi-k2.5, 1/zai.glm-5, 1/deepseek.v3.2, 1/qwen.qwen3-next-80b-a3b). Other models (gpt-4o, Claude, Gemini) are unavailable: say so in the plan and use one of these. Run with disable_remote_inference=True, disable_remote_cache=True.
+
+Files. An attached image or PDF reaches respondents only as Scenario({"ad": FileStore("attachments/<file>")}) used as {{ scenario.ad }}, or ScenarioList.from_pdf("attachments/<file>"). Never put a file path in question text.
+
+Real people. There is no Expected Parrot account: build the survey, survey.save("outputs/human-survey/survey"), run `runway render outputs/human-survey/survey.json.gz -o outputs/human-survey/`, link the page, and say fielding needs an Expected Parrot account.
+
+Gates (every new request, even a small coding task):
+1. First reply: plan only (decision, segments, questions, an available model, sample size). Run nothing that calls .run(). Ask for approval and stop.
+2. After approval: pilot of 10 respondents in total (or 10 records to code). Show answers, the pilot's input and output tokens and its $ cost, ask again, stop.
+3. After approval: the full run at exactly the sample the user set. N respondents per segment means N agents per segment; every price or condition is a scenario each agent answers. Never add respondents.
+Cost comes from the pilot's raw_model_response.*input_tokens and *output_tokens at $/M input/output: llama4-maverick 0.24/0.97, kimi-k2.5 0.60/2.50, glm-5 1.00/3.20, deepseek.v3.2 0.58/1.68, qwen3-next 0.15/1.20, scaled to the full run. Never guess a cost.
+
+Respondents differ. Build each AgentList with varied traits (age, income, habits, attitudes); every agent's traits differ from every other's. Flag near-uniform answers.
+
+Output. Work only in outputs/: save Results with results.save("outputs/<study>/results_pilot") and results.save("outputs/<study>/results_full"), charts (PNG) and tables (CSV) in outputs/graphs/. Show every chart inline:
+  ![Title](/api/build/sessions/<session>/artifacts/outputs/graphs/<file>.png)
+where <session> is basename "$PWD", with its key numbers under it. Never only name a file or folder.
+
+Wording. Respondents are language-model personas, not real people; say so once. Report only numbers the results contain, as shares of simulated respondents ("49% of simulated respondents would buy"). Never report a market share, penetration, adoption, market size or reach, in prose, titles, labels, file names or headers; if asked for one, explain in other words why the study cannot give it.
+
+Before sending results: write the reply to outputs/reply.md, run `ls outputs/graphs/; grep -inE "market share|penetration|adoption|market size|\breach\b" outputs/reply.md`, make every new PNG inline, rewrite every hit, then send reply.md.
+
+Expected Parrot tools. Its application CLIs are on PATH (`ls /opt/ep-tools`, plus uxtest, spence, conversation). When the user names one, run `<tool> --help` (and `guide`) first and use its commands.

deployment/craft-ep/rebuild.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Rebuild the Craft stack and the Expected Parrot sandbox image from this checkout, then apply the admin settings.
+# Usage (on the Craft host, from any directory): deployment/craft-ep/rebuild.sh
+# Needs: Docker with buildx, deployment/docker_compose/.env (see README), and ~/craft-ep/admin.env with
+# CRAFT_ADMIN_EMAIL and CRAFT_ADMIN_PASSWORD. Keeps the database and file store volumes.
+set -euo pipefail
+HERE=$(cd "$(dirname "$0")" && pwd)
+REPO=$(git -C "$HERE" rev-parse --show-toplevel)
+COMPOSE="$REPO/deployment/docker_compose"
+TAG=fork-$(git -C "$REPO" rev-parse --short=9 HEAD)
+AGENT_MODEL=${CRAFT_AGENT_MODEL:-minimax.minimax-m2.5}
+B=http://localhost:${HOST_PORT:-3390}/api
+SANDBOX=craft-ep/sandbox:$TAG-edsl1.0.9
+echo "tag $TAG"
+
+docker build -q -t "craft-ep/onyx-backend:$TAG" -f "$REPO/backend/Dockerfile" "$REPO/backend"
+docker build -q -t "craft-ep/onyx-web-server:$TAG" -f "$REPO/web/Dockerfile" "$REPO/web"
+docker build -q -t "craft-ep/sandbox-base:$TAG" "$REPO/backend/onyx/server/features/build/sandbox/image"
+docker build -q --build-arg "BASE=craft-ep/sandbox-base:$TAG" -t "$SANDBOX" "$HERE/sandbox"
+docker run --rm --entrypoint /workspace/.venv/bin/python "$SANDBOX" -c "import edsl; print('edsl', edsl.__version__)"
+
+# One-time host setup the Compose file expects.
+docker network inspect onyx_craft_sandbox >/dev/null 2>&1 || docker network create onyx_craft_sandbox
+docker volume inspect sandbox_proxy_ca >/dev/null 2>&1 || docker volume create sandbox_proxy_ca
+MINIO=minio/minio:RELEASE.2025-07-23T15-54-02Z-cpuv1
+docker image inspect "$MINIO" >/dev/null 2>&1 || { docker pull onyxdotapp/minio:RELEASE.2025-07-23T15-54-02Z-cpuv1 && docker tag onyxdotapp/minio:RELEASE.2025-07-23T15-54-02Z-cpuv1 "$MINIO"; }
+
+set_env() { if grep -q "^$1=" "$COMPOSE/.env"; then sed -i "s|^$1=.*|$1=$2|" "$COMPOSE/.env"; else echo "$1=$2" >> "$COMPOSE/.env"; fi; }
+set_env ONYX_BACKEND_IMAGE "craft-ep/onyx-backend:$TAG"
+set_env ONYX_WEB_SERVER_IMAGE "craft-ep/onyx-web-server:$TAG"
+set_env SANDBOX_CONTAINER_IMAGE "$SANDBOX"
+# MiniMax M2.5 can go quiet for minutes while it writes a long tool call; the 60 s default cuts it off.
+set_env LLM_SOCKET_READ_TIMEOUT 600
+set_env OPENCODE_PROMPT_INACTIVITY_TIMEOUT_SECONDS 1800
+# docker-compose.craft.yml adds the sandbox egress proxy and the api_server settings that point at it.
+(cd "$COMPOSE" && docker compose -f docker-compose.yml -f docker-compose.craft.yml up -d)
+# nginx resolves api_server once at start; a re-created api_server leaves it returning 502.
+docker restart "$(cd "$COMPOSE" && docker compose ps -q nginx)" >/dev/null
+
+# Sandboxes on an older image are re-created by Craft on the user's next session. This removes every
+# sandbox-* container on the host not on the new image, live sessions included: run it on a quiet stack.
+for c in $(docker ps -a --format '{{.Names}} {{.Image}}' | awk -v img="$SANDBOX" '$1 ~ /^sandbox-/ && $2 != img {print $1}'); do
+  docker rm -f "$c" >/dev/null && echo "removed old sandbox $c"
+done
+
+for i in $(seq 1 60); do curl -sf "$B/health" >/dev/null && break; sleep 5; done
+curl -sf "$B/health" >/dev/null || { echo "API not healthy after 5 minutes"; exit 1; }
+set -a; . ~/craft-ep/admin.env; set +a
+JAR=$(mktemp)
+curl -sf -c "$JAR" -o /dev/null --data-urlencode "username=$CRAFT_ADMIN_EMAIL" --data-urlencode "password=$CRAFT_ADMIN_PASSWORD" "$B/auth/login"
+PROVIDER=$(curl -sf -b "$JAR" "$B/admin/llm/provider" | python3 -c "import sys,json; d=json.load(sys.stdin); d=d.get('providers',d) if isinstance(d,dict) else d; print(next((p['id'] for p in d if p['provider']=='bedrock'),''))")
+[ -n "$PROVIDER" ] || { echo "no Bedrock provider: run test/copy_bedrock.py once (README)"; exit 1; }
+curl -sf -o /dev/null -b "$JAR" -H "Content-Type: application/json" -d "{\"provider_id\":$PROVIDER,\"model_name\":\"$AGENT_MODEL\"}" "$B/admin/llm/default-craft"
+python3 -c "import json,sys; print(json.dumps({'craft_instructions': open(sys.argv[1]).read()}))" "$HERE/craft-instructions.md" \
+  | curl -sf -b "$JAR" -X PATCH -H "Content-Type: application/json" -d @- -o /dev/null "$B/admin/settings"
+rm -f "$JAR"
+echo "DONE $TAG agent=$AGENT_MODEL sandbox=$SANDBOX"

deployment/craft-ep/sandbox/Dockerfile

@@ -0,0 +1,34 @@
+# Onyx Craft sandbox (fork main) plus Expected Parrot (EDSL, MIT) and its ep CLI.
+ARG BASE=craft-ep/sandbox-base:fork-5b4b5ddae
+FROM ${BASE}
+USER root
+RUN /workspace/.venv/bin/pip install --no-cache-dir "edsl==1.0.9" "boto3>=1.39.12" "openai[aiohttp]>=2.44,<3" && chown -R sandbox:sandbox /workspace/.venv
+# Expected Parrot skills (expectedparrot/ep-skills), pinned; copied to /tmp at start
+# so every user can read them without switching shared skills on one by one.
+RUN /workspace/.venv/bin/python -c "import io,tarfile,urllib.request; tarfile.open(fileobj=io.BytesIO(urllib.request.urlopen('https://codeload.github.com/expectedparrot/ep-skills/tar.gz/f8ad7545a6d1280be518f50743c0e3e432426633').read())).extractall('/opt')" && mv /opt/ep-skills-f8ad7545a6d1280be518f50743c0e3e432426633 /opt/ep-skills && chmod -R a+rX /opt/ep-skills
+# Craft starts sandboxes with /workspace/entrypoint.sh (it ignores the image ENTRYPOINT),
+# so the copy into the fresh tmpfs /tmp runs as that script's first step.
+RUN sed -i '2i mkdir -p /tmp/ep-skills \&\& cp -r /opt/ep-skills/plugins/edsl-research/skills/. /tmp/ep-skills/ 2>/dev/null || true' /workspace/entrypoint.sh && sed -n 1,3p /workspace/entrypoint.sh
+# Expected Parrot's runway (MIT): renders the respondent view of an EDSL human survey
+# locally. Own venv because it pins edsl to a git branch.
+RUN python3 -m venv /opt/runway-venv && /opt/runway-venv/bin/pip install --no-cache-dir "git+https://github.com/expectedparrot/runway.git@75f6810ac889aac4d37c0750c37340e40a8d2e4f" && chmod -R a+rX /opt/runway-venv && ln -s /opt/runway-venv/bin/runway /usr/local/bin/runway
+# Expected Parrot's application CLIs (expectedparrot/<repo>, pinned in ep-tools.txt), each in
+# its own venv under /opt/ep-tools because several pin edsl to different git revisions.
+COPY ep-tools.txt install-ep-tools.sh /tmp/ep-tools/
+RUN sh /tmp/ep-tools/install-ep-tools.sh /tmp/ep-tools/ep-tools.txt && cat /opt/ep-tools/install.log
+# uxtest requires Python < 3.13 (the sandbox has 3.13): give it uv's Python 3.12, plus
+# Playwright's Chromium in a shared path the sandbox user can read.
+ENV PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright
+RUN python3 -m venv /opt/uv-venv && /opt/uv-venv/bin/pip install -q --no-cache-dir uv \
+ && UV_PYTHON_INSTALL_DIR=/opt/uv-python /opt/uv-venv/bin/uv python install 3.12 \
+ && UV_PYTHON_INSTALL_DIR=/opt/uv-python /opt/uv-venv/bin/uv venv -q --clear -p 3.12 /opt/ep-tools/uxtest \
+ && /opt/uv-venv/bin/uv pip install -q --python /opt/ep-tools/uxtest/bin/python "git+https://github.com/expectedparrot/uxtest.git@fc7d086365024585be579dc0c2bd258e9c81bda2" \
+ && /opt/ep-tools/uxtest/bin/playwright install --with-deps chromium \
+ && ln -sf /opt/ep-tools/uxtest/bin/uxtest /usr/local/bin/uxtest \
+ && chmod -R a+rX /opt/uv-python /opt/ep-tools/uxtest /opt/ms-playwright
+# Global OpenCode plugin: names each new chart with its inline Markdown and flags
+# banned share wording, the same way Onyx's turn-budget plugin steers the agent.
+COPY --chown=sandbox:sandbox ep-charts.ts /home/sandbox/.config/opencode/plugin/ep-charts.ts
+# 50 concurrent EDSL tasks exhausted the API's Redis pool; 8 is safe.
+ENV EDSL_MAX_CONCURRENT_TASKS=8
+USER sandbox