Repository navigation
Investigate a market from the Beca chat with Expected Parrot #119
Description
Destination
A spec, ready for an unattended build, for the Beca chat on Onyx where someone names a market question and gets the study designed, populated with simulated respondents, run on a real model, charted and explained in that conversation, reliably. Reliable means the eleven Expected Parrot sample flows and three scripted market investigations pass end to end on repeated runs, through the chat, with transcripts and charts as proof.
Notes
- Owner ask, 2026-11-01: a "magical wow experience so that someone can reliably investigate a market using the Expected Parrot tools". Beca already has every Expected Parrot demo tool registered.
- What exists: Plan #109 and PR #110 (branch
agent/claude-ep): the FastMCP wrapperscripts/subconscious/serve_ep.pyover EDSL 1.0.9, persona 3 "EP Research Agent" on the agent-dev stack (Claude Haiku 4.5 on Bedrock), eleven demos passing through the chat on the free test model, the persona promptdocs/subconscious/ep-research-prompt.md, the demo harnessep_demo.py. Test page: https://agent-dev.taila3aadb.ts.net:3011/app?agentId=3 (the persona is public since 5057534;MAX_LLM_CYCLES=20applied on agent-dev 2026-11-01). - Two owner experiences that define the gap. Starter "one open question, run it on the free test model": the answer was the EDSL placeholder "Hello, world X." Starter "choice-based conjoint, run on the test model, show the importance chart": 18 steps, every choice empty, no chart. Both are honest and both are empty. The test model proves mechanics only; the wow needs answers from a real model and sensible defaults the user never has to spell out.
- Credential facts (names only): Doppler project
becaholdsOPENAI_API_KEY,GEMINI_API_KEY,AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY,ANTHROPIC_TOKEN. The agent-dev Onyx Bedrock provider usesAWS_BEARER_TOKEN_BEDROCKin us-east-1 with model configurations includingus.anthropic.claude-haiku-4-5-20251001-v1:0. The wrapper's venv~/ep-venvhas no boto3. Plan Plan: Beca designs, recruits, runs and analyzes Expected Parrot studies from chat #109 states "no Anthropic API key"; agents run on existing subscriptions and recorded exceptions only. - Skills every session consults:
prior-art-checkbefore any new capability;researchfor facts outside the repo;grillingwithdomain-modelingfor decisions;prototypefor the reference transcript. - Standing preferences: never send external messages; secrets from Doppler, never printed; the eleven demos stay green (36 wrapper tests plus the live-chat demo run); no validation caveats in product UI beyond what the owner locks in the answer-style ticket; one request, one change.
- Execution override: a wrapper or prompt defect found while resolving a ticket may be fixed on
agent/claude-ep(PR feat: Beca designs, runs and analyzes Expected Parrot studies from chat (#109) #110) in the same session, with its demo rerun as proof. Decisions still resolve only through their ticket. - Proof for the eventual build: the three scripted market investigations and the eleven demos run through
ep_demo.pyagainst persona 3 with a non-admin account (~/.config/ep-mcp/test-user.envon agent-dev), three consecutive passes, transcripts and charts saved.
Decisions so far
- Find how the test model can answer every question type validly (#122): with no canned answer the test model says "Hello, world X", which only free text accepts; derive a valid per-question canned dict from the saved survey (verified 56/56 cells, conjoint importance drawn).
- Choose the respondent model and the pre-authorized spend (#120): Claude Haiku 4.5 on Bedrock; runs under one dollar go without asking; the test model is only a silent mechanics check.
- Establish what each candidate live model needs on agent-dev and costs (#121): Bedrock Haiku needs boto3>=1.39.12 in the wrapper venv plus a placeholder AWS_SECRET_ACCESS_KEY for EDSL's key lookup; a conjoint of 40 interviews costs about $0.16, a pilot about $0.01.
- Define what Beca does unasked when a user names a market (#123): follow Expected Parrot's own practice; plan approval and cost approval before a live run (supersedes the no-ask spend rule).
- Choose the Onyx surface for the thin layer into Expected Parrot (#130): Onyx Craft with Expected Parrot's
ep, EDSL and skills, Docker sandbox on agent-dev; the custom wrapper retires once Craft passes.
Not yet specified
- Human respondents through EDSL
humanizeand Prolific (family 12): sharp only after the live-model path works and the owner sets a recruiting budget. - PDF and image stimuli (family 13): the chat has no file-path route to the wrapper yet.
- Memory across conversations: whether a study started in one chat can be continued or compared in another, and how results are named and found later.
- Validation against human results: how a simulated finding is graded, once a human path exists.
Out of scope
- The Rehoboam path (#106, PR feat: approve, run and follow an experiment from the Beca chat (#106) #107).
- The Onyx customer-experience redesign (map #84); this effort uses what Onyx renders today (text, tables, Mermaid, images).
- Production deployment and spend beyond pilot runs on the dev stack.
- A separate research application or a second wrapper.
Activity
beca-oc commentedon Nov 2, 2026
Craft stand-up on agent-dev, 2026-11-02 (route from Choose the Onyx surface for the thin layer into Expected Parrot (#130)):
craft-ep(~/craft-ep/onyx/deployment/docker_compose), web on http://agent-dev.taila3aadb.ts.net:3390, API health ok. Sandbox imagecraft-ep/sandbox:v5.0.3-edsl1.0.9= upstream sandbox plusedsl==1.0.9andboto3>=1.39.12;ep inforuns inside it. Test admin in~/craft-ep/admin.env(mode 600).ee/onyx/utils/tier.py_self_hosted_tier). The olderLICENSE_ENFORCEMENT_ENABLED=falsedeveloper bypass is gone in v5.0.3 but still present in our fork. The Onyx enterprise licence (backend/ee/LICENSElines 18-20) permits development and testing without a subscription.beca-oc commentedon Nov 2, 2026
Craft stack rebuilt from fork main (5b4b5dd), 2026-11-02:
craft-fork, http://agent-dev.taila3aadb.ts.net:3390, API healthy. Tier reads Enterprise in developer mode (LICENSE_ENFORCEMENT_ENABLED=false, dev and testing only perbackend/ee/LICENSE), so Craft is available and a sandbox session starts.craft-ep/sandbox:fork-5b4b5ddae-edsl1.0.9(fork sandbox plusedsl==1.0.9,boto3>=1.39.12). Expected Parrot's 15edsl-researchskills imported fromexpectedparrot/ep-skillsat f8ad7545a through Onyx's GitHub skill import, all enabled.INVALID_PAYMENT_INSTRUMENT(AWS Marketplace subscription), from both the dev chat stack and the Craft stack, with the bearer key and with the IAM roles. Open-weight Bedrock models answer from the Craft container:moonshotai.kimi-k2.5,deepseek.v3.2,openai.gpt-oss-120b-1:0.beca-oc commentedon Nov 2, 2026
First end-to-end Expected Parrot study through Onyx Craft, agent-dev, 2026-11-02 (non-admin test user, session 01694b97):
create-studyskill; returned a plan (decision, two segments, questions incl. Van Westendorp, respondent model, sample) and waited.study_survey.py,study_agent_list.py,study_model_list.py,Makefile,create_results.py), ran 10 respondents on Llama 4 Maverick through Onyx's gateway, showed answers and a computed cost ($0.02 pilot, $0.24 projected), waited.results_full.csv, reported actual cost $0.72.What made it work (all configuration, plus one fork fix):
openai_compatibleservice pointed at Onyx's gateway with the sandbox's own token, so no new keys;openai[aiohttp]added to the sandbox image;EDSL_MAX_CONCURRENT_TASKS=8because 50 exhausted the API's Redis pool.tracing.py) and the output folder.OPENCODE_PROMPT_INACTIVITY_TIMEOUT_SECONDS=1800: the 200 s default cut full runs off mid-study.Known gaps: the reply names the chart folder instead of showing charts inline; it called simulated brand shares "market share"; a new user must switch on the 15 shared skills before they apply.
beca-oc commentedon Nov 2, 2026
Three gaps closed and every Expected Parrot demo run through Craft, 2026-11-02. Setup recorded in feat(craft): run Expected Parrot studies in Onyx Craft (#133).
ep-charts.tsOpenCode plugin appends each new chart's inline Markdown to the tool result; instructions require itep-skillscopied into the sandbox/tmpat start (Craft's OpenCode rules allow/tmp)create-studyand 4 more skill files from/tmp/ep-skillsDemos (quickstart, survey_logic, agents_scenarios, import_csv, labeling, pretest, results_cache, nps, two_pass, enrich_agents, conjoint): all pass; labeling, pretest and enrich_agents needed instruction fixes and a rerun (the agent coded by hand, used its own sub-agents as reviewers, or swapped a trait for a scenario). Known weak spot: the conjoint explained a noisy n=52 price dip as a positioning effect.
beca-oc commentedon Nov 2, 2026
Final-image rerun (sandbox image with
ep-charts.tsand/tmp/ep-skills, final Craft instructions), 2026-11-02, every flow three turns through Craft's API as non-admin users:{{ agent.career }}trait)Earlier brand-new-user meal-kit run on the same image: 4 inline charts, 0 banned terms, skills read from
/tmp/ep-skills. Labeling, nps, conjoint and agents_scenarios each needed one rerun because my test script was edited while running; the Craft turns themselves did not fail.beca-oc commentedon Nov 2, 2026
Plan families 12 and 13 through Craft on the final image, 2026-11-02 (three turns each, non-admin users):
.humanize()fails with "The API key you provided is invalid"; no Expected Parrot account or key exists (none in Dopplerbeca). Needs an Expected Parrot account to recruit humans.Banned share terms: 0 in every flow.
beca-oc commentedon Nov 2, 2026
Expected Parrot application tools through Craft, 2026-11-02. Every public package in Expected Parrot's directory except humanize-based
spence(skipped per owner) anduxtest(requires Python < 3.13; sandbox has 3.13) ran one three-turn flow as a non-admin user. Pass means the agent ran the tool's own commands, finished every turn, and hit no errors.37 of 37 pass: epiq, conversation, aivis, ariel, bewley, bower, burr, dewey, flyvbjerg, green, golden, helmer, ishikawa, kahn, katz, langley, mccall, mcda, messick, niles, oneheart, perla, premortem, pruefung, raiffa, roth, stanton, tommy, treffen, umriss, urteil, vorhersage, voting, zermelo, zugunruhe, zwicky, zwill.
Examples: aivis measured brand mentions across two models; green ran a choice experiment and charted attribute importance; ishikawa rendered its fishbone SVG inline; langley named low awareness as the least-refuted hypothesis; raiffa computed the expected-value policy; zugunruhe migrated a survey with
zugunruhe migrate. Inline charts loaded wherever made; the only banned-term hits were stanton citing Grand View Research's real market size as a source.Private packages (AISS, Earnings, Kean, Zitate) could not be tested.
beca-oc commentedon Nov 3, 2026
Reliability pass on Craft + Expected Parrot, 2026-11-03 (agent-dev, non-admin test users, PR #133 head
34aa5f624+ #132).ep run; fixed.ep-charts.ts;rebuild.shrebuilt the running stack; README updated.Findings that change the plan: GLM-5 cost $141 of agent tokens on 2026-11-02 alone; Craft now runs MiniMax M2.5 (about $0.70-1.00 per full study vs about $1.50). Remaining flaky checks are model compliance: the "simulated respondents" caveat on follow-ups and pilot size on conjoint.
beca-oc commentedon Nov 6, 2026
Closing on the owner's instruction. #133 merged as
aae1a142811a89597cec1318d851c5bdaf90ffd3, and #132 as0eb0fc21a. Not deployed to production; the dev stack on agent-dev runs8dba9f98b.Reliability was not fully reached, and that was accepted:
d4977e435d4977e435d4977e435e605485598dba9f98bThe plan gate is red without the gate and green with it. Starter prompts show for a non-admin user. The first chart came 81-367 s after approval in the clean runs, with one miss at 1,097 s against the 900 s limit.
Open gaps: the agent can use its own EDSL script in place of a tool the user names (7 of 39 tool cases), can answer a small coding task itself in plain text, and sometimes leaves out the simulated caveat. The 14 failed demo and tool cases were not rerun on a later commit. A tool case passes on
--help, so it does not prove the tool did the work.Lessons are in
deployment/craft-ep/README.mdunder "What we learned". Bedrock spend for this goal: $69.40.beca-oc commentedon Nov 6, 2026
New finding after the merge of #133, from another session's testing on the final build (
8dba9f98b). No code has changed for it yet.In session
db10138c, turn 4 asked "What market share would BrightBrew get at $2.99?". That session reports a reply that ran no tool and said 71% of office workers (14/20) and 45% of students (9/20) scored 4-5 at $2.99, without the word "simulated" on those shares.I recounted
results_full.csvfor that session: 20/20 office workers and 13/20 students scored 4-5 at $2.99. So the reported numbers do not match the study, and 14/20 is not 71%.The tests do not catch this.
no_banned_termsandsays_simulatedread only the reply's wording, and no check compares a follow-up reply's numbers with the Results file. So the 43/43 runs do not show that follow-up numbers are correct.