Repository navigation
Conversation
One FastMCP server wraps the open-source EDSL library and Expected Parrot's public chart assets so Beca can design, run and analyze a study from the chat without a separate research app. Tools build surveys, agents and scenarios, show the flow and prompts for review, estimate cost, run on the free test model or a named model, read results, and draw distribution and mean charts that return as images. Study objects persist as JSON under a workspace. Eight tests drive it as an MCP client on the keyless test model, starting with Expected Parrot's own quickstart. A persona prompt adapted from ep-skills and a demo harness that drives the chat through Onyx's API come with it. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ort, cache Adds the next Expected Parrot example families to the wrapper: CSV text becomes scenarios or agents (importing_data, files); agents may differ by instruction alone (prompt_sensitivity); checkbox labeling with bounds and targeted memory runs end to end (labeling, qualitative); results export as CSV text (results, dataset); and answers are cached in the workspace so an identical rerun spends nothing new and reports its cache hits (caching). Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… looped questions Expected Parrot's two-pass patterns feed one run into the next. The wrapper now flattens list answers to induce a codebook (results_values), turns chosen columns into the next study's scenarios (results_to_scenarios), adds answers as traits of an enriched audience (results_to_agents), and copies a question once per item with the item filled in (build_survey loop_over), which is how the NPS and concept-test notebooks ask one rating per item. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s, shares Follows Expected Parrot's public conjoint-study skill: balanced choice tasks by random search, one question per task with static option labels, one scenario per design version, counting-based part-worths and importance, segment utilities by agent trait, and logit choice shares. Charts: attribute importance, part-worths, position bias, segment heatmap, price curve. The estimator is a drafting aid and every output is labelled simulated. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
beca-oc
changed the title
Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t; bound errors Review findings, all fixed with a test each: - Cache hits were inferred from interview times questions, which was wrong whenever a stop or skip rule fired. They now come from EDSL's own per-answer cache_used record. - EDSL's cache key does not see the test model's canned answer, so a rerun with a new canned answer returned the old one. Each canned answer now has its own cache file. - A question skipped by a rule was reported as a validation failure. Only interviews with a recorded validation exception are counted now. - Market shares silently scored unknown attributes or levels as zero. A profile outside the design is refused with the allowed levels. - Two overlapping runs could drop each other's cache entries; the cache file is now written under a lock. - Runtime errors reached the model with multi-paragraph messages; they are one bounded line with a next step. - The conjoint arithmetic is tested against hand-built choices under option shuffling, including that segments never include agent names. Also: pasted CSV temp files are removed after loading, a conjoint name is validated for its derived scenario name before anything is saved, and an existing study is never overwritten. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Independent review (fresh reviewer, given the diff and the acceptance criteria, not my conclusions) at
Nits fixed: pasted CSV temp files removed after load, conjoint names validated before saving, existing studies never overwritten, chart tools declared without an output schema. The reviewer also verified that EDSL's Jinja rendering is sandboxed and its filter expressions reject imports and attribute access, and that CSV cells are not resolved as file paths. 27 tests pass on agent-dev. A re-review of the fixes is running. |
…view and its cause Re-review found that EDSL marks a rejected reply as served from the cache, so cache hits were overstated, and that a model error on every interview was reported as one validation failure. Hits now come from answers that exist minus entries the cache gained; failures come from EDSL's task history, with the exception kinds named. The cache lock is now one per cache file, so runs on different files proceed in parallel while runs sharing a file take turns. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Re-review at |
Verification found that an error EDSL fixed by retrying still counted as a failed interview, and that the warning named EDSL's exception collection instead of the exception. Only collections with unfixed exceptions count now, and the kind comes from each entry's exception type. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Verification at |
EDSL types every recorded error as a plain Exception, so the type name could not separate a rejected reply from a model error. The warning now carries the first line of each unfixed error, and the tests check for the validation text and the model error text respectively. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Verification at |
…at all Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Final verification at |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| reaches other private tailnet hosts, so it allows private addresses. | ||
|
|
||
| Paid models need provider keys in the wrapper's environment (`OPENAI_API_KEY`, | ||
| `GEMINI_API_KEY`, or AWS credentials for Bedrock). Without them only the |
There was a problem hiding this comment.
Configuring the documented GEMINI_API_KEY leaves EDSL's Google provider unauthenticated because it reads GOOGLE_API_KEY instead.
| @@ -0,0 +1,6 @@ | |||
| edsl[viz]==1.0.9 | |||
There was a problem hiding this comment.
A clean installation omits the provider SDKs required for local paid-model runs, so supplying the documented API keys does not enable those runs.
| counts = analysis["position_counts"] | ||
| labels = sorted(counts) | ||
| fig, ax = plt.subplots(figsize=(7, 4.5)) | ||
| bars = ax.bar(labels, [counts[level] for level in labels], color=blue) | ||
| ax.bar_label(bars, padding=3) | ||
| ax.axhline( | ||
| sum(counts.values()) / max(1, len(labels)), | ||
| color=red, | ||
| linestyle="--", |
There was a problem hiding this comment.
When a conjoint option is never chosen, the position-bias chart omits it and raises the unbiased baseline, masking the strongest position bias.
|
|
||
| ```bash | ||
| uv venv ~/ep-venv --python 3.12 | ||
| ~/ep-venv/bin/uv pip install -r scripts/subconscious/ep-requirements.txt |
There was a problem hiding this comment.
Following the setup instructions in a fresh environment fails because uv venv does not create the invoked bin/uv executable.
| values[level] = ( | ||
| math.log((count / total) / expected) if total and count else -2.0 | ||
| ) |
There was a problem hiding this comment.
A never-chosen conjoint level receives higher utility than a sufficiently rare chosen level, reversing their predicted market shares.
…ie or key sign-in for demos register_ep.py mirrors configure_executive.py: it acts through an admin's browser cookie jar or an Onyx API key, never a password. It creates or reuses the MCP server record, discovers the wrapper's tools live, saves them, and creates or updates the study persona with the prompt on an existing Bedrock model configuration. The demo harness accepts the same sign-in forms. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
| for cookie in jar: | ||
| self.http.cookies.set(cookie.name, cookie.value, domain=cookie.domain) |
There was a problem hiding this comment.
Importing browser cookies drops their Secure and Path restrictions, allowing session credentials to be sent over plain HTTP or outside their original path.
…s the API expects Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s, sanitise trait keys The live chat demos spent most of their six LLM cycles recovering from EDSL strictness: 'single_choice', 'scale', 'open' as types, opt_out or per-question skip_rules the class rejects, trait keys with spaces, and not-found errors that did not say what exists. The wrapper now maps the guesses, hoists per-question rules, drops unsupported fields with a warning, renames trait keys to identifiers, and lists saved names. Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…X_LLM_CYCLES Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
| for key in list(traits): | ||
| clean = _identifier(key) | ||
| if clean != key: | ||
| renamed[key] = clean | ||
| traits[clean] = traits.pop(key) | ||
| kwargs: dict[str, Any] = {"traits": traits} |
There was a problem hiding this comment.
Trait keys that normalize to the same identifier overwrite each other, saving an incomplete respondent profile while reporting success.
…rio hints, conjoint redesign
run_survey and estimate_cost take canned_response as a dict keyed by
question name so a mixed survey runs clean on the test model. Question text
'{{ answer.q }}' becomes '{{ q.answer }}' and rule expressions name the
question directly. A survey that fills scenario fields warns when run bare
and a conjoint runs on its own version scenarios. conjoint_design may be
redesigned under the same name. Column lookups ignore a wrong prefix.
Errors EDSL raises for scenario and rule mistakes carry a usable hint.
The pretest and results_cache demos are self-contained.
Refs #109
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
| { | ||
| "survey": name, |
There was a problem hiding this comment.
Redesigning a conjoint under an existing name replaces the specification used to analyze older results, producing utilities for levels those respondents never saw.
…booleans; prefer Anthropic on Bedrock Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
| continue | ||
| ends = {options[0], options[-1]} | ||
| if clean and ends <= set(clean): | ||
| spec["option_labels"] = {k: clean[k] for k in sorted(ends)} |
There was a problem hiding this comment.
A linear scale with labels for every value loses all intermediate labels during normalization, changing the survey respondents see.
…nts; demo prompts as requests Refs #109 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A private persona is visible only to its owner (the registering admin), and the dev stack disables the default assistant, so every other account saw "No Agent Available" on /app?agentId=3. Verified on agent-dev with a fresh non-admin account: persona 3 lists with 24 tools and the quickstart demo runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Description
Implements the wrapper side of #109. One FastMCP server,
scripts/subconscious/serve_ep.py, wraps the open-source Expected Parrot library (EDSL, MIT) and its public chart assets so Beca can design, run and analyze a study from the chat. Study objects persist as JSON in a workspace and are addressed by name. Charts return as MCP image content, which native Onyx saves into the chat. No Onyx backend or web code changes.Tools, grouped by Expected Parrot example family:
list_question_types,build_survey(rules, memory, piping,loop_overfor one question per item),build_agents(traits, per-agent instructions, codebook),build_scenarios,build_scenarios_from_csv,build_agents_from_csv.show_flow(Mermaid, which Onyx renders),show_prompts,estimate_cost.run_surveyon the freetestmodel by default or a named model; answers are cached in the workspace and reruns report cache hits; rejected answers are reported, not hidden.results_review,results_select,results_tally,results_values,results_export,results_to_scenariosandresults_to_agentsfor two-pass designs.plot_distribution,plot_means, and for conjointplot_conjoint(importance, part-worths, position bias, segment heatmap, price curve).conjoint_design,conjoint_analyze,conjoint_market_shareinep_conjoint.py, following Expected Parrot'sconjoint-studyskill (counting-based part-worths, a drafting aid, every output labelled simulated).With it: a persona prompt adapted from
ep-skills(docs/subconscious/ep-research-prompt.md), a demo harness that drives a chat through Onyx's API and keeps transcript and images as proof (ep_demo.py, eleven demos, one per family), requirements, anddocs/subconscious/ep-studies.md.Refs #109, notCloses.How Has This Been Tested?
On agent-dev at
ca3b369dc:testmodel, one per example family plus the input tolerance the live chat needed: type aliases, dropped fields, rewritten piping and rule syntax, per-question and list canned answers, scale ranges, trait-key sanitising, column aliases, not-found errors that list saved names, a conjoint that runs on its own version scenarios and can be redesigned.ruff checkandruff formatpass. Scripts are outside the repo'styscope.ep_demo.py; every one completed with the expected tool calls, and the agents_scenarios, nps and conjoint demos returned charts (2, 1 and 6 images). The per-demo table is on Plan: Beca designs, recruits, runs and analyzes Expected Parrot studies from chat #109. Transcripts and PNGs are on agent-dev under~/ep-demosand~/ep-demos-retry.Limits: Onyx's
MAX_LLM_CYCLES(default 6) bounds a study turn; the persona prompt keeps the demos inside it and the doc says how to raise it. No paid model has been called. Humanize/Prolific and PDF stimuli (families 12 and 13) are not covered.Additional Options
🤖 Generated with Claude Code