Repository navigation
Find how the test model can answer every question type validly #122
Description
Question
How can the free test model return a valid placeholder answer for every question type, so a mechanics check never shows "Hello, world X" or empty conjoint choices?
Surface how EDSL's test model handles a missing canned_response per question type (free_text, multiple_choice, checkbox, linear_scale, numerical, rank, list, yes_no), whether EDSL has any built-in random-valid-answer mode, and the smallest wrapper change in serve_ep.py _model_for that derives a valid canned answer per question from the saved survey when the caller gives none. Record what the placeholder path currently returns for the conjoint flow (every choice empty) and why.
Activity
beca-oc commentedon Nov 2, 2026
Resolved. Findings: docs/subconscious/research/test-model-valid-answers.md on research/test-model-answers.
Cause. With no
canned_response, EDSL 1.0.9's test model answers the fixed stringHello, world X(inference_services/services/test_service.py:65-70). Onlyfree_textaccepts it; multiple_choice, checkbox, linear_scale, numerical, rank, list, yes_no, likert_five, dict and top_k all reject it, so the invigilator storesanswer: Noneand an exception (invigilators/invigilators.py:529-533). The conjoint usesmultiple_choiceper task, so every choice wasNoneandep_conjoint.analyzeraised "No answered choice tasks".Built-ins.
QuestionBase._simulate_answerandTestServiceLanguageModel.set_canned_response(survey)exist but nothing wires them intoModel("test"), and both give invalid answers forrankandtop_k(they need option indices).Recommended change. Add
_default_canned(survey)toserve_ep.py: a dict keyed byquestion_namethat cycles options by question position (indices for rank/top_k, labels elsewhere, midpoint numbers, typed dict values, a short sentence for free text), merge it under any caller dict inrun_surveybefore_job, so it also feeds the cache-file digest (a cache primed withHello, world Xotherwise serves stale empties). Verified on agent-dev: 56/56 valid cells over 11 types with scenarios and agents; the real conjoint flow yields importance Price 50.4 / Color 23.8 / Size 25.8.