Add critical evaluation report of the website tutorials

devdocs/tutorial-evaluation/eval-byo-outputs.md

@@ -0,0 +1,60 @@
+# Evaluation: Bring your own outputs (`byo-outputs.qmd`)
+
+**Slot:** Getting started (2 of 3)
+
+**Overall verdict:** A genuinely useful page for the audience it targets, a modeler whose costs and effects already exist in a spreadsheet or a legacy simulator, and its placement right after `full-pipeline` is the correct call: the reader has already seen what `Outcomes` and `run_psa` are for, so being told they can skip the model-building step lands as intended rather than as a mystery. The code runs cleanly and every numeric claim in the prose checks out against the actual computation. Its main shortfall is pedagogical follow-through: several sections state what a plot or record generically shows without reading the specific rendered numbers back to the reader, which is exactly the "show, do not describe" standard the style guide sets. **Rating: Good.**
+
+## Strengths
+
+- **Sequencing is right.** Coming after `full-pipeline` (which already builds a toy model and calls `icer_table`, `evpi`, and `evppi_ranking`), this page's value proposition, "you do not need a model engine to use the analysis layer", is legible. Placed first, it would leave the reader unsure what `Outcomes` is supposed to represent or why `icer_table` exists at all.
+- **Verified correctness.** I ran the tutorial's own code (`N=2_000`, seed `20260704`) end to end. Every claim matches: all three interventions come back `ND` (non-dominated), the frontier ICERs land at approximately 20,971 and 38,074 so the willingness-to-pay threshold of 30,000 does sit between them as the text says, and `evppi_ranking` returns values sorted highest to lowest (`u_gain` first) exactly as claimed in "The ranking is sorted from the highest value downward" (line 101).
+- **Good cross-referencing instead of repetition.** `full-pipeline.qmd` explicitly defers to this page ("[Bring your own outputs] covers the analysis functions in more detail; here the focus is the run itself"), and this page in turn signposts forward to `mdm-cohort.qmd` and sideways to `parameter-inputs.qmd` ("bring your own outputs is the analogue on the outcome side", quoted from `parameter-inputs.qmd`). This is the "link rather than repeat" discipline the style guide asks for, done correctly across three pages.
+- **Named constants up top.** `WTP` and `N` are lifted to the top of the first code block with an inline comment explaining the choice of `WTP`: `WTP = 30_000.0  # sits between the two frontier ratios, so the decision is uncertain` (line 24). That is precisely the "why this threshold" content the guide asks for, verified against the actual output rather than asserted.
+- **Headings are parallel and correctly named.** "Simulating an external results table", "Converting the table to the standard structure", "Ranking parameters by their value of information" are all gerund phrases naming what the section does, matching the guide's rule.
+- **`as_outcomes` claim is accurate.** "`as_outcomes` accepts a `DataFrame` or a CSV path with `intervention`, `iteration`, `cost`, and effect columns, and returns `Outcomes`" (line 55) matches the function's actual signature and docstring in `src/heormodel/run/runner.py` exactly.
+
+## Weaknesses and gaps
+
+1. **Plots are described generically, not interpreted specifically.** The guide requires "a sentence or two after the output that interprets it: what the number or plot means for the decision, not a restatement of what was just computed" (writing_style.md, "What makes a tutorial good," item 3). Every plot in the "Plotting the cost-effectiveness plane, acceptability, and tornado" section gets only a *before*-the-code sentence describing what that chart type shows in general, and no *after*-the-code sentence reading off what this particular rendered figure actually shows:
+   - `plot_ce_plane(...)`: preceded by "the spread of each intervention's regions shows the uncertainty that the table's point estimates alone cannot" (line 105), and then the page moves straight to the acceptability curves with no sentence about what the New drug and Drug + monitoring regions actually look like relative to the 30,000 line.
+   - `plot_ceac(...)`: "making the crossover in the decision visible" (line 113) describes CEACs in general; it does not say where the crossover falls on this run's grid.
+   - `plot_tornado(...)`: "ranking each by how much it moves the incremental net benefit at the threshold" (line 119) is again a description of what a tornado plot does, not of which parameter actually tops this one. Given that `u_gain` already topped the `evppi_ranking` table two paragraphs earlier, one sentence tying the tornado back to that ("`u_gain` again dominates, consistent with the EVPPI ranking above") would cost nothing and would satisfy the guide's own standard: "a plot beats an adjective."
+   This is a real gap, not a nitpick: the style guide gives this exact package's chart types (CEAC) as its worked example of what a good, specific interpretation looks like, and this page does not clear that bar it sets for itself.
+
+2. **The final section ends with no interpretation at all.** After `capture_run` prints the run report (lines 130–135), the page moves directly to "Next: the [Markov cohort model]..." (line 138) with no sentence about what is actually in that printed record or why a reviewer would trust it. Every other section closes with at least one interpretive sentence; this one, the closing section of the tutorial, does not.
+
+3. **No target number stated up front.** The style guide is explicit: "Put a specific outcome in the title and on the first screen... the opening states the result they are working toward, the target number or figure, before the code that produces it." The opening paragraph (line 12) previews the four kinds of output (cost-effectiveness, value of information, plots, a reproducibility record) but no actual number, e.g., that the frontier will run all three interventions non-dominated with an EVPI near 550 per person at the 30,000 threshold. A reader has no way to sanity-check partway through that their run is heading somewhere sensible until the ICER table appears midway down the page.
+
+4. **The linked example script uses banned business jargon.** The page tells the reader to walk through `examples/byoo_example.py` "step by step" (line 12), but that script's own module docstring reads: "This script demonstrates **the adoption wedge** of `heormodel`" (examples/byoo_example.py, line 3). "Adoption wedge" is explicitly listed in the style guide as banned business jargon ("wedge, adoption wedge, go-to-market... describe product intervention, not a cost-effectiveness analysis"). The guide applies to "everything written in this repository," and a reader following the tutorial's own link will land on this line. It is outside the `.qmd` file itself but is a direct, load-bearing dependency of it.
+
+5. **Minor redundancy in framing.** The near-identical phrase "a spreadsheet export or a legacy simulator's output" (or a close variant) appears three times: in the front-matter description, in the opening paragraph (line 12), and again in the first section ("A real analysis would load this table from a spreadsheet or a legacy simulator's output," line 16). None of the three is wrong, but by the third repetition it reads as restating rather than adding. One of the three could be cut without losing anything, in keeping with "if a paragraph restates the previous one, delete it."
+
+## Writing-style issues (with quoted lines)
+
+Overall the prose is clean and largely compliant; I did not find em-dashes, exclamation marks, or any of the explicitly banned words (leverage, delve, seamless, comprehensive, robust, crucial, streamline) inside `byo-outputs.qmd` itself. The specific issues that do stand out:
+
+- **Acronym spelled out inconsistently across related quantities.** EVPI gets the full treatment: "The expected value of perfect information (EVPI) puts a monetary value on resolving every remaining source of uncertainty at once" (line 88). But CEAC and CEAF, the two other named quantities this same package computes and that the guide names as its own worked example ("`Cost-effectiveness acceptability curve (CEAC)`" appears verbatim in writing_style.md as the model heading), never get spelled out anywhere on the page. The prose only ever says "`ceac` and `ceaf` turn that uncertainty into acceptability curves" (line 74), using the lower-case function names as if they were the concept names. A reader who has not seen "cost-effectiveness acceptability curve" spelled out before is left to infer it from the code identifier.
+- **Heading names the action, not the computed quantity, where the guide's own example says to do the opposite.** The section "Ranking parameters by their value of information" (line 82) computes EVPPI (expected value of partial perfect information) via `evppi_ranking`, but the heading paraphrases the action rather than naming the quantity. The guide's exact wording: "Naming the canonical quantity a section computes takes priority over parallelism... a plain 'Calculating the expected value of perfect information (EVPI)' beats an evocative paraphrase of its purpose." A heading such as "Expected value of partial perfect information (EVPPI) by parameter" would both name the quantity and spell out the acronym in the one place the page never does.
+- **Section heading mixes three chart types under one label.** "Plotting the cost-effectiveness plane, acceptability, and tornado" (line 103) is a noun list rather than a single named action, unlike every other heading on the page. It is not wrong, but it is the one heading that reads as an inventory rather than a name.
+
+## Concept introduction and sequencing
+
+- **No forward reference problem.** Everything the page uses, `ParameterSet`, `Beta.from_mean_se`, `Normal`, `Gamma.from_mean_se`, correlated sampling, `SeedManager`, was introduced with at least a sentence of explanation in `full-pipeline.qmd` first. The one exception, the `correlation={("p_response", "u_gain"): 0.3}` argument (line 36), was only *mentioned* in `full-pipeline`'s prose ("a Spearman rank correlation can be imposed on any pair of parameters") without being shown in that page's code; this page is the first to actually use it. That is a small gap but not a serious one, since the prose sentence already primed the concept.
+- **Reasonable overlap with `voi.qmd`, not a duplication.** This page gives the EVPI formula ($\text{EVPI} = \mathbb{E}_\theta[\max_d \text{NMB}_d(\theta)] - \max_d \mathbb{E}_\theta[\text{NMB}_d(\theta)]$, line 90) that `voi.qmd` never repeats (there it stays prose-only: "EVPI is the difference between a decision made with perfect information and one made with current information"). `voi.qmd` in turn gives the EVPPI formula that this page never repeats. The split is deliberate and not redundant, but it does mean a "getting started" tutorial is the first place a reader meets a $\max_d \mathbb{E}_\theta$ expression, ahead of the dedicated `dsa.qmd` and `voi.qmd` pages in the "Sensitivity & VoI" group. That is defensible, since the formula clarifies what the functions compute before the reader uses them repeatedly, but it does mean this page is doing more mathematical lifting than its "getting started" position might suggest to a reader skimming the table of contents.
+- **Combines more than one workflow step, consistent with `full-pipeline`'s own pattern.** The page covers cost-effectiveness analysis, value-of-information ranking, three plot types, and a reproducibility record, four distinct topics in one tutorial. Taken alone this looks like it strains "teach one method per tutorial," but `full-pipeline.qmd` already establishes CEA + VoI as one bundled "standard workflow" rather than as two separate methods, and this page is consistent with that established pattern rather than introducing a new one. No change recommended here; flagged only because it is a real trade-off, not an oversight.
+- **Placement relative to `parameter-inputs.qmd` is correct.** The two pages are explicit analogues, "bring your own outputs is the analogue on the outcome side" (from `parameter-inputs.qmd`), one substituting the outcome table, the other substituting the parameter draws. Putting the outcome-side substitution before the parameter-side one is a reasonable choice, since results computed elsewhere (this page) is the more common practitioner starting point than an externally exported draw matrix (the next page).
+
+## Prioritized recommendations
+
+**Major**
+
+1. Add one interpretive sentence after each plot that reads the actual rendered result, not the general chart type: where the CEAC crossover falls, which region dominates in the cost-effectiveness plane, and which parameter tops the tornado (tie it back to the EVPPI ranking already shown). This is the single highest-value fix; it is the concrete gap between this page and the standard the style guide sets for itself.
+2. Add a closing sentence after the `capture_run` output stating what the printed record actually shows (e.g., which seed and iteration count a reviewer would see) before the "Next" link.
+3. Fix the module docstring in `examples/byoo_example.py` to drop "the adoption wedge of `heormodel`" (banned business jargon per the style guide) in favor of a plain description, since the tutorial sends readers directly to that file.
+
+**Minor**
+
+4. Spell out "cost-effectiveness acceptability curve (CEAC)" and, if used again, "cost-effectiveness acceptability frontier (CEAF)" on first use in prose rather than only ever writing the lower-case function names `ceac`/`ceaf`.
+5. Rename "Ranking parameters by their value of information" to name the computed quantity, e.g. "Expected value of partial perfect information (EVPPI) by parameter", matching the guide's own worked example for this exact package.
+6. State a specific target number in the opening paragraph (for example, the EVPI value or that all three interventions land non-dominated) so the reader has something concrete to check their own run against partway through.
+7. Cut one of the three near-identical repetitions of "a spreadsheet export or a legacy simulator's output" across the description, the opening paragraph, and the first section.