Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 59 additions & 51 deletions docs/tutorials/PROfit_Tutorial_v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,14 @@ numbers should match.
1. [PROfit conceptual introduction](#1-profit-conceptual-introduction)
2. [The PROfit XML](#2-the-profit-xml)
3. [General arguments and how stuff works](#3-general-arguments-and-how-stuff-works)
4. [Subcommand `plot` — exploring your spectra](#4-subcommand-plot--exploring-your-spectra)
5. [Subcommand `global` — fitting and fitter configuration](#5-subcommand-global--fitting-and-fitter-configuration)
6. [Subcommand `profile` — 1D profiled Δχ²](#6-subcommand-profile--1d-profiled-χ²)
7. [Subcommand `surface` — 2D Wilks surfaces and AMR](#7-subcommand-surface--2d-wilks-surfaces-and-amr)
8. [Feldman-Cousins: `fc` and `fc-adaptive`](#8-feldman-cousins-fc-and-fc-adaptive)
9. [PROjector — two-stage pre-fit / projected fits](#9-projector--two-stage-pre-fit--projected-fits)
10. [PROletariat — grid submission: `proletariat`](#10-proletariat--grid-submission-proletariat)
4. [Subcommand `process` — loading your files](#4-subcommand-process--loading-your-files)
5. [Subcommand `plot` — exploring your spectra](#5-subcommand-plot--exploring-your-spectra)
6. [Subcommand `global` — fitting and fitter configuration](#6-subcommand-global--fitting-and-fitter-configuration)
7. [Subcommand `profile` — 1D profiled Δχ²](#7-subcommand-profile--1d-profiled-χ²)
8. [Subcommand `surface` — 2D Wilks surfaces and AMR](#8-subcommand-surface--2d-wilks-surfaces-and-amr)
9. [Feldman-Cousins: `fc` and `fc-adaptive`](#9-feldman-cousins-fc-and-fc-adaptive)
10. [PROjector — two-stage pre-fit / projected fits](#10-projector--two-stage-pre-fit--projected-fits)
11. [PROletariat — grid submission: `proletariat`](#11-proletariat--grid-submission-proletariat)

Appendices:

Expand Down Expand Up @@ -430,7 +431,21 @@ MC event loop (quick tests only).

---

# 4. Subcommand `plot` — exploring your spectra
# 4. Subcommand `process` — loading your files

```
Usage: PROfit process [OPTIONS]

Options:
-h,--help Print this help message and exit
```

The process command loops through events in the input data, prediction, and systematic files and processes them to efficiently store only the necessary information described in the xml.
The output is _prop.bin and _syst.bin files which are efficiently loaded by the remaining PROfit commands.

---

# 5. Subcommand `plot` — exploring your spectra

```
Usage: PROfit plot [OPTIONS]
Expand Down Expand Up @@ -557,7 +572,7 @@ PDFs are directly comparable by eye.

---

# 5. Subcommand `global` — fitting and fitter configuration
# 6. Subcommand `global` — fitting and fitter configuration

`global` performs one full global best fit of all physics + spline parameters
and draws the post-fit results. It takes no subcommand options of its own —
Expand Down Expand Up @@ -700,7 +715,7 @@ is deliberately easy — they all implement the same `PROmetric` interface.

---

# 6. Subcommand `profile` — 1D profiled Δχ²
# 7. Subcommand `profile` — 1D profiled Δχ²

```
Usage: PROfit profile [OPTIONS]
Expand Down Expand Up @@ -798,7 +813,7 @@ PROfit -x tutorial.xml -t TUT -o profso --seed 405 -n 8 --syst-only profile --pr

---

# 7. Subcommand `surface` — 2D Wilks surfaces and AMR
# 8. Subcommand `surface` — 2D Wilks surfaces and AMR

`surface` maps Δχ² over a 2D grid of two physics parameters, profiling over
everything else at each point. Contours at Wilks-theorem critical values
Expand Down Expand Up @@ -870,21 +885,21 @@ PROfit -x tutorial.xml -t TUT -o surfnoflux --seed 405 -n 8 --exclude-systs Flux

### PROcurve: watching the pulls along a 1D path

`--curve-mode x1 y1 x2 y2` (values in the axes' native — here log10 — space)
`--curve-mode param1start param2start param1end param2end` (values in the axes' native — here log10 — space)
replaces the 2D scan with a 1D walk from point A to point B across the
(x, y) plane, fitting the nuisances at each step and plotting how every pull
(param1, param2) plane, fitting the nuisances at each step and plotting how every pull
evolves along the path. It's the quickest way to see *which* systematics
bend to absorb an oscillation signal as you approach it. `-g` sets the
number of points on the path.

```bash
PROfit -x tutorial.xml -t TUT -o curve1 --seed 405 -n 8 \
surface $AXES -g 20 --curve-mode -3 -1 -1 1
surface $AXES -g 20 --curve-mode -1 -3 1 -1
```

<img src="figures/TUT_curve1_PROcurve.png" width="800"/>

*`TUT_curve1_PROcurve.pdf` — path across the plane + every nuisance parameter's best-fit value along it, here from (sin²2θμe, Δm²) = (10⁻³, 0.1 eV²) to (10⁻¹, 10 eV²).*
*`TUT_curve1_PROcurve.pdf` — path across the plane + every nuisance parameter's best-fit value along it, here from (Δm², sin²2θμe) = (0.1 eV², 10⁻³) to (10 eV², 10⁻¹).*

### Adaptive mesh refinement: `--surface-amr`

Expand Down Expand Up @@ -930,10 +945,8 @@ PROfit -x tutorial.xml -t TUT -o surfbrz --seed 405 -n 16 --log surfbrz.log \
surface $AXES --surface-amr --amr-initial 10 --amr-levels 2 --brazil-band
```

> 📷 **PLOT PLACEHOLDER** — `TUT_surfbrz_surface.pdf`
> (median sensitivity with ±1σ/±2σ Brazil bands — regenerate with
> `RUN_EXPENSIVE=1 make_tutorial_plots.sh` + `make_tutorial_figures.sh`, then uncomment:)
> <!-- <img src="figures/TUT_surfbrz_surface.png" width="600"/> -->
This command does not make a Brazil band itself, but outputs information that could be used to make a Brazil band externally.


### A note on models: parameterize in the variable you plot

Expand All @@ -953,7 +966,7 @@ if the end goal is a contour in an effective angle, fit in that angle.

---

# 8. Feldman-Cousins: `fc` and `fc-adaptive`
# 9. Feldman-Cousins: `fc` and `fc-adaptive`

Wilks' theorem (Δχ² cuts of 2.30/5.99/...) assumes Gaussian-land: no physical
boundaries, no degenerate minima. Oscillation fits violate both, so for
Expand Down Expand Up @@ -983,7 +996,7 @@ PROfit -x tutorial.xml -t TUT -o fc1 --seed 405 -n 8 \
```

Output is `TUT_fc1_FC.root` containing a TTree with, per universe, the two
χ² values, Δχ², and the best-fit parameters — from which you extract the
χ² values and the best-fit parameters — from which you extract the
90%/95% quantiles and compare to the Wilks values. This is the honest but
brute-force approach: to calibrate a whole *contour* you would repeat it at
every grid point, which is exactly what `fc-adaptive` automates.
Expand Down Expand Up @@ -1152,44 +1165,38 @@ into unsampled territory, and the top-up is what makes it decidable.

Outputs along the way:

> 📷 **PLOT PLACEHOLDER** — `TUT_afc_metamesh.pdf`
> (the meta-mesh: cell refinement levels, concentrated where throws put the contour —
> regenerate with `make_tutorial_plots.sh` + `make_tutorial_figures.sh`, then uncomment:)
> <!-- <img src="figures/TUT_afc_metamesh.png" width="600"/> -->
<img src="figures/TUT_afc_metamesh.png" width="600"/>
*`TUT_afc_metamesh.pdf` — the meta-mesh: cell refinement levels, concentrated where throws put the contour.*

> 📷 **PLOT PLACEHOLDER** — `TUT_afc_throws.pdf`
> (the Wilks-prepass throw contours that built the mesh)
> <!-- <img src="figures/TUT_afc_throws.png" width="600"/> -->
<img src="figures/TUT_afc_throws.png" width="600"/>
*`TUT_afc_throws.pdf` — the Wilks-prepass throw contours that built the mesh.*

> 📷 **PLOT PLACEHOLDER** — `TUT_afc_bank_summary.pdf` (PE bank occupancy per level)
> <!-- <img src="figures/TUT_afc_bank_summary.png" width="600"/> -->
<img src="figures/TUT_afc_bank_summary.png" width="600"/>
*`TUT_afc_bank_summary.pdf` — PE bank occupancy per level.*

> 📷 **PLOT PLACEHOLDER** — `TUT_afc_asimov_contour.pdf`
> (FC-corrected contour vs the Wilks contour)
> <!-- <img src="figures/TUT_afc_asimov_contour.png" width="600"/> -->
<img src="figures/TUT_afc_asimov_contour.png" width="600"/>
*`TUT_afc_asimov_contour.pdf` — FC-corrected contour vs the Wilks contour.*

> 📷 **PLOT PLACEHOLDER** — `TUT_afc_asimov_verdict.pdf`
> (per-cell FC vs Wilks verdict map)
> <!-- <img src="figures/TUT_afc_asimov_verdict.png" width="600"/> -->
<img src="figures/TUT_afc_asimov_verdict.png" width="600"/>
*`TUT_afc_asimov_verdict.pdf` — per-cell FC vs Wilks verdict map.*

> 📷 **PLOT PLACEHOLDER** — `TUT_afc_brazil_band.pdf`
> (FC-corrected Brazil band from the bank)
> <!-- <img src="figures/TUT_afc_brazil_band.png" width="600"/> -->
<img src="figures/TUT_afc_brazil_band.png" width="600"/>
*`TUT_afc_brazil_band.pdf` — FC-corrected Brazil band from the bank.*

Determinism note: with `-n 1` and a fixed `--seed` the entire pipeline is
bit-reproducible; multithreaded runs are statistically equivalent.

---

# 9. PROjector — two-stage pre-fit / projected fits
# 10. PROjector — two-stage pre-fit / projected fits

PROjector answers "what does my near detector buy me?" properly. Instead of
fitting ND and FD simultaneously every time, you (1) fit **only** the ND
channels once and save the nuisance posterior, then (2) run any FD study with
those channels masked out and the saved posterior installed as a correlated
prior. Same statistical content as the joint fit (to the Gaussian
approximation), at a fraction of the per-fit cost — which matters enormously
for FC studies.
prior. Same statistical content as the joint fit (with a Gaussian
approximation and a no-near-detector-oscillation approximation), at a fraction
of the per-fit cost — which matters enormously for FC studies.

### Stage 1: the pre-fit

Expand Down Expand Up @@ -1237,14 +1244,13 @@ the constraint file), masks the pre-fit channels OUT of the χ² (active-bins
mask + zeroed data), and installs (θ̂, Σ) as a fully correlated Gaussian prior
on the promoted spline parameters.

> 📷 **PLOT PLACEHOLDER** — `TUT_pjprof_PROfile_1sigma.pdf`
> (projected nuisance constraints — compare against the joint-fit `TUT_prof1_PROfile_1sigma.pdf` —
> regenerate with `make_tutorial_plots.sh` + `make_tutorial_figures.sh`, then uncomment:)
> <!-- <img src="figures/TUT_pjprof_PROfile_1sigma.png" width="800"/> -->
<img src="figures/TUT_pjprof_PROfile_1sigma.png" width="800"/>

> 📷 **PLOT PLACEHOLDER** — `TUT_pjsurf_surface.pdf`
> (projected FD-only sensitivity with the ND constraint as prior, vs the joint surface `TUT_surfamr_surface.pdf`)
> <!-- <img src="figures/TUT_pjsurf_surface.png" width="600"/> -->
*`TUT_pjprof_PROfile_1sigma.pdf` — projected nuisance constraints — compare against the joint-fit `TUT_prof1_PROfile_1sigma.pdf`.*

<img src="figures/TUT_pjsurf_surface.png" width="800"/>

*`TUT_pjsurf_surface.pdf` — projected FD-only sensitivity with the ND constraint as prior, vs the joint surface `TUT_surfamr_surface.pdf`.*

### Rules and closure checks

Expand All @@ -1261,9 +1267,11 @@ on the promoted spline parameters.
constraint only — correlations enter the pull term, not the throws (PROfit
prints a runtime warning to remind you).

Note that autocorrelation and harmonic scan plots will not be filled properly with the projector option.

---

# 10. PROletariat — grid submission: `proletariat`
# 11. PROletariat — grid submission: `proletariat`

Everything above runs on one machine. For the artifacts that are genuinely
expensive at scale — FC PE banks above all — the workflow is: ship PROfit and
Expand Down
Binary file added docs/tutorials/figures/TUT_afc_asimov_contour.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/tutorials/figures/TUT_afc_asimov_verdict.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/tutorials/figures/TUT_afc_brazil_band.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/tutorials/figures/TUT_afc_metamesh.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/tutorials/figures/TUT_afc_throws.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_curve1_PROcurve.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_glob1_PROglobal_hists.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/tutorials/figures/TUT_pjsurf_surface.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_plotcv_PROplot_Covar.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_plotcv_PROplot_Variable_0_CV.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_plotcv_fractional_systematics.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_plotinj_PROplot_Osc.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_plotspl_PROplot_Spline.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_probe1_PROfile.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_prof1_PROfile.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_prof1_PROfile_1sigma.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_prof1_PROfile_hists.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/tutorials/figures/TUT_surf1_surface.png
Binary file modified docs/tutorials/figures/TUT_surfamr_amr_mesh.png
Binary file modified docs/tutorials/figures/TUT_surfamr_surface.png
Binary file added docs/tutorials/figures/TUT_surfbrz_surface.png
15 changes: 9 additions & 6 deletions docs/tutorials/make_tutorial_plots.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#
# Environment overrides:
# PROFIT_BIN PROfit executable (default: <repo>/build/bin/PROfit)
# PROFIT_XML Reference XML config (default: <repo>/docs/tutorials/fake_sbn_v2.xml)
# PROFIT_TEST_MCDIR Directory holding fake_sbn_mc_{ND,FD}.root
# (default: directory containing the XML; the XML's
# hardcoded /exp/... path is rewritten to it)
Expand All @@ -32,9 +33,11 @@ set -uo pipefail

REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"

XML_IN="$REPO/working_dir/Neutrino2026/fake_sbn_v2.xml"
XML_IN="${PROFIT_XML:-$REPO/docs/tutorials/fake_sbn_v2.xml}"
BIN="${PROFIT_BIN:-$REPO/build/bin/PROfit}"
MCDIR="${PROFIT_TEST_MCDIR:-$(cd "$(dirname "$XML_IN")" && pwd)}"
# realpath: absolutize before the cd into RUNDIR below, so relative
# PROFIT_TEST_MCDIR values resolve from the invocation directory.
MCDIR="$(realpath "${PROFIT_TEST_MCDIR:-$(dirname "$XML_IN")}")"
RUNDIR="${TUTORIAL_OUTDIR:-$REPO/docs/tutorials/tutorial_run}"
NTHREADS="${NTHREADS:-8}"
SEED="${SEED:-405}"
Expand All @@ -47,9 +50,9 @@ TAG="TUT"
mkdir -p "$RUNDIR/logs"
cd "$RUNDIR"

# Localize the XML: the reference config points its MCFile entries at a fixed
# /exp/... path; rewrite that directory to wherever the fake MC actually lives.
sed "s|/exp/uboone/data/users/markross|$MCDIR|g" "$XML_IN" > tutorial.xml
# Localize the XML: point every MCFile entry at $MCDIR, regardless of what
# path (bare filename, /exp/..., etc.) the source XML uses.
sed -E "s|filename=\"[^\"]*(fake_sbn_mc_[NF]D\.root)\"|filename=\"$MCDIR/\1\"|g" "$XML_IN" > tutorial.xml

for f in "$MCDIR/fake_sbn_mc_ND.root" "$MCDIR/fake_sbn_mc_FD.root"; do
[ -f "$f" ] || { echo "ERROR: fake MC file missing: $f (set PROFIT_TEST_MCDIR)"; exit 99; }
Expand Down Expand Up @@ -116,7 +119,7 @@ run_step surf1 surface -g 30 "${AXES[@]}"
run_step surfstat --statonly surface -g 30 "${AXES[@]}"
run_step surfamr surface "${AXES[@]}" --surface-amr --amr-initial 10 --amr-levels 3 \
--amr-levels-chi2 2.30 5.99
run_step curve1 surface "${AXES[@]}" -g 20 --curve-mode -3 -1 -1 1
run_step curve1 surface "${AXES[@]}" -g 20 --curve-mode -1 -3 1 -1
if [ "$RUN_EXPENSIVE" = "1" ]; then
run_step surfnoflux --exclude-systs Flux1 Flux2 Flux3 surface -g 30 "${AXES[@]}"
run_step surfbrz surface "${AXES[@]}" --surface-amr --amr-initial 10 --amr-levels 2 \
Expand Down
Loading