Skip to content

Add examples/python_fraction: measuring the fraction of run time spent in Python - #825

Open
DLWoodruff wants to merge 9 commits into
Pyomo:mainfrom
DLWoodruff:python_fraction
Open

Add examples/python_fraction: measuring the fraction of run time spent in Python#825
DLWoodruff wants to merge 9 commits into
Pyomo:mainfrom
DLWoodruff:python_fraction

Conversation

@DLWoodruff

@DLWoodruff DLWoodruff commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Adds examples/python_fraction/: an experiment suite and a short writeup measuring
what fraction of an mpi-sppy run is spent in Python as opposed to compiled code
(solver, numpy, MPI). Nothing outside that directory is touched, so there is no
effect on the library.

What is in it

  • run_experiments.bash — runs each case under scalene,
    repeating each case so the writeup can show run-to-run spread rather than a single
    sample. PROFILE=0 reruns the same cases unprofiled, which is what the overhead
    column needs.
  • scalene_wrapper.bash — rank-aware launcher so mpiexec gives each rank its own
    profile file.
  • scalene_totals.py — pulls the Python/native/system split out of the scalene JSON.
  • summarize_reps.py, make_scalene_latex_table.py, make_tables.bash — aggregate the
    repetitions into the LaTeX tables; make_tables.bash regenerates tables from profiles
    already on disk without rerunning anything.
  • python_fraction.tex plus the two generated table files — the writeup. The built
    python_fraction.pdf is included as well, so the results can be read without a LaTeX
    toolchain.
  • readme.rst — how to run it.

Every case runs the same three cylinders (PH hub, lagrangian spoke, xhatshuffle spoke,
one rank each) with the convergence tolerance set to zero, so runs end on the iteration
limit and only the model and instance change between cases.

What it found

There is no single number: the fraction depends almost entirely on how much work the
solver does per subproblem. With gurobi_persistent, sslp_15_45 sits at 8.7–9.6%
Python, while every case with small subproblems sits far above that, up to 76% for
farmer240. sslp_5_25_50 has more scenarios than sslp_15_45_15 and yet spends six
times the fraction of its time in Python, so what matters is the size of each subproblem
and not how many there are.

The mechanism is visible in the line attribution: each solve costs a roughly fixed amount
of Python work (rebuild the proximal objective, hand it to the solver, read the solution
back) that is amortized over the solver's work.

Two secondary results:

  • Bundling should be judged on wall time, not on the Python fraction. It cut wall time in
    both cases tried (farmer240 82.9s → 20.9s; sslp_15_45_15 104.8s → 73.7s) but moved the
    Python share down in the first and up in the second.
  • A non-persistent solver interface adds a lot of per-solve Python work: sslp_5_25_50
    goes from 58.1% to 75.4% Python, and even the solver-bound sslp_15_45_10, whose wall
    time barely moves, goes from 8.7% to 31.8%.

Caveats stated in the writeup

Scalene's overhead falls mostly on Python, so the reported Python percentages are upper
bounds; the overhead column tracks this (0.99–1.11x for the sslp cases, 1.42–1.55x for
the farmer cases). These are Pyomo models and Pyomo time counts as Python, so a good deal
of what is labelled Python is Pyomo rather than mpi-sppy. One machine, one solver, three
repetitions per case.

The suite is likely to be fragile across scalene releases, since scalene changes its
output format; it was last run against scalene 2.0.1.

DLWoodruff and others added 9 commits July 29, 2026 13:32
The profiling scripts take SOLVER from the environment, so the demo
script's default solver is unrelated to this work. Keeps the branch's
net diff confined to examples/python_fraction/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every non-empty Python file needs the header; test_headers.py was
failing on make_scalene_latex_table.py.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The measured Python fraction is not a single number: it tracks how much
work the solver does per subproblem. The sslp_15_45 instances sit at
8.7-9.6% Python, matching the earlier estimate, while cases whose
subproblems are trivial run 58-76% Python. Both regimes attribute
essentially all time to spopt.py:337, the Pyomo solve call, which is
6.6% Python / 88.2% native for sslp_15_45_15 but 48.2% / 5.8% for
farmer60; the per-solve Python cost is roughly fixed, so it is
invisible when the solver is busy and dominant when it is not.

Experiments:
- Eight cases, three repetitions each, so the tables can show
  run-to-run spread instead of a single sample.
- Bundled cases added, since subproblem size drives the answer.
  farmer240 vs farmer240_bun10 is a controlled pair. Bundling is not
  monotone in the Python share: farmer goes 76.1% -> 62.6% but sslp
  goes 9.6% -> 12.5%, because bundling cut solver work faster than
  Python work. It won on wall time in both cases.
- Unprofiled runs (PROFILE=0) give an overhead column; scalene costs
  1.42-1.55x on the Python-heavy cases and 0.99-1.11x on the
  solver-bound ones, confirming its overhead is Python-side.

Tooling:
- Read the Python/native/system split out of the scalene JSON instead
  of scraping `scalene view --cli`. The scraper had stopped working
  entirely: scalene colourizes even when writing to a pipe, so the row
  regex matched nothing and every column came out em-dashed. The JSON
  also avoids --reduced's undercount and the CLI's whole-percent
  rounding. --from-cli keeps the repaired old path for cross-checking.
- Retry a repetition when scalene dies during startup with a KeyError
  out of importlib, which happens intermittently before any mpi-sppy
  code runs.
- run_experiments.bash / make_tables.bash replace tests.sh,
  test_sslp.sh and farmer_summary.bash.

Note that --solver-name gurobi is Pyomo's file-based interface, which
writes an LP file and parses a solution file on every solve; on small
subproblems that Python work dominates. The report uses
gurobi_persistent throughout and notes the contrast.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The writeup is the point of this directory, so ship the PDF rather than
requiring a LaTeX toolchain to read it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.79%. Comparing base (1102c6b) to head (5faa570).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #825      +/-   ##
==========================================
+ Coverage   76.49%   76.79%   +0.30%     
==========================================
  Files         175      177       +2     
  Lines       23236    23456     +220     
==========================================
+ Hits        17774    18014     +240     
+ Misses       5462     5442      -20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant