Add examples/python_fraction: measuring the fraction of run time spent in Python - #825
Open
DLWoodruff wants to merge 9 commits into
Open
Add examples/python_fraction: measuring the fraction of run time spent in Python#825DLWoodruff wants to merge 9 commits into
DLWoodruff wants to merge 9 commits into
Conversation
The profiling scripts take SOLVER from the environment, so the demo script's default solver is unrelated to this work. Keeps the branch's net diff confined to examples/python_fraction/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every non-empty Python file needs the header; test_headers.py was failing on make_scalene_latex_table.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The measured Python fraction is not a single number: it tracks how much work the solver does per subproblem. The sslp_15_45 instances sit at 8.7-9.6% Python, matching the earlier estimate, while cases whose subproblems are trivial run 58-76% Python. Both regimes attribute essentially all time to spopt.py:337, the Pyomo solve call, which is 6.6% Python / 88.2% native for sslp_15_45_15 but 48.2% / 5.8% for farmer60; the per-solve Python cost is roughly fixed, so it is invisible when the solver is busy and dominant when it is not. Experiments: - Eight cases, three repetitions each, so the tables can show run-to-run spread instead of a single sample. - Bundled cases added, since subproblem size drives the answer. farmer240 vs farmer240_bun10 is a controlled pair. Bundling is not monotone in the Python share: farmer goes 76.1% -> 62.6% but sslp goes 9.6% -> 12.5%, because bundling cut solver work faster than Python work. It won on wall time in both cases. - Unprofiled runs (PROFILE=0) give an overhead column; scalene costs 1.42-1.55x on the Python-heavy cases and 0.99-1.11x on the solver-bound ones, confirming its overhead is Python-side. Tooling: - Read the Python/native/system split out of the scalene JSON instead of scraping `scalene view --cli`. The scraper had stopped working entirely: scalene colourizes even when writing to a pipe, so the row regex matched nothing and every column came out em-dashed. The JSON also avoids --reduced's undercount and the CLI's whole-percent rounding. --from-cli keeps the repaired old path for cross-checking. - Retry a repetition when scalene dies during startup with a KeyError out of importlib, which happens intermittently before any mpi-sppy code runs. - run_experiments.bash / make_tables.bash replace tests.sh, test_sslp.sh and farmer_summary.bash. Note that --solver-name gurobi is Pyomo's file-based interface, which writes an LP file and parses a solution file on every solve; on small subproblems that Python work dominates. The report uses gurobi_persistent throughout and notes the contrast. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The writeup is the point of this directory, so ship the PDF rather than requiring a LaTeX toolchain to read it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #825 +/- ##
==========================================
+ Coverage 76.49% 76.79% +0.30%
==========================================
Files 175 177 +2
Lines 23236 23456 +220
==========================================
+ Hits 17774 18014 +240
+ Misses 5462 5442 -20 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
examples/python_fraction/: an experiment suite and a short writeup measuringwhat fraction of an mpi-sppy run is spent in Python as opposed to compiled code
(solver, numpy, MPI). Nothing outside that directory is touched, so there is no
effect on the library.
What is in it
run_experiments.bash— runs each case under scalene,repeating each case so the writeup can show run-to-run spread rather than a single
sample.
PROFILE=0reruns the same cases unprofiled, which is what the overheadcolumn needs.
scalene_wrapper.bash— rank-aware launcher sompiexecgives each rank its ownprofile file.
scalene_totals.py— pulls the Python/native/system split out of the scalene JSON.summarize_reps.py,make_scalene_latex_table.py,make_tables.bash— aggregate therepetitions into the LaTeX tables;
make_tables.bashregenerates tables from profilesalready on disk without rerunning anything.
python_fraction.texplus the two generated table files — the writeup. The builtpython_fraction.pdfis included as well, so the results can be read without a LaTeXtoolchain.
readme.rst— how to run it.Every case runs the same three cylinders (PH hub, lagrangian spoke, xhatshuffle spoke,
one rank each) with the convergence tolerance set to zero, so runs end on the iteration
limit and only the model and instance change between cases.
What it found
There is no single number: the fraction depends almost entirely on how much work the
solver does per subproblem. With
gurobi_persistent,sslp_15_45sits at 8.7–9.6%Python, while every case with small subproblems sits far above that, up to 76% for
farmer240.sslp_5_25_50has more scenarios thansslp_15_45_15and yet spends sixtimes the fraction of its time in Python, so what matters is the size of each subproblem
and not how many there are.
The mechanism is visible in the line attribution: each solve costs a roughly fixed amount
of Python work (rebuild the proximal objective, hand it to the solver, read the solution
back) that is amortized over the solver's work.
Two secondary results:
both cases tried (farmer240 82.9s → 20.9s; sslp_15_45_15 104.8s → 73.7s) but moved the
Python share down in the first and up in the second.
sslp_5_25_50goes from 58.1% to 75.4% Python, and even the solver-bound
sslp_15_45_10, whose walltime barely moves, goes from 8.7% to 31.8%.
Caveats stated in the writeup
Scalene's overhead falls mostly on Python, so the reported Python percentages are upper
bounds; the overhead column tracks this (0.99–1.11x for the sslp cases, 1.42–1.55x for
the farmer cases). These are Pyomo models and Pyomo time counts as Python, so a good deal
of what is labelled Python is Pyomo rather than mpi-sppy. One machine, one solver, three
repetitions per case.
The suite is likely to be fragile across scalene releases, since scalene changes its
output format; it was last run against scalene 2.0.1.