fix(perf): #875 root setup cost and budget — sparse linearizer, deadline-aware tightening

[jkitchin]

Summary

Contributes to #875 — deliberately not "Closes"; see Not verified at the bottom. The issue's acceptance criterion could not be measured in this environment.

Root setup on watercontamination0202 (106,711 vars / 107,209 rows) took 579 s against a 30 s time_limit with nodes=0 after #868 stopped it hanging. The four items the issue lists are two different defects:

Cost bug (~460 s, 23 of 29 stack samples). _fix_single_var_equalities scans == rows for a body affine in a single variable — every such body has one leaf. It cost O(rows × vars) anyway, because _linearize_affine_expr opens with np.zeros(n_vars), allocated and zeroed per call, and the caller then walked all 106,711 entries in Python to find the one nonzero. A deadline would have been the wrong fix. Adds _linearize_affine_expr_sparse as the core, with the dense _linearize_affine_expr kept as a view of it, and routes every per-constraint scan through it. Also replaces two bool(_linear_constraint_forms(...)) calls — which built and kept one dense row vector per constraint (91 GB here) to evaluate a boolean, one of them in the MccormickLPRelaxer constructor — with a short-circuiting _any_linear_constraint_form.

Budget bug (95.6 s). tighten_nonlinear_bounds had no deadline parameter; it now polls between rounds, between rules, and inside a rule's row scan. The innermost is what makes the poll real — one rule's sweep over 107k rows already exceeds a tight budget, so a between-rules-only poll would repeat #868's probing lesson. The convexity classifier's budget is 0.2 × time_limit recomputed per model object, so a reformulation restarts it and the fractions add up; it is now also clamped to the absolute model._solve_deadline.

Plus both independent items from the issue: gap_certified now requires both ends of a gap, and the QCP probe extractor's Hessian is support-restricted and sparse.

Correctness

• Cannot produce a false certificate
  • Sparse linearizer — same recognition rules, same ValueError refusals, same coefficients; the dense form is a literal view of the sparse core. Bound-neutral, and the panel below asserts that empirically.
  • Deadline-aware tightening — every early exit does strictly less work: a dropped round, a skipped rule, or an unscanned row leaves the box looser, never wrong. The one thing a budget can cost is an infeasibility proof the full pass would have found; infeasible then stays unset and B&B discovers it — weaker, never false.
  • The soundness catch worth reviewing. Truncating a row scan is not universally sound, so the API says so rather than leaving it to each rule's author. row_scan_is_anytime defaults to False. Thirteen rules opt in because each row's inference stands alone. PeriodicVariableBoundRule does not: it restricts a variable to one period precisely because nothing else in the model uses it, so a scan cut short could miss the disqualifying row and cut feasible points. Under a budget it is declined whole rather than run on a prefix. The safe default is the one a future rule inherits without thinking about it.
  • gap_certified — downgrade True → False only, so it cannot manufacture a certificate. The dual bound is deliberately kept: a bound with no incumbent is still a valid bound; it is the gap claim that was unfounded.
• No validation, fallback, or safety guard was weakened to make a check pass
  • One existing test changed meaning and was not loosened away: test_finite_zero_bound_is_a_valid_certificate asserted gap_certified is True for objective=None, bound=0.0. Its stated purpose is that a falsy-but-finite bound must not be mistaken for "no bound" — that assertion is kept and strengthened (bound survives), and a companion case with an incumbent now covers the certification half.
  • The dual-side guard was not widened to exempt unbounded; that would have weakened an existing check.

Tests run (state the result — numbers, not adjectives)

• pytest -m smoke → 828 passed, 15 skipped, 2 xpassed (identical to the pre-change baseline on this tree; the earlier "666 passed, 1 skipped" in this body was the template's placeholder example, not a measurement)
• pytest -m slow python/tests/test_adversarial_recent_fixes.py → 10 passed
• cargo test -p discopt-core → N/A, no Rust touched
• ruff check / ruff format --check / mypy → clean

Also ran a targeted nbt / QP / classifier / convexity / RLT / McCormick / relaxation / certification sweep: 2087 passed, 67 skipped, 2 xfailed. Two cyipopt failures in that sweep are pre-existing in my container (Ipopt is not installed) and reproduce on a clean tree.

Regression test

• Added python/tests/test_875_root_setup_budget.py (441 lines) and python/tests/test_875_sparse_qcp_probe_extraction.py (152 lines).

Fail-before / pass-after confirmed by running the assertions against the stashed tree:

defect	before	after
_fix_single_var_equalities scaling (16× vars, fixed rows)	18.8×	1.0×
tighten_nonlinear_bounds accepts a deadline	no	yes
gap_certified with objective=None	True	False
_any_linear_constraint_form exists	no	yes
QCP sparse-probe suite	5 of 7 fail	7 pass

Notable tests: test_a_truncated_pass_only_ever_loosens (anytime contract — a budgeted pass must return a superset of the unbudgeted box), test_a_future_deadline_changes_nothing (a non-binding budget changes nothing — same box, same stats), and test_a_row_scan_that_needs_completeness_is_never_truncated (behavioural, with the disqualifying use in the last row and a positive control). The QCP tests assert sp.issparse on every forced-sparse arm — without that the comparison is dense-vs-dense and proves nothing, the trap #868 recorded.

Performance change — measurement

No new flag: this is a bound-neutral refactor (CLAUDE.md §5 regime 1) plus budget plumbing that only fires when a budget binds, so there is nothing to graduate default-off.

Cost fix, entry experiment (400 single-variable equalities, varying only the variable count):

n_vars	before	after
2,000	0.068 s	0.001 s
8,000	0.240 s	0.001 s
32,000	0.951 s	0.001 s
128,000	3.650 s	0.002 s

Exactly linear in n_vars before (3.84× per 4× vars), flat after (0.94×) — the signature of the dense array rather than of the bodies, and it extrapolates to the measured ~460 s.

Bound-neutrality panel, 66-instance in-repo MINLPLib corpus, HEAD~1 vs HEAD, time_limit=5, arms run sequentially with no concurrent load:

	count	result
terminal in both arms	38	node_count, objective, bound, cert all exactly equal
limit-status in both	28	no objective drift, no bound moved
intended cert downgrade	2	bchoco08, casctanks — both objective=None, nodes=0 exits, dual bound bit-identical
status differences	2	nvs12, tspn10 — both pre-existing flakes, see below

Neither status difference is caused by this change. nvs12 flips optimal/feasible on both trees across 5 repetitions (before 2/5 optimal, after 3/5) with objective, bound and gap bit-identical every time — its certification, not its search, is timing-dependent at a tight budget (one before-run returned 233 nodes instead of 231). tspn10 returns feasible/nodes=1/obj=225.126 5/5 on both trees; the panel's single time_limit reading did not reproduce. nvs12's flakiness may deserve its own issue.

Not verified

The issue's definition of done was not measured. This environment has no MINLPLib snapshot and no network route to minlplib.org, so watercontamination0202 was never solved here. All evidence above is mechanism-level plus the corpus panel; the ~1.25×T claim for T=30 and T=60 is unconfirmed.

The decisive test is written and skips without the snapshot:

pytest -m slow python/tests/test_875_root_setup_budget.py -k watercontamination

Arithmetic from the issue's attributed table suggests ~1.0–1.1×T, but that is an estimate and it does not account for the residual root-relaxation build cost. Please run that test before closing #875 — hence "Contributes to", not "Closes".