feat(#860): widen the LP-per-node engine to mixed-integer and MAXIMIZE models - #874
Conversation
…MAXIMIZE The LP-per-node spatial engine was gated to pure-integer, MINIMIZE models, so the mixed / maximize class could not even be offered to it — no primal heuristic inside it could help instances it refused to accept. Entry experiment first (CLAUDE.md §4), on the in-repo corpus (the issue's named gate probes are not vendored and this environment has no route to minlplib.org). 71 of the 119 instances are mixed; 5 are MAXIMIZE, including the syn/rsyn family. Round 1 found the binding blocker was NOT mixed-ness: 31 of the 71 mixed instances were rejected by the all-variables-finite box test, 28 by the incremental patch declining, and only 11 reached a root LP. Round 2, under the proposed gate: finite root McCormick bound 11 -> 46 of 71 verified feasible point at root 4 -> 13 of 46 (round + LP completion) The kill criterion did not fire. On the maximize family the blocker is again the box: root OBBT finitizes every infinite upper bound in 0.2-0.4s and the LP then returns a valid bound (syn05m -1165.78, syn05hfsg -1335.50 min-equivalent). Engine changes: - scope: ">=1 integer variable, either sense, any continuous mix". Pure-continuous stays out — the convergence argument is that branching the integers drives the products exact. `_is_in_scope(model, mixed=False)` keeps the old gate. - sense: the engine runs in minimize-equivalent space, where both producers already live (uniform_relax negates the LP cost, NLPEvaluator negates the point value), so `sgn` appears exactly once — on the reported objective/bound. `bound <= incumbent` becomes `bound >= incumbent` for a maximize. - continuous branching: integers split disjointly at floor/ceil, continuous bisect at the midpoint with the children SHARING it. Candidates narrower than `_MIN_BRANCH_WIDTH` are unbranchable and fold into `unresolved_lb`. - mixed primal `complete()`: fix integers, re-solve the node LP for the continuous coordinates, verify the completed point against the ground-truth evaluator — the LP completion is a relaxation, so its point is only ever a candidate. - feasibility pump acts on integer coordinates only. - infinite root boxes accepted: OBBT runs whenever the box is infinite, and whatever stays infinite goes to the cold builder (which drops rows it cannot make finite). The incremental patch has no such guard, so it is declined when a column it patches is infinite (`IncrementalMcCormickLP.box_is_patchable`). - cold node builds are deadline-bounded (build + solve), since the widened scope routes far larger models through that path. The default-path fallback reserve for the newly in-scope class is behind DISCOPT_LP_SPATIAL_MIXED (default OFF) pending its graduation panel; the engine itself is widened unconditionally. Also fixes a soundness bug on the DEFAULT path, found while widening: `IncrementalMcCormickLP` solved `min c·x` but dropped the relaxation's `obj_offset`, so its node bound sat on a different origin from the cold build's. On a node whose true McCormick optimum is -92.0 the fast path reported +8.0 — a dual bound ABOVE the true optimum, the false-fathom class. The offset is now carried on the structure and added back at every bound-returning exit including the certificate's safe_bound; `_validate` asserts it is box-independent. Verified: `pytest -m smoke` 837 passed / 15 skipped / 2 xpassed; the lp_spatial and incremental-McCormick suites pass, including new fail-before/pass-after regressions for the objective-offset bound and for mixed/maximize scope and soundness. `test_matches_brute_force` fails on 3 of 4 parameter sets before AND after this change (verified against the base commit) — a pre-existing post-#636 tightness gap. Contributes to #860 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
…over maximize soundness and infinite root boxes - test_844_unresolved_child_soundness: the injected `_relax_bound` now takes `**kw` so it keeps exercising the unresolved-child path instead of failing on the new `deadline` argument. - test_lp_spatial_bb: brute-force maximize soundness (incumbent never above the true max, upper bound never below it) and a mixed model with an infinite continuous upper bound, which the pre-#860 gate declined outright. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
…back result The #844 fallback merged its dual bound with `max`, which is correct for a minimize (larger lower bound is tighter) but keeps the LOOSER of two valid upper bounds for a maximize — reachable now that the engine serves maximize models. Sound either way; this keeps the intent. Also covers the new flag: the mixed model's result is unchanged by DISCOPT_LP_SPATIAL_MIXED under either fallback setting, the flag defaults off, and the engine's (widened) gate and the reserve gate can disagree — which is the point of separating them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
… cold path The cold path re-runs build_milp_relaxation at every node and the lift is box-dependent, so a node's column count can differ from the root's. `info` was captured once at the root and read against every node's `x`, which indexes past the end of that node's solution vector — measured on st_e38: root aux column 18 against a 17-column node LP, IndexError. The caller swallowed it as "engine failed" and silently fell back, so the engine was simply unavailable on such models. `relax`/`node_relax` now return the product map of the LP that produced the solution, and it travels with the node through the frontier. Pre-existing on the cold path; st_e38 reaches it only because #860 brought mixed models into scope. After the fix st_e38 returns feasible 7197.73 with bound 6795.05 in 1.9s. Regression: test_cold_path_product_map_travels_with_its_solution. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
…n, not a day One process per solve is the cleanest isolation, but the fixed startup cost (JAX import + parse + evaluator build) dominated at ~2.3 min per instance against ~1s of actual solving — a 10-hour panel. The worker now takes a batch of instances and emits one RESULT line each; batches stay small so a crash or a leak costs only that batch, and every instance still builds a fresh model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
…d relation The panel analyzer inferred each instance's sense from whether the reported bound sat above or below the objective — which is circular, since that relation is exactly the soundness property under test, and it silently defaulted a bound-only maximize run to the minimize check. Senses now come from a precomputed map read off the models themselves. Includes the in-flight Panel A results; the final panel output lands with the verification write-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
Panel A over all 89 in-scope instances (70 newly in scope, 19 the old gate already served), 20s budget, every incumbent independently re-verified with a fresh evaluator: newly in scope 70 declined 15 errors 0 incumbent 33 certified 12 legacy in scope 19 declined 0 errors 0 incumbent 17 certified 14 unsound bounds / incumbents: 0 false optimality certificates: 0 33 instances the engine previously refused outright now yield a verified feasible point; the legacy pure-integer class is unchanged. Both maximize instances return the bound on the correct side (syn05m / syn05hfsg: objective 837.732, upper bound 868.110, incumbent independently feasible). bchoco06/07/08 decline on an iteration_limit root LP — a conditioning problem, not a scope one. Panel B (the DISCOPT_LP_SPATIAL_MIXED graduation gate) is still running; its section carries a PANEL_B_PLACEHOLDER until those numbers land. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
The graduation-gate panel writes its results file after each batch; this is an in-flight snapshot, superseded by the final results and the verdict write-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
…ive, flag stays OFF
Graduation panel for DISCOPT_LP_SPATIAL_MIXED (20s budget, 70 newly in-scope in-repo
instances, off vs on; the other 49 take a bit-identical path since the reserve gate is
the flag's only consumer):
cert-clean 0 certification regressions, 0 incumbent_verification_failed,
0 unsound bounds, 0 false optimality certificates
net-positive FAILED — gains 1, improved 1, LOST 2
tspn12 no incumbent (30.6s) -> feasible 262.647 (9.3s)
ex1252a feasible 183660.35 (24.5s) -> feasible 149530.99 (14.9s)
tls2 feasible 11.30 (20.1s) -> NO INCUMBENT (13.6s)
st_e31 feasible -2.00 (22.2s) -> NO INCUMBENT (14.8s)
The widening is sound — Panel A already showed that, with 33 newly reachable instances
gaining a verified incumbent and 0 unsound results. What it cannot do is pay for itself
on the default path. The reserve hands 35% of the budget to the fallback BEFORE knowing
whether the engine can serve the model: on tls2 and st_e31 the primary then runs out of
time at 65% and returns nothing, while the fallback declines anyway
(require_incremental=True + an infinite root box). Budget taken from a path that was
going to succeed, given to a path that never runs.
Sound but harmful stays OFF with the measurement recorded — the DISCOPT_CUT_INHERIT
rule. The 0.747 wall ratio is not a win: the on-runs are faster largely because they
gave up earlier, on exactly the two instances that lost their answer. The fix this
points at (make the reserve conditional on buildability) is a separate change with its
own panel.
Also: the analyzer now classifies objective drift by sense, so a better incumbent
(ex1252a) is not scored as harm.
Separately, Panel B's independent verifier rejects nvs22's incumbent at tol 1e-5 under
BOTH settings. Not from this branch: byte-identical against base ac9f5cf (objective
6.058219942618198, max violation 2.641e-4) — above the panel's 1e-5, below the
deliberately loose 1e-3 of discopt's own final guard, so nothing flags it today. Worth
its own investigation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
Review: 🛑 DO NOT MERGE — reproducible false optimality certificate on
|
| status | objective | bound | gap_certified |
|
|---|---|---|---|---|
main (3ea21389) |
optimal | 1.6434284641 | 1.6434274718 | True ✅ |
| PR #874 | optimal | 184279.32477275995 | 184279.32477275995 | True ❌ |
MINIMIZE, so a valid dual bound must satisfy bound <= true_opt. It exceeds it by 1.8e5.
The incumbent itself is feasible (re-verified with a fresh NLPEvaluator + _check_constraint_feasibility; the objective re-evaluates to 184279.32477275995), so this is a false dual bound + certificate, not a false primal. Deterministic: identical on 4 independent runs, node_count=1, at gap_tolerance 1e-4 and 1e-9. lp_spatial=True is a documented public kwarg whose docstring this PR edits (solver.py:4645); DISCOPT_LP_SPATIAL_MIXED does not gate it.
Root cause — a guard downgraded from a decline to a flag, and the PR's own justification for that is empirically false. Main's lp_spatial_bb.py:258-259:
if not (np.all(np.isfinite(lb0)) and np.all(np.isfinite(ub0))):
return None # unbounded integer box: out of scope for this stepThe PR keeps the same predicate but binds it to _inf_box (lp_spatial_bb.py:403), which now only triggers OBBT instead of declining. The accompanying comment states the case for that directly:
"It is not, however, a soundness boundary — see the two paths below."
gear4 is the counterexample. Measured:
| root box | cold _relax_bound root McCormick bound |
|---|---|
raw (ub = [60,60,60,60,inf,inf]) |
184279.32477276 ← above the true optimum |
continuous ub capped at 1e6 |
0.0 (valid) |
continuous ub capped at 1e7 |
0.0 (valid) |
Both stated safeguards fail on it:
- §2.4's claim that "the cold
build_milp_relaxation… self-guards and merely loosens" is falsified: it returns_objective_bound_valid=Truealongside an invalid bound on a partially infinite box. - The comment at
lp_spatial_bb.py:405-409— "OBBT finitizes every infinite upper bound in ~0.2-0.4 s and the McCormick LP then returns a valid bound" — is falsified: neuteringobbt_tighten_rootchanges nothing (bound stays 184279.32477275995 at 1 node). OBBT is load-bearing in the design but does not finitizegear4, and nothing checks that it succeeded before the bound is trusted.
Not a one-off shape. Across both in-repo corpora, 23 of 104 engine-accepted instances have an infinite root endpoint — exactly the class the removed guard excluded. Of the 3 that returned status="optimal", 1 is false:
| instance | objective | bound | oracle | verdict |
|---|---|---|---|---|
gear4 |
184279.32 | 184279.32 | 1.643428474 | UNSOUND |
meanvarx |
14.36926 | 14.36816 | 14.36923211 | ok |
util |
999.57875 | 999.55772 | 999.5787502 | ok |
Why Panel A missed it: it checks only self-consistency — "the bound never crossing the best verified feasible point found by any run of that instance". gear4 is perfectly self-consistent (bound == objective at a genuinely feasible point). Only an external oracle catches it.
Process findings
The oracle corpus this PR says is unavailable is present locally. The PR and docs/dev/issue-860-lp-spatial-mixed-scope.md §0 state the gate probes "are not vendored in this repo and this environment has no network route to minlplib.org". ~/Dropbox/projects/discopt-minlp-benchmark/ — named in CLAUDE.md as the benchmark corpus and oracle — contains all of them plus minlplib.solu (=opt= gastrans040 0.0, rsyn0805m04hfsg 7174.219, lip 5685067.877, …). This is the direct cause of the blocker: both panels ran without an oracle, so a certificate five orders of magnitude off looked clean. Re-running Panel A against minlplib.solu takes ~20 min and finds it immediately.
The unconditional obj_offset fix is a default-path bound-changing change with no corpus differential. It is a genuine and correct soundness fix, but it ships default-ON on mccormick_lp's incremental fast path covered only by unit tests. Blast radius on the 66-instance corpus: 5 instances have a nonzero relaxation objective constant, 1 negative — nvs14 at −40792.141 (true optimum −40358.155), so the false-fathom class was live in-repo. Effect PR vs main: nvs14 still certifies −40358.15477 on both, wall 0.68 s → 1.85 s (expected — the over-optimistic bounds were over-fathoming). Nothing unsound, but §5's differential-panel discipline wasn't applied to the highest-blast-radius change in the PR.
The reserve flag design is right. DISCOPT_LP_SPATIAL_MIXED default-OFF with a recorded "cert-clean but NOT net-positive" Panel B (1 gained + 1 improved vs 2 lost: tspn12/ex1252a vs tls2/st_e31) is exactly the DISCOPT_CUT_INHERIT rule, honestly applied. The problem is that the engine's gate was widened unconditionally, and the ungated half is where the false certificate lives.
Second correctness finding — obj_offset misses one exit
The PR states the offset is "added back at every bound-returning exit". mccormick_lp._reverify_incremental_infeasible (mccormick_lp.py:875) recovers a node bound via inc.solve_assembled_full(..., c_override=c2), and c_override solves deliberately skip the offset. On the PR's own _const_qcqp(-100.0) fixture:
normal path : bound = -92.0000000000001 <- correct (fixed by this PR)
reverify path: bound = 7.999999999999904 <- short by exactly obj_offset
DIFFERENCE = -100.0 == obj_offset
Same class as the bug this PR fixes; for a negative offset it is an over-high node lower bound (false fathom). Pre-existing rather than a regression — but it is in a file this PR edits and the "every exit" claim is inaccurate. Reachable on nvs14 (offset −40792) if the C-38 false-infeasible recovery fires.
What came back clean — the widening machinery itself is well built
I could not break it anywhere except through the removed infinite-box guard.
- Sense handling is correct.
sgnapplied exactly once at the exit; both producers (uniform_relax3090/3208,NLPEvaluator._negate318/344/636) already negate for MAXIMIZE, so the loop legitimately needs no sign. Verified againstminlplib.solu:syn05mcert-optimal 837.7323995 (oracle 837.7324009, bound above it ✓);lipfeasible 5665067.89 ≤ oracle, bound 5769241.72 above it ✓;syn05hfsg837.7324009, bound 868.11 ✓. - MAXIMIZE fallback-merge arithmetic is right — picks
min(tighter upper bound) for maximize,maxfor minimize, never crosses the incumbent. 12 executed assertions, fired-counter proving the branch ran. - Mixed exact-leaf is handled correctly — the collapse test (
lp_spatial_bb.py:880) spans all columns, so a node with a live continuous dimension folds intounresolved_lband cannot certify. Probed on a model whose optimum requires continuous spatial branching (root bound −2, true optimum −1, both senses): closes correctly, never certifies at the loose root bound. - Fuzz, engine-direct: 360 random mixed/maximize models, 991 executed assertions, 0 violations. End-to-end with the flag on: 70 models, 290 assertions, 0 violations.
- Oracle panel: 104 in-scope instances, 104 bound + 78 objective checks, 0 errors, 1 violation (gear4).
- Integer / nonconvex-MINLP primal heuristics workstream (LP-based FP + sub-MIP RENS + structure-aware constructors) #844 regression set (default flags, PR vs main, 20 s):
nvs040.72 /nvs061.7703125 /nvs09−43.134336918 /nvs151.0 — all four bit-identical in objective, bound andgap_certified. tln4/5/6 sound on both. pytest -m smoke844 passed / 1 skipped / 2 xpassed; adversarial 10 passed; ruff + format + mypy clean;gh pr checks 874green; no Rust touched.
Nits
ruff check scratchpad/→ 23 errors on the new harnesses (CI doesn't lintscratchpad/; consistent with existing files).- ~3,072 lines of panel JSON committed — matches repo convention but heavy.
docs/dev/issue-860-lp-spatial-mixed-scope.md§4 flags a pre-existingnvs22incumbent-verification issue (violation 2.641e-4, byte-identical on the base commit) — correctly scoped out, worth its own issue.
What would unblock
- Make
_inf_boxa decline again unless OBBT actually finitized the box — i.e. re-checknp.all(np.isfinite(...))afterobbt_tighten_rootandreturn Noneif it still fails — or gate infinite-box acceptance behind its own default-off flag. The PR already recomputes_inf_boxpost-OBBT at line 440 but never acts on the result; acting on it would declinegear4. - Better: root-cause
build_milp_relaxationreturning_objective_bound_valid=Truewith a bound above the true optimum on a partially infinite box. That is a soundness bug in the relaxation layer, of wider consequence than this engine. - Re-run Panel A against
~/Dropbox/projects/discopt-minlp-benchmark/minlplib.solu(present locally) rather than self-consistency, and re-state the "0 false optimality certificates" claim on that basis. - Add
gear4+solve(lp_spatial=True)as a fail-before/pass-after regression test. - Separately: carry
obj_offsetthrough_reverify_incremental_infeasible, or correct the "every bound-returning exit" claim.
The prior finite-box gate was load-bearing, not conservative decoration. Downgrading it to an OBBT trigger produces a certified optimal at 184279.32 where the true optimum is 1.6434 — through a public API, on an instance this repo already treats as a gate probe. The one-line version of the fix is at line 440, already computed and currently discarded.
|
Update on the blocker: root-caused, and it is not in this PR. The false certificate I reported (
Not gear4-specific: 11 of 174 random LPs with unbounded columns returned a claimed- This corrects my earlier root-cause wording, and one framing in the review. I wrote that both safeguards in §1/§2.4 were "empirically false". The measurements stand, but the reason was not that What this means for this PR. The finite-root-box guard was load-bearing, but not for the reason either of us thought: it was masking a presolve bug for the pure-integer engine, not compensating for a weak relaxation. With #877 merged, Suggested path: rebase on #877 once it lands, then re-run Panel A against Everything else in the review stands as written: sense handling, the MAXIMIZE fallback-merge arithmetic, mixed exact-leaf collapse, and the #844 regression set all came back clean, and the reserve-flag design remains a textbook application of the |
Blocker cleared — measured, not inferred. One new finding.#877 is merged. I rebased this branch onto it (10 commits, clean, no conflicts) and re-ran the blocker case, asserting both markers before measuring: the
The false certificate is gone. 4 executed soundness checks (bound-vs-oracle and objective-vs-oracle at New finding: gear4 is now a quality regression rather than a soundness oneSame instance, same 25 s budget:
So the merge question moves from sound? to net-positive? — CLAUDE.md §5 bar (2). Since the engine's gate widening is unconditional while only the behaviour sits behind
Unchanged from the earlier review
I have not pushed the rebase; the measurement was done in a local worktree. Say the word if you want me to push it. |
Measured: this PR does not close #860 — 0/3 named instances gain an incumbentRebased onto current
6 executed soundness checks, 0 violations — every bound is on the correct side of its oracle, and there is no false primal. The widening itself does what it says: all three are now admitted, where before they failed the gate. But #860's stated gap is that these three "return no incumbent", and all three still return no incumbent. So this PR advances the issue without closing it. One defect visible in the table
The branch no longer merges cleanly — resolution, for the recordMerging
I verified that resolution locally: Where that leaves the merge decisionSoundness is settled (the
With the gate flagged and Instrument caveatMy probe printed |
…_MIXED Review outcome on #874. The widening is sound and stays; what changes is that it no longer ships by DEFAULT, because it is not net-positive on the default path (CLAUDE.md 5 bar 2). The flag already governed the #844 fallback's 35% budget reserve, but the ENGINE gate was unconditional, so solve(lp_spatial=True) -- a documented public kwarg -- picked up the widened scope with no opt-in. Measured on gear4, which is mixed (4 integer, 2 continuous with infinite upper bounds) and therefore admitted only under the widening, at a 25 s budget: DISCOPT_LP_SPATIAL_MIXED=0 (default): optimal, 1.6434284641, certified, 3 nodes, 1.1 s DISCOPT_LP_SPATIAL_MIXED=1: time_limit, 17.514, UNcertified, 2678 nodes, 25.0 s The engine accepts a model the default path already certified in 3 nodes, then spends the whole budget to return an incumbent ~10.7x worse with no certificate. It is SOUND -- the bound never crosses the oracle, and the false certificate seen on gear4 earlier in review was the LP-presolve INF-sentinel bug fixed in #877, not this widening -- but shipping it by default trades a certificate for a worse incumbent. Changes: * solver.py's lp_spatial=True entry point passes mixed=_lp_spatial_mixed_fallback_enabled(); the #844 fallback already did. * solve_lp_spatial_bb(mixed=...) and _is_in_scope(mixed=...) now default to False, so a NEW call site inherits the pre-#860 gate rather than silently shipping the widening. Same rationale as row_scan_is_anytime defaulting to False in the tightening rules. * The flag docstring now describes both entry points it governs and records the gear4 measurement alongside the existing Panel B table. * Five tests in test_lp_spatial_bb.py exercise the widened CAPABILITY, so they now request it explicitly with mixed=True. The two tests that assert the pre-#860 gate declines (mixed=False) are unchanged and still pass. Also in this branch: merged main and resolved two conflicts by COMPOSITION, not by choosing a side (rationale recorded in comments at both sites). * modeling/core.py -- main (#861 review) moved the fallback dual-bound merge out of the `_fb.objective is not None` block so a sound bound is reported even with no incumbent; this branch made the same merge sense-aware (min for MAXIMIZE, where the dual bound is an UPPER bound and max would keep the looser one). Kept main's placement AND this branch's sense-awareness. * _jax/incremental_mccormick.py -- main relaxed _validate to compare shape[1] with vacuity-filtered row sets, bounds checked first, which is what lets a PINNED box validate; this branch added the obj_offset box-independence assertion. Kept main's structure AND this branch's assertion. Tests: python/tests/test_860_mixed_gate_is_opt_in.py -- 6 tests pinning the GATING rather than wall-clock, so they cannot rot into machine-speed assertions: mixed=True admits mixed-minimize and integer-maximize while mixed=False declines both (with pure-integer-minimize as a control that must be admitted either way), both defaults are False, a source-level guard fails if either production call site stops threading the flag, and the env-var helper is off when unset. Fails-before/passes-after verified by stashing the three source files: 2 failed / 4 passed before, 6 passed after. Verified: pytest -m smoke -> 844 passed, 1 skipped, 2 xpassed; pytest -m slow test_adversarial_recent_fixes.py -> 10 passed; ruff check python/ clean; ruff format clean; no Rust touched. Also measured, and the reason this is not "Closes": at time_limit=60 with the flag ON, all three instances #860 names are now IN SCOPE (_is_in_scope True, where they previously failed the gate) but NONE returns an incumbent -- rsyn0805m04hfsg (MAXIMIZE) bound 9741.371 vs oracle 7174.219, gastrans040 bound 0.0, gastrans582_cold13 bound 0.0, objective None on all three, 6 soundness checks and 0 violations. #860's stated gap is that these three return no incumbent, so it remains open for the primal work. Contributes to #860
… the stale gate assertion Two CI failures from the previous commit, both real. 1. My new test called importlib.reload(discopt.solver_tuning) to re-read the env var. That was unnecessary -- _lp_spatial_mixed_fallback_enabled reads os.environ directly with no caching -- and actively harmful: reloading the module replaces the SolverTuning CLASS OBJECT, so unrelated isinstance(..., SolverTuning) assertions later in the same xdist worker fail. It broke test_result_io::test_options_from_payload_rebuilds_tuning and test_solve_daemon::test_solve_request_rebuilds_tuning, neither of which this branch touches. Removed the reload (monkeypatch alone is sufficient and correct), added a "0" case, and recorded why no reload belongs here so it is not reintroduced. 2. test_844_lp_spatial_fallback::test_860_mixed_flag_defaults_to_off_and_gates_the_reserve asserted `_is_in_scope(m) is True # engine: widened unconditionally`. That is precisely the behavior the previous commit changed, so it had to be updated -- updated with the flag's new scope, not loosened. The substantive assertion (mixed=False declines a mixed model) is unchanged; it now additionally asserts that the DEFAULT is the conservative gate, which is the new invariant. The docstring records what it used to say and why it changed. Verified with CI's exact PR-fast selection rather than the smoke subset that missed these: pytest python/tests/ -m "not slow and not correctness and not integration and not amp_benchmark and not requires_cyipopt and not memory_heavy" --ignore=test_correctness.py --ignore=test_amp.py -> 7255 passed, 29 skipped, 5 xfailed, 2 xpassed, 0 failed (622 s) Also ran the three previously-failing files plus the new one in a SINGLE process (-p no:randomly) so the cross-test contamination would reappear if it were still there: 42 passed. Contributes to #860
Summary
Contributes to #860 — not
Closes. Measured after gating: with the flag ON attime_limit=60, all three instances #860 names are now in scope (_is_in_scope=True, where they previously failed the gate) but none returns an incumbent —rsyn0805m04hfsg(MAXIMIZE) bound 9741.371 vs oracle 7174.219,gastrans040bound 0.0,gastrans582_cold13bound 0.0,objective=Noneon all three (6 soundness checks, 0 violations). #860's stated gap is that these three return no incumbent, so it stays open for the primal work.The widening is now opt-in behind
DISCOPT_LP_SPATIAL_MIXEDat both entry points, includingsolve(lp_spatial=True). It is sound but not net-positive on the default path (CLAUDE.md §5 bar 2): ongear4the pre-#860 gate certifies 1.6434284641 in 3 nodes / 1.1 s, while the widened gate returns 17.514 uncertified over 2678 nodes / 25 s. See the last commit for the full rationale.The LP-per-node spatial engine (
_jax/lp_spatial_bb.py) was gated to pure-integer,MINIMIZE models, so the mixed / maximize class could not even be offered to it — no
primal heuristic inside it could help instances it refused to accept. This widens the
scope to "≥1 integer variable, either objective sense, any continuous mix".
The issue's named gate probes (
gastrans040,rsyn0805m04hfsg,gastrans582_*) arenot vendored in this repo and this environment has no network route to minlplib.org, so
every number below comes from the in-repo corpus (
minlplib_nl∪minlplib, 119distinct instances) — which is dominated by exactly this class: 71 mixed, 5 MAXIMIZE,
including
syn05m/syn05hfsg, the samesyn/rsynfamily as the issue's maximizeprobe.
The entry experiment reframed the work. Measuring the existing gate showed
mixed-ness was not the binding constraint: 31 of the 71 mixed instances were rejected
by the all-variables-finite box test (real MINLPs leave continuous columns unbounded
above), 28 by the incremental patch declining, and only 11 reached a root LP at all. A
widening that only relaxed the variable-mix test would have been close to a no-op on the
real class — the #727 RLT lesson. Under the proposed gate, at the root box only, no tree:
The kill criterion did not fire. On the maximize family the blocker is again the box:
both in-repo maximize instances have no valid objective bound over the raw box, and
root OBBT finitizes every infinite upper bound in 0.2–0.4 s (
syn05m−1165.78,syn05hfsg−1335.50, minimize-equivalent).What changed: sense handled in minimize-equivalent space (
sgnapplied exactly once, atthe exit — both
uniform_relaxandNLPEvaluatoralready negate for a MAXIMIZE);continuous spatial bisection with children sharing the midpoint; a mixed-integer
primal
complete()that fixes the integers, re-solves the node LP for the continuouscoordinates and verifies the point exactly; feasibility pump on integer coordinates
only; partially infinite root boxes accepted; deadline-bounded cold node builds.
Full write-up, measurements and known limits:
docs/dev/issue-860-lp-spatial-mixed-scope.md.Correctness
Three properties carry the widened scope, and Panel A (below) measures all three over
every in-scope in-repo instance:
inputs already live. A valid lower bound on
−fis a valid upper bound onf, sobound ≤ incumbentbecomesbound ≥ incumbentfor a maximize.valid outer approximation of its subtree. A candidate narrower than
_MIN_BRANCH_WIDTHis unbranchable and folds intounresolved_lb, so it can neveryield an optimality proof — the exact-leaf test still requires the whole box
collapsed, continuous dimensions included.
relaxation of the continuous restriction, so its point is only ever a candidate,
discarded unless the ground-truth evaluator accepts it.
This PR also fixes two soundness bugs it uncovered:
A false-fathom on the DEFAULT path.
IncrementalMcCormickLPsolvedmin c·xbutdropped the relaxation's
obj_offset, so its node bound sat on a different originfrom the cold build's. On a node whose true McCormick optimum is −92.0 the fast path
reported +8.0 — a dual bound above the true optimum. This is
mccormick_lp'sincremental fast path (default since cert:T1.3), not only the opt-in engine.
The offset is now carried on the structure and added back at every bound-returning
exit including the certificate's
safe_bound;_validateasserts it isbox-independent, so a shape where it were not falls back rather than mis-reporting.
Root product map read against a node's solution. The cold path rebuilds the
relaxation per node and the lift is box-dependent, so a node's column count can differ
from the root's — on
st_e38, root aux column 18 against a 17-column node LP →IndexError, swallowed by the caller as "engine failed". The map now travels with thesolution it describes.
Tests run (state the result — numbers, not adjectives)
pytest -m smoke→ 837 passed, 15 skipped, 2 xpassed (7427 deselected)pytest -m slow python/tests/test_adversarial_recent_fixes.py→ 10 passedcargo test -p discopt-core→ N/A — no Rust touchedruff check/ruff format --checkclean;mypyclean on every changed modulePre-existing failure, unchanged by this PR:
test_lp_spatial_bb.py::test_matches_brute_forcefails on 3 of its 4 parameter setsbefore and after (verified against base
ac9f5cf— identical set): the engine findsthe true optimum but reports
feasiblerather thanoptimal. Documented post-#636conservatism —
a*bis lifted via univariate squares, so it never appears in theengine's
infomap and no node short of a fully collapsed box is treated as exact.Regression test
Added regression tests that fail before / pass after
test_incremental_node_bound_includes_objective_constant— before the fix the fastpath returned 8.0 for every constant; at
const=-100that is a lower bound of +8.0 ona node whose true optimum is −92.0. Plus
test_incremental_solve_bound_matches_cold_builder_with_constantone level down.test_cold_path_product_map_travels_with_its_solution—st_e38raisedIndexErrorbefore; now returns feasible 7197.73 / bound 6795.05 in 1.9 s.
test_mixed_integer_in_scope_and_correct,test_maximize_bound_is_a_valid_upper_bound,test_maximize_never_overstates_the_optimum(brute force),test_infinite_root_box_is_accepted_not_declined— each returnedNonebefore.test_legacy_gate_still_declines_mixed,test_maximize_legacy_gate_returns_none,test_pure_continuous_stays_out_of_scopepin the new boundary;test_860_mixed_flag_defaults_to_off_and_gates_the_reservepins the flag plumbing.Bound-changing / performance change?
DISCOPT_LP_SPATIAL_MIXED(governs only whether the defaultpath reserves budget for the Integer / nonconvex-MINLP primal heuristics workstream (LP-based FP + sub-MIP RENS + structure-aware constructors) #844 fallback on the newly in-scope class; flag-OFF the
reserve gate is the pre-LP-per-node engine is scope-gated to pure-integer MINIMIZE: 3 #844 instances unreachable #860 gate)
evaluator; bound-vs-verified-point crossing checked in both senses
Harness
scratchpad/panel860.py, scoringscratchpad/panel860_analyze.py, 20 s budget.Panel A — engine soundness on the widened class (89 in-scope instances):
33 instances the engine previously refused outright now yield an independently
re-verified feasible point, 12 with an optimality certificate; the legacy pure-integer
class is untouched. Both maximize instances return the bound on the correct side
(
syn05m/syn05hfsg: objective 837.732, upper bound 868.110, incumbentindependently feasible).
bchoco06/07/08decline on aniteration_limitroot LP — aconditioning problem, not a scope one.
Panel B — graduation gate for
DISCOPT_LP_SPATIAL_MIXED(70 newly in-scopeinstances, off vs on): cert-clean, but NOT net-positive → the flag stays default-OFF.
tspn12ex1252atls2st_e31gains=1 improved=1 lost_incumbents=2 cert_regressions=0 incumbent_verification_failed=0 unsound=0.The widening is sound; what it cannot do is pay for itself here. The reserve hands 35%
of the budget to the fallback before knowing whether the engine can serve the model:
on
tls2andst_e31the primary then runs out of time at 65% and returns nothing,while the fallback declines anyway (
require_incremental=Trueplus an infinite rootbox). Budget taken from a path that was going to succeed, given to a path that never
runs. Sound but harmful stays OFF with the measurement recorded — the
DISCOPT_CUT_INHERITrule. The 0.747 total wall ratio is not a win: the on-runs arefaster largely because they gave up earlier, on exactly the two instances that lost
their answer. The fix this points at — make the reserve conditional on the engine
actually being able to build — is a separate change with its own panel.
A pre-existing soundness flag, not from this branch: Panel B's independent verifier
rejects
nvs22's incumbent at tolerance 1e-5 under both settings. Byte-identicalagainst base
ac9f5cf— objective6.058219942618198,gap_certified=True, maxconstraint violation
2.641e-4, above the panel's 1e-5 and below the deliberately loose1e-3 of discopt's own final guard, so nothing flags it today. Worth its own
investigation; out of scope here.
🤖 Generated with Claude Code
https://claude.ai/code/session_019bsy1zfNW8H1hZGg9SKfPu
Generated by Claude Code