Old-frame SL reach-back amplifies exponentially on per-step-moving meshes (free-surface T blow-up; affects movers/adapt identically)

[lmoresi]

Summary

The serial free-surface temperature unboundedness (T reaching [-1.6, 3.2] at ~8% surface
deformation in annulus FS convection) is an exponential instability of the semi-Lagrangian
record -> trace -> solve loop that requires per-step mesh motion and old_frame_traceback=True.
It is not a landing-point problem, not the monotone limiter, not the flux-history frame, and not
the timestep. The free surface itself is exonerated as physics: its only necessary role is
moving the mesh between solves, so MMPDE movers and adapt-on-top workflows are exposed
identically.

Evidence chain (all measured)

1. Stage attribution (annulus FS convection, gap-Ra 3e4, rho_g 2e5, res 0.06): per-step
bounds of T_pre / psi_star-after-clamped-traceback / T_post show the clamp invariant
range(psi*) ⊆ range(T_pre) holds at every step — the traceback contracts the range —
while the FE solve grows BOTH extremes ~8-10%/step. The worst violator is mesh-locked at
r=0.952 for 22 consecutive steps (hot BC is at r=0.5): a local linear instability, not
advected material. Onset at h/R ≈ 5.4%.

2. Lever scan: theta_flux=1.0 (BE) delays onset and saturates at a period-2 limit cycle
([-0.38, 1.51]) instead of running away — an accelerant removed, not a cause. Old-frame for
the flux history: no effect. True dt halving makes it WORSE at matched deformation —
the gain is per loop cycle, not per unit time.

3. Minimal reproducer (slcn_minimal_repro.py, attached path below): no Stokes, no free
surface — prescribed 4-cell streamfunction velocity, sharp (1-2 cell) cold boundary layer,
mode-4 squeezed annulus:

variant	60-step outcome
undeformed, static mesh	bounded [-0.05, 1.02]
squeezed 9%, static	mild linear creep, max -> 1.26
squeezed 9%, moving ±0.09%R/step	exponential runaway, T -> 160

4. Localisation (2x2 under motion):

	theta 0.5	theta 1.0
old_frame_traceback=True	T -> 160	T -> 46
old_frame_traceback=False (standard ALE)	[-0.002, 1.118]	[-0.002, 1.052]

5. Verification on the real problem: annulus FS convection at rho_g 2e5 with the
manager's scalar history flipped to old_frame_traceback=False: T ∈ [-0.001, 1.000] for
all 60 steps to 16.9% deformation (the old-frame run was unusable beyond ~5.4%).

The irony

old_frame_traceback=True was introduced as the FIX for the previous high-Ra free-surface
blow-up (Stage-0 lagged-clone design; its docstring says "cures the high-Ra free-surface
convection blow-up"), validated on a Cartesian box at small deformation. At annulus-scale
deformation it is itself the amplifier. Note the original Ra=1e5 blow-up predates the
deform-aware return_coords_to_bounds, the monotone clamp and the ring filter — its root
cause was plausibly among those (since fixed), leaving old-frame's sampling leg without
its original justification. Its record leg (direct nodal carry, exact) is separable and
remains sound.

Open question (the actual defect)

WHY the old-geometry ephemeral-restore sampling amplifies is not yet localised at source
level. Suspect family: stale navigation/kd-tree structures used by global_evaluate on the
ephemerally-restored geometry (cf. the MMPDE stale-nav-kd-tree class of bugs). The clamp
provably cannot break the loop (the spike is inside its own neighbourhood's data range).

Reproduce

~/+Simulations/FreeSurface/annulus_fs_convection/teaching/slcn_minimal_repro.py
(-squeeze 0.09 -delta 0.03 -move, ~5 min serial) and bounded_probe.py -rg 2e5 [-no_oldframe] for the full free-surface version. Both print the per-stage bounds table.

Underworld development team with AI support from Claude Code