feat: add second order to SScalar - #88
Merged
Merged
Conversation
SScalar carried a value and a gradient keyed by name. This adds the Hessian, so named derivatives reach the same order the indexed DDScalar already offered. The Python types are SScalar (order 1) and SSScalar (order 2), following the one-letter-per-order naming of DScalar and DDScalar. The Hessian is dense over the names a value carries, laid out as the upper triangle in the packing DDScalar uses. That is deliberate: any nonlinear function contributes the full outer product of its own gradient, so the triangle fills up after a couple of operations. What stays sparse is the set of names, which is what makes the type useful -- it does not have to be known in advance. A pair-keyed map would allocate per entry to store what is dense anyway. Order 1 pays nothing for the addition. The Hessian storage sits behind [[no_unique_address]] and collapses to an empty member, so sizeof stays at 32 bytes against 56 for order 2. Second order costs 1.31x the runtime of first order over six names. The expected values in the new tests were generated symbolically, so they share no formulas with the header. Writing them first paid off: it caught four operators (-u, s - u, u * s, u / s) that copied the storage and rewrote the gradient in place without touching the Hessian. Routing them through the existing apply() helper fixes that and is shorter than what it replaces.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the three-step plan for
SScalar: characterize (#84), change the storage (#85), add second order (this).SScalarcarried a value and a gradient keyed by name. This adds the Hessian, so named derivatives reach the same order the indexedDDScalaralready offered. The Python types areSScalar(order 1) andSSScalar(order 2), following the one-letter-per-order naming ofDScalar/DDScalar;dd(a, b)is absent from the first-order class rather than raising.Why dense over the names
The Hessian is dense over the names a value carries, laid out as the upper triangle in the same packing
DDScalaruses. That was measured rather than assumed: any nonlinear function contributes the full outer product of its own gradient, so a simulation of the fill rate reaches 100% after two operations. What stays sparse is the set of names — which is the point of the type, since it does not have to be known in advance. A pair-keyed map would allocate per entry to store what is dense anyway (25 allocations against 0 for k=6).Cost
Order 1 pays nothing. The Hessian storage sits behind
[[no_unique_address]]and collapses to an empty member:sizeofSScalar<1>SScalar<2>Second order costs 1.31× the first — for six names and 21 Hessian entries.
Tests
34 new C++ cases (each checking the value, three gradient entries and all six Hessian entries, plus symmetry) and 8 new Python cases. The expected values were generated symbolically with sympy, so they share no formulas with the header.
Writing them first paid off. Of 413 assertions, 174 failed at the start; the last 8 to fall pointed at four operators —
-u,s - u,u * s,u / s— that copied the storage and rewrote value and gradient in place without ever touching the Hessian. Routing them through the existingapply()helper fixes it and is shorter than what it replaced.Verification
-Werrorjust tidy35 findings against 41 onmain(all pre-existing, none in the new code)