native bootstrapping (Linux, Windows) - #2303
Merged
Merged
Conversation
As a template it was expanded into the (rightly) .inline Cursor/TokenBuf =destroy/=dup hooks, dragging the whole cold ORC free path into every splice site and cascading multiplicatively into every container-hook instantiation — the single largest source of backend IR blowup. As a plain proc the hooks splice as the few tokens they read as: a nil test and a call. Measured on nimsem: IR x8.38 -> x6.52, boot 50.4s -> 42.1s, runtime unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sm__ label An importc proc with neither header nor nodecl used its libc identifier in the generated C, and its extern prototype (types derived from the NIF signature) conflicted with the real header's prototype whenever both met in one TU — 'extern NI64 write(int, void*, NU64)' vs <unistd.h>'s 'ssize_t write(int, const void*, size_t)'. Authors dodged this by hand (see threadpool's reportResidualFailure comment), but inliner splices now carry bare-importc references into arbitrary modules, so the dodge no longer composes. Declare such procs under their mangled C identifier and bind it to the real symbol with an asm label: 'extern NI64 sysWrite_0_... (...) __asm__(NIM_ASM_PREFIX "write");'. The identifier can never collide with a header prototype; NIM_ASM_PREFIX (new in the prelude) supplies Mach-O's leading underscore via string-literal concatenation. Procs with header/nodecl and all exportc handling are unchanged, as is the LLVM backend (IR has no headers to collide with). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The inliner's per-call heuristic rewards literal arguments because they promise post-splice simplification — but nothing downstream delivered it. A spliced 'proc use(c: ptr T) = (if c != nil: ...c.f...)' called with nil kept its dead '(deref (nil))' arm; the C backend swallows such arms, arkham's typed getType rightly cannot. Two pieces make the promise real. New rewrite rules fold what literal substitution makes constant: eq/neq on nil literals, not on bool literals, and-false / or-true (with a purity guard). The new branchprune pass (nifcore, self-tested) then deletes what the folding decided: - (false)-elifs drop, the first (true)-elif becomes the else of any undecided branches before it (or replaces the if outright), a fully decided if vanishes; - a literal (nil) condition reads as false — pointer truthiness, in condition position only; - a branch is only droppable when every label it defines has no uses outside it: hexer's try/except parks the handler in an '(elif (false) (stmts (lab :`exlab.N) ...))' entered by jmp from the try body, which must survive, while the splicer's own returnLabel (def and jumps both in-branch) pins nothing; - an unreachable-statement sweep drops label- and decl-free statements between an unconditional jmp/ret and the next lab — the value splice's dead 'dest = result' self-copy read a never-written var, which nifasm's clobber verifier rejects. Wired into optimizeBody after the rewriter fixpoint, with one refold when a prune fired. Note the pass only runs under the optimizer; non-optimized native builds still ship unpruned guards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The inlining decision no longer follows the .inline annotation (which
keeps its emission meaning): a body of at most 100 tokens always
splices — forwarders, accessors and hooks, annotated or not — .noinline
never does, and bigger bodies go through the per-call-site weighted
score, so a fat body needs genuinely decisive arguments. Both passes
derive the policy from the module file they already parse
(indexProcBodies), ending the pragma transport.
Sound and bounded by construction:
- per-param weights are clamped: the weight walk adds context value per
occurrence, so uncapped weights grew with body size while the
threshold did too — the ratio was size-independent and any body whose
params fed conditions inlined at every site. Multi-GB hexer RSS on a
nimsem build traced back to exactly this (plus the pass's unlimited
depth, now capped at 4 like the cross-module one);
- each caller gets a growth budget (it may roughly double, floor 1000
tokens), charged by every committed splice including those inside
re-walked spliced content — the hard backstop that keeps program
growth linear whatever the per-call heuristic thinks;
- importc procs never inline: their '(stmts .)' body is a placeholder
and splicing it deletes the call (memfiles inlined posix open's empty
shell and never called open(2));
- varargs procs never inline (a spliced '(var :p (varargs) ...)'
binding cannot be sized);
- assembler procs never inline (the splice strands {.register.}-pinned
locals in ordinary code);
- call sites whose arguments contain an aggregate constructor are
declined: the C backend renders those as block-scope compound
literals, and the splice's (scope ...) cut their lifetime short when
the address escaped (static Shape[N] params read dead stack). Lifting
this needs the splicer to hoist constructor temps out of its scope.
Measured on nimsem: cold native build 9.7s -> 5.8s, IR blowup x6.52 ->
x2.4, binary 11MB -> 5.5MB, native boot 42.1s -> 28.7s with stages
1=2=3 byte-identical; full suite green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The staging-exhaustion class that gated the native boot is fixed on the nativenif side (transient-register demand held inside the budget per emit step), and the inliner rework upstream keeps arkham's input sane. Boot verified: stages 1=2=3 byte-identical, 28.7s total. tunion.nim.c picks up the two intended codegen changes: the NIM_ASM_PREFIX prelude macro and the module-init proc now being spliced into main by the size-driven policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Leng has no noreturn pragma of its own, so the fact was dropped at the lengcgen boundary. Carry it as the existing `(attr "noreturn")`: the C backend renders it `__attribute__((noreturn))` (a codegen win by itself), arkham skips pragmas it does not know, and the optimizer's condition-elimination pass reads it back to learn what an assert/panic guard's fall-through proves. NOT for `.raises` procs. Under goto exceptions a raising "noreturn" proc — raiseOSError — RETURNS at the Leng level, handing back an error code for the caller to propagate; `retType` above already rewrites its signature for exactly that. Telling C it never returns made gcc delete the callers' error handling and broke the stage-2 boot, and it would mislead the fall-through learning the same way. Only a proc that genuinely diverges — exits or aborts — gets the attribute. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An inlined accessor re-materializes the same guard over and over: `cursorTagId` alone re-tests `c.p != nil and 0 < c.rem` and `kind == TagLit` several times, each dragging a ~200-token write-stderr-and-quit block behind it. The first guard's fall-through already PROVES the condition; every later re-test is dead weight that no existing pass could see, because the operands are calls and the temps holding them all have different names. The new pass keys facts by a canonical VALUE serialization: single- assignment locals are substituted by their defining expression (so `x.0h379` and `x.0h386` collapse to one key), xelim's and/or diamond is recognized and serialized as `(and C E)`, symbols carry a version bumped on assignment, memory reads carry an epoch bumped by any writing call, and READONLY callees — derived from the callee's own body, since hexer's summaries mark every cross-module caller `callsUnknown` — key like pure expressions. Facts come from diverging guards (a branch whose tail calls a `noreturn` proc) and from single-value case arms, and decompose through not/and/or and eq/neq duals. Matching folds a condition to a literal; `branchprune`, running right after, deletes the branch. Loops and jumped-to labels reset the fact set; branch-local facts and definitions do not survive a join. Two soundness rules the boot fixed point taught us, both with regression tests: - an ADDRESS-TAKEN local is memory, never substituted by its definition: `f.rem > 0` folded across `sigmatchLoop(m, &f, …)`, which advances `f`; - only SINGLE-ASSIGNMENT locals are substituted at all: two mutable `SymId` locals both initialized to 0 shared one key, so `lenSym == 0` being false in an else-branch deleted an unrelated `if dataSym == 0: dataSym = …` lazy-init in expreval. CONDELIM=off / CONDELIM_SKIP=<suffix,…> disable the pass wholesale or per module — that bisection is what found both bugs. Measured on the sem module: 443 guards learned, 24 conditions folded; full suite green and native boot 1=2=3 byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A proc's `result` slot receives `=wasMoved` followed by `=destroy` before its first assignment — `some[T]`/`none[T]` in `std/opt` are the shortest example. Both hooks dispatched on the discriminant of memory that has never been written. The pair is self-neutralizing only while the two loads of that garbage agree: `=wasMoved` clears the payload of whatever branch it read, so the `=destroy` behind it finds an empty one. Nothing guarantees they agree. An uninitialized load is `undef` and may be materialized independently per use; `=wasMoved` reading None while `=destroy` reads Some frees a `string` that was never constructed. That is why this stayed latent on master, on Linux, and under older clang, and why it surfaces now: the inliner splices both hooks into their caller, handing the uninitialized load to SROA. It was found chasing tests/nimony/stdlib/topt.nim, which fails on the macOS CI runner alone — a message-less assert in the joined group, exit 133 (SIGTRAP) standalone — and is consistent with this defect, though the runner's clang is five majors newer than anything available to reproduce on. So write the discriminant instead of reading it: assign the first branch's value (its low bound for an `of lo..hi`), then let the existing dispatch clear exactly that branch. Every op a `=wasMoved` emits is a pure write, so the result is a valid, trivially-destroyable value even on uninitialized memory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… time Deleting the shoggoth optimizers (fd9d6b0) took three duties with them that were correctness obligations of the inliner, not optimizations, and stage 1 of the native boot died on each in turn. They now live where the constant is manufactured: after a splice's nested re-walk, `prunedInto` rewrites the expansion before it reaches the caller's buffer. - An `(elif …)` arm whose guard the substituted literals decide is deleted: `if c != nil: …c.f…` spliced with `c := nil` keeps an ill-typed `(deref (nil))` in its false arm, and arkham (rightly) asserts on it. The guard is judged only after the nested re-walk, because a `!=` forwarder must itself be spliced before the guard is a literal comparison. - An arm that defines a `(lab …)` some outside code jumps to is pinned however its guard folds — hexer's try/except lowering parks the handler in an `(elif (false) (stmts (lab :`exlab.N) …))` entered only via `(jmp …)` — so a pinned arm bails its `if` out to a verbatim copy. - Statements between an unconditional `(jmp …)` and the next `(lab …)` are dropped: the value-splice epilogue's `dest = result` self-copy is dead on a callee whose every path returns explicitly, and it reads a `result` that was never written — nifasm's clobber check rejects that. Splice-time is also where this information can still feed the inlining POLICY (the growth budget sees the true residual size), which a whole-module pass after the fact never could. Boot is green again: stages 1/2 and 2/3 byte-identical; tests/nimony 666/666; full tree walk clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rking lot The handler used to sit in `(if (elif (false) (stmts (lab :`exlab.N) <handler>)) (else <finally>))`, entered only via `(jmp `exlab.N)` from the raise sites. The structure encoded "normal path runs the finally, handler path skips it" — but every consumer paid for the pretense: - arkham cannot know the guard is dead, so every native try/except materialized the `(false)`, compared it against 0 and emitted a never-taken branch; - every branch-pruning pass needs a label-pinning rule to keep it from deleting a "dead" arm that outside code jumps into — the exact trap the optimizer removal (fd9d6b0) walked into. Now the same control flow is spelled directly: <try body> # raises inside became (jmp `exlab.N) <finally> # normal path; raise-site finallys are duplicated (jmp `exend.N) (stmts (lab :`exlab.N) <handler>) (lab :`exend.N) The except cursors are parked while the finally (which follows them in the tree) is emitted first; handler and finally are both translated with the label already popped, so a nested raise propagates past this try as before. Handlers keep their `(stmts …)` wrapper, so the spliced shape downstream is unchanged except for the vanished `if`. The label-pinning bailout in intramodinliner stays as a safety net, but no lowering emits label-carrying dead arms anymore. Gates: boot green (stages 1/2 and 2/3 byte-identical), tests/nimony 666/666, full tree walk clean, zero golden churn; a native try/except/ finally smoke test runs both paths correctly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.