Compiler: experimental ES module output (--esm) [RFC]

[hhugo]

Experiment on top of #2163: emit ECMAScript modules from js_of_ocaml, end to end, to inform the design discussion in #1161 (ESM as build artifact) and #551 (Node-consumable modules). Draft — the main goal is to decide what the result of separate compilation should be: bundled or not.

What works

All four configurations compile and run hello.ml (stdlib + Printf) under node:

# whole program
js_of_ocaml compile --esm hello.byte -o hello.mjs

# separate compilation: module runtime + one file per library/unit
js_of_ocaml build-runtime --esm -o runtime.js
js_of_ocaml compile --esm stdlib.cma -o stdlib.cma.js   # multi-unit container
js_of_ocaml compile --esm hello.cmo -o hello.js

# link, unbundled: entry module importing every unit in link order
js_of_ocaml link --esm <units...> -o entry.js

# link, bundled: single self-contained module, tree-shaken
js_of_ocaml link --esm --bundle <units...> -o bundle.js   # --no-tree-shake to disable

• compile --esm: no IIFE wrap, no globalThis polyfill; units reference runtime primitives as plain identifiers bound by a named import — import { caml_get_global, caml_call_gen, ... } from "./runtime.js" (--esm-runtime-path to override) — instead of reading globalThis.jsoo_runtime. No import *: the output stays statically analyzable by bundlers and tree shakers.
• build-runtime --esm: exports every primitive by name; shims CommonJS require with node:module's createRequire when the runtime needs it.
• link --esm: emits an entry module with one side-effect import per unit — ES evaluation order guarantees units run in link order. --bundle merges everything through Esm_bundle.
• Units reference each other through imports: caml_get_global("X") compiles to a binding introduced by import X from "./X.js", and caml_register_global compiles to export default <module block> — emitted directly by the code generator from the primitives (no JavaScript-level rewriting). Predefined exceptions keep using the runtime table through the distinct *_predef primitives. Cross-unit references are then ES bindings, statically visible: the ESM linker recovers ocamlc's archive semantics (container units linked only when referenced; standalone unit files always linked; --linkall links everything) — which the script linker gets from unitInfo metadata, but which was impossible for the table-based ESM bundle where every unit had to be an entry point. It also means browsers, node and external bundlers see the true module graph without any jsoo-specific metadata. The unbundled entry only imports the root units; dependencies load through the import graph.
• .cma.js files are containers, not modules. A multi-unit .cma.js concatenates one module per unit, with duplicated names across units (caml_call* helpers, runtime import bindings) — importing it directly is a SyntaxError, by design. Like a .cma archive, it is only meant to be consumed by the linker: link --esm --bundle splits it at //# unitInfo: Provides: boundaries in memory (per-module renaming absorbs the duplicates), and unbundled link --esm materializes one module file per unit next to the entry (rebasing relative imports). Files holding a single unit are valid modules and are used in place. The runtime's own unitInfo header has an empty Provides: — the linker treats that as file-level metadata, not a unit boundary, so the runtime links as a plain module referenced in place (an earlier revision misread it as a unit named "", materializing it as a file called .js and evaluating the runtime twice).

Fixes to the #2163 machinery found by the experiment

Four bugs surfaced while driving the tooling end to end; they are folded into the parent branch (#2163) as the four Compiler: esm: commits and are not part of this PR's diff:

• Module evaluation order: the bundler emitted independent sibling modules in alphabetical order (SCC topological sort with module-id tie-break), not the ES-mandated evaluation order (post-order DFS over import declarations).
• Entry side effects: tree shaking seeded liveness only from entry exports — an entry module that exports nothing (any jsoo unit, any program) was shaken to an empty file.
• External imports: unresolvable specifiers (e.g. node:module) failed build_graph; resolve returning None now means "external".
• Namespace imports vs tree shaking: import * as runtime kept every runtime export alive; the ns.field → named-import optimization now runs before shaking.

Numbers (hello world, compact output)

configuration	bytes
whole-program script (status quo)	72,194
whole-program ESM	87,327
separate, script concatenation (status quo)	315,329
separate, ESM bundle, tree-shaken	208,411
separate, ESM bundle, tree-shaken, --linkall	394,768

(Script concatenation and ESM bundle link the same 20 stdlib units; see the decomposition below.)

Both linkers select the same 20-unit Requires-closure — the script linker already drops unused archive units (auto-link), and with import-based unit references the ESM linker matches that (the earlier table-based ESM bundle could not: every unit had to be an entry point, 403K). The 105K win over the script concatenation decomposes as: the runtime is tree-shaken to the primitives actually used (173K → ~80K), and the caml_set_link_info symbol table (~20K, which serves dynlink/toplevel) is not emitted — partially offset by unit code being ~6% larger while module top-level names are not minified. Remaining size gaps are fixable, not inherent to ESM:

1. Top-level names are not minified in modules. Inside the script IIFE every runtime function gets a short name; at module top level, string-named declarations are left alone. This is most of the whole-program gap (72K → 87K) and part of the remaining bundle overhead. Fixing it means teaching the minifier that module top level is a private scope (modulo exported names).
2. Statement-level shaking inside a unit is coarse. A unit that is referenced keeps all its side-effecting statements; per-value shaking across units (the full Node.js compatible modules #551 named-exports design, one export per value instead of one default block) would go further.
3. Js_variable_coalescing is now module-aware (export expressions record uses; by-name exports are live bindings, excluded from coalescing), so unit modules flow through the full JavaScript optimization pipeline, including bundled output.

The design question: should linking bundle?

Findings that frame it:

• Concatenation is dead in ESM — unless .cma.js is a container. A .cma.js holding N concatenated module-units is a syntax error if imported directly (shared top-level scope, duplicate bindings). This PR keeps the one-file-per-library convention by treating it as a container the linker splits back into units (--keep-unit-names remains available to emit per-unit files at compile time). There is no cheap cat-style link for modules — the unbundled link step stays cheap because it only writes the entry plus, for containers, one small file per unit.
• Unbundled preserves today's incrementality (a changed unit only recompiles that unit; "linking" rewrites a 3-line entry), loads natively in node and browsers, and interoperates with external tools — rollup/esbuild can consume the per-unit output directly and do their own bundling/minification. Cost: many files, more total bytes, HTTP round-trips without a bundler.
• Bundled gives a single self-contained artifact and shakes unused runtime primitives, but re-parses and re-analyzes every unit at each link (slower incremental builds).

Measured link-step times (hello world + stdlib, hyperfine, warm cache; process startup alone is ~20 ms):

link step	time
script concatenation (status quo)	16 ms
link --esm, per-unit files already in place	9 ms
link --esm, splitting .cma.js containers	260 ms
link --esm --bundle	295 ms

Concatenation never parses JavaScript — its cost is I/O. The ESM linker pays lex+parse+rename+print for whatever it has to materialize: everything when splitting containers, the whole program when bundling, nothing when the units are already on disk next to the entry — that last configuration beats concatenation, which still copies the full output on every link.

Tentative recommendation: make per-unit modules + generated entry the canonical result of separate compilation, and keep bundling as an optional packaging step (link --esm --bundle) — or leave bundling to external tools, which can now see the whole graph (imports between units, named runtime imports). Whole-program compilation remains the size-optimal single-file path and now emits a proper module with --esm; the bundled separate-compilation path gets surprisingly close (210K vs 72K is the remaining gap to close via minification and per-value exports).

Known gaps (deliberately out of scope for the experiment)

• Source maps are ignored by link --esm (both modes).
• Toplevel/dynlink, --wrap-with-fun, --include-runtime (partial runtime) with --esm: untested / likely broken.
• Browser runtime: --target-env browser should avoid the node:module shim, untested.
• //# unitInfo metadata is lost when bundling.
• No dune rules; wiring --esm into the dune jsoo mode is future work.

🤖 Generated with Claude Code