Skip to content

Feat/new compiler - #92

Merged
rutefig merged 201 commits into
mainfrom
feat/new-compiler
Sep 17, 2026
Merged

rutefig merged 201 commits into
mainfrom
feat/new-compiler

Conversation

@shreyas-londhe

@shreyas-londhe shreyas-londhe commented Mar 28, 2025 •

Copy link
Copy Markdown
Contributor

Note

Replaces the old Rust compiler and extensive Circom tests with a new Bun/TypeScript scripts suite (gen-regex, gen-inputs), shared utilities, and Jest-based tests.

  • Tooling/Scripts (new):
    • Add Bun/TypeScript scripts for generating regex artifacts: circom/scripts/gen-regex.ts, noir/scripts/gen-regex.ts, noir/scripts/gen-inputs.ts.
    • Introduce shared utilities (utils/*) for logging, file ops, subprocess, and types.
    • Add project config: tsconfig.json, jest.config.ts, jest.setup.ts, bun.lock, and package.json.
  • Tests (new):
    • Add Jest integration and unit tests under scripts/__tests__ for script workflows and utilities.
  • Compiler (removed):
    • Remove legacy Rust compiler package (packages/compiler/*).
  • Circom Tests (removed):
    • Remove numerous legacy Circom test circuits and test cases under packages/circom/tests/*.

Written by Cursor Bugbot for commit 3d37c31. This will update automatically on new commits. Configure here.

Summary by CodeRabbit

  • New Features

    • Many new regex verification circuits for email headers/addresses, plus a compiler and WASM bindings to generate Circom/Noir circuits with capture extraction.
  • Infrastructure & Build

    • CI workflow replaced; new test workflow and pre-push hook to keep generated templates in sync; new project/package and TypeScript configs for tooling.
  • Documentation

    • Major README rewrite and new docs for compiler, Circom, and Noir with usage, troubleshooting, and developer guidance.
  • Tests

    • End-to-end circuit tests, sample inputs, and generation scripts for multiple regexes.
  • Chores

    • Lint/format configs removed and .gitignore/workspace manifests updated.

…shutdown

Replace per-process SIGINT/SIGTERM handlers with a centralized
AbortController. All Bun.spawn calls now receive the abort signal,
benchmark loops check isAborted() to break early, and the first
Ctrl+C triggers graceful unwinding through finally blocks.
… at all sizes

Replace hardcoded sample inputs with a configurable scaling system
that fills each target input length with regex-matching content.
Three strategies: repeat (tile template), extend (grow match portion),
and pad-with-match (anchored template + filler).

Each pattern in patterns.json now declares inputTemplate and
scalingStrategy. All three providers use generateScaledInput()
instead of static lookup tables. Output reports include scaling
methodology, actual content lengths, and per-pattern strategy tables.

Also improves memory measurement: separates command stderr from
/usr/bin/time output for reliable RSS parsing, records data from
non-zero exits (common with nested shell wrapping), and adds
--no-memory flag support via BENCH_NO_MEMORY env var.
Use accurate compiler backend names (DFA/NFA) instead of version
numbers (v1/v2) throughout benchmark configs, output generation,
and type definitions for clarity in academic context.
Increase minRuns to 5 and maxRuns to 10 for better statistical
significance. Add missing generateScaledInput imports to circom
and noir providers. Add docs/ to gitignore.
Add compilerCommitHash and toolVersions fields to individual benchmark
result files for reproducibility. Rename all provider identifiers from
v1/v2 to DFA/NFA to better describe the algorithm difference.

- Extend BenchmarkProvider interface with getCommitHash() and getToolVersions()
- Move BenchmarkResult type from bench.ts to shared types.ts
- Rename providers: circom-v1→circom-dfa, circom-v2→circom-nfa, noir-v2→noir-nfa
- Update all type fields, config keys, CLI args, scripts, and README
- Add compilerVersions aggregation to collect-results.ts
Remove ~120 lines of dead code: isOk, isErr, unwrap, unwrapOr, map,
flatMap from errors.ts; runHyperfineBatch, measureSync, hashFile,
proveAndVerify from utils. None had any callers in the codebase.
…code

Create two new shared modules (exec.ts, project.ts) and centralize
duplicated functions across 7 files (~19 duplicate instances removed):

- execAsync/execCommand: unified in utils/exec.ts with configurable
  options (useNvm, errorFactory) to preserve each call site's behavior
- wrapCommandWithNvm/getNvmWrappedCommand: unified with bashWrap option
- getProjectRoot/getGitCommitHash: extracted to utils/project.ts
- defaultMemoryStats: removed local copy from collect-results.ts
- getToolVersions: Circom sync version centralized in hardware.ts
- measureWitnessGeneration: promoted to shared utility in snarkjs.ts

Net reduction: ~300 lines. No behavioral changes.
…teStats

Variance was dividing by n (population variance) instead of n-1 (sample
variance). With minRuns=5 this underestimates stddev by ~12%. Added n>1
guard to avoid division by zero on single measurements.
Validate git ref against a strict pattern and quote it in the shell
command to prevent injection via a malicious benchmark.json value.
Rename remaining v1/v2 references in generate-outputs.ts comments and
reductionPct parameter names to use DFA/NFA terminology, and update
the module docstring in src/index.ts.
The PTAU file download performed no integrity verification after
fetching a ~2GB file. Add streaming SHA256 verification for both
fresh downloads and cached files, with automatic re-download on
mismatch.
Replace the abstract class with direct interface implementation since all
methods were abstract except an empty cleanup() that every provider overrides.
Also consolidate NoirCircuitInput and MemoryStats into types.ts.
…benchmarks-package

feat(benchmarks): add benchmarking suite for DFA vs NFA comparison
The linkage check stopped at maxMatchBytes-2, so when matchLength ==
maxMatchBytes the final transition was not tied to the rest of the path
and a prover could jump straight into an accept state. Adds a forged
witness regression test and regenerates the common circuits.
Lookup keys pack state + byte*257 + next*257^2, which stops being
injective past 257 states. Detect real aliasing at codegen time and fail
with a clear error instead of emitting an unsound circuit.
Merge prep for #92: 3.0.0-alpha.1, soundness fixes, migration guide
@socket-security

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn Critical
Critical CVE: Handlebars.js has JavaScript Injection via AST Type Confusion

CVE: GHSA-2w6w-674q-4c4q Handlebars.js has JavaScript Injection via AST Type Confusion (CRITICAL)

Affected versions: >= 4.0.0 < 4.7.9

Patched version: 4.7.9

From: scripts/package.json → npm/ts-jest@29.4.1 → npm/handlebars@4.7.8

ℹ Read more on: This package | This alert | What is a critical CVE?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Remove or replace dependencies that include known critical CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/handlebars@4.7.8. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn High
Obfuscated code: npm @noble/hashes is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: benchmarks/package.json → npm/snarkjs@0.7.6 → npm/@noble/hashes@1.8.0

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/@noble/hashes@1.8.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn High
Obfuscated code: npm @noble/hashes is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: benchmarks/package.json → npm/snarkjs@0.7.6 → npm/@noble/hashes@1.8.0

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/@noble/hashes@1.8.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

@rutefig
rutefig merged commit 89a4adb into main Sep 17, 2026
6 checks passed
@rutefig
rutefig deleted the feat/new-compiler branch September 17, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants