Skip to content

perf(kotlin): resolve operand paths lazily, and fix numeric comparison #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapatao merged 4 commits into from
Aug 31, 2026
Merged

perf(kotlin): resolve operand paths lazily, and fix numeric comparison #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e58b76e
perf(kotlin): resolve operands without compiling a regex
rapatao Aug 31, 2026
269d231
fix(kotlin): resolve the elements of a list operand
rapatao Aug 31, 2026
4f61c19
perf(benchmark): add allocation and gc tracking
rapatao Aug 31, 2026
a416dac
fix: normalize numbers by value, not scale
rapatao Aug 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 94 additions & 29 deletions BENCHMARKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,20 @@ Every evaluator module ships a `bench` task that replays the full test rule set
./gradlew :rhino-evaluator:bench
./gradlew :graaljs-evaluator:bench -PbenchIterations=5000
./gradlew :graaljs-evaluator:bench -PbenchIterations=5000 -PbenchReuse=true
./gradlew :kotlin-evaluator:bench -PbenchWide=200
```

Each iteration evaluates the 147 expressions from `com.rapatao.projects.ruleset.engine.cases.TestData` against the same
Each iteration evaluates the 173 expressions from `com.rapatao.projects.ruleset.engine.cases.TestData` against the same
input object, after 100 warmup iterations. Results are printed and written to `bench_<engine>.txt`.

Each run reports throughput, the latency distribution of an iteration (avg, stddev, min, max, p50 to p99), the bytes
allocated per evaluation, and the garbage collections that ran during the measured loop. Allocation is read from the
JVM's per-thread counter on the benchmark thread, and reports `n/a` on a JVM that does not expose it.

`-PbenchWide=N` runs the same rules against the same `item`, under a root carrying `N` extra scalar fields and an `N`
element list. Nothing the rules read changes, only how much input surrounds it, which separates a per-call cost that
scales with the input from one that scales with the rule.

Two things to set up before trusting a run:

* Run at full power. On a laptop in a power saving mode the whole suite lands 25 to 30% low, uniformly across engines.
Expand All @@ -25,30 +34,78 @@ Two things to set up before trusting a run:

## Results

2000 iterations (294,000 evaluations per engine), Apple M3 Pro, Amazon Corretto 21.0.11. These are relative
magnitudes, not absolute figures: the harness is a simple timing loop, not JMH, and the GraalJS run is interpreter-only
because Corretto is not a GraalVM JDK.
2000 iterations of the 173 expression suite, 346,000 evaluations per engine, Apple M3 Pro, Amazon Corretto 21.0.11,
three runs per configuration in one session at full power, medians below. These are relative magnitudes, not absolute figures: the harness is a
simple timing loop, not JMH, and the GraalJS run is interpreter-only because Corretto is not a GraalVM JDK.

| engine | ops/s | avg per iteration | stddev | p50 | p99 | relative cost |
|----------------------|-----------|-------------------|--------|---------|---------|---------------|
| Kotlin | 1,506,886 | 115us | 93us | 83us | 425us | 1x |
| Rhino | 288,132 | 600us | 240us | 490us | 1.41ms | ~5.2x |
| GraalJS (reused ctx) | 240,662 | 719us | 439us | 580us | 2.58ms | ~6.3x |
| GraalJS | 8,959 | 19.31ms | 640us | 19.09ms | 21.08ms | ~168x |

Run-to-run spread differs by engine, and sets how large a difference has to be before it means anything:

| engine | observed across runs | p99 vs p50 |
|----------------------|------------------------|------------|
| Kotlin | 1,454,000 to 1,621,000 | ~5.1x |
| Rhino | 287,000 to 320,000 | ~2.9x |
| GraalJS (reused ctx) | 237,000 to 257,000 | ~4.4x |
| GraalJS | 8,771 to 9,176 | ~1.1x |

The three fast configurations move by about 10% across runs and their p99 is 3 to 5 times their p50. The tail is GC
and JIT, not the engine, so read a difference smaller than that as noise. Default GraalJS is the exception on both
counts: an iteration is so dominated by context creation that nothing else is visible in it.

The Kotlin engine on a wide input is the least stable measurement here, spanning 1,409,000 to 2,195,000 across its
three runs. Its iteration is short enough that the loop measures the JVM more than the engine.

### Allocation

Bytes allocated per `evaluate`, counted on the benchmark thread by the JVM's own allocation counter, and the garbage
collections that ran during the measured loop:

| engine | alloc per evaluation | vs Kotlin | gc during the run |
|----------------------|----------------------|-----------|-------------------|
| Kotlin | 893 B | 1x | 2, 3ms |
| GraalJS (reused ctx) | 5,850 B | ~6.6x | 15, 19ms |
| Rhino | 11,741 B | ~13.1x | 17, 22ms |
| GraalJS | 128,763 B | ~144x | 425, 186ms |

This is the steadiest number the harness produces: it varies by under 0.5% across runs, where throughput varies by 5%.

The order is not the throughput order. Reused-context GraalJS allocates less per evaluation than Rhino and is still
slower, so Rhino's cost is not allocation-bound: it compiles a fresh script per operator invocation, and compilation
is work rather than garbage. Default GraalJS allocates a whole polyglot `Context` per call, which is the 144x.

### Input width

The same run with `-PbenchWide=200`: identical rules reading identical fields, under a root carrying 200 extra scalar
fields and a 200 element list.

| engine | ops/s default | ops/s wide(200) | wide is | alloc default | alloc wide(200) | wide allocates |
|----------------------|---------------|-----------------|---------------|---------------|-----------------|----------------|
| Kotlin | 1,506,886 | 1,768,634 | unchanged | 893 B | 872 B | unchanged |
| Rhino | 288,132 | 138,658 | ~2.1x slower | 11,741 B | 23,788 B | ~2.0x more |
| GraalJS (reused ctx) | 240,662 | 16,313 | ~14.8x slower | 5,850 B | 85,596 B | ~14.6x more |
| GraalJS | 8,959 | 5,875 | ~1.5x slower | 128,763 B | 208,161 B | ~1.6x more |

| engine | ops/s | avg per iteration | p50 | p99 | relative cost |
|----------------------|----------|-------------------|-----------|-----------|---------------|
| Kotlin | 782,637 | 188us | 166us | 331us | 1x |
| Rhino | 352,564 | 417us | 340us | 1.18ms | ~2.2x |
| GraalJS (reused ctx) | ~240,000 | ~590us | ~500us | ~2.0ms | ~3.3x |
| GraalJS | 9,391 | 15.65ms | 15.57ms | 17.36ms | ~83x |
Each factor compares the two columns to its left, within the same row. A row varies the input only: the engine and its
configuration are held constant across it, so `reuseContextPerThread` is on in both columns of the reused row and off
in both columns of the row below it. The Results table above prices the reuse setting.

Each row is one representative run. Run-to-run spread differs sharply by engine, and sets how large a difference has to
be before it means anything:
For both JS engines the allocation factor tracks the throughput factor, which identifies the cost: they inject every
top-level entry of the input into the scope on every `evaluate`, and pay for it whether a rule reads it or not.
Neither pays for *depth*, since a nested object is handed over whole and JS walks into it lazily. Reused-context
GraalJS runs 157 collections over the wide input against 15 over the default one, and Rhino 34 against 17.

| engine | observed across runs | p99 vs p50 |
|----------------------|----------------------|------------|
| Kotlin | 778,000 to 791,000 | ~2x |
| Rhino | 286,000 to 394,000 | ~3.5x |
| GraalJS (reused ctx) | 185,000 to 294,000 | ~4x |
| GraalJS | stable within a few % | ~1.1x |
Default GraalJS shows the smallest factor because context creation, at ~19ms per iteration, dominates the injection.
In the reused-context mode the injection is the dominant remaining cost.

The two fastest configurations are the least stable. Once the per-evaluation context cost is gone, an iteration is
short enough that the loop measures JIT and GC noise as much as the engine. Default GraalJS is the opposite: an
iteration is so dominated by context creation that nothing else is visible.
The Kotlin engine resolves the paths a rule names and never visits the rest, so its cost tracks the rule. Its two
columns overlap across runs, with the wide one reading slightly faster: that input roots at a `Map` while the default
one roots at a data class, so one hash lookup replaces one reflective property read.

`GraalJS (reused ctx)` is the same engine with `reuseContextPerThread = true`. Closing the per-call context and
injecting the input into a per-evaluation object costs the default mode about 4% (9,750 to 9,391 ops/s), and buys
Expand All @@ -68,29 +125,37 @@ The `Evaluator` contract sets up a fresh evaluation context on every `evaluate`

These rows come from one tight loop over a single rule, after 50,000 warmup calls, so they isolate the steady-state
cost. They are not comparable to the suite numbers above, which include cold and JIT-transient iterations. That loop is
not part of this repository and the `bench` tasks do not reproduce it. The Kotlin row predates the current operand
parsing, which cut per-operand work and not context setup, so its real setup share is above the ~82% shown.
not part of this repository and the `bench` tasks do not reproduce it. The Kotlin row predates both the current operand
parsing and the removal of input flattening.

Reading of the table:

* **Kotlin**: the fixed cost is flattening the input graph, and it is nearly the whole cost. It scales with the size of
the input object, not with the rule, so a wide input evaluated against a two-field rule pays for every other field
* **Kotlin**: the 0.88us of setup was flattening the whole input graph into a map of every path, which scaled with the
size of the input rather than the rule. That step is gone. Setup is now a constructor call, operand paths are
resolved on demand, and the cost tracks the rule: widening the input to 200 extra fields and a 200 element list cost
11.7x under flattening and costs nothing measurable now
* **Rhino**: setup is entering a `Context`, creating a child scope and injecting the input, because the standard
objects are shared. Before that change the same two columns read 20.5us and 13.0us, a ~63% share. What is left is
compiling and running one small script per operator, which is why deep rule trees cost more than the numbers for a
single rule suggest
single rule suggest. The injection half of that setup is what the input width table above prices
* **GraalJS**: context creation dominates almost entirely. On this setup the rule itself is nearly free compared to the
polyglot context it runs in, which is what `reuseContextPerThread = true` removes
polyglot context it runs in, which is what `reuseContextPerThread = true` removes. What remains once it is removed is
injecting the input, the cost that grows with the input

## Practical guidance

* Reuse the evaluator instance. Operators are resolved once in the constructor, and for GraalJS the shared `Engine`
caches parsed sources across contexts, so a new evaluator per request throws that away
* On GraalJS, set `reuseContextPerThread = true` unless rules are untrusted or deliberately write globals. It is the
single largest win available on that engine
* Pass the narrowest input object that satisfies the rule. All three engines materialise the whole input per call
* On the JS engines, pass the narrowest input object that satisfies the rule: both inject every top-level entry per
call, worth 2.1x on Rhino and 14.8x on reused-context GraalJS for 200 extra fields. Nesting the parts a rule does not
read one level deeper avoids it. The Kotlin engine reads only the paths a rule names and is flat here
* Watch allocation, not just throughput, if the service is latency-sensitive: an evaluation costs 893 B on the Kotlin
engine and 128 KB on default GraalJS, and that is what fills the nursery and sets the GC rate under load
* Prefer `Map` inputs over arbitrary objects when the data is already in that shape: the object path goes through
Kotlin reflection
Kotlin reflection. On the Kotlin engine this is now a small difference, since the properties of each class are
reflected once and cached
* Order `anyMatch` cheaply-first and `allMatch` most-selective-first. Evaluation short-circuits, and with the JS engines
every skipped expression is a script that is never compiled
* On GraalJS, run on a GraalVM JDK (or put the Graal compiler on the runtime classpath) before drawing conclusions from
Expand Down
208 changes: 208 additions & 0 deletions JSON.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2014,3 +2014,211 @@ To see more details, check its source: [here](src/test/kotlin/com/rapatao/projec
}
```

```json
{
"left" : [ "\"item1\"", "\"item2\"" ],
"operator" : "contains",
"right" : "\"item1\""
}
```

```json
{
"left" : [ "\"item1\"", "\"item2\"" ],
"operator" : "not_contains",
"right" : "\"item3\""
}
```

```json
{
"left" : [ "item.name", "\"something else\"" ],
"operator" : "contains",
"right" : "\"product name\""
}
```

```json
{
"left" : "item.weight",
"operator" : "equals",
"right" : "1.5"
}
```

```json
{
"left" : "item.weight",
"operator" : "equals",
"right" : "1.9"
}
```

```json
{
"left" : "item.weight",
"operator" : "equals",
"right" : "1.0"
}
```

```json
{
"left" : "item.weight",
"operator" : "equals",
"right" : "1"
}
```

```json
{
"left" : "item.weight",
"operator" : "not_equals",
"right" : "1.9"
}
```

```json
{
"left" : "item.weight",
"operator" : "greater_than",
"right" : "1.4"
}
```

```json
{
"left" : "item.weight",
"operator" : "greater_than",
"right" : "1.6"
}
```

```json
{
"left" : "item.weight",
"operator" : "less_than",
"right" : "1.6"
}
```

```json
{
"left" : "item.weight",
"operator" : "less_than",
"right" : "1.4"
}
```

```json
{
"left" : "item.weight",
"operator" : "greater_or_equal_than",
"right" : "1.5"
}
```

```json
{
"left" : "item.scaled",
"operator" : "equals",
"right" : 10
}
```

```json
{
"left" : "item.scaled",
"operator" : "equals",
"right" : "10.0"
}
```

```json
{
"left" : "item.scaled",
"operator" : "equals",
"right" : "10.00"
}
```

```json
{
"left" : "item.scaled",
"operator" : "not_equals",
"right" : 10
}
```

```json
{
"left" : "item.scaled",
"operator" : "equals",
"right" : "10.01"
}
```

```json
{
"left" : "item.price",
"operator" : "equals",
"right" : "10.00"
}
```

```json
{
"left" : [ 1, 2 ],
"operator" : "contains",
"right" : 1
}
```

```json
{
"left" : [ 1, 2 ],
"operator" : "contains",
"right" : 3
}
```

```json
{
"left" : [ 1, 2 ],
"operator" : "not_contains",
"right" : 3
}
```

```json
{
"left" : "item.quantities",
"operator" : "contains",
"right" : "1"
}
```

```json
{
"left" : "item.quantities",
"operator" : "contains",
"right" : "3"
}
```

```json
{
"left" : "item.quantities",
"operator" : "not_contains",
"right" : "3"
}
```

```json
{
"left" : "item.quantities",
"operator" : "contains",
"right" : "1.0"
}
```

Loading
Loading