perf(graaljs): opt-in per-thread context reuse, plus benchmark tooling and engine guide

README.md

@@ -155,9 +155,30 @@ val evaluator = com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSEva
 
 #### How it works
 
-The evaluator holds a shared polyglot `Engine` and builds a new `Context` per `evaluate` call. Input data is injected as
-context bindings (maps by key, other objects by Kotlin reflection with `HostAccess.ALL`), and each operator evaluates a
-JavaScript `Source` against that context, so both operands are arbitrary JavaScript.
+The evaluator holds a shared polyglot `Engine` and, by default, builds a new `Context` per `evaluate` call and closes it
+afterwards. Input data is injected into a fresh JavaScript object created for that evaluation (maps by key, other
+objects by Kotlin reflection with `HostAccess.ALL`), and each operator evaluates a JavaScript `Source` resolved against
+that object, so both operands are arbitrary JavaScript.
+
+#### Reusing the context
+
+Building a `Context` is what the engine spends almost all of its time on. `reuseContextPerThread` keeps one context per
+thread instead, which is roughly 25x faster on the benchmark below:
+
+```kotlin
+val evaluator = GraalJSEvaluator(reuseContextPerThread = true)
+```
+
+| | default (`false`) | `reuseContextPerThread = true` |
+|---------------------------------|--------------------------------|-------------------------------------------------|
+| context lifetime | built and closed per `evaluate` | one per thread, alive as long as the thread |
+| concurrent `evaluate` | safe | safe, each thread has its own context |
+| input bindings between calls | isolated | isolated, the input object is replaced per call |
+| globals a rule writes | discarded with the context | visible to later calls on the same thread |
+
+So the reused mode is safe to call concurrently and never leaks input data between evaluations, but a rule that writes
+to `globalThis` or redefines a builtin affects later evaluations on the same thread. Use it with a bounded thread pool:
+per-thread contexts are not closed, so unbounded thread creation retains them.
 
 Both the `Engine` and the `Context.Builder` are constructor parameters, which is where the engine is tuned:
 
@@ -182,8 +203,9 @@ pass a restricted `Context.Builder`.
 
 #### Trade-offs
 
-* The slowest of the three engines in the measurement below, and most of that cost is context creation rather than the
-  rule itself
+* The slowest of the three engines in the measurement below when left on the default context handling, and most of that
+  cost is context creation rather than the rule itself. `reuseContextPerThread = true` removes it, at the cost of the
+  global-state isolation described above
 * On a stock (non-GraalVM) JDK, Truffle runs in interpreter-only mode. The evaluator sets
   `engine.WarnInterpreterOnly=false`, so the usual warning is not printed. Running on a GraalVM JDK, or adding the Graal
   compiler to the runtime classpath, is what unlocks its performance
@@ -216,6 +238,7 @@ Every evaluator module ships a `bench` task that replays the full test rule set
 ./gradlew :kotlin-evaluator:bench
 ./gradlew :rhino-evaluator:bench
 ./gradlew :graaljs-evaluator:bench -PbenchIterations=5000
+./gradlew :graaljs-evaluator:bench -PbenchIterations=5000 -PbenchReuse=true
 ```
 
 Each iteration evaluates the 147 expressions from `com.rapatao.projects.ruleset.engine.cases.TestData` against the same
@@ -227,14 +250,21 @@ Numbers below come from that benchmark, 2000 iterations (294,000 evaluations per
 Amazon Corretto 21.0.11. Treat them as relative magnitudes, not absolute figures: the harness is a simple timing loop,
 not JMH, and the GraalJS run is interpreter-only because Corretto is not a GraalVM JDK.
 
-| engine  | ops/s   | avg per iteration | p50       | p99       | relative cost |
-|---------|---------|-------------------|-----------|-----------|---------------|
-| Kotlin  | 566,752 | 259us             | 210us     | 638us     | 1x            |
-| Rhino   | 47,827  | 3.07ms            | 2.99ms    | 4.60ms    | ~12x          |
-| GraalJS | 9,688   | 15.17ms           | 14.97ms   | 17.15ms   | ~58x          |
+| engine               | ops/s    | avg per iteration | p50       | p99       | relative cost |
+|----------------------|----------|-------------------|-----------|-----------|---------------|
+| Kotlin               | 566,752  | 259us             | 210us     | 638us     | 1x            |
+| GraalJS (reused ctx) | ~240,000 | ~590us            | ~500us    | ~2.0ms    | ~2x           |
+| Rhino                | 47,827   | 3.07ms            | 2.99ms    | 4.60ms    | ~12x          |
+| GraalJS              | 9,391    | 15.65ms           | 15.57ms   | 17.36ms   | ~60x          |
+
+Most engines are stable under load, with the p99 within 1.2x to 3x of the median. The reused-context GraalJS row is the
+exception: it varied between 185,000 and 294,000 ops/s across runs here, so it is quoted as an approximation. Once the
+context cost is gone, an iteration is short enough that the timing loop measures JIT and GC noise as much as the
+engine.
 
-All three engines are stable under load: the p99 stays within 1.2x to 3x of the median, so there are no pathological
-outliers, only different constant costs.
+`GraalJS (reused ctx)` is the same engine with `reuseContextPerThread = true`. Closing the per-call context and
+injecting the input into a per-evaluation object costs the default mode about 4% (9,750 to 9,391 ops/s here), and buys
+deterministic context release plus binding isolation that holds under reuse.
 
 ### Where the time goes
 
@@ -255,12 +285,14 @@ Reading of the table:
   running one small script per operator, which is why deep rule trees cost more than the numbers for a single rule
   suggest
 * **GraalJS**: context creation dominates almost entirely. On this setup the rule itself is nearly free compared to the
-  polyglot context it runs in
+  polyglot context it runs in, which is what `reuseContextPerThread = true` removes
 
 ### Practical guidance
 
 * Reuse the evaluator instance. Operators are resolved once in the constructor, and for GraalJS the shared `Engine`
   caches parsed sources across contexts, so a new evaluator per request throws that away
+* On GraalJS, set `reuseContextPerThread = true` unless rules are untrusted or deliberately write globals. It is the
+  single largest win available on that engine
 * Pass the narrowest input object that satisfies the rule. All three engines materialise the whole input per call
 * Prefer `Map` inputs over arbitrary objects when the data is already in that shape: the object path goes through
   Kotlin reflection
@@ -271,10 +303,11 @@ Reading of the table:
 * On Rhino, keep the default `interpretedMode = true`. Compiled mode measures about 10x slower here, because each
   operator compiles a new script that is thrown away
 
-Both JS engines rebuild their context per `evaluate` call, and that dominates their cost. Overriding `call` to reuse a
-context makes the benchmark suite about 48x faster on GraalJS (16.6ms to 0.35ms) and about 11x faster on Rhino (3.5ms
-to 0.30ms), at the price of thread safety and binding isolation between evaluations. See
-[docs/tasks](docs/tasks) for the analysis and the trade-offs.
+Both JS engines rebuild their context per `evaluate` call, and that dominates their cost. On GraalJS this is now an
+opt-in: `reuseContextPerThread = true` keeps one context per thread and runs the suite roughly 25x faster (15.6ms to
+about 0.6ms) while staying thread-safe and isolating input bindings. Rhino still rebuilds its scope per call; reusing it
+there is about 11x faster (3.5ms to 0.30ms) but is not implemented. See [docs/tasks](docs/tasks) for the analysis and
+the trade-offs.
 
 ## Get started
 

graaljs-evaluator/build.gradle

@@ -8,8 +8,11 @@ dependencies {
 
 tasks.register("bench", JavaExec) {
     group = "verification"
-    description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000"
+    description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000, reused context: -PbenchReuse=true"
     classpath = sourceSets.test.runtimeClasspath
     mainClass = "com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSBenchmarkKt"
-    args = [providers.gradleProperty("benchIterations").getOrElse("1000")]
+    args = [
+        providers.gradleProperty("benchIterations").getOrElse("1000"),
+        providers.gradleProperty("benchReuse").getOrElse("false"),
+    ]
 }

graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSEvaluator.kt

@@ -27,6 +27,9 @@ import org.graalvm.polyglot.Value
  *
  * @property engine The GraalVM Polyglot Engine instance used by this evaluator engine.
  * @property contextBuilder A builder instance used to create a JavaScript Context with custom options and settings.
+ * @property reuseContextPerThread When `false` (the default), every evaluation builds a new Context and closes it
+ * afterwards. When `true`, each thread keeps one Context and reuses it across evaluations, which removes the context
+ * construction cost from every call. See the README for the trade-offs of the reused mode.
  *
  * @see org.graalvm.polyglot.Context.Builder
  * @see org.graalvm.polyglot.Engine
@@ -41,6 +44,7 @@ open class GraalJSEvaluator(
         .allowHostAccess(HostAccess.ALL).allowHostClassLookup { true }
         .option("js.nashorn-compat", "true").allowExperimentalOptions(true),
     operators: List<Operator> = listOf(),
+    private val reuseContextPerThread: Boolean = false,
 ) : Evaluator(
     operators = listOf(
         Equals(),
@@ -58,17 +62,30 @@ open class GraalJSEvaluator(
     ) + operators,
 ) {
 
+    private val threadContext = ThreadLocal.withInitial { contextBuilder.build() }
+
     override fun <T> call(inputData: Any, block: EvalContext.() -> T): T =
-        createContext().let {
-            parseParameters(
-                it.getBindings("js"),
-                inputData,
-            )
-            block(GraalJSContext(this, it))
+        if (reuseContextPerThread) {
+            evaluateWith(threadContext.get(), inputData, block)
+        } else {
+            contextBuilder.build().use { evaluateWith(it, inputData, block) }
         }
 
-    private fun createContext(): Context {
-        return contextBuilder.build()
+    private fun <T> evaluateWith(context: Context, inputData: Any, block: EvalContext.() -> T): T {
+        val scope = context.eval("js", "({})")
+
+        parseParameters(scope, inputData)
+
+        val bindings = context.getBindings("js")
+        // an operator may evaluate another expression through EvalContext.engine(), which reuses this context
+        val outerScope = bindings.getMember(INPUT_SCOPE)
+        bindings.putMember(INPUT_SCOPE, scope)
+
+        return try {
+            block(GraalJSContext(this, context))
+        } finally {
+            outerScope?.let { bindings.putMember(INPUT_SCOPE, it) }
+        }
     }
 
     /**
@@ -81,6 +98,9 @@ open class GraalJSEvaluator(
     /**
      * Parses parameters and injects them into the given scope based on the input data.
      *
+     * The scope is a fresh JavaScript object created for the evaluation, and it is replaced on every call, so the
+     * injected members are never visible to another evaluation.
+     *
      * @param bindings the values object where the parameters will be injected
      * @param inputData the input data containing the parameters
      */
@@ -90,4 +110,11 @@ open class GraalJSEvaluator(
             else -> TypedInjector.inject(bindings, inputData)
         }
     }
+
+    internal companion object {
+        /**
+         * Name of the global member holding the input data of the current evaluation.
+         */
+        const val INPUT_SCOPE = "__ruleset_input__"
+    }
 }

graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/operator/extensions.kt

@@ -2,6 +2,7 @@ package com.rapatao.projects.ruleset.engine.evaluator.graaljs.operator
 
 import com.rapatao.projects.ruleset.engine.context.EvalContext
 import com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSContext
+import com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSEvaluator.Companion.INPUT_SCOPE
 import org.graalvm.polyglot.Source
 
 internal fun EvalContext.evaluate(content: String): Boolean {
@@ -10,7 +11,7 @@ internal fun EvalContext.evaluate(content: String): Boolean {
     return graalJSContext.context().eval(
         Source.newBuilder(
             "js",
-            "true == ($content)",
+            "(function() { with ($INPUT_SCOPE) { return true == ($content) } })()",
             content,
         ).buildLiteral()
     ).asBoolean()

graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt

@@ -3,5 +3,7 @@ package com.rapatao.projects.ruleset.engine.evaluator.graaljs
 import com.rapatao.projects.ruleset.engine.BaseEngineBenchmark
 
 fun main(args: Array<String>) {
-    BaseEngineBenchmark(GraalJSEvaluator()).main(args)
+    BaseEngineBenchmark(
+        GraalJSEvaluator(reuseContextPerThread = args.getOrNull(1).toBoolean())
+    ).main(args)
 }

graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSReusedContextEvaluatorTest.kt

@@ -0,0 +1,93 @@
+package com.rapatao.projects.ruleset.engine.evaluator.graaljs
+
+import com.rapatao.projects.ruleset.engine.BaseEvaluatorTest
+import com.rapatao.projects.ruleset.engine.cases.TestData
+import com.rapatao.projects.ruleset.engine.context.EvalContext
+import com.rapatao.projects.ruleset.engine.helper.ExposeEngineTestOperator
+import com.rapatao.projects.ruleset.engine.types.Expression
+import com.rapatao.projects.ruleset.engine.types.builder.MatcherBuilder.allMatch
+import com.rapatao.projects.ruleset.engine.types.builder.extensions.equalsTo
+import com.rapatao.projects.ruleset.engine.types.operators.Operator
+import org.hamcrest.MatcherAssert.assertThat
+import org.hamcrest.Matchers.equalTo
+import org.junit.jupiter.api.DisplayName
+import org.junit.jupiter.api.Test
+import org.junit.jupiter.api.assertThrows
+import java.util.concurrent.Executors
+import java.util.concurrent.TimeUnit
+
+class GraalJSReusedContextEvaluatorTest : BaseEvaluatorTest(evaluator) {
+
+    companion object {
+        private const val THREADS = 8
+
+        private val evaluator = GraalJSEvaluator(
+            operators = listOf(
+                ExposeEngineTestOperator(),
+            ),
+            reuseContextPerThread = true,
+        )
+    }
+
+    @Test
+    @DisplayName("should not expose bindings from a previous evaluation")
+    fun assertBindingsAreNotSharedBetweenEvaluations() {
+        val rule = "b" equalsTo 2
+
+        assertThat(
+            evaluator.evaluate(rule, mapOf("a" to 1, "b" to 2)),
+            equalTo(true),
+        )
+
+        assertThrows<Exception> {
+            evaluator.evaluate(rule, mapOf("a" to 1))
+        }
+    }
+
+    @Test
+    @DisplayName("should restore the input of the outer evaluation when an operator evaluates another expression")
+    fun assertNestedEvaluationDoesNotReplaceTheInput() {
+        val nesting = NestedEvaluationOperator()
+        val evaluator = GraalJSEvaluator(operators = listOf(nesting), reuseContextPerThread = true)
+
+        val rule = allMatch(
+            Expression(left = "outer", operator = nesting.name(), right = "1"),
+            "outer" equalsTo 1,
+        )
+
+        assertThat(evaluator.evaluate(rule, mapOf("outer" to 1)), equalTo(true))
+    }
+
+    /**
+     * Evaluates an unrelated expression against a different input, the way a custom operator may do through
+     * [com.rapatao.projects.ruleset.engine.context.EvalContext.engine].
+     */
+    private class NestedEvaluationOperator : Operator {
+        override fun process(context: EvalContext, left: Any?, right: Any?): Boolean =
+            context.engine().evaluate("inner" equalsTo 2, mapOf("inner" to 2))
+
+        override fun name(): String = "nested_evaluation"
+    }
+
+    @Test
+    @DisplayName("should evaluate concurrently without sharing the context between threads")
+    fun assertConcurrentEvaluation() {
+        val cases = TestData.cases().map { it.get() }
+            .map { it[0] as Expression to it[1] as Boolean }
+
+        val pool = Executors.newFixedThreadPool(THREADS)
+        try {
+            val results = (1..THREADS).map {
+                pool.submit {
+                    cases.forEach { (expression, expected) ->
+                        assertThat(evaluator.evaluate(expression, TestData.inputData), equalTo(expected))
+                    }
+                }
+            }
+
+            results.forEach { it.get(1, TimeUnit.MINUTES) }
+        } finally {
+            pool.shutdownNow()
+        }
+    }
+}