diff --git a/README.md b/README.md index 7d4ce98..baa2a71 100644 --- a/README.md +++ b/README.md @@ -155,9 +155,30 @@ val evaluator = com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSEva #### How it works -The evaluator holds a shared polyglot `Engine` and builds a new `Context` per `evaluate` call. Input data is injected as -context bindings (maps by key, other objects by Kotlin reflection with `HostAccess.ALL`), and each operator evaluates a -JavaScript `Source` against that context, so both operands are arbitrary JavaScript. +The evaluator holds a shared polyglot `Engine` and, by default, builds a new `Context` per `evaluate` call and closes it +afterwards. Input data is injected into a fresh JavaScript object created for that evaluation (maps by key, other +objects by Kotlin reflection with `HostAccess.ALL`), and each operator evaluates a JavaScript `Source` resolved against +that object, so both operands are arbitrary JavaScript. + +#### Reusing the context + +Building a `Context` is what the engine spends almost all of its time on. `reuseContextPerThread` keeps one context per +thread instead, which is roughly 25x faster on the benchmark below: + +```kotlin +val evaluator = GraalJSEvaluator(reuseContextPerThread = true) +``` + +| | default (`false`) | `reuseContextPerThread = true` | +|---------------------------------|--------------------------------|-------------------------------------------------| +| context lifetime | built and closed per `evaluate` | one per thread, alive as long as the thread | +| concurrent `evaluate` | safe | safe, each thread has its own context | +| input bindings between calls | isolated | isolated, the input object is replaced per call | +| globals a rule writes | discarded with the context | visible to later calls on the same thread | + +So the reused mode is safe to call concurrently and never leaks input data between evaluations, but a rule that writes +to `globalThis` or redefines a builtin affects later evaluations on the same thread. Use it with a bounded thread pool: +per-thread contexts are not closed, so unbounded thread creation retains them. Both the `Engine` and the `Context.Builder` are constructor parameters, which is where the engine is tuned: @@ -182,8 +203,9 @@ pass a restricted `Context.Builder`. #### Trade-offs -* The slowest of the three engines in the measurement below, and most of that cost is context creation rather than the - rule itself +* The slowest of the three engines in the measurement below when left on the default context handling, and most of that + cost is context creation rather than the rule itself. `reuseContextPerThread = true` removes it, at the cost of the + global-state isolation described above * On a stock (non-GraalVM) JDK, Truffle runs in interpreter-only mode. The evaluator sets `engine.WarnInterpreterOnly=false`, so the usual warning is not printed. Running on a GraalVM JDK, or adding the Graal compiler to the runtime classpath, is what unlocks its performance @@ -216,6 +238,7 @@ Every evaluator module ships a `bench` task that replays the full test rule set ./gradlew :kotlin-evaluator:bench ./gradlew :rhino-evaluator:bench ./gradlew :graaljs-evaluator:bench -PbenchIterations=5000 +./gradlew :graaljs-evaluator:bench -PbenchIterations=5000 -PbenchReuse=true ``` Each iteration evaluates the 147 expressions from `com.rapatao.projects.ruleset.engine.cases.TestData` against the same @@ -227,14 +250,21 @@ Numbers below come from that benchmark, 2000 iterations (294,000 evaluations per Amazon Corretto 21.0.11. Treat them as relative magnitudes, not absolute figures: the harness is a simple timing loop, not JMH, and the GraalJS run is interpreter-only because Corretto is not a GraalVM JDK. -| engine | ops/s | avg per iteration | p50 | p99 | relative cost | -|---------|---------|-------------------|-----------|-----------|---------------| -| Kotlin | 566,752 | 259us | 210us | 638us | 1x | -| Rhino | 47,827 | 3.07ms | 2.99ms | 4.60ms | ~12x | -| GraalJS | 9,688 | 15.17ms | 14.97ms | 17.15ms | ~58x | +| engine | ops/s | avg per iteration | p50 | p99 | relative cost | +|----------------------|----------|-------------------|-----------|-----------|---------------| +| Kotlin | 566,752 | 259us | 210us | 638us | 1x | +| GraalJS (reused ctx) | ~240,000 | ~590us | ~500us | ~2.0ms | ~2x | +| Rhino | 47,827 | 3.07ms | 2.99ms | 4.60ms | ~12x | +| GraalJS | 9,391 | 15.65ms | 15.57ms | 17.36ms | ~60x | + +Most engines are stable under load, with the p99 within 1.2x to 3x of the median. The reused-context GraalJS row is the +exception: it varied between 185,000 and 294,000 ops/s across runs here, so it is quoted as an approximation. Once the +context cost is gone, an iteration is short enough that the timing loop measures JIT and GC noise as much as the +engine. -All three engines are stable under load: the p99 stays within 1.2x to 3x of the median, so there are no pathological -outliers, only different constant costs. +`GraalJS (reused ctx)` is the same engine with `reuseContextPerThread = true`. Closing the per-call context and +injecting the input into a per-evaluation object costs the default mode about 4% (9,750 to 9,391 ops/s here), and buys +deterministic context release plus binding isolation that holds under reuse. ### Where the time goes @@ -255,12 +285,14 @@ Reading of the table: running one small script per operator, which is why deep rule trees cost more than the numbers for a single rule suggest * **GraalJS**: context creation dominates almost entirely. On this setup the rule itself is nearly free compared to the - polyglot context it runs in + polyglot context it runs in, which is what `reuseContextPerThread = true` removes ### Practical guidance * Reuse the evaluator instance. Operators are resolved once in the constructor, and for GraalJS the shared `Engine` caches parsed sources across contexts, so a new evaluator per request throws that away +* On GraalJS, set `reuseContextPerThread = true` unless rules are untrusted or deliberately write globals. It is the + single largest win available on that engine * Pass the narrowest input object that satisfies the rule. All three engines materialise the whole input per call * Prefer `Map` inputs over arbitrary objects when the data is already in that shape: the object path goes through Kotlin reflection @@ -271,10 +303,11 @@ Reading of the table: * On Rhino, keep the default `interpretedMode = true`. Compiled mode measures about 10x slower here, because each operator compiles a new script that is thrown away -Both JS engines rebuild their context per `evaluate` call, and that dominates their cost. Overriding `call` to reuse a -context makes the benchmark suite about 48x faster on GraalJS (16.6ms to 0.35ms) and about 11x faster on Rhino (3.5ms -to 0.30ms), at the price of thread safety and binding isolation between evaluations. See -[docs/tasks](docs/tasks) for the analysis and the trade-offs. +Both JS engines rebuild their context per `evaluate` call, and that dominates their cost. On GraalJS this is now an +opt-in: `reuseContextPerThread = true` keeps one context per thread and runs the suite roughly 25x faster (15.6ms to +about 0.6ms) while staying thread-safe and isolating input bindings. Rhino still rebuilds its scope per call; reusing it +there is about 11x faster (3.5ms to 0.30ms) but is not implemented. See [docs/tasks](docs/tasks) for the analysis and +the trade-offs. ## Get started diff --git a/graaljs-evaluator/build.gradle b/graaljs-evaluator/build.gradle index a4f4f65..d7b26da 100644 --- a/graaljs-evaluator/build.gradle +++ b/graaljs-evaluator/build.gradle @@ -8,8 +8,11 @@ dependencies { tasks.register("bench", JavaExec) { group = "verification" - description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000" + description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000, reused context: -PbenchReuse=true" classpath = sourceSets.test.runtimeClasspath mainClass = "com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSBenchmarkKt" - args = [providers.gradleProperty("benchIterations").getOrElse("1000")] + args = [ + providers.gradleProperty("benchIterations").getOrElse("1000"), + providers.gradleProperty("benchReuse").getOrElse("false"), + ] } diff --git a/graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSEvaluator.kt b/graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSEvaluator.kt index 570f8a0..ed0fc90 100644 --- a/graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSEvaluator.kt +++ b/graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSEvaluator.kt @@ -27,6 +27,9 @@ import org.graalvm.polyglot.Value * * @property engine The GraalVM Polyglot Engine instance used by this evaluator engine. * @property contextBuilder A builder instance used to create a JavaScript Context with custom options and settings. + * @property reuseContextPerThread When `false` (the default), every evaluation builds a new Context and closes it + * afterwards. When `true`, each thread keeps one Context and reuses it across evaluations, which removes the context + * construction cost from every call. See the README for the trade-offs of the reused mode. * * @see org.graalvm.polyglot.Context.Builder * @see org.graalvm.polyglot.Engine @@ -41,6 +44,7 @@ open class GraalJSEvaluator( .allowHostAccess(HostAccess.ALL).allowHostClassLookup { true } .option("js.nashorn-compat", "true").allowExperimentalOptions(true), operators: List = listOf(), + private val reuseContextPerThread: Boolean = false, ) : Evaluator( operators = listOf( Equals(), @@ -58,17 +62,30 @@ open class GraalJSEvaluator( ) + operators, ) { + private val threadContext = ThreadLocal.withInitial { contextBuilder.build() } + override fun call(inputData: Any, block: EvalContext.() -> T): T = - createContext().let { - parseParameters( - it.getBindings("js"), - inputData, - ) - block(GraalJSContext(this, it)) + if (reuseContextPerThread) { + evaluateWith(threadContext.get(), inputData, block) + } else { + contextBuilder.build().use { evaluateWith(it, inputData, block) } } - private fun createContext(): Context { - return contextBuilder.build() + private fun evaluateWith(context: Context, inputData: Any, block: EvalContext.() -> T): T { + val scope = context.eval("js", "({})") + + parseParameters(scope, inputData) + + val bindings = context.getBindings("js") + // an operator may evaluate another expression through EvalContext.engine(), which reuses this context + val outerScope = bindings.getMember(INPUT_SCOPE) + bindings.putMember(INPUT_SCOPE, scope) + + return try { + block(GraalJSContext(this, context)) + } finally { + outerScope?.let { bindings.putMember(INPUT_SCOPE, it) } + } } /** @@ -81,6 +98,9 @@ open class GraalJSEvaluator( /** * Parses parameters and injects them into the given scope based on the input data. * + * The scope is a fresh JavaScript object created for the evaluation, and it is replaced on every call, so the + * injected members are never visible to another evaluation. + * * @param bindings the values object where the parameters will be injected * @param inputData the input data containing the parameters */ @@ -90,4 +110,11 @@ open class GraalJSEvaluator( else -> TypedInjector.inject(bindings, inputData) } } + + internal companion object { + /** + * Name of the global member holding the input data of the current evaluation. + */ + const val INPUT_SCOPE = "__ruleset_input__" + } } diff --git a/graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/operator/extensions.kt b/graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/operator/extensions.kt index bcb7e29..cd0976d 100644 --- a/graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/operator/extensions.kt +++ b/graaljs-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/operator/extensions.kt @@ -2,6 +2,7 @@ package com.rapatao.projects.ruleset.engine.evaluator.graaljs.operator import com.rapatao.projects.ruleset.engine.context.EvalContext import com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSContext +import com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSEvaluator.Companion.INPUT_SCOPE import org.graalvm.polyglot.Source internal fun EvalContext.evaluate(content: String): Boolean { @@ -10,7 +11,7 @@ internal fun EvalContext.evaluate(content: String): Boolean { return graalJSContext.context().eval( Source.newBuilder( "js", - "true == ($content)", + "(function() { with ($INPUT_SCOPE) { return true == ($content) } })()", content, ).buildLiteral() ).asBoolean() diff --git a/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt b/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt index 224047d..9981975 100644 --- a/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt +++ b/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt @@ -3,5 +3,7 @@ package com.rapatao.projects.ruleset.engine.evaluator.graaljs import com.rapatao.projects.ruleset.engine.BaseEngineBenchmark fun main(args: Array) { - BaseEngineBenchmark(GraalJSEvaluator()).main(args) + BaseEngineBenchmark( + GraalJSEvaluator(reuseContextPerThread = args.getOrNull(1).toBoolean()) + ).main(args) } diff --git a/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSReusedContextEvaluatorTest.kt b/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSReusedContextEvaluatorTest.kt new file mode 100644 index 0000000..408ab63 --- /dev/null +++ b/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSReusedContextEvaluatorTest.kt @@ -0,0 +1,93 @@ +package com.rapatao.projects.ruleset.engine.evaluator.graaljs + +import com.rapatao.projects.ruleset.engine.BaseEvaluatorTest +import com.rapatao.projects.ruleset.engine.cases.TestData +import com.rapatao.projects.ruleset.engine.context.EvalContext +import com.rapatao.projects.ruleset.engine.helper.ExposeEngineTestOperator +import com.rapatao.projects.ruleset.engine.types.Expression +import com.rapatao.projects.ruleset.engine.types.builder.MatcherBuilder.allMatch +import com.rapatao.projects.ruleset.engine.types.builder.extensions.equalsTo +import com.rapatao.projects.ruleset.engine.types.operators.Operator +import org.hamcrest.MatcherAssert.assertThat +import org.hamcrest.Matchers.equalTo +import org.junit.jupiter.api.DisplayName +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows +import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit + +class GraalJSReusedContextEvaluatorTest : BaseEvaluatorTest(evaluator) { + + companion object { + private const val THREADS = 8 + + private val evaluator = GraalJSEvaluator( + operators = listOf( + ExposeEngineTestOperator(), + ), + reuseContextPerThread = true, + ) + } + + @Test + @DisplayName("should not expose bindings from a previous evaluation") + fun assertBindingsAreNotSharedBetweenEvaluations() { + val rule = "b" equalsTo 2 + + assertThat( + evaluator.evaluate(rule, mapOf("a" to 1, "b" to 2)), + equalTo(true), + ) + + assertThrows { + evaluator.evaluate(rule, mapOf("a" to 1)) + } + } + + @Test + @DisplayName("should restore the input of the outer evaluation when an operator evaluates another expression") + fun assertNestedEvaluationDoesNotReplaceTheInput() { + val nesting = NestedEvaluationOperator() + val evaluator = GraalJSEvaluator(operators = listOf(nesting), reuseContextPerThread = true) + + val rule = allMatch( + Expression(left = "outer", operator = nesting.name(), right = "1"), + "outer" equalsTo 1, + ) + + assertThat(evaluator.evaluate(rule, mapOf("outer" to 1)), equalTo(true)) + } + + /** + * Evaluates an unrelated expression against a different input, the way a custom operator may do through + * [com.rapatao.projects.ruleset.engine.context.EvalContext.engine]. + */ + private class NestedEvaluationOperator : Operator { + override fun process(context: EvalContext, left: Any?, right: Any?): Boolean = + context.engine().evaluate("inner" equalsTo 2, mapOf("inner" to 2)) + + override fun name(): String = "nested_evaluation" + } + + @Test + @DisplayName("should evaluate concurrently without sharing the context between threads") + fun assertConcurrentEvaluation() { + val cases = TestData.cases().map { it.get() } + .map { it[0] as Expression to it[1] as Boolean } + + val pool = Executors.newFixedThreadPool(THREADS) + try { + val results = (1..THREADS).map { + pool.submit { + cases.forEach { (expression, expected) -> + assertThat(evaluator.evaluate(expression, TestData.inputData), equalTo(expected)) + } + } + } + + results.forEach { it.get(1, TimeUnit.MINUTES) } + } finally { + pool.shutdownNow() + } + } +}