diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 6effdf9..7a325ce 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -12,11 +12,20 @@ Every evaluator module ships a `bench` task that replays the full test rule set ./gradlew :rhino-evaluator:bench ./gradlew :graaljs-evaluator:bench -PbenchIterations=5000 ./gradlew :graaljs-evaluator:bench -PbenchIterations=5000 -PbenchReuse=true +./gradlew :kotlin-evaluator:bench -PbenchWide=200 ``` -Each iteration evaluates the 147 expressions from `com.rapatao.projects.ruleset.engine.cases.TestData` against the same +Each iteration evaluates the 173 expressions from `com.rapatao.projects.ruleset.engine.cases.TestData` against the same input object, after 100 warmup iterations. Results are printed and written to `bench_.txt`. +Each run reports throughput, the latency distribution of an iteration (avg, stddev, min, max, p50 to p99), the bytes +allocated per evaluation, and the garbage collections that ran during the measured loop. Allocation is read from the +JVM's per-thread counter on the benchmark thread, and reports `n/a` on a JVM that does not expose it. + +`-PbenchWide=N` runs the same rules against the same `item`, under a root carrying `N` extra scalar fields and an `N` +element list. Nothing the rules read changes, only how much input surrounds it, which separates a per-call cost that +scales with the input from one that scales with the rule. + Two things to set up before trusting a run: * Run at full power. On a laptop in a power saving mode the whole suite lands 25 to 30% low, uniformly across engines. @@ -25,30 +34,78 @@ Two things to set up before trusting a run: ## Results -2000 iterations (294,000 evaluations per engine), Apple M3 Pro, Amazon Corretto 21.0.11. These are relative -magnitudes, not absolute figures: the harness is a simple timing loop, not JMH, and the GraalJS run is interpreter-only -because Corretto is not a GraalVM JDK. +2000 iterations of the 173 expression suite, 346,000 evaluations per engine, Apple M3 Pro, Amazon Corretto 21.0.11, +three runs per configuration in one session at full power, medians below. These are relative magnitudes, not absolute figures: the harness is a +simple timing loop, not JMH, and the GraalJS run is interpreter-only because Corretto is not a GraalVM JDK. + +| engine | ops/s | avg per iteration | stddev | p50 | p99 | relative cost | +|----------------------|-----------|-------------------|--------|---------|---------|---------------| +| Kotlin | 1,506,886 | 115us | 93us | 83us | 425us | 1x | +| Rhino | 288,132 | 600us | 240us | 490us | 1.41ms | ~5.2x | +| GraalJS (reused ctx) | 240,662 | 719us | 439us | 580us | 2.58ms | ~6.3x | +| GraalJS | 8,959 | 19.31ms | 640us | 19.09ms | 21.08ms | ~168x | + +Run-to-run spread differs by engine, and sets how large a difference has to be before it means anything: + +| engine | observed across runs | p99 vs p50 | +|----------------------|------------------------|------------| +| Kotlin | 1,454,000 to 1,621,000 | ~5.1x | +| Rhino | 287,000 to 320,000 | ~2.9x | +| GraalJS (reused ctx) | 237,000 to 257,000 | ~4.4x | +| GraalJS | 8,771 to 9,176 | ~1.1x | + +The three fast configurations move by about 10% across runs and their p99 is 3 to 5 times their p50. The tail is GC +and JIT, not the engine, so read a difference smaller than that as noise. Default GraalJS is the exception on both +counts: an iteration is so dominated by context creation that nothing else is visible in it. + +The Kotlin engine on a wide input is the least stable measurement here, spanning 1,409,000 to 2,195,000 across its +three runs. Its iteration is short enough that the loop measures the JVM more than the engine. + +### Allocation + +Bytes allocated per `evaluate`, counted on the benchmark thread by the JVM's own allocation counter, and the garbage +collections that ran during the measured loop: + +| engine | alloc per evaluation | vs Kotlin | gc during the run | +|----------------------|----------------------|-----------|-------------------| +| Kotlin | 893 B | 1x | 2, 3ms | +| GraalJS (reused ctx) | 5,850 B | ~6.6x | 15, 19ms | +| Rhino | 11,741 B | ~13.1x | 17, 22ms | +| GraalJS | 128,763 B | ~144x | 425, 186ms | + +This is the steadiest number the harness produces: it varies by under 0.5% across runs, where throughput varies by 5%. + +The order is not the throughput order. Reused-context GraalJS allocates less per evaluation than Rhino and is still +slower, so Rhino's cost is not allocation-bound: it compiles a fresh script per operator invocation, and compilation +is work rather than garbage. Default GraalJS allocates a whole polyglot `Context` per call, which is the 144x. + +### Input width + +The same run with `-PbenchWide=200`: identical rules reading identical fields, under a root carrying 200 extra scalar +fields and a 200 element list. + +| engine | ops/s default | ops/s wide(200) | wide is | alloc default | alloc wide(200) | wide allocates | +|----------------------|---------------|-----------------|---------------|---------------|-----------------|----------------| +| Kotlin | 1,506,886 | 1,768,634 | unchanged | 893 B | 872 B | unchanged | +| Rhino | 288,132 | 138,658 | ~2.1x slower | 11,741 B | 23,788 B | ~2.0x more | +| GraalJS (reused ctx) | 240,662 | 16,313 | ~14.8x slower | 5,850 B | 85,596 B | ~14.6x more | +| GraalJS | 8,959 | 5,875 | ~1.5x slower | 128,763 B | 208,161 B | ~1.6x more | -| engine | ops/s | avg per iteration | p50 | p99 | relative cost | -|----------------------|----------|-------------------|-----------|-----------|---------------| -| Kotlin | 782,637 | 188us | 166us | 331us | 1x | -| Rhino | 352,564 | 417us | 340us | 1.18ms | ~2.2x | -| GraalJS (reused ctx) | ~240,000 | ~590us | ~500us | ~2.0ms | ~3.3x | -| GraalJS | 9,391 | 15.65ms | 15.57ms | 17.36ms | ~83x | +Each factor compares the two columns to its left, within the same row. A row varies the input only: the engine and its +configuration are held constant across it, so `reuseContextPerThread` is on in both columns of the reused row and off +in both columns of the row below it. The Results table above prices the reuse setting. -Each row is one representative run. Run-to-run spread differs sharply by engine, and sets how large a difference has to -be before it means anything: +For both JS engines the allocation factor tracks the throughput factor, which identifies the cost: they inject every +top-level entry of the input into the scope on every `evaluate`, and pay for it whether a rule reads it or not. +Neither pays for *depth*, since a nested object is handed over whole and JS walks into it lazily. Reused-context +GraalJS runs 157 collections over the wide input against 15 over the default one, and Rhino 34 against 17. -| engine | observed across runs | p99 vs p50 | -|----------------------|----------------------|------------| -| Kotlin | 778,000 to 791,000 | ~2x | -| Rhino | 286,000 to 394,000 | ~3.5x | -| GraalJS (reused ctx) | 185,000 to 294,000 | ~4x | -| GraalJS | stable within a few % | ~1.1x | +Default GraalJS shows the smallest factor because context creation, at ~19ms per iteration, dominates the injection. +In the reused-context mode the injection is the dominant remaining cost. -The two fastest configurations are the least stable. Once the per-evaluation context cost is gone, an iteration is -short enough that the loop measures JIT and GC noise as much as the engine. Default GraalJS is the opposite: an -iteration is so dominated by context creation that nothing else is visible. +The Kotlin engine resolves the paths a rule names and never visits the rest, so its cost tracks the rule. Its two +columns overlap across runs, with the wide one reading slightly faster: that input roots at a `Map` while the default +one roots at a data class, so one hash lookup replaces one reflective property read. `GraalJS (reused ctx)` is the same engine with `reuseContextPerThread = true`. Closing the per-call context and injecting the input into a per-evaluation object costs the default mode about 4% (9,750 to 9,391 ops/s), and buys @@ -68,19 +125,22 @@ The `Evaluator` contract sets up a fresh evaluation context on every `evaluate` These rows come from one tight loop over a single rule, after 50,000 warmup calls, so they isolate the steady-state cost. They are not comparable to the suite numbers above, which include cold and JIT-transient iterations. That loop is -not part of this repository and the `bench` tasks do not reproduce it. The Kotlin row predates the current operand -parsing, which cut per-operand work and not context setup, so its real setup share is above the ~82% shown. +not part of this repository and the `bench` tasks do not reproduce it. The Kotlin row predates both the current operand +parsing and the removal of input flattening. Reading of the table: -* **Kotlin**: the fixed cost is flattening the input graph, and it is nearly the whole cost. It scales with the size of - the input object, not with the rule, so a wide input evaluated against a two-field rule pays for every other field +* **Kotlin**: the 0.88us of setup was flattening the whole input graph into a map of every path, which scaled with the + size of the input rather than the rule. That step is gone. Setup is now a constructor call, operand paths are + resolved on demand, and the cost tracks the rule: widening the input to 200 extra fields and a 200 element list cost + 11.7x under flattening and costs nothing measurable now * **Rhino**: setup is entering a `Context`, creating a child scope and injecting the input, because the standard objects are shared. Before that change the same two columns read 20.5us and 13.0us, a ~63% share. What is left is compiling and running one small script per operator, which is why deep rule trees cost more than the numbers for a - single rule suggest + single rule suggest. The injection half of that setup is what the input width table above prices * **GraalJS**: context creation dominates almost entirely. On this setup the rule itself is nearly free compared to the - polyglot context it runs in, which is what `reuseContextPerThread = true` removes + polyglot context it runs in, which is what `reuseContextPerThread = true` removes. What remains once it is removed is + injecting the input, the cost that grows with the input ## Practical guidance @@ -88,9 +148,14 @@ Reading of the table: caches parsed sources across contexts, so a new evaluator per request throws that away * On GraalJS, set `reuseContextPerThread = true` unless rules are untrusted or deliberately write globals. It is the single largest win available on that engine -* Pass the narrowest input object that satisfies the rule. All three engines materialise the whole input per call +* On the JS engines, pass the narrowest input object that satisfies the rule: both inject every top-level entry per + call, worth 2.1x on Rhino and 14.8x on reused-context GraalJS for 200 extra fields. Nesting the parts a rule does not + read one level deeper avoids it. The Kotlin engine reads only the paths a rule names and is flat here +* Watch allocation, not just throughput, if the service is latency-sensitive: an evaluation costs 893 B on the Kotlin + engine and 128 KB on default GraalJS, and that is what fills the nursery and sets the GC rate under load * Prefer `Map` inputs over arbitrary objects when the data is already in that shape: the object path goes through - Kotlin reflection + Kotlin reflection. On the Kotlin engine this is now a small difference, since the properties of each class are + reflected once and cached * Order `anyMatch` cheaply-first and `allMatch` most-selective-first. Evaluation short-circuits, and with the JS engines every skipped expression is a script that is never compiled * On GraalJS, run on a GraalVM JDK (or put the Graal compiler on the runtime classpath) before drawing conclusions from diff --git a/JSON.md b/JSON.md index 6d15ccb..9c0d2b3 100644 --- a/JSON.md +++ b/JSON.md @@ -2014,3 +2014,211 @@ To see more details, check its source: [here](src/test/kotlin/com/rapatao/projec } ``` +```json +{ + "left" : [ "\"item1\"", "\"item2\"" ], + "operator" : "contains", + "right" : "\"item1\"" +} +``` + +```json +{ + "left" : [ "\"item1\"", "\"item2\"" ], + "operator" : "not_contains", + "right" : "\"item3\"" +} +``` + +```json +{ + "left" : [ "item.name", "\"something else\"" ], + "operator" : "contains", + "right" : "\"product name\"" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "equals", + "right" : "1.5" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "equals", + "right" : "1.9" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "equals", + "right" : "1.0" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "equals", + "right" : "1" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "not_equals", + "right" : "1.9" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "greater_than", + "right" : "1.4" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "greater_than", + "right" : "1.6" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "less_than", + "right" : "1.6" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "less_than", + "right" : "1.4" +} +``` + +```json +{ + "left" : "item.weight", + "operator" : "greater_or_equal_than", + "right" : "1.5" +} +``` + +```json +{ + "left" : "item.scaled", + "operator" : "equals", + "right" : 10 +} +``` + +```json +{ + "left" : "item.scaled", + "operator" : "equals", + "right" : "10.0" +} +``` + +```json +{ + "left" : "item.scaled", + "operator" : "equals", + "right" : "10.00" +} +``` + +```json +{ + "left" : "item.scaled", + "operator" : "not_equals", + "right" : 10 +} +``` + +```json +{ + "left" : "item.scaled", + "operator" : "equals", + "right" : "10.01" +} +``` + +```json +{ + "left" : "item.price", + "operator" : "equals", + "right" : "10.00" +} +``` + +```json +{ + "left" : [ 1, 2 ], + "operator" : "contains", + "right" : 1 +} +``` + +```json +{ + "left" : [ 1, 2 ], + "operator" : "contains", + "right" : 3 +} +``` + +```json +{ + "left" : [ 1, 2 ], + "operator" : "not_contains", + "right" : 3 +} +``` + +```json +{ + "left" : "item.quantities", + "operator" : "contains", + "right" : "1" +} +``` + +```json +{ + "left" : "item.quantities", + "operator" : "contains", + "right" : "3" +} +``` + +```json +{ + "left" : "item.quantities", + "operator" : "not_contains", + "right" : "3" +} +``` + +```json +{ + "left" : "item.quantities", + "operator" : "contains", + "right" : "1.0" +} +``` + diff --git a/README.md b/README.md index 6dd849b..dc7944c 100644 --- a/README.md +++ b/README.md @@ -10,14 +10,14 @@ Below are the available engines that can be used to evaluate expressions. All of `com.rapatao.projects.ruleset.engine.Evaluator` contract and accept the same `Expression` tree, so switching engine is a matter of changing the dependency and the instantiation line. -A quick comparison, measured with the benchmark shipped in this repository (details in -[BENCHMARKS.md](BENCHMARKS.md)): +| engine | operands | best fit | +|---------|------------------|----------------------------------------------------| +| Kotlin | field paths only | high volume, plain comparison rules | +| Rhino | JavaScript | rules that need scripting, high volume | +| GraalJS | JavaScript | modern ECMAScript, GraalVM deployments, low volume | -| engine | operands | throughput (ops/s) | relative | best fit | -|-----------|---------------------|--------------------|----------|------------------------------------------------------| -| Kotlin | field paths only | ~783,000 | 1x | high volume, plain comparison rules | -| Rhino | JavaScript | ~350,000 | ~2.2x | rules that need scripting, high volume | -| GraalJS | JavaScript | ~9,700 | ~81x | modern ECMAScript, GraalVM deployments, low volume | +Throughput, run-to-run spread, where the time goes inside each engine and tuning guidance are in +[BENCHMARKS.md](BENCHMARKS.md). ### Kotlin engine implementation @@ -39,16 +39,36 @@ val evaluator = com.rapatao.projects.ruleset.engine.evaluator.kotlin.KotlinEvalu #### How it works -On each `evaluate` call the input data is flattened into a `Map` keyed by dotted field paths -(`item.price`, `item.tags[0]`, ...). Maps are walked by key, arbitrary objects by Kotlin reflection -(`memberProperties`). Operands are then resolved against that map, with numbers normalised to `BigDecimal` so that an -`Int` operand and a `BigDecimal` field compare as expected. Operators are plain Kotlin functions (`==`, `>`, -`String.contains`, `Collection.contains`, ...). +Operands that are field paths (`item.price`, `item.tags[0]`, ...) are resolved against the input on demand, one path +at a time: maps are read by key, collections and arrays by index, and arbitrary objects by Kotlin reflection +(`memberProperties`, reflected once per class and cached). Only the nodes a path names are visited, so the cost tracks +the rule rather than the input. Operators are plain Kotlin functions (`==`, `>`, `String.contains`, ...). + +Numbers are normalised to `BigDecimal`, elements of a list included, so an `Int` operand and a `BigDecimal` field +compare as expected and `listOf(1, 2) expContains 1` matches. They compare by value rather than by representation, so +`10` and `10.00` are the same number and a fraction is never truncated. + +A path that does not exist throws, which `onFailure` turns into a rule result. A path exists when every step of it +does: a map holds the key, an object has the property, an index is in range. Nothing exists below a `null`, a string +or a number, so `item.name.length` throws rather than resolving through reflection. Operands are literals or field paths only. A quoted operand (`"\"value\""`) is a string literal, an unquoted one is first tried as a number or boolean literal and then as a field path. There is no expression language, so `item.price * 2` is not supported: model it as a field on the input, or as a custom operator. +A list written in an expression holds operands, and each element is resolved by those same rules: + +```kotlin +// the literal "test", and whatever item.name holds +listOf("\"test\"", "item.name") expContains "\"product name\"" + +// looks for fields named test and brand-new, and throws when the input has neither +listOf("test", "brand-new") expContains "\"test\"" +``` + +An unquoted string element is a field path, exactly as an unquoted scalar operand is. Quote it to compare against the +text itself. + #### Best for * Rule sets made of comparisons over data you already have in memory @@ -59,9 +79,9 @@ first tried as a number or boolean literal and then as a field path. There is no #### Trade-offs * No expressions in operands -* Flattening walks the whole input graph on every evaluation, not just the fields the rule touches. Cost grows with the - size of the input object, not with the size of the rule, so prefer passing a narrow input object over a wide one -* Reflection is used for non-map inputs; passing a `Map` avoids it +* Reflection is used for non-map inputs; passing a `Map` avoids it. The properties of each class are reflected once + and cached, so the cost falls on the first evaluation against a given input type +* The cache is keyed by `Class` and never evicted, which pins classloaders in a container that redeploys #### Gradle @@ -115,9 +135,8 @@ val evaluator = RhinoEvaluator( `interpretedMode` defaults to `true`, and that default is the fast one for this engine. Because a fresh snippet is compiled per operator invocation and never cached, bytecode generation cost is paid on every evaluation and never -amortised: measured on the benchmark rule set, `interpretedMode = false` is about 100x slower (37.7ms vs 0.33ms per -iteration). The gap widened with the shared standard scope, which sped up the interpreted path without touching the -bytecode generation cost. Leave it as is unless you have measured your own workload. +amortised, which makes `interpretedMode = false` far slower on the benchmark rule set +(see [BENCHMARKS.md](BENCHMARKS.md)). Leave it as is unless you have measured your own workload. #### Best for @@ -128,7 +147,7 @@ bytecode generation cost. Leave it as is unless you have measured your own workl #### Trade-offs -* Around 1.6x the cost of the Kotlin engine +* Slower than the Kotlin engine, faster than GraalJS (see [BENCHMARKS.md](BENCHMARKS.md)) * JavaScript language support is behind GraalJS; set `languageVersion` explicitly if you need ES6 syntax * The standard objects are sealed, so a rule cannot monkey-patch a builtin (`Array.prototype.foo = ...` throws) * The whole input is injected per `evaluate` call, even when the rule reads a single field @@ -171,7 +190,8 @@ that object, so both operands are arbitrary JavaScript. #### Reusing the context Building a `Context` is what the engine spends almost all of its time on. `reuseContextPerThread` keeps one context per -thread instead, which is roughly 25x faster on the benchmark below: +thread instead, which is the single largest change available on this engine +(see [BENCHMARKS.md](BENCHMARKS.md)): ```kotlin val evaluator = GraalJSEvaluator(reuseContextPerThread = true) @@ -211,9 +231,9 @@ pass a restricted `Context.Builder`. #### Trade-offs -* The slowest of the three engines in the measurement below when left on the default context handling, and most of that - cost is context creation rather than the rule itself. `reuseContextPerThread = true` removes it, at the cost of the - global-state isolation described above +* The slowest of the three engines when left on the default context handling, and most of that cost is context creation + rather than the rule itself. `reuseContextPerThread = true` removes it, at the cost of the global-state isolation + described above * On a stock (non-GraalVM) JDK, Truffle runs in interpreter-only mode. The evaluator sets `engine.WarnInterpreterOnly=false`, so the usual warning is not printed. Running on a GraalVM JDK, or adding the Graal compiler to the runtime classpath, is what unlocks its performance @@ -236,12 +256,6 @@ implementation "com.rapatao.ruleset:graaljs-evaluator:$rulesetVersion" ``` -## Performance - -The Kotlin engine runs the benchmark suite at about 783,000 ops/s, Rhino at about 350,000, and GraalJS at about 9,700 -on a stock JDK. Full results, how to reproduce them, where the time goes inside each engine, and tuning guidance are in -[BENCHMARKS.md](BENCHMARKS.md). - ## Get started After adding the desired engine as the application dependency, copy and past the following code, replacing diff --git a/graaljs-evaluator/build.gradle b/graaljs-evaluator/build.gradle index d7b26da..3870e22 100644 --- a/graaljs-evaluator/build.gradle +++ b/graaljs-evaluator/build.gradle @@ -8,11 +8,13 @@ dependencies { tasks.register("bench", JavaExec) { group = "verification" - description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000, reused context: -PbenchReuse=true" + description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000, reused context: -PbenchReuse=true, " + + "wide input: -PbenchWide=200" classpath = sourceSets.test.runtimeClasspath mainClass = "com.rapatao.projects.ruleset.engine.evaluator.graaljs.GraalJSBenchmarkKt" args = [ providers.gradleProperty("benchIterations").getOrElse("1000"), + providers.gradleProperty("benchWide").getOrElse("0"), providers.gradleProperty("benchReuse").getOrElse("false"), ] } diff --git a/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt b/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt index 9981975..4a8ce15 100644 --- a/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt +++ b/graaljs-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/graaljs/GraalJSBenchmark.kt @@ -4,6 +4,7 @@ import com.rapatao.projects.ruleset.engine.BaseEngineBenchmark fun main(args: Array) { BaseEngineBenchmark( - GraalJSEvaluator(reuseContextPerThread = args.getOrNull(1).toBoolean()) + evaluator = GraalJSEvaluator(reuseContextPerThread = args.getOrNull(2).toBoolean()), + wide = args.getOrNull(1)?.toIntOrNull() ?: 0, ).main(args) } diff --git a/kotlin-evaluator/build.gradle b/kotlin-evaluator/build.gradle index b924142..0285940 100644 --- a/kotlin-evaluator/build.gradle +++ b/kotlin-evaluator/build.gradle @@ -6,8 +6,12 @@ dependencies { tasks.register("bench", JavaExec) { group = "verification" - description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000" + description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000, wide input: -PbenchWide=200" classpath = sourceSets.test.runtimeClasspath mainClass = "com.rapatao.projects.ruleset.engine.evaluator.kotlin.KotlinBenchmarkKt" - args = [providers.gradleProperty("benchIterations").getOrElse("1000")] + args = [ + providers.gradleProperty("benchIterations").getOrElse("1000"), + providers.gradleProperty("benchWide").getOrElse("0"), + providers.gradleProperty("benchReuse").getOrElse("false"), + ] } diff --git a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/InputPath.kt b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/InputPath.kt new file mode 100644 index 0000000..888055a --- /dev/null +++ b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/InputPath.kt @@ -0,0 +1,111 @@ +package com.rapatao.projects.ruleset.engine.evaluator.kotlin + +import java.util.concurrent.ConcurrentHashMap +import kotlin.reflect.KProperty1 +import kotlin.reflect.full.memberProperties + +/** + * Resolves an operand path such as `item.tags[0]` against the input object, walking only the nodes the path names. + * + * This replaced a flat map of every path in the input, built once per evaluation, so the rules about what exists are + * inherited from it: a map entry exists when the map holds the key or a key whose `toString` matches it, an object + * property exists when Kotlin reflection reports it, an index exists when the node is a collection or an array and + * the position is in range, and nothing exists below a value, a null, a collection or an array. + */ +internal object InputPath { + + /** Returned when a step of the path does not exist, which is distinct from a step resolving to `null`. */ + val ABSENT = Any() + + // ponytail: unbounded and keyed by Class, so it pins classloaders in a redeploy container. + // Swap for a weak-keyed cache if that ever matters. + private val PROPERTIES = ConcurrentHashMap, Map>>() + + @Suppress("ReturnCount") + fun resolve(root: Any, path: String): Any? { + if (path.isEmpty()) { + return if (root.isValue() || root is Collection<*> || root is Array<*>) root else ABSENT + } + + var current: Any? = root + var index = 0 + + while (index < path.length && current !== ABSENT) { + when (path[index]) { + '.' -> index++ + + '[' -> { + val close = path.indexOf(']', index + 1) + if (close < 0) return ABSENT + + current = current.atIndex(path.substring(index + 1, close)) + index = close + 1 + } + + else -> { + val end = path.nameEnd(index) + + current = current.member(path.substring(index, end)) + index = end + } + } + } + + return current + } + + private fun String.nameEnd(from: Int): Int { + var end = from + + while (end < this.length && this[end] != '.' && this[end] != '[') { + end++ + } + + return end + } + + private fun Any?.member(name: String): Any? = when { + this == null || this.isValue() -> ABSENT + this is Map<*, *> -> lookup(name) + this is Collection<*> || this is Array<*> -> ABSENT + else -> { + val property = propertiesOf(this.javaClass)[name] + if (property == null) ABSENT else property.get(this) + } + } + + private fun Map<*, *>.lookup(name: String): Any? { + if (this.containsKey(name)) { + return this[name] + } + + // The flat map keyed entries by key.toString(), so keys that are not strings stay reachable. + val entry = this.entries.firstOrNull { it.key.toString() == name } + + return if (entry == null) ABSENT else entry.value + } + + private fun Any?.atIndex(text: String): Any? { + val position = text.toIntOrNull() ?: return ABSENT + + return when { + // Collection, not List: the flat map indexed by iteration order. + this is Collection<*> -> if (position >= 0 && position < this.size) this.elementAt(position) else ABSENT + this is Array<*> -> if (position in this.indices) this[position] else ABSENT + else -> ABSENT + } + } + + private fun Any?.isValue(): Boolean = + this == null || + this.javaClass.isPrimitive || + this is Boolean || + this is String || + this is Number + + @Suppress("UNCHECKED_CAST") + private fun propertiesOf(type: Class<*>): Map> = + PROPERTIES.computeIfAbsent(type) { + (it.kotlin.memberProperties as Collection>).associateBy { property -> property.name } + } +} diff --git a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinContext.kt b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinContext.kt index 87ccb26..902808a 100644 --- a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinContext.kt +++ b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinContext.kt @@ -4,17 +4,20 @@ import com.rapatao.projects.ruleset.engine.Evaluator import com.rapatao.projects.ruleset.engine.context.EvalContext import com.rapatao.projects.ruleset.engine.types.operators.Operator import java.math.BigDecimal +import java.math.BigInteger /** * KotlinContext is a class that implements the EvalContext interface. * It provides the ability to process expressions using Kotlin operations. * + * Operand paths are resolved by [InputPath] on demand, so only the nodes a path names are visited. + * * @param evaluator the evaluator implementation instance - * @param inputData the map containing the input data to be used during expression evaluation + * @param inputData the input data to be used during expression evaluation */ class KotlinContext( private val evaluator: Evaluator, - private val inputData: Map + private val inputData: Any ) : EvalContext { override fun process(left: Any?, operator: Operator, right: Any?): Boolean { @@ -23,46 +26,53 @@ class KotlinContext( override fun engine(): Evaluator = evaluator - private fun Any?.asValue(): Any? { - val result = when { - this !is String -> this - this == "null" -> null - else -> { - val trimmed = this.trim() - if (QUOTED.matches(trimmed)) trimmed.unwrap() else trimmed.rawValue() - } - } + private fun Any?.asValue(): Any? = this.resolved().normalized() + + /** + * Numbers become `BigDecimal` so that an `Int` operand and a `BigDecimal` field compare as equal. + * + * A `BigDecimal` is already one and is kept as it is, fraction included. A `BigInteger` is converted directly, + * because going through `toLong` overflows above `Long.MAX_VALUE`. + * + * A collection is normalized element by element, and only when it holds a number, so that + * `listOf(1, 2) expContains 1` matches. The scan keeps a collection of non-numbers as it is, which is the + * common case and the one on the hot path. + */ + private fun Any?.normalized(): Any? = when { + this is BigDecimal -> this + this is BigInteger -> BigDecimal(this) + this is Number && (this is Double || this is Float) -> BigDecimal(this.toDouble()) + this is Number && this !is Byte -> BigDecimal.valueOf(this.toLong()) + this is Collection<*> && this.any { it is Number || it is Collection<*> } -> this.map { it.normalized() } + else -> this + } - return when { - result is Number && (result is Double || result is Float) -> BigDecimal(result.toDouble()) - result is Number && result !is Byte -> BigDecimal.valueOf(result.toLong()) - else -> result + private fun Any?.resolved(): Any? = when { + // A list written in the expression holds operands, so each element is resolved on its own. + this is Collection<*> -> this.map { it.resolved() } + this !is String -> this + this == "null" -> null + else -> { + val trimmed = this.trim() + if (QUOTED.matches(trimmed)) trimmed.unwrap() else trimmed.rawValue() } } + @Suppress("ReturnCount") private fun String.rawValue(): Any? { val key = this.unwrap() - return listOf( - { - key.toBigIntegerOrNull() - }, - { - key.toBigDecimalOrNull() - }, - { - key.toBooleanStrictOrNull() - }, - { - inputData[key] - }, - { - if (!inputData.containsKey(key)) { - throw NoSuchElementException("$key not found") - } - null - } - ).firstNotNullOfOrNull { it() } + key.toBigIntegerOrNull()?.let { return it } + key.toBigDecimalOrNull()?.let { return it } + key.toBooleanStrictOrNull()?.let { return it } + + val resolved = InputPath.resolve(inputData, key) + + if (resolved === InputPath.ABSENT) { + throw NoSuchElementException("$key not found") + } + + return resolved } private fun String.unwrap() = this.trim() diff --git a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinEvaluator.kt b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinEvaluator.kt index ef0eeb1..1ee8974 100644 --- a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinEvaluator.kt +++ b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinEvaluator.kt @@ -15,7 +15,6 @@ import com.rapatao.projects.ruleset.engine.evaluator.kotlin.operator.NotEquals import com.rapatao.projects.ruleset.engine.evaluator.kotlin.operator.NotStartsWith import com.rapatao.projects.ruleset.engine.evaluator.kotlin.operator.StartsWith import com.rapatao.projects.ruleset.engine.types.operators.Operator -import kotlin.reflect.full.memberProperties /** * An evaluator engine implementation that uses Kotlin to process expressions. @@ -41,68 +40,7 @@ open class KotlinEvaluator( ) + operators, ) { - override fun call(inputData: Any, block: (context: EvalContext) -> T): T { - return block(KotlinContext( - this, - mutableMapOf().apply { - parseKeys("", inputData) - } - )) - } + override fun call(inputData: Any, block: (context: EvalContext) -> T): T = block(KotlinContext(this, inputData)) override fun name(): String = "KotlinEval" - - private fun MutableMap.parseKeys(node: String, input: Any?) { - when { - input.isValue() -> { - this[node] = input - } - - input is Collection<*> -> { - this[node] = input - - input.forEachIndexed { index, value -> - parseKeys("${node}[$index]", value) - } - } - - input is Array<*> -> { - @Suppress("DuplicatedCode") - this[node] = input - - input.forEachIndexed { index, value -> - parseKeys("${node}[$index]", value) - } - } - - input is Map<*, *> -> { - val currNode = node.childNode() - - input.forEach { key, value -> - this["${currNode}${key}"] = value - - parseKeys("${currNode}${key}", value) - } - } - - else -> { - val currNode = node.childNode() - - input?.javaClass?.kotlin?.memberProperties?.forEach { - this["${currNode}${it.name}"] = it.get(input) - - parseKeys("${currNode}${it.name}", it.get(input)) - } - } - } - } - - private fun String.childNode() = if (this.isNotBlank()) "${this}." else this - - private fun Any?.isValue(): Boolean = - this == null || - this.javaClass.isPrimitive || - this is Boolean || - this is String || - this is Number } diff --git a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/Contains.kt b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/Contains.kt index c23ec5c..2a6a81f 100644 --- a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/Contains.kt +++ b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/Contains.kt @@ -10,8 +10,8 @@ internal class Contains : ContainsOperator() { private fun Any?.checkContains(value: Any?): Boolean { return when { this is String && value is String -> this.contains(value) - this is Collection<*> -> this.contains(value) - this is Array<*> -> this.contains(value) + this is Collection<*> -> this.any { it.matches(value) } + this is Array<*> -> this.any { it.matches(value) } else -> throw UnsupportedOperationException("contains doesn't support ${this?.javaClass} type") } } diff --git a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/Equals.kt b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/Equals.kt index 5aa0cba..c6c2418 100644 --- a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/Equals.kt +++ b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/Equals.kt @@ -5,5 +5,5 @@ import com.rapatao.projects.ruleset.engine.types.operators.EqualsOperator internal class Equals : EqualsOperator() { override fun process(context: EvalContext, left: Any?, right: Any?): Boolean = - left == right + left.matches(right) } diff --git a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/NotEquals.kt b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/NotEquals.kt index 15c3aa3..81f7fe8 100644 --- a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/NotEquals.kt +++ b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/NotEquals.kt @@ -5,5 +5,5 @@ import com.rapatao.projects.ruleset.engine.types.operators.NotEqualsOperator internal class NotEquals : NotEqualsOperator() { override fun process(context: EvalContext, left: Any?, right: Any?): Boolean = - left != right + !left.matches(right) } diff --git a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/extensions.kt b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/extensions.kt index 416e5e4..079c7e7 100644 --- a/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/extensions.kt +++ b/kotlin-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/operator/extensions.kt @@ -1,5 +1,21 @@ package com.rapatao.projects.ruleset.engine.evaluator.kotlin.operator +import java.math.BigDecimal + @Suppress("UNCHECKED_CAST") internal fun T.comparable() = this as Comparable +/** + * Equality that compares numbers by value. + * + * Every number reaches an operator as a `BigDecimal`, and `BigDecimal.equals` is scale sensitive, so `10` and `10.00` + * are not equal to it while `compareTo` reports them as the same number. A collection is compared element by element + * under the same rule. + */ +internal fun Any?.matches(other: Any?): Boolean = when { + this is BigDecimal && other is BigDecimal -> this.compareTo(other) == 0 + this is Collection<*> && other is Collection<*> -> + this.size == other.size && this.asSequence().zip(other.asSequence()).all { (a, b) -> a.matches(b) } + + else -> this == other +} diff --git a/kotlin-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinBenchmark.kt b/kotlin-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinBenchmark.kt index 7f43192..f1df62d 100644 --- a/kotlin-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinBenchmark.kt +++ b/kotlin-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinBenchmark.kt @@ -3,5 +3,8 @@ package com.rapatao.projects.ruleset.engine.evaluator.kotlin import com.rapatao.projects.ruleset.engine.BaseEngineBenchmark fun main(args: Array) { - BaseEngineBenchmark(KotlinEvaluator()).main(args) + BaseEngineBenchmark( + evaluator = KotlinEvaluator(), + wide = args.getOrNull(1)?.toIntOrNull() ?: 0, + ).main(args) } diff --git a/kotlin-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinPathResolutionTest.kt b/kotlin-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinPathResolutionTest.kt new file mode 100644 index 0000000..71bda85 --- /dev/null +++ b/kotlin-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/kotlin/KotlinPathResolutionTest.kt @@ -0,0 +1,108 @@ +package com.rapatao.projects.ruleset.engine.evaluator.kotlin + +import com.rapatao.projects.ruleset.engine.types.builder.extensions.equalsTo +import org.hamcrest.MatcherAssert.assertThat +import org.hamcrest.Matchers.equalTo +import org.junit.jupiter.api.DisplayName +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows + +/** + * Locks the path resolution semantics the flat-map implementation had: which paths resolve, which resolve to null, + * and which throw. `OnFailure` turns the throw into a rule result, so the difference is observable. + */ +class KotlinPathResolutionTest { + + data class Holder(val value: String?, val nested: Holder? = null) + + private val evaluator = KotlinEvaluator() + + private fun resolves(path: String, expected: Any?, input: Any) = + assertThat(path, evaluator.evaluate(path equalsTo expected, input), equalTo(true)) + + private fun absent(path: String, input: Any) = + assertThrows(path) { evaluator.evaluate(path equalsTo 1, input) } + + @Test + @DisplayName("resolves a map value, a nested map value and the map itself") + fun assertMapPaths() { + val input = mapOf("a" to mapOf("b" to "c")) + + resolves("a.b", "\"c\"", input) + resolves("a", mapOf("b" to "c"), input) + } + + @Test + @DisplayName("a map key that is not a string stays reachable by its string form") + fun assertNonStringMapKey() { + resolves("a.1", "\"one\"", mapOf("a" to mapOf(1 to "one"))) + } + + @Test + @DisplayName("a present key holding null resolves to null instead of throwing") + fun assertPresentNullValue() { + resolves("a", null, mapOf("a" to null)) + resolves("value", null, Holder(value = null)) + } + + @Test + @DisplayName("a missing key throws") + fun assertMissingKey() { + absent("a", mapOf("b" to 1)) + absent("missing", Holder(value = "x")) + } + + @Test + @DisplayName("a path continuing past a value or a null throws") + fun assertPathPastLeaf() { + absent("a.length", mapOf("a" to "text")) + absent("value.length", Holder(value = "text")) + absent("a.b", mapOf("a" to null)) + absent("nested.value", Holder(value = "x")) + } + + @Test + @DisplayName("a named step under a collection or an array throws") + fun assertNamedStepUnderCollection() { + absent("a.size", mapOf("a" to listOf(1, 2))) + absent("a.size", mapOf("a" to arrayOf(1, 2))) + } + + @Test + @DisplayName("indexes lists, arrays, sets and nested lists") + fun assertIndexedPaths() { + resolves("a[0]", "\"first\"", mapOf("a" to listOf("first", "second"))) + resolves("a[1]", "\"second\"", mapOf("a" to arrayOf("first", "second"))) + resolves("a[0]", "\"only\"", mapOf("a" to setOf("only"))) + resolves("a[0][1]", "\"inner\"", mapOf("a" to listOf(listOf("outer", "inner")))) + resolves("a[0].value", "\"deep\"", mapOf("a" to listOf(Holder(value = "deep")))) + } + + @Test + @DisplayName("an out of range, malformed or unsupported index throws") + fun assertInvalidIndex() { + absent("a[2]", mapOf("a" to listOf(1, 2))) + absent("a[2]", mapOf("a" to arrayOf(1, 2))) + absent("a[x]", mapOf("a" to listOf(1, 2))) + absent("a[0", mapOf("a" to listOf(1, 2))) + absent("a[0]", mapOf("a" to "text")) + } + + @Test + @DisplayName("the empty path addresses the root only when the root is a value, collection or array") + fun assertRootPath() { + assertThat(evaluator.evaluate("" equalsTo listOf(1), listOf(1)), equalTo(true)) + resolves("[0]", 1, listOf(1)) + resolves("[0]", 1, arrayOf(1)) + absent("", mapOf("a" to 1)) + absent("", Holder(value = "x")) + } + + @Test + @DisplayName("cached reflection resolves the same path across evaluations") + fun assertRepeatedReflectionResolution() { + val input = Holder(value = "x", nested = Holder(value = "y")) + + repeat(2) { resolves("nested.value", "\"y\"", input) } + } +} diff --git a/rhino-evaluator/build.gradle b/rhino-evaluator/build.gradle index ef602b2..f6198f6 100644 --- a/rhino-evaluator/build.gradle +++ b/rhino-evaluator/build.gradle @@ -7,8 +7,12 @@ dependencies { tasks.register("bench", JavaExec) { group = "verification" - description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000" + description = "Runs the engine benchmark. Iterations: -PbenchIterations=1000, wide input: -PbenchWide=200" classpath = sourceSets.test.runtimeClasspath mainClass = "com.rapatao.projects.ruleset.engine.evaluator.rhino.RhinoBenchmarkKt" - args = [providers.gradleProperty("benchIterations").getOrElse("1000")] + args = [ + providers.gradleProperty("benchIterations").getOrElse("1000"), + providers.gradleProperty("benchWide").getOrElse("0"), + providers.gradleProperty("benchReuse").getOrElse("false"), + ] } diff --git a/rhino-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/rhino/operator/Contains.kt b/rhino-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/rhino/operator/Contains.kt index 194b991..6f3644b 100644 --- a/rhino-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/rhino/operator/Contains.kt +++ b/rhino-evaluator/src/main/kotlin/com/rapatao/projects/ruleset/engine/evaluator/rhino/operator/Contains.kt @@ -7,13 +7,22 @@ internal class Contains : ContainsOperator() { override fun process(context: EvalContext, left: Any?, right: Any?): Boolean = context.evaluate( """ - (function() { - if (Array.isArray(${left})) { - return ${left}.includes(${right}) - } else { - return ${left}.indexOf(${right}) !== -1 + (function(source, value) { + if (typeof source === 'string') { + return source.indexOf(value) !== -1 } - })() + if (source == null || typeof source.length !== 'number') { + throw new TypeError('contains does not support ' + typeof source) + } + // Scanned rather than delegated to indexOf, because a java.util.List compares its elements by + // Java equality and a JS number never equals a boxed Integer under it. + for (var i = 0; i < source.length; i++) { + if (source[i] == value) { + return true + } + } + return false + })($left, $right) """.trimIndent() ) } diff --git a/rhino-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/rhino/RhinoBenchmark.kt b/rhino-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/rhino/RhinoBenchmark.kt index 15deaf9..5da948b 100644 --- a/rhino-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/rhino/RhinoBenchmark.kt +++ b/rhino-evaluator/src/test/kotlin/com/rapatao/projects/ruleset/engine/evaluator/rhino/RhinoBenchmark.kt @@ -3,5 +3,8 @@ package com.rapatao.projects.ruleset.engine.evaluator.rhino import com.rapatao.projects.ruleset.engine.BaseEngineBenchmark fun main(args: Array) { - BaseEngineBenchmark(RhinoEvaluator()).main(args) + BaseEngineBenchmark( + evaluator = RhinoEvaluator(), + wide = args.getOrNull(1)?.toIntOrNull() ?: 0, + ).main(args) } diff --git a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/BaseEngineBenchmark.kt b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/BaseEngineBenchmark.kt index da708ca..1fb0693 100644 --- a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/BaseEngineBenchmark.kt +++ b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/BaseEngineBenchmark.kt @@ -2,31 +2,55 @@ package com.rapatao.projects.ruleset.engine import com.rapatao.projects.ruleset.engine.cases.TestData import com.rapatao.projects.ruleset.engine.types.Expression +import java.lang.management.ManagementFactory import java.nio.file.Paths import kotlin.io.path.appendText import kotlin.io.path.createFile import kotlin.io.path.exists import kotlin.io.path.writeText +import kotlin.math.sqrt import kotlin.time.Duration import kotlin.time.DurationUnit import kotlin.time.measureTimedValue - -class BaseEngineBenchmark(private val evaluator: Evaluator) { +import com.sun.management.ThreadMXBean as HotSpotThreadMXBean + +/** + * Replays the full rule set from [TestData] against one engine. + * + * Every `bench` task passes the same arguments in the same order, and each benchmark reads the ones its engine + * supports: + * + * | index | property | meaning | + * |-------|--------------------|--------------------------------------------------------| + * | 0 | `benchIterations` | iterations of the whole rule set | + * | 1 | `benchWide` | extra fields and list elements around the input, 0 off | + * | 2 | `benchReuse` | GraalJS only, `reuseContextPerThread` | + * + * Reported per run: throughput and the latency distribution of an iteration, the bytes each engine allocates per + * evaluation, and the garbage collections that happened while the measured loop ran. + */ +class BaseEngineBenchmark( + private val evaluator: Evaluator, + private val wide: Int = 0, +) { private val benchOut = Paths.get("bench_${evaluator.name()}.txt") + private val input = if (wide > 0) TestData.wideInput(wide) else TestData.inputData + @Suppress("MagicNumber") fun main(args: Array) { - cleanup() val cases = TestData.cases() .map { it.get().first { arg -> arg is Expression } } .map { it as Expression } + appendLine("${evaluator.name()}> input: " + if (wide > 0) "wide($wide)" else "default") + // ini: warmup appendLine("warmup ${evaluator.name()}: start") - repeat(100) { cases.forEach { expression -> evaluator.evaluate(expression, TestData.inputData) } } + repeat(100) { cases.forEach { expression -> evaluator.evaluate(expression, input) } } appendLine("warmup ${evaluator.name()}: done") // end: warmup @@ -36,41 +60,96 @@ class BaseEngineBenchmark(private val evaluator: Evaluator) { appendLine() + val before = usage() + repeat(iterations) { val time = measureTimedValue { - cases.forEach { expression -> evaluator.evaluate(expression, TestData.inputData) } + cases.forEach { expression -> evaluator.evaluate(expression, input) } } print("\r${evaluator.name()}: ${it + 1}") times.add(time.duration) } - appendLine() + + val consumed = usage() - before appendLine() + appendLine() + + report(iterations = iterations, perIteration = cases.size, times = times, consumed = consumed) + } + @Suppress("MagicNumber") + private fun report(iterations: Int, perIteration: Int, times: List, consumed: Usage) { val total = times.reduce { acc, duration -> acc + duration } + val ops = iterations.toLong() * perIteration appendLine("${evaluator.name()}> iterations: $iterations") - appendLine(" ops: " + (iterations * cases.size)) - appendLine(" ops/s: " + ((iterations * cases.size) / total.toDouble(DurationUnit.SECONDS))) - appendLine(" total: $total") - appendLine(" max: " + times.max()) - appendLine(" min: " + times.min()) - appendLine(" avg: " + (total / times.size)) - - val sortedResults = times.sorted() + appendLine(" ops: $ops") + appendLine(" ops/s: " + (ops / total.toDouble(DurationUnit.SECONDS))) + appendLine(" total: $total") + appendLine(" max: " + times.max()) + appendLine(" min: " + times.min()) + appendLine(" avg: " + (total / times.size)) + appendLine(" stddev: " + stdDev(times, total / times.size)) + + val allocated = consumed.allocatedBytes + + appendLine(" alloc/op: " + if (allocated < 0) "n/a" else "%,d B".format(allocated / ops)) + appendLine(" alloc: " + if (allocated < 0) "n/a" else "%,.1f MB".format(allocated / MB)) + appendLine(" gc: ${consumed.gcCollections} collections, ${consumed.gcMillis}ms") + + val sorted = times.sorted() listOf(0.50, 0.75, 0.90, 0.95, 0.99).forEach { p -> appendLine( - " p${(p * 100).toInt()}: " + sortedResults[(sortedResults.size * p).toInt() - .coerceAtMost(sortedResults.lastIndex)] + " p${(p * 100).toInt()}: " + sorted[(sorted.size * p).toInt().coerceAtMost(sorted.lastIndex)] ) } appendLine() } + private fun stdDev(times: List, avg: Duration): Duration { + val mean = avg.toDouble(DurationUnit.MICROSECONDS) + val variance = times.sumOf { + val diff = it.toDouble(DurationUnit.MICROSECONDS) - mean + diff * diff + } / times.size + + return Duration.parse("${sqrt(variance)}us") + } + + /** + * Allocation is counted on this thread only, which is where every engine runs the evaluation, and garbage + * collections are counted JVM wide. + */ + private fun usage(): Usage { + val gc = ManagementFactory.getGarbageCollectorMXBeans() + .fold(0L to 0L) { acc, bean -> + acc.first + bean.collectionCount.coerceAtLeast(0) to acc.second + bean.collectionTime.coerceAtLeast(0) + } + + return Usage( + allocatedBytes = allocatedBytes(), + gcCollections = gc.first, + gcMillis = gc.second, + ) + } + + private fun allocatedBytes(): Long { + val bean = (ManagementFactory.getThreadMXBean() as? HotSpotThreadMXBean) + ?.takeIf { it.isThreadAllocatedMemorySupported } + ?: return UNSUPPORTED + + if (!bean.isThreadAllocatedMemoryEnabled) { + bean.isThreadAllocatedMemoryEnabled = true + } + + return bean.getThreadAllocatedBytes(Thread.currentThread().threadId()) + } + private fun append(value: String) { benchOut.appendText(value) print(value) @@ -86,4 +165,25 @@ class BaseEngineBenchmark(private val evaluator: Evaluator) { } benchOut.writeText("") } + + private data class Usage( + val allocatedBytes: Long, + val gcCollections: Long, + val gcMillis: Long, + ) { + operator fun minus(other: Usage) = Usage( + allocatedBytes = if (allocatedBytes < 0 || other.allocatedBytes < 0) { + UNSUPPORTED + } else { + allocatedBytes - other.allocatedBytes + }, + gcCollections = gcCollections - other.gcCollections, + gcMillis = gcMillis - other.gcMillis, + ) + } + + private companion object { + private const val UNSUPPORTED = -1L + private const val MB = 1024.0 * 1024.0 + } } diff --git a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/BaseEvaluatorTest.kt b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/BaseEvaluatorTest.kt index 849ae5b..22edbcc 100644 --- a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/BaseEvaluatorTest.kt +++ b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/BaseEvaluatorTest.kt @@ -24,11 +24,15 @@ import org.junit.jupiter.params.provider.Arguments import org.junit.jupiter.params.provider.MethodSource import kotlin.reflect.full.memberProperties +// The shared contract every engine has to satisfy, so assertions accumulate here by design. +@Suppress("TooManyFunctions") abstract class BaseEvaluatorTest( private val evaluator: Evaluator ) { companion object { + private const val WIDE_INPUT_SIZE = 20 + @JvmStatic fun tests() = TestData.cases() @@ -74,6 +78,18 @@ abstract class BaseEvaluatorTest( ) } + @Test + @DisplayName("the wide benchmark input evaluates every case to the same result as the default one") + fun assertWideInputMatchesDefaultInput() { + val wide = TestData.wideInput(WIDE_INPUT_SIZE) + + val differences = tests() + .map { it.get().first { arg -> arg is Expression } as Expression } + .filter { evaluator.evaluate(it, wide) != evaluator.evaluate(it, TestData.inputData) } + + assertThat(differences, equalTo(emptyList())) + } + @Test @DisplayName("should support map as input data") fun assertMapAsInputDataSupport() { diff --git a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/ContainsCases.kt b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/ContainsCases.kt index 647694e..5aa8e3c 100644 --- a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/ContainsCases.kt +++ b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/ContainsCases.kt @@ -54,5 +54,17 @@ object ContainsCases { "item.tags" expNotContains "\"different value\"", true, ), + Arguments.of( + listOf("\"item1\"", "\"item2\"") expContains "\"item1\"", + true, + ), + Arguments.of( + listOf("\"item1\"", "\"item2\"") expNotContains "\"item3\"", + true, + ), + Arguments.of( + listOf("item.name", "\"something else\"") expContains "\"product name\"", + true, + ), ) } diff --git a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/NumberCases.kt b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/NumberCases.kt new file mode 100644 index 0000000..74c4d53 --- /dev/null +++ b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/NumberCases.kt @@ -0,0 +1,60 @@ +package com.rapatao.projects.ruleset.engine.cases + +import com.rapatao.projects.ruleset.engine.types.builder.extensions.equalsTo +import com.rapatao.projects.ruleset.engine.types.builder.extensions.expContains +import com.rapatao.projects.ruleset.engine.types.builder.extensions.expNotContains +import com.rapatao.projects.ruleset.engine.types.builder.extensions.greaterOrEqualThan +import com.rapatao.projects.ruleset.engine.types.builder.extensions.greaterThan +import com.rapatao.projects.ruleset.engine.types.builder.extensions.lessThan +import com.rapatao.projects.ruleset.engine.types.builder.extensions.notEqualsTo +import org.junit.jupiter.params.provider.Arguments + +/** + * Numbers that are not whole, and numbers inside a list. + * + * Every engine has to agree on these: a decimal field compares by its full value, two numbers that differ only in + * scale are the same number, and a list of numbers is searched by value. + */ +object NumberCases { + + fun cases(): List = fractionCases() + scaleCases() + numberListCases() + + @Suppress("MagicNumber") + private fun fractionCases(): List = listOf( + // item.weight is 1.5 + Arguments.of("item.weight" equalsTo "1.5", true), + Arguments.of("item.weight" equalsTo "1.9", false), + Arguments.of("item.weight" equalsTo "1.0", false), + Arguments.of("item.weight" equalsTo "1", false), + Arguments.of("item.weight" notEqualsTo "1.9", true), + Arguments.of("item.weight" greaterThan "1.4", true), + Arguments.of("item.weight" greaterThan "1.6", false), + Arguments.of("item.weight" lessThan "1.6", true), + Arguments.of("item.weight" lessThan "1.4", false), + Arguments.of("item.weight" greaterOrEqualThan "1.5", true), + ) + + @Suppress("MagicNumber") + private fun scaleCases(): List = listOf( + // item.scaled is 10.00, and differs from 10 only in scale + Arguments.of("item.scaled" equalsTo 10, true), + Arguments.of("item.scaled" equalsTo "10.0", true), + Arguments.of("item.scaled" equalsTo "10.00", true), + Arguments.of("item.scaled" notEqualsTo 10, false), + Arguments.of("item.scaled" equalsTo "10.01", false), + Arguments.of("item.price" equalsTo "10.00", true), + ) + + @Suppress("MagicNumber") + private fun numberListCases(): List = listOf( + // written in the expression + Arguments.of(listOf(1, 2) expContains 1, true), + Arguments.of(listOf(1, 2) expContains 3, false), + Arguments.of(listOf(1, 2) expNotContains 3, true), + // read from the input data, item.quantities is [1, 2] + Arguments.of("item.quantities" expContains "1", true), + Arguments.of("item.quantities" expContains "3", false), + Arguments.of("item.quantities" expNotContains "3", true), + Arguments.of("item.quantities" expContains "1.0", true), + ) +} diff --git a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/TestData.kt b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/TestData.kt index c6e4f60..bd796bb 100644 --- a/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/TestData.kt +++ b/tests/src/main/kotlin/com/rapatao/projects/ruleset/engine/cases/TestData.kt @@ -16,6 +16,9 @@ object TestData { val tags: List, val arrTags: Array, val nullableStr: String? = null, + val weight: BigDecimal = BigDecimal("1.5"), + val scaled: BigDecimal = BigDecimal("10.00"), + val quantities: List = listOf(1, 2), ) val inputData = RequestData( @@ -29,9 +32,21 @@ object TestData { ) ) + /** + * The same [inputData] item under a deliberately wide root: [size] extra scalar fields and a [size] element list. + * + * Every rule in [cases] roots at `item.*`, so the suite runs unchanged against it and the only difference is how + * much input surrounds the fields the rules read. + */ + fun wideInput(size: Int): Map = + mapOf("item" to inputData.item) + + (1..size).associate { "pad$it" to "value$it" } + + mapOf("padList" to (1..size).map { "element$it" }) + fun cases() = ExpressionCases.cases() + MatcherCases.cases() + OperatorWithCases.cases() + - ContainsCases.cases() + ContainsCases.cases() + + NumberCases.cases() }