Opteryx: rebuild the entry after the engine refactor, add native-format variant, refresh results - #1416
Merged
Merged
Conversation
…ive-format variant The previous entry targeted Python 3.11 and no longer ran. Opteryx has changed substantially since: the planning engine is largely as it was, still in Python, while the execution engine has been significantly refactored. It now ships as `opteryx-core` (previously `opteryx`), with a different execution API. install: stock CPython 3.14 and opteryx-core. Not the free-threaded 3.14t build -- the free-threaded experiment was abandoned in June 2026 (the parallelism target is native C++ threads under a released GIL) and opteryx-core stopped publishing cp314t wheels after 0.9.16, so a 3.14t interpreter falls through to the sdist and tries to build Rust/C++ on the box. The wheel has no runtime dependencies, so this is a single binary download. Ends with a version banner so a fallback to a source build fails loudly there rather than 43 queries later. check: `version()` is no longer a function -- the version is the system variable `@@version` -- and the old `python -m opteryx` path caught the resulting error and still exited 0, a false green that let a broken install through. Probe `@@version` through the Python API and assert the shape of the result. This matters because bench_load() calls ./check unconditionally, after the ~14 GB download. query: rewritten for the current surface (session.execute_to_morsels); opteryx.query() / Cursor.arrow() are gone. Times the morsel drain, which is the execution, and emits the full result as TSV afterwards. queries.sql: adapt 8 queries to the dialect -- explicit casts on the EventDate/EventTime comparisons, TRUNC for DATE_TRUNC, and b''/r'' literals in the REGEXP_REPLACE. Syntactic only; the unadapted forms produce null rows for exactly those 8 rather than wrong numbers. template.json: rename the system to "Opteryx (Parquet, partitioned)", following duckdb-parquet-partitioned and chdb-parquet-partitioned, which frees the unqualified name for the native-format entry. Historical result files keep "Opteryx": they were accurate when recorded and are not rewritten. Add the "C++" tag, matching the implementation-language tag carried by comparable embedded engines. opteryx-skene/ is that native-format entry. skene is Opteryx's own columnar format, so it takes the unqualified name, as `duckdb` does against `duckdb-parquet-partitioned`. install, check, query, queries.sql, start and stop are shared with the Parquet entry verbatim -- query needs no change at all, since a skene dataset is addressed by the same directory resolution and `FROM hits` finds ./hits/*.skene. What differs is the load. ClickBench distributes Parquet, so the dataset must be converted before it can be queried, and that conversion is the load step -- Load time is its wall-clock, as for any entry ingesting source data into a native store. convert.py is self-contained rather than reusing the engine repo's dev tooling, which a ClickBench run does not have; everything it needs (skene, draken, rugo) is in the one wheel. Conversion runs across processes, which costs layout: row groups do not span chunk boundaries, so a 12-worker build holds 28 files against 24 for a single worker and measures ~7% slower to query. Row count is verified per chunk and in total, and a mismatch fails the run. Written lz4 with read acceleration, the engine's read-first posture.
opteryx-core 0.9.69 on CPython 3.14, three tries per query with the page cache dropped before each first try, run through the shared driver. Parquet, eight machines spanning 2 to 16 vCPUs on both architectures: c6a.large, c6a.xlarge, c6a.2xlarge, c6a.4xlarge, c8g.large, c8g.xlarge, c8g.2xlarge, t3a.small. skene on c8g.4xlarge. Every query returns on every machine -- 387 results, no nulls. The published 0.26.x runs nulled 17 of 43 on t3a.small, 15 on c6a.large and 12 on c6a.xlarge; those all now produce numbers. The smaller instances are slow rather than failing: the heavy GROUP BYs run out of swap (10 GB against 2 GB of RAM on t3a.small) using the 16 GB swapfile ClickBench's cloud-init configures for every system. The c8g.* machines are the first AArch64 results for this entry, which 0.9.69 made possible by publishing manylinux aarch64 wheels. Measured at matched sizes, Graviton4 leads the equivalent c6a by 50% at 16 vCPU, 30% at 8, 25% at 4 and 18% at 2 -- the advantage narrowing as the workload moves from CPU-bound to memory-bound. Machines above 32 vCPUs are absent: the account running these is capped at 32 concurrent on-demand vCPUs, so the 192-vCPU metal instances could not be launched. Both READMEs say so.
Contributor
|
Results for Logs:
|
robot-clickhouse
had a problem deploying
to
benchmark-approval
August 18, 2026 01:43 — with 
GitHub Actions
Error
GitHub Actions
Error
…large, c6a.large, c6a.metal, c6a.xlarge, c7a.metal-48xl, c8g.4xlarge, c8g.metal-48xl)
Contributor
|
Results for Logs:
|
robot-clickhouse
had a problem deploying
to
benchmark-approval
August 27, 2026 22:56 — with
GitHub Actions
Error
GitHub Actions
Error
Contributor
|
Results for Logs:
|
robot-clickhouse
had a problem deploying
to
benchmark-approval
August 28, 2026 01:48 — with
GitHub Actions
Error
GitHub Actions
Error
Contributor
|
Results for Logs:
|
robot-clickhouse
requested a deployment
to
benchmark-approval
August 28, 2026 04:54 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The previous entry targeted Python 3.11 and no longer ran. Opteryx has changed substantially since: the planning engine is largely as it was, still in Python, while the execution engine has been significantly refactored. It now ships as
opteryx-core(previouslyopteryx), with a different execution API.opteryx/— rebuilt foropteryx-coreon CPython 3.14.install,checkandqueryrewritten; 8 queries adapted to the dialect; renamed toOpteryx (Parquet, partitioned)and taggedC++.opteryx-skene/— new entry for skene, Opteryx's native format, so it takes the unqualified name (asduckdbdoes againstduckdb-parquet-partitioned). Shares the harness; the load step converts the Parquet corpus, soLoad timeis that conversion.Results: 0.9.69, nine machines, three tries per query, 387 results with no nulls. Hot totals run from 25.9 s (skene, c8g.4xlarge) and 38.7 s (Parquet, same box) to 2821 s on t3a.small. The previous 0.26.x runs nulled 17 of 43 on t3a.small, 15 on c6a.large and 12 on c6a.xlarge; all now return numbers. The
c8g.*results are the first AArch64 numbers for this entry.Notes:
EventDate/EventTimecomparisons,TRUNCforDATE_TRUNC,b''/r''regex literals. Both READMEs document them."system": "Opteryx"; they were accurate when recorded.c6a.metalandc7a.metal-48xlstill appear under it until re-run.index.htmluntouched; the website workflow regenerates it.