Merge pull request #6 from Query-farm/feat/database-worker-packages

feat: add database-backed worker packages

Author: rustyconover

.github/workflows/integration.yml

@@ -118,6 +118,9 @@ jobs:
             --allow-skip 'require spatial' \
             --allow-skip 'require-env VGI_DOCKER_IMAGE' \
             --allow-skip 'require-env VGI_DOCKER_TCP_IMAGE' \
+            --allow-skip 'require-env VGI_DATABASE_BUN_WORKER' \
+            --allow-skip 'require-env VGI_DATABASE_PYTHON_WORKER' \
+            --allow-skip 'require-env VGI_DATABASE_RUST_WORKER' \
             --allow-skip 'require-env VGI_GITHUB_NETWORK_TESTS' \
             --allow-skip 'require-env VGI_TEST_ICEBERG' \
             --allow-skip 'require-env VGI_TEST_COMPANION_TARGET' \

CLAUDE.md

@@ -92,6 +92,20 @@ the matrix had the machinery but nothing ran it, so the failures accumulated
 silently. It is not wired into CI yet (four toolchains on one runner); until it
 is, running it locally is the guardrail.
 
+### Database-packaged worker artifacts (`make test_database_workers`)
+
+The ordinary `database_worker/package.test` uses a tiny executable wrapper so
+the default regression stays fast. The explicit heavyweight target builds and
+packages three real workers, then restores and executes each one through
+`database://`: a PyInstaller-frozen Python/PyArrow worker, Bun's standalone
+build of `~/Development/vgi-open-meteo`, and a `.tar.gz` containing the native
+Rust example worker plus a metadata file. It expects the sibling vgi-python,
+vgi-open-meteo, and vgi-rust repositories and the `uv`, `bun`, `cargo`, and
+`tar` tools. Repository paths and cross-build targets are overrideable; see
+`docs/database-worker-transport.md`. The target finishes with a two-process
+restart test: one DuckDB process persists all three BLOBs, then a fresh process
+with an empty cache reopens that registry and executes every worker.
+
 ### HTTP Transport
 
 ```bash
@@ -1041,6 +1055,12 @@ transport in `src/query_farm_telemetry.cpp`. Full field reference: [docs/telemet
 | `vgi_trust_empty_kinds` | BOOLEAN | true | Trust worker assertions that `estimated_object_count[kind] == 0` means the kind is empty (skip `catalog_schema_contents_*` RPC). Set to false to force every RPC to fire — debug escape hatch for diagnosing worker bugs |
 | `vgi_secret_default_ttl_seconds` | BIGINT | 300 | Default cache TTL for credentials fetched from an Orchard remote secret provider, when the server suggests none. Further capped per-credential by the credential's own expiry. Read at `ATTACH`, frozen per-provider. See *Remote Secret Provider* |
 | `vgi_github_cache_dir` | VARCHAR | `""` | Cache directory for worker binaries downloaded from GitHub releases (`github://` / `github-auto://`). Empty → `${XDG_CACHE_HOME:-~/.cache}/vgi/releases`. Must be on an exec-capable filesystem. See [docs/github-transport.md](docs/github-transport.md) |
+| `vgi_worker_cache_dir` | VARCHAR | `""` | Cache directory for immutable `database://` worker packages. Empty follows `vgi_github_cache_dir`, then `${XDG_CACHE_HOME:-~/.cache}/vgi/releases`. Must be exec-capable. See [docs/database-worker-transport.md](docs/database-worker-transport.md) |
+| `vgi_worker_cache_max_bytes` | UBIGINT | 5368709120 (5 GiB) | Managed database-worker artifact budget. Opportunistic cleanup evicts least-recently-used, unleased entries to an 80% low-water mark; 0 disables size eviction. |
+| `vgi_worker_cache_ttl_seconds` | UBIGINT | 2592000 (30 days) | Evict database-worker artifacts unused for this long; 0 disables TTL eviction. Catalogs and running/pooled workers hold cross-process leases and cannot be evicted. |
+| `vgi_worker_package_max_bytes` | UBIGINT | 536870912 (512 MiB) | Maximum stored/compressed `contents` BLOB accepted from a package table. |
+| `vgi_worker_package_max_extracted_bytes` | UBIGINT | 1073741824 (1 GiB) | Maximum decompressed size accepted while installing a database worker package. |
+| `vgi_worker_package_max_files` | UBIGINT | 10000 | Maximum archive entries accepted while installing a database worker package. |
 | `vgi_result_cache` | BOOLEAN | true | Master switch for the **table-function result cache**. When a worker advertises `vgi.cache.*` metadata on its result's first batch, the complete result is cached and identical future scans are served without the worker round-trip. Double-gated: only advertised results are stored, and a catalog opts out via the `cache` ATTACH option |
 | `vgi_result_cache_max_bytes` | UBIGINT | 268435456 (256 MB) | Global byte cap for the in-memory result cache (LRU eviction above it). Read per-query, so `SET` is runtime-effective |
 | `vgi_result_cache_max_entry_bytes` | UBIGINT | 67108864 (64 MB) | Per-entry RAM cap: capture buffers up to this in RAM, then — if the disk tier is on (`vgi_result_cache_dir`+`disk_max_bytes`) — **spills the rest to a streaming disk blob** (RAM stays flat at ~this cap, result cached to disk); if the disk tier is **off**, capture aborts and the result streams uncached (`result_cache.abort reason=entry_too_large`). Also the serve threshold: a disk entry larger than this is served by streaming (never materialized), smaller ones are materialized + adopted into memory. See *Spill-to-disk capture* |
@@ -1097,6 +1117,7 @@ Catalogs may register additional settings at `ATTACH` time (e.g., `greeting`, `m
 - `launch:<argv>` → **AF_UNIX** transport with the worker spawned via the launcher
 - `oci://<image>[:tag]` (or `docker://` alias) → **container** transport: the worker runs inside an OCI container via the host runtime (docker/podman/nerdctl/Apple `container`), wired over stdin/stdout like a subprocess and pooled the same way. See [docs/container-transport.md](docs/container-transport.md)
 - `github://owner/repo@tag/asset[#sha256=][#path=]` and `github-auto://owner/repo@tag[/prefix]` → **GitHub-release** transport (any subprocess-capable platform — POSIX **and** Windows; not Emscripten): download a worker executable from a GitHub release, SHA256-verify it, extract the full archive (`.tar.gz`/`.tar.zst`/`.zst`/`.gz`/bare via in-process Decompress; `.zip` via vendored miniz) into `${XDG_CACHE_HOME:-~/.cache}/vgi/releases`, and run it over the subprocess transport. All file I/O goes through DuckDB's cross-platform `FileSystem` (cross-process write-lock + atomic dir install); only the exec bit (`chmod`), tar symlinks, and macOS codesign stay POSIX. `github://` names the asset explicitly (+ optional `#sha256=` pin, enforced even on cache hit); `github-auto://` builds `{prefix=repo}-{tag}-{DuckDB-platform}.tar.gz` (`.zip` on Windows) and verifies against the published `.sha256` sidecar. Entrypoint = the single exec-bit member (tar) / single `.exe` (zip), or `#path=`. A DX convenience (overlaps `oci://`); SHA256-only (cosign provenance deferred); no allowlist; cwd inherited (publisher builds relocatable). See [docs/github-transport.md](docs/github-transport.md)
+- `database://catalog/schema/table/worker?package_version=version[#sha256=digest]` → **database-package resolver** (native subprocess platforms): query one committed row matching worker/platform/version through a dedicated DuckDB connection, verify the `contents` BLOB's SHA-256, safely extract/copy it into the shared immutable worker cache, then directly execute its entrypoint over the subprocess transport. Identifiers are quoted and row keys bound; URI components are percent-decoded. The content/format/entrypoint-keyed cache uses cross-process shared leases held by catalogs and running/pooled workers, so TTL/LRU/explicit cleanup cannot remove live code. `vgi_worker_package(...)` packages a file with `read_blob()` + `pragma_platform()`. See [docs/database-worker-transport.md](docs/database-worker-transport.md)
 - `worker:<url>` → **in-browser SAB** transport (Emscripten/DuckDB-WASM only): the worker runs client-side in a Web Worker, exchanging Arrow batches over a SharedArrayBuffer duplex-ring channel in DuckDB's linear memory — **no server**. Reuses the subprocess `FunctionConnection` streaming paths (only the byte transport changes: `Atomics`-blocked SAB rings vs an OS pipe). Parallel scans work (`max_workers>1` → one serve pthread per slot; browser E2E proves `maxConcurrency=4`). Two emscripten-pthread footguns are load-bearing — JS ring stubs must read `wasmMemory.buffer` (not the stale cached `Module.HEAPU8` view) and re-publish the per-realm channel offset before each ring op — else intermittent scheduler-dependent hangs. Files: `src/vgi_webworker_function_connection.cpp`, `src/vgi_sab_stream.cpp`, `src/include/vgi_sab_abi.hpp`; host glue in `haybarn-wasm` (`lib/js-stubs.js` + `vgi-webworker-bridge.ts`); worker via `vgi-rust`. See [docs/wasm-worker-transport.md](docs/wasm-worker-transport.md) + [docs/sab_transport_abi.md](docs/sab_transport_abi.md)
 
 The `launch:` and `unix://` paths share one warm worker process across every DuckDB instance that points at the same `(cmd, args, cwd, VGI_RPC_*-env)` tuple — coordinated system-wide via per-hash flock + AF_UNIX socket.  See `docs/launcher-protocol.md` for the wire-protocol contract (state-dir layout, hash inputs, lockfile semantics) shared with the Python reference launcher in `vgi-rpc/vgi_rpc/launcher.py`.
@@ -1113,6 +1134,10 @@ The `launch:` and `unix://` paths share one warm worker process across every Duc
 | `vgi_worker_pool_flush()` | Table | Clear all subprocess-pooled workers; returns one row with the count flushed (`flushed`). Has no effect on `launch:` / `unix://` workers. |
 | `vgi_github_cache()` | Table | Diagnostic: one row per worker binary cached from a `github://` / `github-auto://` release LOCATION. Columns: `owner`, `repo`, `tag`, `asset`, `digest` (archive SHA256), `dir` (extracted directory), `entrypoint`, `age_seconds`. See [docs/github-transport.md](docs/github-transport.md) |
 | `vgi_github_cache_flush()` | Table | Delete the on-disk GitHub-release worker cache; returns one row with the count of cached releases removed (`flushed`). |
+| `vgi_worker_package(path, worker_name, package_version, package_format := 'auto', entrypoint := NULL, platform := NULL)` | Table macro | Produce one canonical package-table row using `read_blob()` and `pragma_platform()`, including SHA-256 and `created_at`. Archives require an entrypoint. See [docs/database-worker-transport.md](docs/database-worker-transport.md) |
+| `vgi_worker_cache()` | Table | List database worker artifacts: identity/source/digest/format/entrypoint/directory, size, unused age, and whether a cross-process lease is active. |
+| `vgi_worker_cache_prune()` | Table | Synchronously apply database-worker cache TTL and size limits. Returns `removed`, `bytes_removed`, `skipped_in_use`. |
+| `vgi_worker_cache_flush()` | Table | Attempt to remove all database-worker artifacts. Lease-protected artifacts are retained and counted in `skipped_in_use`. |
 | `vgi_clear_cache()` | Table | Clear cached catalog metadata (schemas, tables, functions, statistics) for all attached VGI catalogs. Also drops that catalog's **result-cache** entries |
 | `vgi_result_cache(include_disk := false)` | Table | Diagnostic: one row per cached table-function result. Columns: `catalog`, `function`, `key_hash`, `scope`, `attached_data_version`, `implementation_version`, `catalog_version`, `at_unit`, `at_value`, `num_batches`, `num_substreams` (capture-thread count — `> 1` proves parallel capture across workers), `num_rows`, `total_bytes`, `age_seconds`, `ttl_seconds`, `stale`, `tier` (`memory`/`disk`), `etag`, `last_modified`, `revalidatable`, `hits`, `codec` (on-disk compression codec `none`/`zstd`/`lz4`; always `none` for memory-tier rows), `partition_label` (per-partition entry's tuple, e.g. `country=US`; empty for non-partition entries — see *Per-Partition Result Cache*). Defaults to the **in-memory** tier; `include_disk := true` also walks the on-disk refs so **spilled/disk-only** entries (invisible to the memory index until adopted) are listed with `tier='disk'`. See *Table-Function Result Cache* |
 | `vgi_result_cache_flush()` | Table | Clear the in-memory result cache; returns one row with the count of entries flushed (`flushed`) |
@@ -1147,6 +1172,8 @@ The `launch:` and `unix://` paths share one warm worker process across every Duc
 | `vgi_worker_pool_functions.cpp` | Pool diagnostic SQL functions |
 | `vgi_github.cpp` | `github://` / `github-auto://` transport (subprocess-capable platforms incl. Windows): coordinate parsing, `github-auto://` convention name-building from `DuckDB::Platform()` (`.zip` on Windows, `.tar.gz` else), authenticated GitHub API GET (+ reuse of `HttpGetBytes` for CDN downloads), SHA256 verify-before-extract, full-tree USTAR + miniz `.zip` extractors with path sanitization, and the content-digest-keyed atomic-directory cache. File I/O via DuckDB's cross-platform `FileSystem` (`CreateLocal()`; cross-process write-lock); only `chmod`/tar-symlink/macOS-codesign stay POSIX (`#if VGI_POSIX_TRANSPORT`). Exposes `ResolveWorkerPath()` (called at both spawn sites: `EnsureWorkerSpawned` + `AttemptUnaryRpc`). See [docs/github-transport.md](docs/github-transport.md) |
 | `vgi_github_functions.cpp` | `vgi_github_cache()` / `vgi_github_cache_flush()` SQL diagnostics |
+| `vgi_database_worker.cpp` | `database://` coordinate parser/resolver, parameterized committed-table lookup, SHA-256 verification, immutable atomic artifact install, cross-process leases, last-use metadata, and TTL/LRU/explicit cleanup. See [docs/database-worker-transport.md](docs/database-worker-transport.md) |
+| `vgi_worker_package_functions.cpp` | `vgi_worker_package(...)` packaging macro plus `vgi_worker_cache()` / `_prune()` / `_flush()` diagnostics. |
 | `vgi_table_function_impl.cpp` | Shared table function logic (bind/init/scan) |
 | `vgi_scalar_function_impl.cpp` | Scalar function bind/execute with dynamic types and const params |
 | `vgi_aggregate_function_impl.cpp` | Aggregate function bind / update / combine / finalize / destructor RPC client |

CMakeLists.txt

@@ -130,6 +130,8 @@ src/vgi_input_dedup.cpp
 src/vgi_result_cache_functions.cpp
 src/vgi_cached_replay_connection.cpp
 src/vgi_github.cpp
+src/vgi_database_worker.cpp
+src/vgi_worker_package_functions.cpp
 src/vgi_global_functions.cpp
 src/vgi_github_functions.cpp
 src/vgi_wasm_async_pool.cpp
@@ -217,6 +219,7 @@ if((BUILD_VGI_UNIT_TESTS OR DEFINED ENV{BUILD_VGI_UNIT_TESTS}) AND NOT WIN32)
     test/cpp/test_sab_conn.cpp
     test/support/vgi_sab_native_ring.cpp
     test/support/vgi_sab_test_stubs.cpp
+    test/support/vgi_database_worker_test_stubs.cpp
     src/vgi_sab_stream.cpp
     src/vgi_webworker_function_connection.cpp
     src/vgi_rpc_client.cpp

Makefile

@@ -446,7 +446,43 @@ VGI_JAVA_DIR   ?= $(HOME)/vgi-java
 VGI_RUST_DIR   ?= $(HOME)/Development/vgi-rust
 VGI_CSHARP_DIR ?= $(HOME)/Development/vgi-csharp
 
-.PHONY: test_python test_python_crash test_go test_typescript test_java test_rust test_csharp test_languages
+.PHONY: test_python test_python_crash test_go test_typescript test_java test_rust test_csharp test_languages \
+	build_database_workers test_database_workers test_database_worker_restart
+
+# Real database-package demonstrations. These are intentionally separate from
+# the fast default suite: PyInstaller and Bun each produce a ~100 MiB native
+# executable, and Rust compiles the full example worker. The resulting tests do
+# not delegate to VGI_TEST_WORKER; the BLOB restored from DuckDB is the worker.
+VGI_DATABASE_WORKER_FIXTURE_DIR ?= $(PROJ_DIR)build/database-worker-fixtures
+VGI_DATABASE_PYTHON_DIR ?= $(HOME)/Development/vgi-python
+VGI_DATABASE_RUST_DIR ?= $(HOME)/Development/vgi-rust
+VGI_DATABASE_OPEN_METEO_DIR ?= $(HOME)/Development/vgi-open-meteo
+
+build_database_workers:
+	VGI_DATABASE_WORKER_FIXTURE_DIR="$(VGI_DATABASE_WORKER_FIXTURE_DIR)" \
+	VGI_DATABASE_PYTHON_DIR="$(VGI_DATABASE_PYTHON_DIR)" \
+	VGI_DATABASE_RUST_DIR="$(VGI_DATABASE_RUST_DIR)" \
+	VGI_DATABASE_OPEN_METEO_DIR="$(VGI_DATABASE_OPEN_METEO_DIR)" \
+	./scripts/build_database_worker_fixtures.sh
+
+test_database_workers: build_database_workers
+	database_worker_cache="$$(mktemp -d)"; \
+	trap 'rm -rf "$$database_worker_cache"' 0; \
+	XDG_CACHE_HOME="$$database_worker_cache" \
+	VGI_DATABASE_PYTHON_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-python-worker" \
+	VGI_DATABASE_BUN_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-open-meteo" \
+	VGI_DATABASE_RUST_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-rust-worker.tar.gz" \
+	./build/release/test/unittest "test/sql/integration/database_worker/real_*"
+	VGI_DATABASE_PYTHON_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-python-worker" \
+	VGI_DATABASE_BUN_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-open-meteo" \
+	VGI_DATABASE_RUST_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-rust-worker.tar.gz" \
+	./scripts/test_database_worker_restart.sh
+
+test_database_worker_restart: build_database_workers
+	VGI_DATABASE_PYTHON_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-python-worker" \
+	VGI_DATABASE_BUN_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-open-meteo" \
+	VGI_DATABASE_RUST_WORKER="$(VGI_DATABASE_WORKER_FIXTURE_DIR)/vgi-rust-worker.tar.gz" \
+	./scripts/test_database_worker_restart.sh
 
 # Python uses this repo's default worker set. It does NOT just chain to
 # test_launcher, which invokes `unittest` bare: a lane that stops running tests
@@ -563,6 +599,9 @@ VGI_EXPECTED_SKIPS := \
 	--allow-skip 'require spatial' \
 	--allow-skip 'require-env VGI_DOCKER_IMAGE' \
 	--allow-skip 'require-env VGI_DOCKER_TCP_IMAGE' \
+	--allow-skip 'require-env VGI_DATABASE_BUN_WORKER' \
+	--allow-skip 'require-env VGI_DATABASE_PYTHON_WORKER' \
+	--allow-skip 'require-env VGI_DATABASE_RUST_WORKER' \
 	--allow-skip 'require-env VGI_GITHUB_NETWORK_TESTS' \
 	--allow-skip 'require-env VGI_TEST_ICEBERG' \
 	--allow-skip 'require-env VGI_TEST_COMPANION_TARGET' \

docs/README.md

@@ -108,7 +108,8 @@ if __name__ == "__main__":
 - **Multiple transports** — `LOCATION` accepts a bare command (subprocess, pooled per
   DuckDB process), `http(s)://` (HTTP), `tcp://host:port` (raw TCP, including DNS
   and bracketed IPv6), `unix:///path/to.sock` (AF_UNIX), or `launch:<argv>`
-  (launcher-managed shared worker).
+  (launcher-managed shared worker), plus cached workers from GitHub releases,
+  OCI images, or a [`database://` package table](database-worker-transport.md).
 - **Pushdown** — projection, filter, `ORDER BY` + `LIMIT`, and join-key pushdown to
   workers that opt in.
 - **Worker pooling** — subprocess workers are pooled and reused across queries, with
@@ -123,6 +124,7 @@ if __name__ == "__main__":
 
 See the [`docs/`](.) directory for deep dives on
 [multi-branch tables](multi_branch.md),
+[database-backed worker packages](database-worker-transport.md),
 [the launcher protocol](launcher-protocol.md),
 [catalog profiling](catalog_profiling.md), and more.