Distributed research compute mesh. Single Go binary — drop it on any machine, it finds the cluster (or starts one), advertises capabilities, and begins accepting work.
Tasks are arbitrary commands: any script, binary, or pipeline that runs on the worker's OS. Ziggurat manages workspaces, fetches inputs from content-addressed storage, executes the command, captures output, and uploads results. No SDK required.
Runs on Linux, macOS (Apple Silicon), and Windows.
Trusted-LAN by default. Out of the box, TLS, join tokens, and API auth are off and the node binds all interfaces (
0.0.0.0) — any peer that can reach the ports can join the cluster and run commands on your machines. This is intended for a trusted LAN. Enable mTLS + a join token + an API token (and firewall the ports) before exposing a node to an untrusted network — see Configuration and SECURITY.md.
Download a prebuilt binary for your platform from the latest release — linux/amd64, linux/arm64, darwin/arm64, and windows/amd64 are published, with SHA256SUMS to verify:
# Example: Linux amd64
curl -LO https://github.com/syzygyhack/ziggurat/releases/latest/download/ziggurat-linux-amd64
chmod +x ziggurat-linux-amd64
sudo mv ziggurat-linux-amd64 /usr/local/bin/ziggurat
ziggurat versionOr install from source with Go 1.24+:
go install github.com/syzygyhack/ziggurat/cmd/ziggurat@latestTo build from a checkout instead, see Building.
# Build
go build -o ziggurat ./cmd/ziggurat
# Start a node — works immediately with no config
./ziggurat start
# In another terminal: submit a task
./ziggurat run -- echo "hello world"
# Check status
./ziggurat status
# Wait for a task to complete
./ziggurat wait <task-id>To generate a config file with LAN defaults (optional):
./ziggurat init # writes ~/.ziggurat/ziggurat.yaml# On the first machine
./ziggurat start
# On additional machines — join via seed address
./ziggurat start --join 10.0.0.5:7102Nodes discover each other via gossip (memberlist). By default every node runs as hybrid (coordinator + worker). Use --role to run dedicated roles:
# Dedicated coordinator (schedules tasks, doesn't execute them)
./ziggurat start --role coordinator
# Dedicated worker (executes tasks, doesn't schedule)
./ziggurat start --role worker --join 10.0.0.5:7102One node per machine. ziggurat start refuses to launch if another node is already running on the same OS (a node manages all of the machine's cores, RAM, and GPUs by itself). For local multi-node development, run nodes on Windows + WSL or on separate machines — both use distinct lock paths and join normally.
Ziggurat works out of the box with no config file. To customize, run ziggurat init to generate a commented template at ~/.ziggurat/ziggurat.yaml, then edit as needed.
Config file search order (first found wins):
--config <path>flag./ziggurat.yaml(current directory)~/.ziggurat/ziggurat.yaml(data directory)- Built-in defaults (if no file exists)
node:
name: gpu-workstation
role: hybrid # hybrid (default), coordinator, worker
tags: [gpu, cuda, python3]
data_dir: ~/.ziggurat
capabilities:
custom.key: value
network:
bind: 0.0.0.0
http_port: 7100 # REST API
grpc_port: 7101 # inter-node transport
gossip_port: 7102 # memberlist gossip
cluster:
name: default
seeds: [10.0.0.5:7102]
storage:
replication_factor: 2
capacity: 53687091200 # 50 GB
gc_grace_period: 1h
erasure:
enabled: true
data_shards: 4
parity_shards: 2
compute:
concurrency: 4 # max parallel tasks (0 = NumCPU)
task_timeout: 5m
max_output_size: 1073741824 # 1 GB
cancel_grace: 10s
max_retained_workspaces: 20
env_max_age: 168h # 7 days
env_max_count: 50
resilience:
task_retries: 2
dead_letter: true
max_queue_depth: 0 # 0 = unlimited
security:
tls:
enabled: false # mTLS for inter-node gRPC
join_token: "" # shared secret for cluster join
api_token: "" # bearer token for HTTP API
log:
format: text # text | json
level: info # debug | info | warn | errorConnection resolution for client commands: --addr flag > ZIGGURAT_ADDR env > config client.addr > 127.0.0.1:7100.
Before leaving the LAN, set all three. The security defaults above (
tls.enabled: false, emptyjoin_token, emptyapi_token) are for a trusted local network. To harden a node: enable mTLS (security.tls.enabled: true), generate a cluster join token (ziggurat token generate), set an API bearer token, and firewall the gossip/gRPC/HTTP ports to known peers. See SECURITY.md.
Each node advertises capabilities — auto-detected facts plus operator-declared values — and tasks are routed only to nodes that satisfy their requirements. Matching happens both at the coordinator (candidate selection) and at the receiving worker, so a task never runs on an ineligible node.
Auto-detected (no config needed):
| Capability | Example | Source |
|---|---|---|
os, arch, cpu.cores |
linux, amd64, 24 |
runtime (cpu.cores honors cgroup CPU quota) |
mem.total, disk.avail, storage.class |
bytes; ssd |
OS probes (mem.total honors cgroup limit) |
gpu.count, gpu.vram, gpu.vram.max, gpu.model, gpu.vendor, gpu.cuda, gpu.driver |
2, total bytes, largest-device bytes, NVIDIA …, nvidia |
nvidia-smi/nvcc; AMD rocm-smi, Intel xpu-smi (best-effort) |
gpu.<i>.model, gpu.<i>.vram |
per-device | as above |
container.runtime |
podman / docker |
PATH lookup |
compute.concurrency |
16 |
task slots |
ziggurat.version |
0.3.0 |
build version (tagged builds only) |
python.version, node.version, go.version, java.version, ruby.version, rust.version |
3.12.1, 20.11.0, … |
runtime --version probes |
Gate work on the agent's build version with --constraint "ziggurat.version >= 0.3.0" — useful when a task relies on a feature added in a specific release. Only tagged builds (semver from git describe) advertise ziggurat.version; dev/untagged builds omit it, so version-gated tasks safely never land on them.
gpu.vram is the sum across devices; use gpu.vram.max (largest single device) to require one GPU big enough for a job, e.g. --constraint "gpu.vram.max >= 16GB".
Operator-declared node.tags and node.capabilities are merged in and override auto-detected values for the same key. For anything the built-in detectors don't cover (installed packages, custom tools), declare capability probes — commands whose output becomes a capability:
node:
capability_probes:
- capability: torch.version
command: ["python3", "-c", "import torch; print(torch.__version__)"]
version: true # extract a dotted version from the output
- capability: ffmpeg # without `version`, the first output line is used
command: ["ffmpeg", "-version"]
version: trueThen route with --constraint "torch.version >= 2.1". Container-image tasks (--image) are automatically restricted to nodes advertising container.runtime.
Three ways to require capabilities when submitting a task:
# Tags (presence flags) — node must carry every required tag
ziggurat run --require gpu --require python3 -- python train.py
# Capability constraints (expressions: == != >= <= > <; version- and byte-size-aware)
ziggurat run --constraint "python.version >= 3.10" -- python script.py
ziggurat run --constraint "gpu.vram >= 16GB" -- ./infer
# Resources (admission + live capacity) — routes to nodes with enough free CPU/mem/GPU
ziggurat run --gpus 1 --memory 8GB -- ./gpu-jobA GPU job (--gpus N) is placed only on nodes with enough free GPUs, which then reserve devices and expose them via CUDA_VISIBLE_DEVICES. A task whose requirements no node satisfies stays queued until an eligible node appears.
Note: capability requirements are explicit — Ziggurat does not infer them from the command (a
.pyargument does not auto-require Python). Declare what a task needs with--require/--constraint/--gpus.
Ziggurat dispatches arbitrary commands and guarantees at-least-once execution — not exactly-once. Before submitting destructive or stateful work, understand what this means.
At-least-once delivery. When a node leaves the cluster, the coordinator re-queues the tasks it was running so they retry on another capable node. On a clean crash the departed node has genuinely stopped, so the task runs exactly once in practice. But during a network partition the original node may still be alive and still executing while unreachable from the coordinator — which cannot tell a partition apart from a crash, so it re-dispatches the task. The task can then run a second time, concurrently, on a different node. Task retries (resilience.task_retries, default 2) can likewise re-run a task that already produced side effects. Exactly-once execution would require execution leases / fencing tokens and is not currently implemented (Phase 2 roadmap).
Write idempotent tasks. Design every task to tolerate running more than once:
- Prefer content-addressed outputs. Writing results back through Ziggurat's store keys objects by content hash, so duplicate runs producing identical bytes deduplicate naturally.
- Make external side effects idempotent. Use deterministic output keys/paths, upserts instead of appends, conditional writes, or a unique task-scoped key so a re-run overwrites rather than doubles up.
- Assume concurrency. A duplicate may run at the same time as the original, not just after it. Guard shared external resources accordingly.
Warning — non-idempotent and destructive commands. Commands whose effect changes when run twice are not safe to submit as-is: appending to a file/ledger, incrementing counters, sending notifications/emails, charging/billing, irreversible deletes (rm -rf, DROP TABLE), or non-transactional multi-step migrations. A duplicate run can corrupt data or produce wrong results. If a task cannot be made idempotent, gate it behind your own external locking or run it outside the mesh.
| Command | Description |
|---|---|
ziggurat init |
Generate default config at ~/.ziggurat/ziggurat.yaml |
ziggurat start |
Start a node (hybrid role) |
ziggurat start --role worker --join <addr> |
Start a dedicated worker and join cluster |
ziggurat drain |
Stop accepting work, finish in-flight tasks |
ziggurat resume |
Resume task dequeuing after drain |
ziggurat status |
Cluster dashboard |
ziggurat nodes |
List cluster members |
ziggurat version |
Binary version |
ziggurat token generate |
Generate a cluster join token |
| Command | Description |
|---|---|
ziggurat run -- <cmd> |
Submit a task |
ziggurat run --image <ref> -- <cmd> |
Submit a task in an OCI container |
ziggurat run --wait -- <cmd> |
Submit and wait for result |
ziggurat tasks |
List tasks (filterable: --status running) |
ziggurat task <id> |
Task detail + stdout/stderr |
ziggurat logs <id> |
Stream live stdout/stderr (SSE) |
ziggurat cancel <id> |
Cancel a task |
ziggurat wait <id> |
Block until task completes |
ziggurat batch --from <file> |
Submit batch from JSON/YAML |
ziggurat sweep --grid k=v1,v2 -- <cmd with ${k}> |
Fan out a command template over a parameter grid |
ziggurat dead-letter |
List dead-lettered tasks |
ziggurat run --env <name> --env-setup <cmd> |
Submit with persistent environment |
| Command | Description |
|---|---|
ziggurat pipeline submit <file> |
Submit a pipeline definition |
ziggurat pipeline status <id> |
Pipeline status + stage details |
ziggurat pipeline cancel <id> |
Cancel all stages |
ziggurat pipeline retry <id> |
Retry from first failed stage |
| Command | Description |
|---|---|
ziggurat env list |
List persistent environments on this node |
ziggurat env prune |
Remove stale environments |
| Command | Description |
|---|---|
ziggurat put <key> <file> |
Upload object |
ziggurat get <key> [dest] |
Download object |
ziggurat ls [prefix] |
List objects |
ziggurat rm <key> |
Delete object |
ziggurat pin <key> |
Pin object (prevent GC) |
ziggurat unpin <key> |
Unpin object (allow GC) |
ziggurat mount <path> |
FUSE-mount the store at a directory |
| Command | Description |
|---|---|
ziggurat top |
Live cluster dashboard (nodes, tasks, load) |
ziggurat top --once |
Single snapshot, no refresh |
ziggurat benchmark |
CPU, memory, disk I/O, GPU detection, and peer latency benchmarks |
ziggurat benchmark --skip-network |
Local benchmarks only |
All commands support --json for machine-parseable output.
ziggurat/
cmd/ziggurat/ # main entry point
internal/
api/ # HTTP REST API (chi router)
benchmark/ # Local + network benchmarks, GPU detection
cluster/ # Gossip membership (memberlist)
cmd/ # CLI commands (cobra)
config/ # YAML config loading + defaults
coord/ # Coordinator: task queue, scheduling, pipelines
metrics/ # Prometheus metric definitions
model/ # Shared types (Task, Node, Pipeline, etc.)
node/ # Node lifecycle, capability detection
scheduler/ # Locality + load-balanced scoring
store/ # Content-addressed storage (BLAKE3, BoltDB)
transport/ # gRPC inter-node transport (protobuf)
worker/ # Process execution engine
proto/ # Protobuf definitions
docs/ # Spec and UX design docs
- Content-addressed storage: BLAKE3 hashing, integrity verified on read (verifyingReader), deduplication via refcounting
- Deterministic tar: Sorted entries, normalized metadata (uid/gid 0, epoch mtime, fixed mode) — identical content always produces identical hashes across platforms
- Pull-based local scheduling: Workers poll the coordinator via Dequeue; natural load balancing without push complexity
- Push-based cross-node dispatch: Coordinator dispatches tasks to remote workers via gRPC, collects results, and replicates output back to the origin node. A hybrid node keeps runnable work local for locality, but offloads overflow — when its local worker is saturated and a peer has spare capacity, queued tasks spill to the peer, so a single submit point scales across the mesh.
- Persistent environments: Fingerprint-based reuse of task environments (venvs, node_modules, etc.) — same deps = same env across tasks, rebuilt only when fingerprint changes
- Pipeline DAGs: Kahn's algorithm cycle detection,
$stage.outputreference resolution, transitive failure cancellation - Platform-split process management:
process_unix.go(SIGTERM/SIGKILL via process groups) andprocess_windows.go(CREATE_NEW_PROCESS_GROUP + TerminateProcess)
See Task Safety & Execution Semantics for the at-least-once / duplicate-execution guidance task authors need; this section covers the cluster-internal mechanics.
When a node leaves the cluster, the coordinator re-queues the tasks it was running (RUNNING/SCHEDULED) so they can be retried on another capable node; its load tracking and stale shard placements are cleaned up and a repair pass restores replication. Graceful shutdowns broadcast a clean departure; crashes are caught by the gossip failure detector. Because re-queuing can re-run a task that was still executing during a partition, execution is at-least-once — see Task Safety & Execution Semantics.
Ziggurat runs on Linux, macOS (Apple Silicon), and Windows. macOS and Linux share the Unix code paths; only memory/disk detection differs. Platform-specific code is isolated via build tags:
| Concern | Linux | macOS | Windows |
|---|---|---|---|
| Process groups | Setpgid + SIGTERM/SIGKILL |
Setpgid + SIGTERM/SIGKILL |
CREATE_NEW_PROCESS_GROUP + TerminateProcess |
| Graceful shutdown signals | SIGINT + SIGTERM |
SIGINT + SIGTERM |
SIGINT only |
| Memory detection | syscall.Sysinfo (cgroup-aware) |
sysctl hw.memsize |
GlobalMemoryStatusEx |
| Disk space detection | syscall.Statfs |
unix.Statfs |
GetDiskFreeSpaceExW |
File permissions (0o755/0o644) are specified but only enforced on Linux and macOS; Windows ignores them. Tar archives use forward-slash paths regardless of OS for cross-platform determinism. The FUSE mount (ziggurat mount) requires a kernel FUSE provider — built in on Linux, macFUSE on macOS; it is not available on Windows.
- Commands must be real executables. Task commands are executed directly, not through a shell. On Windows, shell built-ins like
echo,dir,set, andtypeare not programs and will fail with "executable file not found". Wrap them:ziggurat run -- cmd /c echo hello(or invoke an interpreter, e.g.python script.py). - Firewall. Zero-config LAN discovery needs inbound traffic allowed for the gossip port (TCP+UDP
7102), the gRPC port (7101), the HTTP API (7100), and mDNS (UDP5353). On first run, allowziggurat.exethrough Windows Defender Firewall on private networks, or peers won't discover/reach each other. - Advertise address. If the machine has Docker or a WSL2 bridge on a
172.16–31.x.xsubnet, the node may auto-select that address — unreachable from other machines. Ziggurat warns when it detects this range; setnetwork.advertiseto the machine's real LAN IP in~/.ziggurat/ziggurat.yaml. - Persistent environments. Python venvs on Windows place executables in
Scripts\(vsbin/on Unix); both are added toPATHautomatically.
By default WSL2 uses NAT networking: the Linux instance sits on a private subnet (e.g. 192.168.239.x / 172.x) behind the Windows host, not on your LAN. A node started inside WSL2 can reach the mesh outbound (so --join and even task execution appear to work), but it advertises its private WSL address, which other physical machines cannot route to. The node ends up only half-connected: reachable from its own host, invisible to the rest of the LAN. mDNS auto-discovery also won't cross the NAT boundary.
Note that the private subnet is often a 192.168.x range indistinguishable from a real LAN by address alone, so Ziggurat cannot reliably warn about it — the symptom is simply that peers on other machines never see the WSL node.
To make a WSL2 node a real LAN member, choose one:
- Mirrored networking (recommended, Windows 11 22H2+). Give WSL the host's LAN identity. In
C:\Users\<you>\.wslconfig:Then[wsl2] networkingMode=mirrored
wsl --shutdownand restart. The WSL node now joins and is discovered like a native host. - Run the node natively on Windows instead of inside WSL — simplest, and the intended deployment.
- Manual port-forward (advanced). Keep NAT, set
network.advertiseto the Windows host's LAN IP, and forward the gossip/gRPC/HTTP ports from the host into WSL withnetsh interface portproxy. Fiddly and easy to get wrong; prefer options 1–2.
Prometheus metrics are served at /metrics on the HTTP port (default 7100):
ziggurat_tasks_submitted_total— counterziggurat_tasks_completed_total{status}— counter by terminal statusziggurat_task_duration_seconds— histogram (1s to ~4.5h buckets)ziggurat_task_queue_depth— gaugeziggurat_workers_active— gaugeziggurat_store_objects_total— gaugeziggurat_store_bytes_total— gaugeziggurat_nodes_total{role}— gauge by role (hybrid/coordinator/worker/all)
Task execution, storage, API, CLI.
Gossip discovery, replication, distributed scheduling, gRPC transport, failure detection + task requeue.
Streaming I/O, storage repair loop, dead letter queue, batch submission, Prometheus metrics, pipelines, cross-node dispatch with output replication.
The following features are implemented and exercised on a trusted LAN: shard rebalancing on join, drain with shard migration, resource-aware scheduling (memory/CPU/GPU admission), work stealing from overloaded workers, mDNS auto-discovery (_ziggurat._tcp.local), remote cancel propagation via gRPC, schema versioning for BoltDB, integration test harness, task log streaming (SSE), container execution (OCI via Podman/Docker), mTLS + join tokens + API bearer auth.
Parametric fan-out (ziggurat sweep) is complete. Remaining work, grouped by theme:
- Data & reproducibility (P0): global consumables/semaphores, provenance receipts, opt-in memoization (content-addressed result cache), reduce/gather primitive.
- Scavenging & reactivity: opportunistic idle-harvest + owner-yield, reactive watch-triggered tasks.
- Infrastructure & hardening: coordinator failover (Raft), speculative execution, cross-cluster federation + burst, Python client, encryption at rest, cgroup limits + isolation hardening (gates untrusted/multi-tenant use).
See docs/spec.md § Phase 2 Planning for rationale, prioritization, and the "smarter mesh, not bigger nodes" direction.
go build -o ziggurat ./cmd/ziggurat
# With version info (also sets the ziggurat.version capability for version gating)
go build -ldflags "-X github.com/syzygyhack/ziggurat/internal/version.Version=$(git describe --tags --always) -X github.com/syzygyhack/ziggurat/internal/version.Commit=$(git rev-parse --short HEAD)" -o ziggurat ./cmd/zigguratOr use the Makefile (injects version/commit automatically):
make build # build for the host OS (ziggurat, or ziggurat.exe on Windows)
make install # build and copy to ~/.local/bin (%USERPROFILE%\.local\bin on Windows)
make windows # cross-compile ziggurat.exe for Windows amd64
make dist # cross-compile all release targets into dist/make dist produces binaries for linux/amd64, linux/arm64, darwin/arm64, and
windows/amd64 — the same set published as assets on each tagged GitHub Release.
The Makefile works both under a POSIX shell (Linux, macOS, Git Bash/MSYS2) and under
cmd.exe/PowerShell on Windows. FUSE mount is excluded from Windows builds.
Requires Go 1.24+. No CGo dependencies.
go test ./...Licensed under the Apache License, Version 2.0. See LICENSE for the full text.