Skip to content

Latest commit

 

History

History
589 lines (395 loc) · 40.9 KB

ops.md

File metadata and controls

589 lines (395 loc) · 40.9 KB

Operating Kazari

Kazari keeps all its state in one directory and runs its own maintenance jobs. This page covers the data layout, backing up and restoring, upgrading, logs, metrics and tracing, the scheduled jobs and the page lifecycle, quotas, disabling users, responding to abuse, the audit log and troubleshooting. It assumes an instance set up as in the installation guide.

Data layout

Everything Kazari keeps lives under data_dir, which the systemd unit and the Docker image set to /var/lib/kazari. Under systemd's DynamicUser, the real path is /var/lib/private/kazari. The directory holds:

Path Contents
kazari.db, kazari.db-wal, kazari.db-shm SQLite in WAL mode: users, keys, artifacts, versions, file lists, OAuth state, audit log
blobs/<aa>/<sha256> File contents, named by their SHA-256; aa is its first two hex digits
uploads/<id>/<n> Chunked uploads in progress. Disposable
certmagic/ ACME account, certificates and locks (tls.mode: acme only)
admin.sock The admin socket, mode 0600; recreated at every start
tmp/ Multipart bundles spilled to disk while they are published, removed afterwards. Disposable

Kazari writes each blob to a temporary .put-* file and renames it, so a partial blob is never visible.

Backing up and restoring

The database and the blobs are the state. Back up the database first, with SQLite's online backup, which is consistent while Kazari is running, then copy the blobs straight after:

cd /var/lib/private/kazari
sqlite3 kazari.db ".backup /backup/kazari/kazari.db"
rsync -a --delete blobs/ /backup/kazari/blobs/
rsync -a certmagic/ /backup/kazari/certmagic/     # optional; saves a re-issue after a restore

uploads/ and tmp/ need no backup.

Warning

Do not copy kazari.db with cp or rsync while the server runs. The WAL makes that copy inconsistent; use SQLite's .backup as above.

The database first and the blobs second gives a complete backup. Blobs are immutable and deleted only once no version references them, so a later blob copy holds everything the database snapshot refers to. The only gap is blobs of artifacts purged in between, which the snapshot no longer needs once it is restored and swept.

Restoring from a backup

Restore with the server stopped, the blobs first and the database second:

sudo systemctl stop kazari
cd /var/lib/private/kazari
rsync -a --delete /backup/kazari/blobs/ blobs/
rm -f kazari.db-wal kazari.db-shm
cp /backup/kazari/kazari.db kazari.db
rsync -a /backup/kazari/certmagic/ certmagic/
sudo systemctl start kazari

Restoring the blobs before the database means no restored row ever points at a missing file. Extra blobs are harmless: the gc job removes the ones nothing references.

Upgrading

Kazari applies schema migrations at start. They are embedded in the binary, each runs in its own transaction, and they only go forward: to downgrade, restore the backup taken before the upgrade.

To upgrade an instance that runs under systemd, back up as in backing up and restoring, then install the new release the way you installed the old one:

# from the package: the service restarts itself
sudo apt install ./kazari_<version>_amd64.deb

# from the install script, then restart
curl -fsSL https://kazari.io/install.sh | sh
sudo systemctl restart kazari

The package keeps /etc/kazari/kazari.yaml and /etc/kazari/kazari.env as they are. Installing by hand follows installing the binary, with the checksums and signature verified, then a restart.

Upgrading under Docker Compose

With KAZARI_VERSION pinned in .env, set it to the new version; with the default latest, nothing changes. Then pull and recreate:

docker compose pull
docker compose up -d

The container keeps its data in the volume kazari_data.

Logs

Kazari logs to stderr through log/slog; under systemd, that is the journal. log.format sets the format, text by default or json, and log.level sets the level, info by default. Every request produces one line when it completes:

time=2026-10-24T09:12:03Z level=INFO msg=request method=GET host=q3-report.art.example.com path=/ status=200 bytes=5120 duration_ms=1.8 request_id=6f1c0e9b2a7d4c33a1e0f5b6c7d8e9f0 ip=198.51.100.7 user_agent="Mozilla/5.0 ..."

Each line carries method, host, path, status, bytes, duration_ms, request_id, ip and user_agent. In proxy mode, ip comes from X-Forwarded-For or X-Real-IP. user_agent is cut to 200 bytes.

5xx responses log at ERROR, the rest at INFO. API 5xx errors also log their cause on a separate line (httpapi: request failed); the client only ever sees internal error.

A successful request for /health, /v1/health or /metrics logs at DEBUG instead, so health checks and scrapes every few seconds do not bury the other requests. A failed one logs as usual, and on artifact hosts these paths are pages like any other.

Request ids

The request id comes from an incoming X-Request-ID header of 8 to 128 safe characters, or Kazari generates one. Either way, it is echoed in the X-Request-ID response header. Quote it when reporting a problem.

Trace ids

When tracing is on (tracing), every line logged while a sampled request or job runs ends with its trace_id and span_id, so a log line leads to its trace and a trace to its lines:

time=2026-11-04T09:12:03Z level=INFO msg=request method=POST host=mcp.art.example.com path=/ status=200 ... trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7

Lines of requests that were not sampled carry neither.

Exporting logs over OTLP

To send a copy of the log to the OpenTelemetry endpoint as well, set telemetry.logs.otlp: true. Records at log.level and above are exported, or at telemetry.logs.level when it is set; stderr is unchanged. Each record carries its trace and span ids as fields, so the backend links it to its trace.

Exported records leave out the keys that name a person or a credential: ip, email, subject, handle, user, actor and family. On the admin API, the handle and credential id in path read REDACTED, as does text quoted in an error message, which is where errors name a user or a key. The journal keeps all of them. User agents are still exported in the access log's user_agent; drop the attribute in your collector if it should not reach a hosted backend.

Messages from libraries

The OpenTelemetry SDK and gRPC, which export telemetry over OTLP, log through the same handler in the same format, marked logger=otel or logger=grpc. Their errors log at WARN, like Kazari's own telemetry problems. Their warnings, such as each failed attempt to reach a gRPC collector, log at DEBUG.

Finding errors

Under systemd, each line starts with its syslog priority, which the journal records and removes, so journalctl -p filters by level. To list the errors, every 5xx response among them, then the warnings as well, run:

journalctl -u kazari -p err
journalctl -u kazari -p warning

To find the certificate messages, filter by text:

journalctl -u kazari -o cat | grep -i certificate

Kazari adds the priority only when stderr is the journal stream that systemd names in JOURNAL_STREAM. Elsewhere, such as under Docker, filter on the level instead:

docker compose logs kazari | grep -E 'level=(WARN|ERROR)'

Metrics

Kazari serves Prometheus metrics at /metrics on the admin host. The names follow the OpenTelemetry semantic conventions, translated the Prometheus way: dots become underscores, the unit becomes a _seconds or _bytes suffix, and counters end in _total.

Where /metrics is served depends on admin.listen:

  • With admin.listen set, it is served at http://<admin.listen>/metrics with no credential, and not on the public listener.
  • Without it, https://admin.<base>/metrics needs Authorization: Bearer <telemetry.metrics.token>. While the token is empty, the public listener does not serve /metrics at all.
  • The admin socket always serves it with no credential, and so does a localhost base domain.

The token keeps the metrics, and the trace ids attached to them as exemplars, for your own scraper. To generate one, run:

openssl rand -hex 32

Set the result as KAZARI_TELEMETRY_METRICS_TOKEN in the dotenv file and restart.

Available metrics

Kazari exports the metrics below, under their Prometheus names. Over OTLP they keep the OpenTelemetry names, with dots: http.server.request.duration, kazari.artifact.publishes and so on. Every label comes from a fixed set, so a client cannot create new series; a value outside the set is _OTHER.

These describe requests on every surface:

Metric Type Labels
http_server_request_duration_seconds histogram kazari_surface, http_request_method, http_response_status_code, http_route, url_scheme, network_protocol_version, error_type
http_server_active_requests gauge kazari_surface, http_request_method, url_scheme
http_server_response_body_size_bytes histogram As http_server_request_duration_seconds; its sum is each surface's bandwidth
kazari_api_errors_total counter kazari_surface (api, admin), kazari_error_code
kazari_auth_requests_total counter kazari_surface, kazari_credential_type (api_key, oauth, admin_token, local), kazari_auth_result (ok, missing, invalid, forbidden)
kazari_rate_limit_rejected_requests_total counter kazari_surface, kazari_rate_limit_scope (ip, credential)

These describe MCP and what agents do:

Metric Type Labels
mcp_server_operation_duration_seconds histogram mcp_method_name, gen_ai_tool_name, gen_ai_operation_name, error_type, rpc_response_status_code, mcp_protocol_version, kazari_error_code, kazari_credential_type, kazari_mcp_input_required
kazari_mcp_tool_arguments_size_bytes histogram gen_ai_tool_name: what clients send inline instead of uploading
kazari_mcp_tool_result_size_bytes histogram gen_ai_tool_name: what a call adds to the model's context
kazari_artifact_publishes_total counter kazari_publish_kind (create, update, edit, import, rollback), kazari_surface, error_type
kazari_artifact_publish_size_bytes histogram kazari_publish_kind
kazari_artifact_actions_total counter kazari_audit_action, kazari_surface (api, mcp, admin, scheduler)
kazari_artifact_edits_total counter kazari_edit_kind (text, anchor), kazari_edit_match (exact, trailing_whitespace, indentation, anchor)
kazari_artifact_edit_retries_total counter None: edits applied again after a concurrent publish
kazari_artifact_imports_total counter error_type
kazari_share_link_operations_total counter kazari_share_link_operation (create, revoke, open), error_type
kazari_upload_chunks_total counter error_type
kazari_fetch_duration_seconds histogram kazari_fetch_purpose (file, import, cimd, oidc), error_type (a status code, refused, timeout, too_large)
kazari_fetch_body_size_bytes histogram kazari_fetch_purpose

These describe sign-in and access:

Metric Type Labels
kazari_gate_requests_total counter kazari_gate_result (public, member, share_link, redirect, denied)
kazari_user_sign_ins_total counter kazari_sign_in_method (oidc, api_key), kazari_sign_in_result (ok, not_admitted, failed)
kazari_oauth_token_requests_total counter kazari_oauth_grant_type, error_type (the RFC 6749 error code)
kazari_oauth_client_registrations_total counter kazari_oauth_registration_method (dcr, cimd), error_type

These describe the instance's state, read from the database at most every 30 seconds:

Metric Type Labels
kazari_artifact_count gauge kazari_artifact_status (published, stale, unpublished, deleted, purged), kazari_artifact_visibility
kazari_artifact_version_count gauge None: every version of every artifact
kazari_artifact_version_usage_bytes gauge None: their logical size, which grows with every edit
kazari_blob_count, kazari_blob_usage_bytes gauge None: the blobs on disk, stored once however many versions share them
kazari_user_count gauge kazari_user_state (active, disabled), kazari_user_role (admin, member)
kazari_upload_count gauge None: chunked uploads in progress
db_client_connection_count gauge db_client_connection_state (idle, used), db_client_connection_pool_name
db_client_connection_max gauge db_client_connection_pool_name

These describe the server itself:

Metric Type Labels
kazari_job_duration_seconds histogram kazari_job_name, error_type
kazari_job_last_success_timestamp_seconds gauge kazari_job_name
kazari_certificate_expiry_timestamp_seconds gauge None: the served certificate; absent in proxy mode
otel_sdk_exporter_span_exported_total, otel_sdk_exporter_metric_data_point_exported_total, otel_sdk_exporter_log_exported_total counter otel_component_type, server_address, error_type: items exported over OTLP, and failures
go_* various The Go runtime: go_memory_used_bytes, go_goroutine_count, go_schedule_duration_seconds and others
process_* various CPU, memory and file descriptors of the process
target_info gauge The service: service_name, service_version, service_instance_id, host_name

kazari_surface is site (artifact pages), api, mcp, admin, apex (the base domain itself) or redirect (the plain-HTTP listener that redirects to HTTPS). http_route is the route that matched, such as /v1/artifacts/{slug} or /{path...} for an artifact's files; it is absent when no route matched. error_type is set on failures only, and kazari_error_code is Kazari's error code, such as version_conflict or quota_exceeded.

A histogram's _count counts requests, calls or runs, so no separate counter exists: the request rate is rate(http_server_request_duration_seconds_count[5m]).

To push the same metrics over OTLP as well, set telemetry.metrics.otlp: true (telemetry.metrics). Process metrics stay Prometheus-only; a collector's hostmetrics receiver covers them for an OTLP backend.

Scraping

On the public listener, the Prometheus scrape job reads the token from a file:

scrape_configs:
  - job_name: kazari
    scheme: https
    authorization:
      credentials_file: /etc/prometheus/kazari-token
    static_configs:
      - targets: ["admin.art.example.com"]

With admin.listen set to "127.0.0.1:9443", no token is needed:

scrape_configs:
  - job_name: kazari
    static_configs:
      - targets: ["127.0.0.1:9443"]      # admin.listen: "127.0.0.1:9443"

To keep the trace ids that link a latency bucket to a trace, start Prometheus with --enable-feature=exemplar-storage.

Alerts

These alerts catch what usually goes wrong:

groups:
  - name: kazari
    rules:
      - alert: KazariCertificateExpiring
        expr: kazari_certificate_expiry_timestamp_seconds - time() < 14 * 86400
      - alert: KazariJobNotSucceeding
        # three intervals of the slowest job, gc, which runs hourly
        expr: time() - kazari_job_last_success_timestamp_seconds > 3 * 3600
      - alert: KazariServerErrors
        expr: sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) > 0.1
      - alert: KazariRefusedAuthentications
        expr: sum(rate(kazari_auth_requests_total{kazari_auth_result="invalid"}[15m])) > 1
      - alert: KazariToolErrors
        expr: |
          sum by (gen_ai_tool_name) (rate(mcp_server_operation_duration_seconds_count{error_type!=""}[1h]))
            / sum by (gen_ai_tool_name) (rate(mcp_server_operation_duration_seconds_count[1h])) > 0.2
      - alert: KazariTelemetryExportFailing
        expr: sum(rate(otel_sdk_exporter_span_exported_total{error_type!=""}[15m])) > 0

KazariJobNotSucceeding uses the hourly gc job; with other lifecycle intervals, use three of the longest. A tool's error ratio is worth reading by kazari_error_code: version_conflict means agents edit concurrently, anchor_mismatch that they edit from stale reads.

Upgrading from 0.2

Kazari 0.3 renamed every metric, and https://admin.<base>/metrics now needs the token described above. Update the scrape job and these queries:

0.2 0.3
kazari_http_requests_total{surface, method, status} http_server_request_duration_seconds_count{kazari_surface, http_request_method, http_response_status_code}
kazari_http_request_duration_seconds{surface} http_server_request_duration_seconds, with the buckets of the semantic conventions
kazari_http_requests_in_flight{surface} http_server_active_requests{kazari_surface, http_request_method, url_scheme}
kazari_job_runs_total{job, result} kazari_job_duration_seconds_count{kazari_job_name, error_type}
kazari_job_duration_seconds{job} The same name, with the label kazari_job_name
go_* from the Prometheus client go_* with semantic-convention names, such as go_memory_used_bytes and go_goroutine_count

The old job label collided with Prometheus's own target label, which stored it as exported_job; kazari_job_name does not.

Tracing

Kazari traces each request and background job with OpenTelemetry and exports the spans over OTLP to a collector or tracing vendor, such as Grafana Tempo, Honeycomb or Datadog. A trace shows what an agent's request did, where the time went and why it failed.

Turning tracing on

Tracing starts as soon as an endpoint is set. To send spans to a collector on the same Docker network, set:

telemetry:
  otlp:
    endpoint: http://otel-collector:4318

A vendor usually needs an API token as a header and an https endpoint (telemetry.otlp). The standard variables work too: with telemetry.otlp.endpoint unset, Kazari exports to OTEL_EXPORTER_OTLP_ENDPOINT, as any OpenTelemetry service does.

To check the export, look for the line telemetry: exporting over OTLP at start, which names each signal's endpoint and protocol. Then open a page and an API route, and search the backend for the service kazari. Spans leave in batches every few seconds.

What is traced

Every request gets a server span named after its method and route, such as GET /v1/artifacts/{slug} or GET /{path...} for an artifact's files. The span carries the status code, the route, the surface (kazari.surface), the request id and, on the control plane, the credential type and OAuth client. Scheduled jobs get a span named job <name>, and startup a kazari.start span with the database migration and OIDC discovery under it.

On mcp., each JSON-RPC method gets its own span under the request's, named as the MCP semantic conventions do: tools/call create_artifact for a tool call, or the method alone, such as tools/list, resources/read or server/discover, the call a client of protocol version 2026-07-28 opens with. Kazari advertises no list_changed notifications, so such a client never holds a subscriptions/listen stream open; a request on mcp. lasts as long as its tool call. A tool span carries the tool, the negotiated protocol version, the sizes of the arguments and the result, and the outcome: a tool that failed is error.type=tool_error with Kazari's error code in kazari.error.code; one that asked the user for confirmation has a kazari.mcp.input_request event. Tool arguments and results themselves are never recorded.

Below the request, each change or lookup has a span named for the operation, the same whether it came over REST or MCP: artifact.create, artifact.edit, upload.append_chunk, oauth.token and so on, with the slug, version, file count and size. Under those come a span per store method, such as ArtifactStore.AddVersion, and per blob read or write (blob.put, blob.open), which show where a request's time went. A fetch for an import, a file named by URL, a client metadata document or the identity provider adds a GET client span per attempt, redirects included, with the host and the URL's query values hidden.

An MCP client can also pass traceparent in a request's params._meta. The tool span then joins that trace, with a link to the HTTP request's span.

A server error (5xx) marks the span as failed, with error.type set to Kazari's error code. A panic adds an exception event. Requests for /health, /v1/health and /metrics are counted but not traced.

Sampling

By default, Kazari keeps every trace that starts on api., mcp., admin. or the admin socket, and one in a hundred on artifact hosts and the base domain. Errors on artifact hosts are still counted in full by the metrics; to keep every failing page view as a trace, sample in the collector instead (telemetry.traces).

A trace is kept or dropped whole: a span's children follow its decision.

Running a collector

A collector between Kazari and the backend batches the export, can keep every failing request's trace however Kazari samples, and can drop attributes before they leave your network. To start from an example, fetch the configuration for the OpenTelemetry Collector's contrib distribution:

curl -fsSLO https://kazari.io/deploy/otel-collector.yaml

It receives OTLP on ports 4317 (gRPC) and 4318 (HTTP), keeps every trace with an error or a span over two seconds plus one in ten of the rest, and prints what it receives until you add your backend's exporter. Its attributes/drop-paths processor removes slugs and paths from spans, for a hosted backend.

Joining a caller's trace

On api., mcp. and admin., a request with a W3C traceparent header continues the caller's trace, so an agent runtime or proxy that traces its own requests sees Kazari's spans under them. Artifact hosts and the base domain ignore the header: their visitors are anonymous, and a parent they sent would decide sampling. Kazari never sends traceparent to the sites it fetches.

What spans never contain

Spans never contain who made a request: no handle, email, OIDC subject, client IP or credential id (API key prefix, OAuth grant id). On the admin API, the handle and key prefix in url.path read REDACTED. Query strings keep their keys and hide every value, because share tokens and OAuth codes travel there. Request and response bodies, file contents and credentials are never recorded. The audit log and the access log keep who did what, on the server.

Spans do contain slugs, versions, file paths, OAuth client ids and control-plane user agents. Traces often go to a hosted vendor, so treat them as you would the access log.

Scheduled jobs

Kazari runs four maintenance jobs on its own:

Job Interval What it does
sweep lifecycle.sweep_interval, 15 minutes by default Moves artifacts along the lifecycle: soft-deletes expired stale ones, purges soft-deleted ones past retention and removes tombstones past the tombstone period
gc lifecycle.gc_interval, 1 hour by default Deletes blobs that no version references, in batches of 1000, sparing blobs written in the last hour
uploads-expire 10 minutes Removes chunked uploads past limits.upload_ttl and staging directories left by a crash
oauth-cleanup 1 hour Deletes expired OAuth authorization requests, codes and tokens, expired browser sign-ins, sign-in codes and sessions, and OAuth clients that completed no grant within a day of registering

gc spares recent blobs so that a publish in progress never loses one. The unreferenced blobs it finds are those of purged or hard-deleted artifacts and of a publish that a crash interrupted, since a failed publish deletes its own. It also removes temporary files a crash left behind: blob writes and spilled bundles older than an hour.

Each interval is jittered by ±10%, and the first run comes after a random delay of up to 10% of the interval. A job never runs twice at once.

Checking the jobs

To see each job's interval, last run, next run and last error, list them:

kazari admin jobs list

A sweep or gc that changed something also logs a summary line, such as sweep soft_deleted=1 purged=0 freed=0.

Running a job now

To run a job now, name it. The job runs synchronously, and the command returns when it finishes:

kazari admin jobs run sweep

# the same through the admin API
curl -X POST -H "Authorization: Bearer $KAZARI_ADMIN_TOKEN" https://admin.art.example.com/v1/jobs/gc/run

Lifecycle

An artifact that is no longer needed moves through these states until its slug is free again:

published ──stale──▶ stale ──grace──▶ soft-deleted ──retention──▶ purged ──tombstone──▶ slug free
    ▲                  │                   │
    └──── unstale ─────┘                   └── restore (PUT/PATCH {"restore": true})

Each state works as follows:

  • Stale: kazari stale <slug> [--grace 72h] sets stale_at, and sets expires_at to now plus the grace period. The page keeps serving. Publishing a new version or restoring the artifact clears the mark unless keep_stale is set. A credential without artifacts:delete can mark only its own user's artifacts.
  • Soft-deleted: by rm, or by the sweep after expires_at. The page answers 404 and the slug stays taken. Until lifecycle.retention has passed since deleted_at, the owner can still read the files and restore the artifact.
  • Purged: by the sweep after retention, or at once by rm --hard. Versions and file lists are deleted, and the blobs go at the next gc. The page answers 410 Gone, and the slug stays reserved for lifecycle.tombstone.
  • Freed: the tombstone row is deleted, and the slug can be used again.

Each step lands in the audit log.

Timings

Timings run from the transition, and the sweep acts at its next run. With the default grace of 14 days, retention of 30 days and tombstone of 90 days, an artifact marked stale today is soft-deleted after 14 days, purged 30 days later, and its slug freed 90 days after that. That is 134 days in all, plus up to one sweep interval at each step.

Tuning the lifecycle

Each period is a lifecycle setting in the configuration. Restart after editing. What each one trades:

  • Shorten retention to reclaim space sooner. It is the window in which a mistaken delete can be undone.
  • Shorten tombstone to free slugs sooner. It is the window in which old links answer 410 rather than someone else's page.
  • Change default_grace to set the grace a stale call gets when it names none.

Unpublished pages

unpublished is separate from this chain. It comes from kazari visibility <slug> unpublished or admin hide. The page answers 404, nothing is deleted, and nothing moves on until someone deletes it.

Quotas

Each user has a quota, quota_bytes. A new user gets limits.default_user_quota, 2 GiB by default, and 0 means unlimited.

Usage is the sum of the sizes of every version of every artifact the user owns, counted logically, plus 4 KiB for each file of each version, whatever its size, because every version keeps a database row per file. A 5 MiB site of 20 files with ten versions uses 50.8 MiB of quota, even though unchanged files are stored once. Soft-deleted artifacts keep counting until they are purged; a hard delete frees the space at once.

Kazari checks the owner's quota whenever a version is added, whoever adds it. Under the shared write policy, a version that another member adds to someone else's artifact counts against the owner's quota too, and a version that would exceed it gets 403 quota_exceeded. An admin transfer moves the artifact's usage to the new owner.

A publish that fails keeps nothing: Kazari deletes the blobs it wrote before it answers, except any that a version uses, so a refused publish takes no disk once it has finished. While it runs, it can hold up to the owner's remaining quota or max_version_bytes, whichever is smaller.

Changing a quota

To see every user's usage and change one user's quota:

kazari admin users list                      # every user, with usage and quota
kazari admin users update ana --quota 5GiB
kazari --json admin users list | jq -r '.[].handle'   # handles only, for scripts

kazari me shows users their own usage. The per-version limits max_version_bytes, max_file_bytes and max_files apply to everyone, admins included.

Disabling users and revoking access

To block a user, disable them:

kazari admin users update ana --disable

Disabling a user revokes every API key at once, makes every OAuth access and refresh token fail, revokes every share link they created on any artifact, and refuses sign-in with the "No access" page. Links they gave to outside reviewers end too: a reviewer who still needs the page needs a new link from someone else. --enable lets them sign in again, but revoked keys and links stay revoked.

Revoking one API key

To revoke a single key, list the user's keys and revoke one by its prefix:

kazari admin keys list ana
kazari admin keys revoke ana abcdefgh

To revoke one OAuth grant instead, which also revokes every token rotated from it, use the admin API as in Revoking one client's access.

Handing over a disabled user's pages

A disabled user's artifacts keep serving. To hand one over to another user, transfer it:

kazari admin transfer q3-report peter

Responding to abuse

To take a page down at once, hide it and disable its owner:

kazari admin hide q3-report                 # unpublishes: the page answers 404 immediately
kazari admin users update ana --disable

hide deletes nothing, and the audit log records it as artifact.visibility. It is not a lock: the owner, or under the shared policy any writer, can publish the page again. That is why disabling the owner usually goes with it.

Removing the content for good

To remove the content for good, use an admin user's key, with the admin scope, on the user API, then collect the blobs:

KAZARI_TOKEN=kzk_admin_key kazari rm q3-report --hard
kazari admin jobs run gc

Blobs written in the last hour survive one more gc run.

Warning

Version hosts are served with Cache-Control: public, max-age=<lifecycle.tombstone, at most a year>, immutable, 90 days by default. Browsers and any CDN in front can keep copies of a pinned version for that long; purge them there if needed.

Audit log

Kazari records every change with its time, actor, action, slug and details. The actor is a handle, admin for the admin token and the admin socket, or scheduler for jobs; no user can have either of those handles.

Artifact events also carry credential, the credential that acted: api_key:<prefix>, oauth:<grant> or admin_token. An oauth: credential has one id per grant, the same across refreshes.

The actions, with the details each records:

Action When
artifact.create Created (details: version, size)
artifact.update Fields changed or version added (version, size when a version)
artifact.publish Published version changed (version)
artifact.stale Marked stale (expires_at)
artifact.unstale Stale mark cleared
artifact.visibility Visibility set, including unpublish and hide (visibility)
artifact.delete Soft-deleted, by a user or by the sweep (reason: expired)
artifact.restore Restored from soft delete
artifact.purge Purged, by rm --hard or the sweep (reason: retention)
artifact.free Tombstone removed, slug free
artifact.transfer Owner changed (from, to)
user.create User created by an admin, or provisioned at sign-in (via: identity)
user.update User changed, identity linked or unlinked
key.issue API key issued (prefix, label, scopes)
key.revoke API key revoked
oauth.grant Consent approved (client_id, scopes, via: oidc or api_key, and key_prefix for api_key)
oauth.revoke OAuth grant revoked
oauth.signin A browser signed in to view members pages (via: oidc or api_key, and key_prefix for api_key)
share_link.create Share link created (prefix, expires_at)
share_link.revoke Share link revoked (prefix; created_by when disabling its creator did it)

An event is written just after the change it describes, outside its transaction, so a crash in between loses that one row. The log is append-only and never pruned.

Reading the log

To read the log, filter it by time, slug or actor:

kazari admin audit --since 24h
kazari admin audit --slug q3-report --limit 50
curl -H "Authorization: Bearer $KAZARI_ADMIN_TOKEN" "https://admin.art.example.com/v1/audit?actor=ana&since=2026-10-01T00:00:00Z"

Troubleshooting

Each entry is named for the symptom. Certificate and DNS problems are in DNS and certificates, and kazari check reproduces most of them in one run.

401 Unauthorized

Kazari did not accept the credential. Check that:

  • the header is Authorization: Bearer <token>;
  • the admin token is used only on admin.<base>, not on api. or mcp.;
  • the key is not revoked or expired (kazari admin keys list <handle>), and its user is not disabled;
  • an OAuth client can refresh its access token, which lives one hour. A client that cannot refresh must sign in again.

A 403 instead means the credential is valid but lacks a scope: artifacts:delete for rm or for stale on an artifact you do not own, or the admin scope for the admin API.

413 Payload Too Large

The message names the limit that was hit:

  • a file over limits.max_file_bytes;
  • a version over limits.max_version_bytes, which counts all files of the version, not only the ones sent;
  • a chunk over limits.max_chunk_bytes;
  • the request body.

Raise the limit and restart, or split the bundle across a chunked upload.

In proxy mode, the proxy has its own limit. nginx defaults to client_max_body_size 1m, which answers 413 with an HTML page before Kazari sees the request.

404 or 410 on a page

404 with the instance page means there is no such slug, the artifact is unpublished or soft-deleted, or there is no such version on a --v<N> host. 410 Gone means the artifact was purged and its slug is still in the tombstone period.

kazari get <slug> shows which timestamps are set.

401 or a sign-in redirect on a page

The artifact's visibility is members. A browser is sent to sign in and other clients get 401; signed-in members and holders of a share link see the page.

To let in someone outside the instance, create a share link with kazari share create <slug>. To open the page to everyone, run kazari visibility <slug> public. If the redirect never ends, see A members page keeps redirecting to sign-in.

A CDN serves a members page without sign-in

Responses of members pages are Cache-Control: private, no-cache, so a shared cache does not keep them. Copies it took while the page was still public stay, though, and pinned versions are cached for up to lifecycle.tombstone.

When you switch a page from public to members, purge https://<slug>.<base>/ and https://<slug>--v*.<base>/ at the CDN.