This document covers the high-level design of Veloria, a code search engine for WordPress extensions (plugins, themes, and cores).
Veloria is a single binary (cmd/veloria/) dispatched by Kong. Running veloria with no subcommand defaults to serve.
Start the HTTP server (default command).
veloria [serve]
Loads configuration from environment variables (and .env in development), connects to PostgreSQL and S3/MinIO, starts the background indexer, and serves the HTTP API and web UI. Shuts down gracefully on SIGINT/SIGTERM.
Download a ZIP, extract source files, and build a trigram search index.
veloria index --repo <type> --slug <name> --zipurl <url>
| Flag | Required | Description |
|---|---|---|
--repo |
No | Repository type: plugins, themes, or cores (default: plugins) |
--slug |
Yes | Extension slug — used as the folder name under source/ and index/ |
--zipurl |
Yes | URL of the ZIP file to download |
This command is not meant to be run manually. The server invokes it as a subprocess for process isolation. It downloads the ZIP (with retry on 429), extracts text files into <datadir>/<repo>/source/<slug>/, builds a trigram index at <datadir>/<repo>/index/<slug>/trigrams, then gzip-compresses the source files. Outputs INDEX_READY:<path> and EXTRACT_STATS:<json> on stdout for the parent process.
Exit code 2 signals a permanent download failure (400/403/404/410), telling the server not to retry.
Run database migrations via goose.
veloria migrate <command> [args...]
| Command | Description |
|---|---|
up |
Apply all pending migrations |
down |
Roll back the last migration |
status |
Show migration status |
version |
Print the current migration version |
redo |
Roll back and re-apply the last migration |
reset |
Roll back all migrations |
up-by-one |
Apply the next pending migration |
up-to <version> |
Migrate up to a specific version |
down-to <version> |
Roll back down to a specific version |
create <name> |
Create a new migration file |
fix |
Fix migration version ordering |
Always loads .env from the current directory so it works standalone on production hosts.
Destructive commands for wiping data. Each subcommand prompts for confirmation unless --force / -f is passed.
veloria wipe data-sources [--force]
Wipes all extension data: truncates plugins, themes, cores, largest_repo_files, and index_events tables, resets datasource scan timestamps, and removes all source and index files from disk.
veloria wipe searches [--force]
Wipes all search data: deletes all search result objects from S3/MinIO and truncates the searches and search_reports tables.
Toggle maintenance mode on the running server via its Unix control socket.
veloria maintenance on
veloria maintenance off
Connects to the server's control socket at <datadir>/veloria.sock and sends a JSON command. The server must be running.
Print version information.
veloria version
internal/
├── admin/ # Admin web handlers (reindex, maintenance)
├── api/ # Shared JSON response helpers (WriteJSON, APIError, pagination)
├── app/ # Application lifecycle (New, Start, Shutdown)
├── auth/ # OAuth providers, session management
├── cache/ # Cache interface + Ristretto implementation
├── client/ # Shared HTTP client with User-Agent
├── codesearch/ # Low-level regexp search (port of Google codesearch)
├── config/ # Environment-based configuration
├── core/ # Core (WordPress versions) web + API handlers
├── health/ # Health/readiness endpoint
├── image/ # OG image generation
├── index/ # Search index: build, read, search operations
├── log/ # Zap logger setup
├── manager/ # Orchestrates all extension stores; owns the updater loop
├── mcp/ # MCP (Model Context Protocol) service
├── middleware/ # Custom HTTP middleware (recoverer, access logging)
├── plugin/ # Plugin web + API handlers
├── repo/ # Extension stores, data models, API client, SVN discovery
├── report/ # Search report (flagging) handlers
├── router/ # Chi router setup and middleware
├── search/ # Search web + API handlers, search model
├── server/ # HTTP server with TLS (certmagic) support
├── service/ # Service registry for dynamic dependency resolution
├── storage/ # S3/MinIO client for search result storage
├── tasks/ # Scheduled background tasks (e.g. cleanup)
├── telemetry/ # OpenTelemetry setup (metrics, tracing, logging)
├── testutil/ # Test fixtures, hand-written fakes
├── theme/ # Theme web + API handlers
├── types/ # Protobuf-generated types for search results
├── ui/ # Templ components (layouts, pages, partials, icons, badges)
├── user/ # User model
├── web/ # Shared web deps, interfaces
Veloria manages three types of WordPress extensions, identified by repo.ExtensionType:
TypePlugins("plugins")TypeThemes("themes")TypeCores("cores")
Each type has a concrete store (PluginStore, ThemeStore, CoreStore) that embeds the generic ExtensionStore[T].
The type system uses layered interfaces to keep boundaries clean:
Extension (7 methods) -- data contract for handlers and templates
^
|
Indexable (9 methods) -- adds index wiring (GetIndexedExtension, SetIndexedExtension)
^
|
ExtensionStore[T] (generic) -- in-memory store with search, load, index management
^
|
DataSource (12 methods) -- what the Manager needs from each store
Extension (internal/repo/extension.go) is the narrow read-only interface used everywhere outside the indexing subsystem:
type Extension interface {
GetSlug() string
GetSource() string
GetName() string
GetVersion() string
GetDownloadLink() string
GetActiveInstalls() int
GetDownloaded() int
}Indexable extends Extension with index lifecycle methods. It is the constraint for the generic ExtensionStore[T Indexable].
DataSource (internal/repo/datasource.go) abstracts what the Manager needs from each store:
type DataSource interface {
Type() ExtensionType
Load() error
Stats() (total int, indexed int)
IndexStatus() map[string]bool
Search(term string, opt *index.SearchOptions, progressFn func(searched, total int)) ([]*SearchResult, error)
PrepareUpdates() ([]IndexTask, error)
ResumeUnindexed() []IndexTask
GetExtension(slug string) (Extension, bool)
MakeReindexTaskBySlug(slug string) (IndexTask, bool)
ResolveSourceDir(slug string) (string, error)
RecordIndexSuccess(slug string)
RecordIndexFailure(slug string)
}The service.Registry (internal/service/registry.go) is a thread-safe container for mutable service references. It allows handlers to resolve dependencies at request time rather than at route-registration time, enabling dynamic reconnection after startup:
type Registry struct {
mu sync.RWMutex
db *gorm.DB
s3 storage.ResultStorage
manager *manager.Manager
tasks *tasks.Tasks
apiClient *repo.APIClient
session *auth.SessionStore
authHandler *auth.Handler
mcpHandler http.Handler
maintenance bool
}The Manager (internal/manager/manager.go) is the central coordinator:
- Holds a
map[ExtensionType]DataSource— one entry per extension type - Runs a single background updater loop that collects
IndexTaskitems from all sources and executes them through a shared semaphore (controlled byINDEXER_CONCURRENCY) - Accepts ad-hoc reindex requests via a buffered channel (
adhocCh) - Exposes
Search(),GetExtension(),ResolveSourceDir(),Stats(),IndexStatus(),SubmitReindex()— all dispatching to the correctDataSourceby type string
The Manager satisfies several interfaces consumed by different subsystems:
| Interface | Package | Methods |
|---|---|---|
SearchService |
web |
Search(repoType, term, params) |
ReindexService |
web |
SubmitReindex(repoType, slug) error |
SourceResolver |
web |
ResolveSourceDir(repoType, slug) |
StatsProvider |
web |
Stats(repoType), IndexStatus(repoType) |
The web.Deps struct (internal/web/deps.go) is the shared dependency container for all web handlers. It holds a *service.Registry and resolves services dynamically:
type Deps struct {
Registry *service.Registry
Cache cache.Cache
Config *config.Config
Progress *ProgressStore
}Services are accessed via accessor methods that resolve from the Registry at call time:
func (d *Deps) DB() *gorm.DB { return d.Registry.DB() }
func (d *Deps) S3() storage.ResultStorage { return d.Registry.S3() }
func (d *Deps) Search() SearchService { ... } // nil-safe
func (d *Deps) Reindex() ReindexService { ... } // nil-safe
func (d *Deps) Sources() SourceResolver { ... } // nil-safe
func (d *Deps) Stats() StatsProvider { ... } // nil-safeAny of these may return nil when the corresponding subsystem is unavailable (e.g., no database connection). Handlers check for nil before use.
The router (internal/router/router.go) uses a RouterDeps struct to receive all dependencies:
type RouterDeps struct {
Logger *zap.Logger
Registry *service.Registry
WebDeps *web.Deps
OGGen *ogimage.Generator
PrometheusHandler http.Handler
HealthHandler http.HandlerFunc
Options Options
}The APIClient (internal/repo/api_client.go) wraps the HTTP client, API key, and circuit breaker for AspireCloud API calls. It is injected into store constructors via StoreConfig, eliminating package-level globals:
apiClient := repo.NewAPIClient(c.AspireCloudAPIKey)
pr := repo.NewPluginStore(ctx, db, cfg, logger, cache, apiClient)The circuit breaker (sony/gobreaker) trips after 5 consecutive failures and recovers after 30 seconds.
app.New()loads config, connects to PostgreSQL and S3/MinIO, sets up OpenTelemetry- Creates a
service.Registryfor dynamic dependency resolution - Creates an
APIClientwith the AspireCloud API key - Creates three stores:
PluginStore,ThemeStore,CoreStore manager.NewManager()loads all stores concurrently from DB + disk indexes, then starts the background updater- Registers all services in the Registry
- Builds the router with all dependencies and starts the HTTP server
The updater loop in Manager.startUpdater():
- On first run, resumes any extensions saved to DB but not yet indexed
- Calls
PrepareUpdates()on each DataSource to discover new/changed extensions - Runs all returned
IndexTaskitems through a shared semaphore - Drains any ad-hoc reindex requests from
adhocCh - Waits 5 minutes (
UpdateInterval) or until an ad-hoc request arrives
Each IndexTask.Run() closure handles the full pipeline: download ZIP, extract source, invoke the veloria index subprocess to build the index, and swap the new index into the store.
- HTTP request arrives at search handler
- Handler calls
SearchService.Search(repoType, term, params) - Manager dispatches to the correct DataSource
ExtensionStore.Search()fans out across all indexed extensions usingSEARCH_CONCURRENCYworkers- Each worker calls
IndexedExtension.SearchCompiled()on the extension's in-memory index - Results are collected, sorted by active installs (descending), and returned
Veloria discovers extensions through two mechanisms:
- SVN scanning (
internal/repo/svn.go): Periodically listsplugins.svn.wordpress.organdthemes.svn.wordpress.orgto find new slugs - Incremental API updates: Fetches recently-updated extensions from the AspireCloud API (
FetchPluginsUpdatedWithinLastHour, etc.)
New slugs are fetched individually via FetchPluginInfo/FetchThemeInfo and saved to the database. A full SVN scan runs every 7 days.
Veloria is designed to run in degraded mode when dependencies are unavailable:
- No database: Server starts without search, repos, or auth. Health endpoint still works.
- No S3: Server starts without search result storage. Search is disabled.
- No manager: Web UI shows "search disabled" messaging. API list endpoints still work (direct DB queries).
- No auth config: Server starts without OAuth/session support. Admin routes are hidden.