feat(index): bound indexing resources

[liuchong]

What does this PR do?

Adds one immutable, configurable resource policy for ordinary indexing requests and enforces it across discovery, supervised workers, and persistence.

• rejects dangerous exact repository roots and supports operator-defined aggregate roots
• bounds files, directories, entries, depth, source bytes, and discovery time
• caps worker CPU parallelism, physical concurrency, memory, temporary output, and duration
• checks database, staging, cache admission, and free disk while preserving the previous published index on failure
• returns structured resource_limit_exceeded errors
• prevents callers from weakening the trusted worker policy
• documents defaults and keeps cross-repo-intelligence on its existing database-only path

Related to #1347

Validation

• make -f Makefile.cbm test: 6791 passed, 4 skipped
• Focused CLI suite: 257 passed
• Focused ASan/UBSan suites: 640 passed, 3 platform skips
• make -f Makefile.cbm lint-ci: passed
• Production binary build: passed
• config --help: verified all resource keys

Checklist

• Every commit is signed off (git commit -s)
• Tests pass locally (make -f Makefile.cbm test)
• Lint passes (make -f Makefile.cbm lint-ci)
• New behavior is covered by tests

[DeusData]

Reviewed in full — every limit traced to its enforcement point and the defaults cross-checked against our recorded benchmarks. The mechanics are strong and the concern is entirely about defaults and surface size. Those are separate, and I do not want the second to overshadow the first.

What is genuinely good. The trust boundary is the best part of this PR: caller-supplied _cbm_index_limits is stripped and replaced with the server's own config policy before the worker spawns, a worker refuses a missing or incomplete policy, and there is a dedicated test that passes {"index_max_files":"999999"} and asserts the config limit still wins. test_worker_watchdog.sh then pins the full-policy contract so adding a key without updating the worker contract fails CI. That is how a supervised-worker boundary should be built.

The discovery semantics are also exactly right for this project: a limit violation aborts the request rather than returning a truncated file list — a partial index is never published as a complete one. Graph-quality-first is the rule here, and you landed on the correct side of it without being told.

Where it cannot ship as-is: the defaults contradict this project's own reference workload.

Default	Our recorded reality	Effect
index_memory_limit_mb 8192, worker killed on breach	linux-kernel index peaks 16.4–33.6 GB RSS	our flagship benchmark becomes impossible by default
index_cpu_cores 4	src/foundation/system_info.c:300 — "Use all cores for initial indexing — user is waiting"	multi-fold index-time regression for everyone with >4 cores
index_max_files 100000	our own bench is ~94,500 files at 1×	1× squeaks through at 94.5%; 2× hard-fails
index_max_file_mb 64	src/foundation/limits.c:11-14 sets a deliberate 512 MiB cap, documented as "generous"	an 8× tightening; files that index today start being skipped
index_scan_timeout_seconds 30	no discovery timeout today	cold-cache, NFS and monorepo discovery can now fail the whole request

A resource policy that kills our own reference index out of the box is not a shape we can adopt. If the direction is approved, the version I would argue for is unlimited (off) by default, with the policy opt-in — the enforcement machinery keeps all its value and nothing regresses for existing users.

Two more things worth surfacing louder:

1. cbm_pipeline_new now bakes default limits into every pipeline, so watcher background reindexes are bound too. A repo that grows past a cap after initial indexing has its refreshes fail into logs and index_status only — the user quietly keeps a stale graph without an obvious signal. Worth making noisy, e.g. a stale-index warning on the next MCP response.
2. 18 permanent config keys is a very large one-way door for one feature. If this is adopted, collapsing to something like one index_resource_profile plus a few targeted overrides would age much better.

Security is clean: no new dependencies, no CI or workflow changes, no prompt-injection surface, and the two raw fopen calls are Linux-only /proc/<pid>/stat reads behind #elif defined(__linux__), which is fine — fixed ASCII procfs paths do not need cbm_fopen. New file writes use cbm_fopen correctly. The root-safety hardening (refusing /, $HOME and the cache dir as exact canonical roots while leaving descendants indexable) is a sensible touch.

Whether this project wants a resource-policy subsystem at all is the maintainer's call, not mine, so I have escalated it with the above attached rather than deciding in review. You will also need a rebase — the PR is CONFLICTING and touches mcp.c, pipeline.c and cli.c, all of which are moving.

Thank you for the care in this one; the parts that are good are good for the right reasons.