diff --git a/Makefile.cbm b/Makefile.cbm index 200ee61c7..7ade7f729 100644 --- a/Makefile.cbm +++ b/Makefile.cbm @@ -191,6 +191,7 @@ FOUNDATION_SRCS = \ src/foundation/profile.c \ src/foundation/dump_verify.c \ src/foundation/limits.c \ + src/foundation/index_limits.c \ src/foundation/subprocess.c \ src/foundation/sha256.c \ src/foundation/macos_acl.c \ @@ -430,6 +431,7 @@ TEST_FOUNDATION_SRCS = \ tests/test_platform.c \ tests/test_diagnostics.c \ tests/test_dump_verify.c \ + tests/test_index_limits.c \ tests/test_subprocess.c \ tests/test_private_file_lock.c \ tests/test_lock_registry.c diff --git a/README.md b/README.md index b542178a0..5e7c7c046 100644 --- a/README.md +++ b/README.md @@ -661,6 +661,12 @@ codebase-memory-mcp config set auto_watch false # don't register backgr codebase-memory-mcp config reset auto_index # reset to default ``` +Repository indexing also has bounded defaults for root breadth, discovery, +worker count, memory, duration, temporary files, database size, cache admission, +and reserved free disk. See +[docs/INDEX_RESOURCE_LIMITS.md](docs/INDEX_RESOURCE_LIMITS.md) for all keys and +failure semantics. + ### Environment Variables | Variable | Default | Description | diff --git a/docs/INDEX_RESOURCE_LIMITS.md b/docs/INDEX_RESOURCE_LIMITS.md new file mode 100644 index 000000000..c71c3b3c8 --- /dev/null +++ b/docs/INDEX_RESOURCE_LIMITS.md @@ -0,0 +1,81 @@ +# Index Resource Limits + +Indexing is intentionally bounded. Before parsing begins, the server validates +the canonical repository root and performs a bounded discovery walk. The worker +and persistence stages continue enforcing the same resolved policy. + +If a hard boundary is reached, `index_repository` returns +`resource_limit_exceeded` with the boundary name, observed value, configured +limit, and a remediation hint. A partial index is never published as a complete +one. + +## Defaults + +| Configuration key | Default | Protects | +|---|---:|---| +| `index_max_files` | `100000` | Accepted source-file count | +| `index_max_directories` | `20000` | Traversed directory count | +| `index_max_entries` | `500000` | Examined filesystem entries | +| `index_max_depth` | `64` | Directory nesting depth | +| `index_max_source_mb` | `4096` | Aggregate accepted source size | +| `index_max_file_mb` | `64` | One source file | +| `index_scan_timeout_seconds` | `30` | Discovery time | +| `index_cpu_cores` | `4` | Indexing worker threads | +| `index_concurrent_jobs` | `1` | Simultaneous physical index workers | +| `index_memory_limit_mb` | `8192` | Worker resident memory, capped by detected budget | +| `index_max_db_mb` | `16384` | Published SQLite database | +| `index_max_staging_mb` | `20480` | Staging database and sidecars | +| `index_max_task_temp_mb` | `24576` | Staging, worker log, and response temporary files | +| `index_cache_max_mb` | `32768` | Total index cache admission | +| `index_min_free_disk_mb` | `4096` | Free disk kept in reserve | +| `index_max_duration_seconds` | `3600` | Worker wall-clock time | +| `index_low_priority` | `true` | Best-effort reduced scheduling priority | +| `index_denied_roots` | empty | Additional exact canonical roots | + +Automatic indexing keeps its separate, stricter `auto_index_limit` default of +`50000`. + +Set a value with: + +```sh +codebase-memory-mcp config set index_max_files 75000 +codebase-memory-mcp config set index_memory_limit_mb 4096 +``` + +Separate additional denied roots with semicolons on every platform: + +```sh +codebase-memory-mcp config set index_denied_roots '/path/to/aggregate;/path/to/archive' +``` + +Numeric values must be positive and within the ranges shown by +`codebase-memory-mcp config --help`. `index_denied_roots` may be empty. Invalid +or unknown keys are rejected. + +## Root safety + +The filesystem root, current user's home directory, and the Codebase Memory +cache directory are always refused as repository roots. This built-in check +cannot be disabled. `index_denied_roots` adds exact canonical roots; it does not +deny all descendants, so individual repositories below an aggregation +directory can still be indexed. + +`CBM_ALLOWED_ROOT` remains available for deployments that also need an outer +containment boundary. + +`cross-repo-intelligence` is not a source-indexing mode: it reads and updates +already published project databases and does not walk `repo_path`. It therefore +keeps the session containment and cross-repository target-count checks, but does +not apply source-root or discovery limits. + +## Failure and recovery + +Discovery violations stop before parsing. Worker memory, time, and temporary +output violations terminate the contained process tree. Persistence violations +remove the current operation's staging files and preserve the previously +published database. Reduced scheduling priority is applied where the operating +system permits it; a refusal is logged and does not silently change the other +limits. + +The cache limit is an admission check, not an eviction policy. The indexer never +deletes unrelated project indexes automatically. diff --git a/docs/changes/index-resource-guards.md b/docs/changes/index-resource-guards.md new file mode 100644 index 000000000..ef2cba15d --- /dev/null +++ b/docs/changes/index-resource-guards.md @@ -0,0 +1,171 @@ +# Index Resource Guard Design + +## Problem + +An explicit `index_repository` request currently trusts the requested directory +and lets discovery, parsing, and persistence grow until the operation finishes +or the host runs out of resources. This is unsafe for aggregation directories, +accidental filesystem roots, and repositories with unexpectedly large generated +trees. + +## Behavior contract + +The indexer resolves one immutable resource policy at admission time. The same +policy is passed to discovery, the worker process, and persistence. A limit +failure is reported as `resource_limit_exceeded`; it is never reported as a +successful or merely empty index. + +### BDD scenarios + +1. **Dangerous root** + - Given an explicit request whose canonical path is the filesystem root, + current user's home directory, or the cache directory + - When `index_repository` validates the request + - Then it refuses the request before discovery or a worker is started + - And the response identifies the `repository_root` policy + +2. **Bounded discovery** + - Given a tree that exceeds the configured file, directory, entry, depth, or + aggregate indexable-source-byte limit + - When discovery reaches that boundary + - Then discovery stops immediately and identifies the exact boundary + - And no partial file list is passed to parsing + +3. **Bounded compute** + - Given default configuration on a large host + - When an index job starts + - Then at most four indexing workers are used + - And only one physical indexing job runs at a time + - And the worker runs at reduced scheduling priority where supported + +4. **Bounded memory and time** + - Given a supervised worker that stays above its hard memory limit or runs + longer than its duration limit + - When the supervisor polls it + - Then the process tree is terminated + - And the operation returns `resource_limit_exceeded`, including the limit + name and configured value + +5. **Bounded persistence** + - Given an index whose staging database, final database, temporary task + footprint, or cache footprint would exceed policy + - When persistence checks the next publish step + - Then the new index is not published + - And the previously published database remains usable + - And abandoned staging and sidecar files for this operation are removed + +6. **Configuration validation** + - Given an unknown resource key, a non-integer value, zero, or a value outside + the supported range + - When `config set` is invoked + - Then configuration is rejected without changing the previous value + +7. **Public-repository hygiene** + - Given the final change + - When tracked content is scanned + - Then it contains only generic examples such as `/path/to/repo` + - And it contains no developer usernames, workstation paths, private + repository names, tokens, or index contents + +8. **Cross-repository database linking** + - Given `mode="cross-repo-intelligence"` and existing source and target + databases + - When `index_repository` handles the request + - Then it preserves session containment and target validation + - And it does not apply source-root or discovery limits because it does not + walk the source directory + +## Resource policy + +All byte-size settings use MiB at the user-facing configuration boundary and +are converted with overflow checks. + +| Key | Default | Decision | +|---|---:|---| +| `index_max_files` | 100,000 | Accepted source files | +| `auto_index_limit` | 50,000 | Stricter automatic-admission file limit | +| `index_max_directories` | 20,000 | Traversed directories, including root | +| `index_max_entries` | 500,000 | Filesystem entries examined | +| `index_max_depth` | 64 | Root is depth 0 | +| `index_max_source_mb` | 4,096 | Aggregate accepted source bytes | +| `index_max_file_mb` | 64 | Oversized files are skipped and reported | +| `index_scan_timeout_seconds` | 30 | Discovery deadline | +| `index_cpu_cores` | 4 | Maximum indexing worker threads | +| `index_concurrent_jobs` | 1 | Maximum physical index workers | +| `index_memory_limit_mb` | 8,192 | Also capped by detected available memory | +| `index_max_db_mb` | 16,384 | Published database ceiling | +| `index_max_staging_mb` | 20,480 | Staging DB plus SQLite sidecars | +| `index_max_task_temp_mb` | 24,576 | Staging plus worker log/response footprint | +| `index_cache_max_mb` | 32,768 | Admission ceiling; no automatic eviction | +| `index_min_free_disk_mb` | 4,096 | Reserved free space | +| `index_max_duration_seconds` | 3,600 | Wall-clock worker deadline | +| `index_low_priority` | `true` | Best-effort lower scheduling priority | +| `index_denied_roots` | empty | Additional canonical roots, semicolon-separated | + +Built-in dangerous roots cannot be disabled. Operators may lower or raise +numeric values within documented validation ranges. `0` never means unlimited; +it is rejected. + +## Data model and state transitions + +`cbm_index_limits_t` is the resolved, byte-based policy. Discovery additionally +returns counters and a stable violation enum. The index request progresses: + +`admitted -> preflight discovery -> worker -> staging persistence -> publish` + +Any resource violation transitions to `resource_limit_exceeded`. Before publish, +failure removes only files owned by the current staging operation. After a +successful atomic rename, the old database is already replaced and later +best-effort artifact export cannot claim rollback. + +## Shared entry paths + +- MCP and one-shot CLI both enter through `handle_index_repository`. +- Daemon admission resolves concurrency and memory caps once. +- Supervised workers receive only resolved internal numeric arguments, not + caller-controlled overrides. +- Automatic indexing uses the same policy with the stricter + `auto_index_limit`. +- `cross-repo-intelligence` remains a database-linking path with its existing + target-count and mutation guards; it does not enter source discovery. + +## Tests and fixtures + +- Discovery fixtures use temporary generic directory trees. +- MCP integration tests exercise dangerous-root rejection and structured error + output without spawning a worker. +- Pipeline tests create a valid old database, force a small staging/DB limit, + and verify the old database remains. +- Supervisor tests use deterministic process/resource probes rather than large + real allocations, including worker-owned temporary output. +- A real supervised-worker integration test proves the parent's resolved policy + reaches the child and replaces caller-supplied internal policy data. +- Configuration tests verify valid overrides and fail-closed parsing. +- Existing cross-repository integration tests verify target validation, + cancellation, deduplication, missing databases, and source-name overrides. + +## Installation and migration + +No database migration is required. Existing indexes remain readable. The new +limits affect the next indexing operation. Existing environment variables remain +compatible, but the resolved policy always chooses the safer lower effective +limit. + +## Non-goals + +- Exact percentage CPU throttling. Worker-count tokens, single-job admission, + and lower process priority provide portable bounded CPU use. +- Automatic deletion of unrelated cache entries. Cache excess blocks new work + and reports remediation; it does not delete another project's data. +- Following symlinks, junctions, or paths outside the canonical requested root. +- Per-request public overrides that let an untrusted MCP caller weaken operator + policy. + +## Directly applicable repository rules + +- "All code must be properly formatted before committing." +- "Maintain appropriate tests for your changes following the test hierarchy." +- "Prefer explicit error handling." +- "Do NOT create directories or files directly in the repository root." +- Behavior changes follow red, green, refactor: tests precede production code. +- Tracked content must be suitable for a public open-source repository. diff --git a/src/cli/cli.c b/src/cli/cli.c index 8010e2258..724f68f17 100644 --- a/src/cli/cli.c +++ b/src/cli/cli.c @@ -6407,10 +6407,42 @@ int cbm_config_get_int(cbm_config_t *cfg, const char *key, int default_val) { return (int)v; } +bool cbm_config_load_index_limits(cbm_config_t *cfg, cbm_index_limits_t *limits, char *error, + size_t error_size) { + if (!limits) { + if (error && error_size > 0) { + (void)snprintf(error, error_size, "missing index resource policy"); + } + return false; + } + cbm_index_limits_defaults(limits); + if (!cfg) { + return true; + } + for (size_t i = 0; i < cbm_index_limits_config_key_count(); i++) { + const char *key = cbm_index_limits_config_key_at(i); + const char *value = cbm_config_get(cfg, key, NULL); + if (value && !cbm_index_limits_set(limits, key, value, error, error_size)) { + return false; + } + } + return true; +} + int cbm_config_set(cbm_config_t *cfg, const char *key, const char *value) { if (!cfg || !key || !value) { return CLI_ERR; } + if (cbm_index_limits_is_config_key(key)) { + cbm_index_limits_t validated; + char error[CLI_BUF_256]; + cbm_index_limits_defaults(&validated); + if (!cbm_index_limits_set(&validated, key, value, error, sizeof(error))) { + return CLI_ERR; + } + } else if (strncmp(key, "index_", sizeof("index_") - 1U) == 0) { + return CLI_ERR; + } sqlite3_stmt *stmt = NULL; if (sqlite3_prepare_v2(cfg->db, "INSERT OR REPLACE INTO config (key, value) VALUES (?, ?)", @@ -6461,6 +6493,15 @@ int cbm_cmd_config(int argc, char **argv) { "Register background git watcher on session connect"); printf(" %-25s default=%-10s %s\n", CBM_CONFIG_UI_LANG, "auto", "Pin graph UI language: en, zh, or auto"); + printf("\nIndex resource keys (sizes are MiB, durations are seconds):\n"); + for (size_t i = 0; i < cbm_index_limits_config_key_count(); i++) { + const char *key = cbm_index_limits_config_key_at(i); + char default_value[CLI_BUF_256]; + if (cbm_index_limits_default_value(key, default_value, sizeof(default_value))) { + printf(" %-32s default=%s\n", key, default_value[0] ? default_value : "(empty)"); + } + } + printf("See docs/INDEX_RESOURCE_LIMITS.md for boundary semantics and ranges.\n"); return 0; } @@ -6490,6 +6531,13 @@ int cbm_cmd_config(int argc, char **argv) { cbm_config_get(cfg, CBM_CONFIG_AUTO_WATCH, "true")); printf(" %-25s = %-10s\n", CBM_CONFIG_UI_LANG, cbm_config_get(cfg, CBM_CONFIG_UI_LANG, "auto")); + for (size_t i = 0; i < cbm_index_limits_config_key_count(); i++) { + const char *key = cbm_index_limits_config_key_at(i); + char default_value[CLI_BUF_256]; + if (cbm_index_limits_default_value(key, default_value, sizeof(default_value))) { + printf(" %-32s = %s\n", key, cbm_config_get(cfg, key, default_value)); + } + } } else if (strcmp(argv[0], "get") == 0) { if (argc < MIN_ARGC_GET) { (void)fprintf(stderr, "Usage: config get \n"); @@ -6502,7 +6550,24 @@ int cbm_cmd_config(int argc, char **argv) { (void)fprintf(stderr, "Usage: config set \n"); rc = CLI_TRUE; } else { - if (cbm_config_set(cfg, argv[CLI_SKIP_ONE], argv[CLI_PAIR_LEN]) == 0) { + const char *key = argv[CLI_SKIP_ONE]; + const char *value = argv[CLI_PAIR_LEN]; + char validation_error[CLI_BUF_256] = {0}; + bool resource_value_valid = true; + if (cbm_index_limits_is_config_key(key)) { + cbm_index_limits_t validated; + cbm_index_limits_defaults(&validated); + resource_value_valid = cbm_index_limits_set( + &validated, key, value, validation_error, sizeof(validation_error)); + } else if (strncmp(key, "index_", sizeof("index_") - 1U) == 0) { + resource_value_valid = false; + (void)snprintf(validation_error, sizeof(validation_error), + "unknown index resource key: %s", key); + } + if (!resource_value_valid) { + (void)fprintf(stderr, "error: %s\n", validation_error); + rc = CLI_TRUE; + } else if (cbm_config_set(cfg, key, value) == 0) { printf("%s = %s\n", argv[CLI_SKIP_ONE], argv[CLI_PAIR_LEN]); } else { (void)fprintf(stderr, "error: failed to set %s\n", argv[CLI_SKIP_ONE]); diff --git a/src/cli/cli.h b/src/cli/cli.h index 27ef4e3ec..264c9e6b5 100644 --- a/src/cli/cli.h +++ b/src/cli/cli.h @@ -9,6 +9,8 @@ #ifndef CBM_CLI_H #define CBM_CLI_H +#include "foundation/index_limits.h" + #include #include #include @@ -384,6 +386,11 @@ bool cbm_config_get_bool(cbm_config_t *cfg, const char *key, bool default_val); /* Get a config value as int. Returns default_val if not found or invalid. */ int cbm_config_get_int(cbm_config_t *cfg, const char *key, int default_val); +/* Load the complete index resource policy from defaults plus persisted + * overrides. Returns false on the first invalid stored value. */ +bool cbm_config_load_index_limits(cbm_config_t *cfg, cbm_index_limits_t *limits, char *error, + size_t error_size); + /* Set a config value. Returns 0 on success. */ int cbm_config_set(cbm_config_t *cfg, const char *key, const char *value); diff --git a/src/daemon/application.c b/src/daemon/application.c index 540749a75..504c9f6bd 100644 --- a/src/daemon/application.c +++ b/src/daemon/application.c @@ -8,6 +8,7 @@ #include "foundation/compat.h" #include "foundation/compat_fs.h" #include "foundation/compat_thread.h" +#include "foundation/index_limits.h" #include "foundation/log.h" #include "foundation/mem.h" #include "foundation/platform.h" @@ -50,7 +51,6 @@ enum { APPLICATION_JOB_THREAD_STACK = 256 * 1024, APPLICATION_JOB_POLL_US = 10000, APPLICATION_COORDINATION_CLEANUP_MS = 500, - APPLICATION_DEFAULT_PHYSICAL_JOB_LIMIT = 4, APPLICATION_DEFAULT_MAX_RESTARTS = 100, APPLICATION_MARKER_MAX_BYTES = 64 * 1024 * 1024, APPLICATION_MAX_SUSPECTS = 65536, @@ -178,6 +178,10 @@ struct cbm_daemon_application { cbm_project_lock_manager_t *project_locks; size_t physical_job_limit; size_t worker_memory_budget_bytes; + size_t worker_memory_limit_bytes; + uint64_t worker_task_temp_limit_bytes; + uint64_t worker_max_duration_ms; + bool worker_low_priority; size_t active_mutations; size_t update_owners; cbm_daemon_application_update_worker_t update_worker; @@ -297,10 +301,18 @@ static int application_worker_start_default(void *context, const char *args_json size_t memory_budget_bytes, const char *marker_file, const char *quarantine_file, cbm_daemon_application_worker_t *worker_out) { - (void)context; + cbm_daemon_application_t *application = context; cbm_index_worker_handle_t *worker = NULL; int result = cbm_index_worker_start(args_json, memory_budget_bytes, false, marker_file, quarantine_file, &worker); + if (result == 0 && worker && application) { + cbm_index_worker_set_memory_limit(worker, application->worker_memory_limit_bytes); + cbm_index_worker_set_task_temp_limit(worker, application->worker_task_temp_limit_bytes); + cbm_index_worker_set_max_duration(worker, application->worker_max_duration_ms); + if (!cbm_index_worker_set_low_priority(worker, application->worker_low_priority)) { + cbm_log_warn("daemon.index.priority_not_lowered", "action", "continue"); + } + } *worker_out = worker; return result; } @@ -1129,6 +1141,14 @@ static char *application_job_failure_response(const cbm_index_worker_result_t *r (void)snprintf(message, sizeof(message), "index worker containment failed (%s); inspect log: %s", cbm_proc_outcome_str(result->outcome), log_path ? log_path : "unavailable"); + } else if (result && result->resource_limit != CBM_INDEX_RESOURCE_LIMIT_NONE) { + (void)snprintf( + message, sizeof(message), + "index resource limit exceeded: %s observed=%llu limit=%llu; the contained worker " + "tree was terminated", + cbm_index_resource_limit_name(result->resource_limit), + (unsigned long long)result->resource_observed, + (unsigned long long)result->resource_limit_value); } else if (result) { (void)snprintf(message, sizeof(message), "index worker ended with %s (exit=%d, signal=%d); inspect log: %s", @@ -2731,7 +2751,22 @@ cbm_daemon_application_t *cbm_daemon_application_new( return NULL; } cbm_mutex_init(&application->mutex); - application->physical_job_limit = APPLICATION_DEFAULT_PHYSICAL_JOB_LIMIT; + cbm_index_limits_t index_limits; + char index_limits_error[256] = {0}; + bool index_limits_valid = + cbm_config_load_index_limits(config ? config->config : NULL, &index_limits, + index_limits_error, sizeof(index_limits_error)); + if (!index_limits_valid) { + cbm_index_limits_defaults(&index_limits); + cbm_log_warn("daemon.index.config_invalid", "reason", index_limits_error); + } + application->physical_job_limit = (size_t)index_limits.concurrent_jobs; + application->worker_memory_limit_bytes = index_limits.memory_limit_bytes > SIZE_MAX + ? SIZE_MAX + : (size_t)index_limits.memory_limit_bytes; + application->worker_task_temp_limit_bytes = index_limits.max_task_temp_bytes; + application->worker_max_duration_ms = index_limits.max_duration_ms; + application->worker_low_priority = index_limits.low_priority; size_t aggregate_memory_budget_bytes = cbm_mem_budget(); if (config) { application->watcher = config->watcher; @@ -2762,9 +2797,12 @@ cbm_daemon_application_t *cbm_daemon_application_new( application->worker_memory_budget_bytes = aggregate_memory_budget_bytes / application->physical_job_limit; } + application->worker_memory_limit_bytes = + (size_t)cbm_index_effective_memory_limit((uint64_t)application->worker_memory_limit_bytes, + (uint64_t)application->worker_memory_budget_bytes); if (!application->worker_ops.start) { application->worker_ops = (cbm_daemon_application_worker_ops_t){ - .context = NULL, + .context = application, .start = application_worker_start_default, .poll = application_worker_poll_default, .cancel = application_worker_cancel_default, diff --git a/src/discover/discover.c b/src/discover/discover.c index b2169f1aa..927b50710 100644 --- a/src/discover/discover.c +++ b/src/discover/discover.c @@ -18,6 +18,7 @@ #include "foundation/win_utf8.h" #endif #include +#include #include // int64_t #include #include @@ -413,6 +414,11 @@ typedef struct { int capacity; int max_files; uint64_t deadline_ms; + const cbm_discover_limits_t *limits; + cbm_discover_report_t *report; + uint64_t directories; + uint64_t entries; + uint64_t source_bytes; bool count_only; bool collect_excluded; bool limit_exceeded; @@ -434,12 +440,29 @@ typedef struct { int ignored_total; } file_list_t; +static void file_list_violate(file_list_t *fl, cbm_discover_limit_t violation, uint64_t observed, + uint64_t limit) { + if (!fl || fl->limit_exceeded || fl->failed) { + return; + } + fl->limit_exceeded = true; + if (fl->report) { + fl->report->violation = violation; + fl->report->observed = observed; + fl->report->limit = limit; + } +} + static bool file_list_should_stop(file_list_t *fl) { if (!fl) { return true; } if (!fl->failed && fl->deadline_ms != 0 && cbm_now_ms() >= fl->deadline_ms) { - fl->failed = true; + if (fl->limits) { + file_list_violate(fl, CBM_DISCOVER_LIMIT_DEADLINE, cbm_now_ms(), fl->deadline_ms); + } else { + fl->failed = true; + } } return fl->failed || fl->limit_exceeded; } @@ -500,9 +523,22 @@ static void file_list_add_ignored(file_list_t *fl, const char *rel_path, const c static void fl_add(file_list_t *fl, const char *abs_path, const char *rel_path, CBMLanguage lang, int64_t size) { if (fl->max_files >= 0 && fl->count >= fl->max_files) { - fl->limit_exceeded = true; + file_list_violate(fl, CBM_DISCOVER_LIMIT_FILES, (uint64_t)fl->count + 1U, + (uint64_t)fl->max_files); + return; + } + uint64_t source_size = size > 0 ? (uint64_t)size : 0; + if (fl->limits && fl->limits->max_source_bytes > 0 && + (source_size > fl->limits->max_source_bytes || + fl->source_bytes > fl->limits->max_source_bytes - source_size)) { + uint64_t observed = source_size > UINT64_MAX - fl->source_bytes + ? UINT64_MAX + : fl->source_bytes + source_size; + file_list_violate(fl, CBM_DISCOVER_LIMIT_SOURCE_BYTES, observed, + fl->limits->max_source_bytes); return; } + fl->source_bytes += source_size; if (fl->count_only) { fl->count++; return; @@ -761,6 +797,7 @@ static void walk_dir_process_file(const char *abs_path, const char *rel_path, co typedef struct { char dir[CBM_SZ_4K]; char prefix[CBM_SZ_4K]; + uint64_t depth; cbm_gitignore_t *local_gi; /* nested .gitignore for this subtree */ char local_gi_prefix[CBM_SZ_4K]; /* rel_prefix when local_gi was loaded */ } walk_frame_t; @@ -814,6 +851,7 @@ static void walk_push_subdir(walk_stack_t *ws, const char *abs_path, const char return; } slot->local_gi = parent->local_gi; + slot->depth = parent->depth + 1U; int local_prefix_length = snprintf(slot->local_gi_prefix, CBM_SZ_4K, "%s", parent->local_gi_prefix); if (local_prefix_length < 0 || local_prefix_length >= CBM_SZ_4K) { @@ -829,6 +867,12 @@ static void walk_dir_process_entry(cbm_dirent_t *entry, const walk_frame_t *fram const cbm_gitignore_t *global_gi, const cbm_gitignore_t *cbmignore, walk_stack_t *ws, file_list_t *out) { + out->entries++; + if (out->limits && out->limits->max_entries > 0 && out->entries > out->limits->max_entries) { + file_list_violate(out, CBM_DISCOVER_LIMIT_ENTRIES, out->entries, out->limits->max_entries); + return; + } + char abs_path[CBM_SZ_4K]; char rel_path[CBM_SZ_4K]; int absolute_length = snprintf(abs_path, sizeof(abs_path), "%s/%s", frame->dir, entry->name); @@ -855,6 +899,20 @@ static void walk_dir_process_entry(cbm_dirent_t *entry, const walk_frame_t *fram if (S_ISDIR(st.st_mode)) { if (!should_skip_directory(entry->name, rel_path, opts, gitignore, global_gi, cbmignore, frame->local_gi, frame->local_gi_prefix)) { + uint64_t child_depth = frame->depth + 1U; + if (out->limits && out->limits->max_depth > 0 && child_depth > out->limits->max_depth) { + file_list_violate(out, CBM_DISCOVER_LIMIT_DEPTH, child_depth, + out->limits->max_depth); + return; + } + uint64_t next_directory = out->directories + 1U; + if (out->limits && out->limits->max_directories > 0 && + next_directory > out->limits->max_directories) { + file_list_violate(out, CBM_DISCOVER_LIMIT_DIRECTORIES, next_directory, + out->limits->max_directories); + return; + } + out->directories = next_directory; walk_push_subdir(ws, abs_path, rel_path, frame, out); } else { /* Record the excluded subtree root so callers can report it (#411). */ @@ -896,6 +954,7 @@ static void walk_dir(const char *dir_path, const char *rel_prefix, const cbm_dis out->failed = true; return; } + out->directories = 1; /* Collect all owned gitignores — freed at the end because child frames * on the stack hold borrowed pointers to them. */ cbm_gitignore_t **owned_gis = NULL; @@ -1158,12 +1217,23 @@ static cbm_discover_status_t discover_impl(const char *repo_path, const cbm_disc /* Walk */ file_list_t fl = { - .max_files = count_only ? max_files : -1, - .deadline_ms = count_only ? deadline_ms : 0, + .max_files = count_only + ? max_files + : (opts && opts->limits && opts->limits->max_files > 0 + ? (opts->limits->max_files > INT_MAX ? INT_MAX + : (int)opts->limits->max_files) + : -1), + .deadline_ms = + count_only ? deadline_ms : (opts && opts->limits ? opts->limits->deadline_ms : 0), + .limits = count_only || !opts ? NULL : opts->limits, + .report = count_only || !opts ? NULL : opts->report, .count_only = count_only, .collect_excluded = !count_only && excluded_out != NULL, .collect_ignored = !count_only && ignored_out != NULL, }; + if (fl.report) { + memset(fl.report, 0, sizeof(*fl.report)); + } walk_dir(repo_path, "", opts, gitignore, global_gi, cbmignore, &fl); /* Cleanup */ @@ -1181,11 +1251,11 @@ static cbm_discover_status_t discover_impl(const char *repo_path, const cbm_disc } return fl.limit_exceeded ? CBM_DISCOVER_LIMIT_EXCEEDED : CBM_DISCOVER_OK; } - if (fl.failed) { + if (fl.failed || fl.limit_exceeded) { cbm_discover_free(fl.files, fl.count); cbm_discover_free_excluded(fl.excluded, fl.excluded_count); cbm_discover_free_ignored(fl.ignored, fl.ignored_count); - return CBM_DISCOVER_ERROR; + return fl.limit_exceeded ? CBM_DISCOVER_LIMIT_EXCEEDED : CBM_DISCOVER_ERROR; } *out = fl.files; @@ -1216,6 +1286,26 @@ static cbm_discover_status_t discover_impl(const char *repo_path, const cbm_disc return CBM_DISCOVER_OK; } +const char *cbm_discover_limit_name(cbm_discover_limit_t limit) { + switch (limit) { + case CBM_DISCOVER_LIMIT_FILES: + return "files"; + case CBM_DISCOVER_LIMIT_DIRECTORIES: + return "directories"; + case CBM_DISCOVER_LIMIT_ENTRIES: + return "entries"; + case CBM_DISCOVER_LIMIT_DEPTH: + return "depth"; + case CBM_DISCOVER_LIMIT_SOURCE_BYTES: + return "source_bytes"; + case CBM_DISCOVER_LIMIT_DEADLINE: + return "deadline"; + case CBM_DISCOVER_LIMIT_NONE: + default: + return "none"; + } +} + int cbm_discover_ex2(const char *repo_path, const cbm_discover_opts_t *opts, cbm_file_info_t **out, int *count, char ***excluded_out, int *excluded_count_out, cbm_ignored_file_t **ignored_out, int *ignored_count_out, diff --git a/src/discover/discover.h b/src/discover/discover.h index 80db1a7e8..0a7f5e4cd 100644 --- a/src/discover/discover.h +++ b/src/discover/discover.h @@ -116,10 +116,37 @@ typedef struct { int64_t size; /* file size in bytes */ } cbm_file_info_t; +typedef enum { + CBM_DISCOVER_LIMIT_NONE = 0, + CBM_DISCOVER_LIMIT_FILES, + CBM_DISCOVER_LIMIT_DIRECTORIES, + CBM_DISCOVER_LIMIT_ENTRIES, + CBM_DISCOVER_LIMIT_DEPTH, + CBM_DISCOVER_LIMIT_SOURCE_BYTES, + CBM_DISCOVER_LIMIT_DEADLINE, +} cbm_discover_limit_t; + +typedef struct { + uint64_t max_files; /* 0 = disabled */ + uint64_t max_directories; /* root counts as one; 0 = disabled */ + uint64_t max_entries; /* all examined directory entries; 0 = disabled */ + uint64_t max_depth; /* root is depth zero; 0 = disabled */ + uint64_t max_source_bytes; /* accepted source files only; 0 = disabled */ + uint64_t deadline_ms; /* absolute cbm_now_ms deadline; 0 = disabled */ +} cbm_discover_limits_t; + typedef struct { - cbm_index_mode_t mode; /* CBM_MODE_FULL or CBM_MODE_FAST */ - const char *ignore_file; /* path to .cbmignore file, or NULL */ - int64_t max_file_size; /* 0 = no limit */ + cbm_discover_limit_t violation; + uint64_t observed; + uint64_t limit; +} cbm_discover_report_t; + +typedef struct { + cbm_index_mode_t mode; /* CBM_MODE_FULL or CBM_MODE_FAST */ + const char *ignore_file; /* path to .cbmignore file, or NULL */ + int64_t max_file_size; /* 0 = no limit */ + const cbm_discover_limits_t *limits; /* NULL = legacy unbounded walk */ + cbm_discover_report_t *report; /* optional exact limit diagnostic */ } cbm_discover_opts_t; typedef enum { @@ -128,6 +155,9 @@ typedef enum { CBM_DISCOVER_LIMIT_EXCEEDED = 1, } cbm_discover_status_t; +/* Stable machine-readable name for a discovery limit. */ +const char *cbm_discover_limit_name(cbm_discover_limit_t limit); + /* Walk a repository directory tree and discover all source files. * Applies hardcoded filters, gitignore patterns, and language detection. * Returns 0 on success, -1 on error. diff --git a/src/foundation/compat_fs.c b/src/foundation/compat_fs.c index 09392097d..9666aee50 100644 --- a/src/foundation/compat_fs.c +++ b/src/foundation/compat_fs.c @@ -7,6 +7,7 @@ #include "foundation/constants.h" #include "foundation/compat_fs.h" #include "foundation/compat_fs_internal.h" +#include "foundation/platform.h" #include #include @@ -116,6 +117,7 @@ cbm_dirent_t *cbm_readdir(cbm_dir_t *d) { d->entry.name[nlen] = '\0'; free(u8); d->entry.is_dir = (d->find_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0; + d->entry.is_symlink = (d->find_data.dwFileAttributes & FILE_ATTRIBUTE_REPARSE_POINT) != 0; d->entry.d_type = 0; return &d->entry; } @@ -714,6 +716,7 @@ cbm_dirent_t *cbm_readdir(cbm_dir_t *d) { memcpy(d->entry.name, de->d_name, nlen); d->entry.name[nlen] = '\0'; d->entry.is_dir = (de->d_type == DT_DIR); + d->entry.is_symlink = (de->d_type == DT_LNK); d->entry.d_type = de->d_type; return &d->entry; } @@ -1004,3 +1007,68 @@ int cbm_remove_db_sidecars(const char *db_path) { } return result; } + +static bool directory_size_walk(const char *path, uint64_t stop_after, unsigned depth, + uint64_t *total) { + enum { DIRECTORY_SIZE_MAX_DEPTH = 64 }; + if (depth > DIRECTORY_SIZE_MAX_DEPTH) { + return false; + } + cbm_dir_t *directory = cbm_opendir(path); + if (!directory) { + return false; + } + bool ok = true; + cbm_dirent_t *entry; + while (ok && (entry = cbm_readdir(directory)) != NULL) { + if (entry->is_symlink) { + continue; + } + char child[CBM_SZ_4K]; + int written = snprintf(child, sizeof(child), "%s/%s", path, entry->name); + if (written <= 0 || (size_t)written >= sizeof(child)) { + ok = false; + break; + } +#ifdef _WIN32 + bool child_is_directory = entry->is_dir; +#else + /* d_type may be DT_UNKNOWN (notably on network and FUSE filesystems). + * lstat keeps the documented no-follow contract authoritative instead + * of letting a later stat-based directory probe follow a link. */ + struct stat child_state; + if (lstat(child, &child_state) != 0) { + ok = false; + break; + } + if (S_ISLNK(child_state.st_mode)) { + continue; + } + bool child_is_directory = S_ISDIR(child_state.st_mode); +#endif + if (child_is_directory) { + ok = directory_size_walk(child, stop_after, depth + 1U, total); + } else { + int64_t file_size = cbm_file_size(child); + if (file_size < 0) { + ok = false; + break; + } + uint64_t size = (uint64_t)file_size; + *total = size > UINT64_MAX - *total ? UINT64_MAX : *total + size; + } + if (*total > stop_after) { + break; + } + } + cbm_closedir(directory); + return ok; +} + +bool cbm_directory_size_bounded(const char *path, uint64_t stop_after, uint64_t *bytes_out) { + if (!path || !bytes_out) { + return false; + } + *bytes_out = 0; + return directory_size_walk(path, stop_after, 0, bytes_out); +} diff --git a/src/foundation/compat_fs.h b/src/foundation/compat_fs.h index 92b6ca799..936096a3c 100644 --- a/src/foundation/compat_fs.h +++ b/src/foundation/compat_fs.h @@ -9,6 +9,7 @@ #include #include +#include #include /* ── Directory iteration ──────────────────────────────────────── */ @@ -21,6 +22,7 @@ typedef struct cbm_dir cbm_dir_t; typedef struct { char name[CBM_DIRENT_NAME_MAX]; bool is_dir; + bool is_symlink; unsigned char d_type; /* DT_REG, DT_DIR, DT_LNK, etc. (POSIX only, 0 on Windows) */ } cbm_dirent_t; @@ -34,6 +36,10 @@ cbm_dirent_t *cbm_readdir(cbm_dir_t *d); /* Close directory handle. */ void cbm_closedir(cbm_dir_t *d); +/* Recursively sum regular files below path without following links or + * junctions. Stops once the total exceeds stop_after. */ +bool cbm_directory_size_bounded(const char *path, uint64_t stop_after, uint64_t *bytes_out); + /* ── Portable popen/pclose ────────────────────────────────────── */ FILE *cbm_popen(const char *cmd, const char *mode); diff --git a/src/foundation/index_limits.c b/src/foundation/index_limits.c new file mode 100644 index 000000000..50188b482 --- /dev/null +++ b/src/foundation/index_limits.c @@ -0,0 +1,368 @@ +/* + * index_limits.c — Unified, validated resource policy for repository indexing. + */ +#include "foundation/index_limits.h" + +#include +#include +#include +#include +#include +#include +#include + +#define CBM_MIB (UINT64_C(1024) * UINT64_C(1024)) + +typedef enum { + INDEX_LIMIT_U64, + INDEX_LIMIT_INT, + INDEX_LIMIT_BOOL, + INDEX_LIMIT_STRING, +} index_limit_value_type_t; + +typedef struct { + const char *key; + size_t offset; + index_limit_value_type_t type; + uint64_t scale; + uint64_t maximum; +} index_limit_spec_t; + +#define U64_SPEC(config_key, field, multiplier, max_value) \ + {config_key, offsetof(cbm_index_limits_t, field), INDEX_LIMIT_U64, multiplier, max_value} +#define INT_SPEC(config_key, field, max_value) \ + {config_key, offsetof(cbm_index_limits_t, field), INDEX_LIMIT_INT, 1, max_value} +#define BOOL_SPEC(config_key, field) \ + {config_key, offsetof(cbm_index_limits_t, field), INDEX_LIMIT_BOOL, 1, 1} +#define STRING_SPEC(config_key, field) \ + {config_key, offsetof(cbm_index_limits_t, field), INDEX_LIMIT_STRING, 1, \ + CBM_INDEX_DENIED_ROOTS_CAP - 1} + +static const index_limit_spec_t INDEX_LIMIT_SPECS[] = { + U64_SPEC("index_max_files", max_files, 1, UINT64_C(10000000)), + U64_SPEC("index_max_directories", max_directories, 1, UINT64_C(1000000)), + U64_SPEC("index_max_entries", max_entries, 1, UINT64_C(100000000)), + U64_SPEC("index_max_depth", max_depth, 1, UINT64_C(1024)), + U64_SPEC("index_max_source_mb", max_source_bytes, CBM_MIB, UINT64_C(1048576)), + U64_SPEC("index_max_file_mb", max_file_bytes, CBM_MIB, UINT64_C(1048576)), + U64_SPEC("index_scan_timeout_seconds", scan_timeout_ms, UINT64_C(1000), UINT64_C(86400)), + INT_SPEC("index_cpu_cores", cpu_cores, 256), + INT_SPEC("index_concurrent_jobs", concurrent_jobs, 32), + U64_SPEC("index_memory_limit_mb", memory_limit_bytes, CBM_MIB, UINT64_C(1048576)), + U64_SPEC("index_max_db_mb", max_db_bytes, CBM_MIB, UINT64_C(1048576)), + U64_SPEC("index_max_staging_mb", max_staging_bytes, CBM_MIB, UINT64_C(1048576)), + U64_SPEC("index_max_task_temp_mb", max_task_temp_bytes, CBM_MIB, UINT64_C(1048576)), + U64_SPEC("index_cache_max_mb", cache_max_bytes, CBM_MIB, UINT64_C(1048576)), + U64_SPEC("index_min_free_disk_mb", min_free_disk_bytes, CBM_MIB, UINT64_C(1048576)), + U64_SPEC("index_max_duration_seconds", max_duration_ms, UINT64_C(1000), UINT64_C(604800)), + BOOL_SPEC("index_low_priority", low_priority), + STRING_SPEC("index_denied_roots", denied_roots), +}; + +static const index_limit_spec_t *find_spec(const char *key) { + if (!key) { + return NULL; + } + for (size_t i = 0; i < sizeof(INDEX_LIMIT_SPECS) / sizeof(INDEX_LIMIT_SPECS[0]); i++) { + if (strcmp(key, INDEX_LIMIT_SPECS[i].key) == 0) { + return &INDEX_LIMIT_SPECS[i]; + } + } + return NULL; +} + +static void set_error(char *error, size_t error_size, const char *message, const char *key) { + if (!error || error_size == 0) { + return; + } + (void)snprintf(error, error_size, message, key ? key : "(null)"); +} + +void cbm_index_limits_defaults(cbm_index_limits_t *limits) { + if (!limits) { + return; + } + *limits = (cbm_index_limits_t){ + .max_files = UINT64_C(100000), + .max_directories = UINT64_C(20000), + .max_entries = UINT64_C(500000), + .max_depth = UINT64_C(64), + .max_source_bytes = UINT64_C(4096) * CBM_MIB, + .max_file_bytes = UINT64_C(64) * CBM_MIB, + .scan_timeout_ms = UINT64_C(30000), + .cpu_cores = 4, + .concurrent_jobs = 1, + .memory_limit_bytes = UINT64_C(8192) * CBM_MIB, + .max_db_bytes = UINT64_C(16384) * CBM_MIB, + .max_staging_bytes = UINT64_C(20480) * CBM_MIB, + .max_task_temp_bytes = UINT64_C(24576) * CBM_MIB, + .cache_max_bytes = UINT64_C(32768) * CBM_MIB, + .min_free_disk_bytes = UINT64_C(4096) * CBM_MIB, + .max_duration_ms = UINT64_C(3600000), + .low_priority = true, + }; +} + +bool cbm_index_limits_is_config_key(const char *key) { + return find_spec(key) != NULL; +} + +size_t cbm_index_limits_config_key_count(void) { + return sizeof(INDEX_LIMIT_SPECS) / sizeof(INDEX_LIMIT_SPECS[0]); +} + +const char *cbm_index_limits_config_key_at(size_t index) { + return index < cbm_index_limits_config_key_count() ? INDEX_LIMIT_SPECS[index].key : NULL; +} + +bool cbm_index_limits_format_value(const cbm_index_limits_t *limits, const char *key, char *value, + size_t value_size) { + const index_limit_spec_t *spec = find_spec(key); + if (!limits || !spec || !value || value_size == 0) { + return false; + } + const unsigned char *field = (const unsigned char *)limits + spec->offset; + int written = -1; + if (spec->type == INDEX_LIMIT_BOOL) { + written = snprintf(value, value_size, "%s", *(const bool *)field ? "true" : "false"); + } else if (spec->type == INDEX_LIMIT_STRING) { + written = snprintf(value, value_size, "%s", (const char *)field); + } else if (spec->type == INDEX_LIMIT_INT) { + written = snprintf(value, value_size, "%d", *(const int *)field); + } else { + written = snprintf(value, value_size, "%llu", + (unsigned long long)(*(const uint64_t *)field / spec->scale)); + } + return written >= 0 && (size_t)written < value_size; +} + +bool cbm_index_limits_default_value(const char *key, char *value, size_t value_size) { + cbm_index_limits_t defaults; + cbm_index_limits_defaults(&defaults); + return cbm_index_limits_format_value(&defaults, key, value, value_size); +} + +static bool parse_positive_u64(const char *value, uint64_t maximum, uint64_t *parsed) { + if (!value || !value[0] || value[0] == '-') { + return false; + } + errno = 0; + char *end = NULL; + unsigned long long number = strtoull(value, &end, 10); + if (errno != 0 || end == value || *end != '\0' || number == 0 || (uint64_t)number > maximum) { + return false; + } + *parsed = (uint64_t)number; + return true; +} + +static bool parse_bool(const char *value, bool *parsed) { + if (value && (strcmp(value, "true") == 0 || strcmp(value, "1") == 0)) { + *parsed = true; + return true; + } + if (value && (strcmp(value, "false") == 0 || strcmp(value, "0") == 0)) { + *parsed = false; + return true; + } + return false; +} + +bool cbm_index_limits_set(cbm_index_limits_t *limits, const char *key, const char *value, + char *error, size_t error_size) { + if (!limits) { + set_error(error, error_size, "missing resource policy for %s", key); + return false; + } + const index_limit_spec_t *spec = find_spec(key); + if (!spec) { + set_error(error, error_size, "unknown index resource key: %s", key); + return false; + } + + cbm_index_limits_t updated = *limits; + unsigned char *field = (unsigned char *)&updated + spec->offset; + if (spec->type == INDEX_LIMIT_BOOL) { + bool parsed = false; + if (!parse_bool(value, &parsed)) { + set_error(error, error_size, "%s must be true or false", key); + return false; + } + *(bool *)field = parsed; + } else if (spec->type == INDEX_LIMIT_STRING) { + size_t length = value ? strlen(value) : 0; + if (!value || length > spec->maximum) { + set_error(error, error_size, "%s must be a bounded path list", key); + return false; + } + (void)memcpy(field, value, length + 1); + } else { + uint64_t parsed = 0; + if (!parse_positive_u64(value, spec->maximum, &parsed)) { + set_error(error, error_size, "%s must be a positive integer within its supported range", + key); + return false; + } + if (parsed > UINT64_MAX / spec->scale) { + set_error(error, error_size, "%s is too large after unit conversion", key); + return false; + } + uint64_t resolved = parsed * spec->scale; + if (spec->type == INDEX_LIMIT_INT) { + if (resolved > INT_MAX) { + set_error(error, error_size, "%s exceeds the integer range", key); + return false; + } + *(int *)field = (int)resolved; + } else { + *(uint64_t *)field = resolved; + } + } + + *limits = updated; + if (error && error_size > 0) { + error[0] = '\0'; + } + return true; +} + +uint64_t cbm_index_effective_memory_limit(uint64_t configured_bytes, + uint64_t detected_budget_bytes) { + if (detected_budget_bytes == 0 || configured_bytes <= detected_budget_bytes) { + return configured_bytes; + } + return detected_budget_bytes; +} + +static bool path_separator(char value) { + return value == '/' || value == '\\'; +} + +static size_t comparable_path_length(const char *path) { + size_t length = path ? strlen(path) : 0; + while (length > 1 && path_separator(path[length - 1])) { +#ifdef _WIN32 + if (length == 3 && isalpha((unsigned char)path[0]) && path[1] == ':') { + break; + } +#endif + length--; + } + return length; +} + +static bool path_character_equal(char left, char right) { + if (path_separator(left) && path_separator(right)) { + return true; + } +#ifdef _WIN32 + return tolower((unsigned char)left) == tolower((unsigned char)right); +#else + return left == right; +#endif +} + +static bool same_canonical_path(const char *left, const char *right) { + if (!left || !left[0] || !right || !right[0]) { + return false; + } + size_t left_length = comparable_path_length(left); + size_t right_length = comparable_path_length(right); + if (left_length != right_length) { + return false; + } + for (size_t i = 0; i < left_length; i++) { + if (!path_character_equal(left[i], right[i])) { + return false; + } + } + return true; +} + +static bool filesystem_root(const char *path) { + size_t length = comparable_path_length(path); + if (length == 1 && path_separator(path[0])) { + return true; + } +#ifdef _WIN32 + if (length == 3 && isalpha((unsigned char)path[0]) && path[1] == ':' && + path_separator(path[2])) { + return true; + } + if (length > 2 && path_separator(path[0]) && path_separator(path[1])) { + int components = 0; + bool in_component = false; + for (size_t i = 2; i < length; i++) { + if (path_separator(path[i])) { + in_component = false; + } else if (!in_component) { + components++; + in_component = true; + } + } + return components <= 2; + } +#endif + return false; +} + +static void set_root_reason(char *reason, size_t reason_size, const char *value) { + if (!reason || reason_size == 0) { + return; + } + (void)snprintf(reason, reason_size, "%s", value); +} + +static bool configured_root_matches(const char *repo_root, const char *configured_roots) { + if (!configured_roots || !configured_roots[0]) { + return false; + } + + char roots[CBM_INDEX_DENIED_ROOTS_CAP]; + size_t length = strlen(configured_roots); + if (length >= sizeof(roots)) { + return true; + } + (void)memcpy(roots, configured_roots, length + 1); + + char *save = NULL; + for (char *root = strtok_r(roots, ";", &save); root; root = strtok_r(NULL, ";", &save)) { + while (isspace((unsigned char)*root)) { + root++; + } + char *end = root + strlen(root); + while (end > root && isspace((unsigned char)end[-1])) { + *--end = '\0'; + } + if (root[0] && same_canonical_path(repo_root, root)) { + return true; + } + } + return false; +} + +bool cbm_index_root_allowed(const char *repo_root, const char *home_root, const char *cache_root, + const char *configured_roots, char *reason, size_t reason_size) { + set_root_reason(reason, reason_size, ""); + if (!repo_root || !repo_root[0]) { + set_root_reason(reason, reason_size, "invalid_root"); + return false; + } + if (filesystem_root(repo_root)) { + set_root_reason(reason, reason_size, "filesystem_root"); + return false; + } + if (same_canonical_path(repo_root, home_root)) { + set_root_reason(reason, reason_size, "home_directory"); + return false; + } + if (same_canonical_path(repo_root, cache_root)) { + set_root_reason(reason, reason_size, "cache_directory"); + return false; + } + if (configured_root_matches(repo_root, configured_roots)) { + set_root_reason(reason, reason_size, "configured_root"); + return false; + } + return true; +} diff --git a/src/foundation/index_limits.h b/src/foundation/index_limits.h new file mode 100644 index 000000000..3e281e072 --- /dev/null +++ b/src/foundation/index_limits.h @@ -0,0 +1,63 @@ +/* + * index_limits.h — Unified, validated resource policy for repository indexing. + */ +#ifndef CBM_INDEX_LIMITS_H +#define CBM_INDEX_LIMITS_H + +#include +#include +#include + +enum { CBM_INDEX_DENIED_ROOTS_CAP = 4096 }; + +typedef struct { + uint64_t max_files; + uint64_t max_directories; + uint64_t max_entries; + uint64_t max_depth; + uint64_t max_source_bytes; + uint64_t max_file_bytes; + uint64_t scan_timeout_ms; + int cpu_cores; + int concurrent_jobs; + uint64_t memory_limit_bytes; + uint64_t max_db_bytes; + uint64_t max_staging_bytes; + uint64_t max_task_temp_bytes; + uint64_t cache_max_bytes; + uint64_t min_free_disk_bytes; + uint64_t max_duration_ms; + bool low_priority; + char denied_roots[CBM_INDEX_DENIED_ROOTS_CAP]; +} cbm_index_limits_t; + +/* Populate the safe, platform-independent defaults documented in + * docs/INDEX_RESOURCE_LIMITS.md. */ +void cbm_index_limits_defaults(cbm_index_limits_t *limits); + +/* True only for public index-resource configuration keys. */ +bool cbm_index_limits_is_config_key(const char *key); +size_t cbm_index_limits_config_key_count(void); +const char *cbm_index_limits_config_key_at(size_t index); +bool cbm_index_limits_format_value(const cbm_index_limits_t *limits, const char *key, char *value, + size_t value_size); +bool cbm_index_limits_default_value(const char *key, char *value, size_t value_size); + +/* Parse and atomically apply one public configuration value. MiB and seconds + * keys are converted to bytes and milliseconds with overflow checks. On + * failure, limits is unchanged and error receives a user-facing explanation. */ +bool cbm_index_limits_set(cbm_index_limits_t *limits, const char *key, const char *value, + char *error, size_t error_size); + +/* Clamp the configured per-worker ceiling to a detected allocator/host budget. + * A zero detected budget means detection is unavailable, not zero capacity. */ +uint64_t cbm_index_effective_memory_limit(uint64_t configured_bytes, + uint64_t detected_budget_bytes); + +/* Reject roots that are too broad to be repositories. All paths must already + * be absolute and canonical. configured_roots is a semicolon-separated list + * of exact roots; descendants remain eligible for indexing. */ +bool cbm_index_root_allowed(const char *repo_root, const char *home_root, const char *cache_root, + const char *configured_roots, char *reason, size_t reason_size); + +#endif /* CBM_INDEX_LIMITS_H */ diff --git a/src/foundation/platform.c b/src/foundation/platform.c index ce4c43dc6..d09425e7c 100644 --- a/src/foundation/platform.c +++ b/src/foundation/platform.c @@ -200,12 +200,30 @@ char *cbm_normalize_path_sep(char *path) { return path; } +bool cbm_disk_free_bytes(const char *path, uint64_t *bytes_out) { + if (!path || !bytes_out) { + return false; + } + wchar_t *wide_path = cbm_path_to_wide(path); + if (!wide_path) { + return false; + } + ULARGE_INTEGER available; + bool ok = GetDiskFreeSpaceExW(wide_path, &available, NULL, NULL) != 0; + free(wide_path); + if (ok) { + *bytes_out = available.QuadPart; + } + return ok; +} + #else /* POSIX (macOS + Linux) */ /* ── POSIX implementation ────────────────────────────────── */ #include #include +#include #include #include @@ -336,6 +354,21 @@ char *cbm_normalize_path_sep(char *path) { return path; } +bool cbm_disk_free_bytes(const char *path, uint64_t *bytes_out) { + if (!path || !bytes_out) { + return false; + } + struct statvfs info; + if (statvfs(path, &info) != 0) { + return false; + } + uint64_t blocks = (uint64_t)info.f_bavail; + uint64_t block_size = (uint64_t)info.f_frsize; + *bytes_out = + block_size > 0 && blocks > UINT64_MAX / block_size ? UINT64_MAX : blocks * block_size; + return true; +} + #endif /* _WIN32 */ /* ── Environment variables ──────────────────────────── */ diff --git a/src/foundation/platform.h b/src/foundation/platform.h index 2511a060e..8bc3d1f87 100644 --- a/src/foundation/platform.h +++ b/src/foundation/platform.h @@ -114,6 +114,14 @@ cbm_system_info_t cbm_system_info(void); * initial=false: max(1, perf_cores-1) (leave headroom for user apps) */ int cbm_default_worker_count(bool initial); +/* Set a cap for worker-count decisions made by the current controller thread. + * A non-positive value clears the cap. Returns the previous value so callers + * can restore nested scopes. */ +int cbm_worker_limit_set_for_thread(int max_workers); + +/* Available bytes on the filesystem containing an existing path. */ +bool cbm_disk_free_bytes(const char *path, uint64_t *bytes_out); + /* ── Environment variables ──────────────────────────────────────── */ /* Thread-safe getenv: copies the value into a caller-provided buffer. diff --git a/src/foundation/subprocess.c b/src/foundation/subprocess.c index 5f3530f1f..f07a269ed 100644 --- a/src/foundation/subprocess.c +++ b/src/foundation/subprocess.c @@ -24,9 +24,17 @@ #include #include #include +#include #include #include #include +#ifdef __APPLE__ +#include +#elif defined(__linux__) +#include +#include +#include +#endif #endif /* NTSTATUS severity ERROR (top two bits set) covers the Windows crash exception @@ -1191,6 +1199,169 @@ bool cbm_subprocess_request_cancel(cbm_subprocess_t *process) { } } +#if defined(__APPLE__) || defined(__linux__) +static bool cbm_memory_add(size_t *total, uint64_t value) { + if (value > SIZE_MAX || *total > SIZE_MAX - (size_t)value) { + *total = SIZE_MAX; + return false; + } + *total += (size_t)value; + return true; +} +#endif + +#ifdef __APPLE__ +static bool cbm_subprocess_memory_macos(cbm_subprocess_t *process, size_t *bytes_out) { + int required = proc_listpids(PROC_ALL_PIDS, 0, NULL, 0); + if (required <= 0) { + return false; + } + pid_t *pids = calloc((size_t)required / sizeof(pid_t) + 1, sizeof(pid_t)); + if (!pids) { + return false; + } + int received = proc_listpids(PROC_ALL_PIDS, 0, pids, required); + if (received <= 0) { + free(pids); + return false; + } + + size_t total = 0; + bool observed = false; + size_t count = (size_t)received / sizeof(pid_t); + for (size_t i = 0; i < count; i++) { + if (pids[i] <= 0) { + continue; + } + struct proc_taskallinfo info; + if (proc_pidinfo(pids[i], PROC_PIDTASKALLINFO, 0, &info, sizeof(info)) != + (int)sizeof(info) || + (pid_t)info.pbsd.pbi_pgid != process->pgid) { + continue; + } + observed = true; + (void)cbm_memory_add(&total, info.ptinfo.pti_resident_size); + } + free(pids); + *bytes_out = total; + return observed; +} +#elif defined(__linux__) +static bool cbm_linux_process_group_and_rss(const char *pid_text, pid_t expected_group, + size_t page_size, size_t *rss_out) { + char path[PATH_MAX]; + int written = snprintf(path, sizeof(path), "/proc/%s/stat", pid_text); + if (written <= 0 || (size_t)written >= sizeof(path)) { + return false; + } + FILE *stat_file = fopen(path, "r"); + if (!stat_file) { + return false; + } + char stat_line[4096]; + bool read_stat = fgets(stat_line, sizeof(stat_line), stat_file) != NULL; + (void)fclose(stat_file); + char *command_end = read_stat ? strrchr(stat_line, ')') : NULL; + char state = '\0'; + long parent = 0; + long process_group = 0; + if (!command_end || + sscanf(command_end + 1, " %c %ld %ld", &state, &parent, &process_group) != 3 || + (pid_t)process_group != expected_group) { + return false; + } + + written = snprintf(path, sizeof(path), "/proc/%s/statm", pid_text); + if (written <= 0 || (size_t)written >= sizeof(path)) { + return false; + } + FILE *memory_file = fopen(path, "r"); + if (!memory_file) { + return false; + } + unsigned long ignored_pages = 0; + unsigned long resident_pages = 0; + bool read_memory = fscanf(memory_file, "%lu %lu", &ignored_pages, &resident_pages) == 2; + (void)fclose(memory_file); + if (!read_memory) { + return false; + } + if (resident_pages > SIZE_MAX / page_size) { + *rss_out = SIZE_MAX; + } else { + *rss_out = (size_t)resident_pages * page_size; + } + return true; +} + +static bool cbm_subprocess_memory_linux(cbm_subprocess_t *process, size_t *bytes_out) { + DIR *directory = opendir("/proc"); + if (!directory) { + return false; + } + long configured_page_size = sysconf(_SC_PAGESIZE); + size_t page_size = configured_page_size > 0 ? (size_t)configured_page_size : 4096U; + size_t total = 0; + bool observed = false; + struct dirent *entry; + while ((entry = readdir(directory)) != NULL) { + bool numeric = entry->d_name[0] != '\0'; + for (const char *cursor = entry->d_name; numeric && *cursor; cursor++) { + numeric = isdigit((unsigned char)*cursor) != 0; + } + if (!numeric) { + continue; + } + size_t rss = 0; + if (cbm_linux_process_group_and_rss(entry->d_name, process->pgid, page_size, &rss)) { + observed = true; + (void)cbm_memory_add(&total, rss); + } + } + (void)closedir(directory); + *bytes_out = total; + return observed; +} +#endif + +bool cbm_subprocess_memory_bytes(cbm_subprocess_t *process, size_t *bytes_out) { + if (!process || !bytes_out) { + return false; + } + *bytes_out = 0; +#ifdef _WIN32 + JOBOBJECT_EXTENDED_LIMIT_INFORMATION information; + memset(&information, 0, sizeof(information)); + if (!process->job || !QueryInformationJobObject(process->job, JobObjectExtendedLimitInformation, + &information, sizeof(information), NULL)) { + return false; + } + *bytes_out = + information.PeakJobMemoryUsed > SIZE_MAX ? SIZE_MAX : (size_t)information.PeakJobMemoryUsed; + return true; +#elif defined(__APPLE__) + return cbm_subprocess_memory_macos(process, bytes_out); +#elif defined(__linux__) + return cbm_subprocess_memory_linux(process, bytes_out); +#else + return false; +#endif +} + +bool cbm_subprocess_set_low_priority(cbm_subprocess_t *process, bool low_priority) { + if (!process) { + return false; + } + if (!low_priority) { + return true; + } +#ifdef _WIN32 + return process->process && SetPriorityClass(process->process, BELOW_NORMAL_PRIORITY_CLASS) != 0; +#else + return setpriority(PRIO_PROCESS, process->pid, 10) == 0; +#endif +} + void cbm_subprocess_destroy(cbm_subprocess_t *process) { if (!process || atomic_load_explicit(&process->lifecycle, memory_order_acquire) != CBM_SUBPROCESS_TERMINAL) { diff --git a/src/foundation/subprocess.h b/src/foundation/subprocess.h index f6ff7f7fd..dd9010a08 100644 --- a/src/foundation/subprocess.h +++ b/src/foundation/subprocess.h @@ -125,6 +125,16 @@ cbm_proc_poll_t cbm_subprocess_poll(cbm_subprocess_t *process, cbm_proc_result_t * performs signal delivery/escalation. */ bool cbm_subprocess_request_cancel(cbm_subprocess_t *process); +/* Return the resident bytes owned by the contained process tree. On Windows + * this is the Job Object's peak committed memory; on macOS/Linux it is the + * current summed RSS of processes in the owned process group. */ +bool cbm_subprocess_memory_bytes(cbm_subprocess_t *process, size_t *bytes_out); + +/* Lower the contained worker's scheduling priority when enabled. false is a + * no-op that succeeds. This is intentionally separate from spawn so callers + * can apply a resolved per-request policy before their first poll. */ +bool cbm_subprocess_set_low_priority(cbm_subprocess_t *process, bool low_priority); + /* Release a terminal handle. This never waits or implicitly cancels; passing a * still-running handle violates the API contract. NULL is a no-op. */ void cbm_subprocess_destroy(cbm_subprocess_t *process); diff --git a/src/foundation/system_info.c b/src/foundation/system_info.c index 8ba237f6c..359057905 100644 --- a/src/foundation/system_info.c +++ b/src/foundation/system_info.c @@ -11,6 +11,7 @@ * Results are cached after first call (immutable hardware properties). */ #include "foundation/constants.h" +#include "foundation/compat.h" enum { DEFAULT_CORES = 1, MIN_WORKERS = 1, CBM_WORKERS_MAX = 256 }; #include "foundation/log.h" @@ -263,6 +264,7 @@ static cbm_system_info_t detect_system_windows(void) { static int info_cached = 0; static cbm_system_info_t cached_info; +static CBM_TLS int worker_limit; cbm_system_info_t cbm_system_info(void) { if (!info_cached) { @@ -287,20 +289,34 @@ int cbm_default_worker_count(bool initial) { * Same precedence shape as other CBM_* env overrides: * explicit override > implicit detection. */ char buf[CBM_SZ_32]; + int workers = 0; if (cbm_safe_getenv("CBM_WORKERS", buf, sizeof(buf), NULL) != NULL) { long n = strtol(buf, NULL, CBM_DECIMAL_BASE); if (n >= MIN_WORKERS && n <= CBM_WORKERS_MAX) { - return (int)n; + workers = (int)n; + } else { + cbm_log_warn("workers.env.invalid", "value", buf, "fallback", "sysconf"); } - cbm_log_warn("workers.env.invalid", "value", buf, "fallback", "sysconf"); } - cbm_system_info_t info = cbm_system_info(); - if (initial) { - /* Use all cores for initial indexing — user is waiting */ - return info.total_cores; + if (workers == 0) { + cbm_system_info_t info = cbm_system_info(); + if (initial) { + /* Use all cores for initial indexing — user is waiting */ + workers = info.total_cores; + } else { + /* Incremental: leave headroom for user's apps */ + workers = info.perf_cores - SKIP_ONE; + if (workers < MIN_WORKERS) { + workers = MIN_WORKERS; + } + } } - /* Incremental: leave headroom for user's apps */ - int workers = info.perf_cores - SKIP_ONE; - return workers > 0 ? workers : MIN_WORKERS; + return worker_limit > 0 && workers > worker_limit ? worker_limit : workers; +} + +int cbm_worker_limit_set_for_thread(int max_workers) { + int previous = worker_limit; + worker_limit = max_workers > 0 ? max_workers : 0; + return previous; } diff --git a/src/mcp/index_supervisor.c b/src/mcp/index_supervisor.c index f3ca4dffe..700896f35 100644 --- a/src/mcp/index_supervisor.c +++ b/src/mcp/index_supervisor.c @@ -374,6 +374,14 @@ struct cbm_index_worker_handle { bool process_terminal; cbm_proc_result_t process_result; atomic_bool terminal; + atomic_bool explicit_cancel_requested; + size_t memory_limit_bytes; + uint64_t task_temp_limit_bytes; + uint64_t max_duration_ms; + uint64_t started_ms; + cbm_index_resource_limit_t resource_limit; + uint64_t resource_observed; + uint64_t resource_limit_value; cbm_index_worker_result_t result; }; @@ -550,7 +558,22 @@ static bool worker_unique_file(char *out, size_t out_size, const char *kind) { static bool worker_result_succeeded(const cbm_index_worker_result_t *result) { return result && result->outcome == CBM_PROC_CLEAN && !result->cancellation_requested && - result->tree_quiesced && !result->supervision_failed; + result->resource_limit == CBM_INDEX_RESOURCE_LIMIT_NONE && result->tree_quiesced && + !result->supervision_failed; +} + +const char *cbm_index_resource_limit_name(cbm_index_resource_limit_t limit) { + switch (limit) { + case CBM_INDEX_RESOURCE_LIMIT_NONE: + return "none"; + case CBM_INDEX_RESOURCE_LIMIT_MEMORY: + return "memory_bytes"; + case CBM_INDEX_RESOURCE_LIMIT_TASK_TEMP: + return "task_temp_bytes"; + case CBM_INDEX_RESOURCE_LIMIT_DURATION: + return "duration_ms"; + } + return "unknown"; } static void worker_terminal_log(cbm_index_worker_handle_t *handle) { @@ -566,6 +589,16 @@ static void worker_terminal_log(cbm_index_worker_handle_t *handle) { } else if (handle->result.supervision_failed || !handle->result.tree_quiesced) { cbm_log_error("index.supervisor.containment_failed", "outcome", cbm_proc_outcome_str(handle->result.outcome), "log", handle->log_path); + } else if (handle->result.resource_limit != CBM_INDEX_RESOURCE_LIMIT_NONE) { + char observed_text[32]; + char limit_text[32]; + (void)snprintf(observed_text, sizeof(observed_text), "%llu", + (unsigned long long)handle->result.resource_observed); + (void)snprintf(limit_text, sizeof(limit_text), "%llu", + (unsigned long long)handle->result.resource_limit_value); + cbm_log_warn("index.supervisor.resource_limit", "limit", + cbm_index_resource_limit_name(handle->result.resource_limit), "observed", + observed_text, "configured", limit_text); } else if (handle->result.cancellation_requested) { cbm_log_warn("index.supervisor.worker_cancelled", "outcome", cbm_proc_outcome_str(handle->result.outcome), "log", handle->log_path); @@ -616,6 +649,8 @@ int cbm_index_worker_start_with_log(const char *args_json, size_t memory_budget_ return -1; } atomic_init(&handle->terminal, false); + atomic_init(&handle->explicit_cancel_requested, false); + handle->started_ms = cbm_now_ms(); handle->log_callback = log_callback; handle->log_context = log_context; worker_result_init(&handle->result); @@ -686,6 +721,89 @@ int cbm_index_worker_start(const char *args_json, size_t memory_budget_bytes, bo marker_file, quarantine_file, NULL, NULL, handle_out); } +void cbm_index_worker_set_max_duration(cbm_index_worker_handle_t *handle, + uint64_t max_duration_ms) { + if (handle && !atomic_load_explicit(&handle->terminal, memory_order_acquire)) { + handle->max_duration_ms = max_duration_ms; + } +} + +void cbm_index_worker_set_memory_limit(cbm_index_worker_handle_t *handle, + size_t memory_limit_bytes) { + if (handle && !atomic_load_explicit(&handle->terminal, memory_order_acquire)) { + handle->memory_limit_bytes = memory_limit_bytes; + } +} + +void cbm_index_worker_set_task_temp_limit(cbm_index_worker_handle_t *handle, + uint64_t task_temp_limit_bytes) { + if (handle && !atomic_load_explicit(&handle->terminal, memory_order_acquire)) { + handle->task_temp_limit_bytes = task_temp_limit_bytes; + } +} + +bool cbm_index_worker_set_low_priority(cbm_index_worker_handle_t *handle, bool low_priority) { + if (!handle || atomic_load_explicit(&handle->terminal, memory_order_acquire)) { + return false; + } + return cbm_subprocess_set_low_priority(handle->process, low_priority); +} + +static uint64_t worker_task_temp_bytes(const cbm_index_worker_handle_t *handle) { + uint64_t total = 0; + const char *paths[] = {handle->response_path, handle->log_path}; + for (size_t i = 0; i < sizeof(paths) / sizeof(paths[0]); i++) { + int64_t size = cbm_file_size(paths[i]); + if (size <= 0) { + continue; + } + uint64_t add = (uint64_t)size; + total = UINT64_MAX - total < add ? UINT64_MAX : total + add; + } + return total; +} + +static void worker_enforce_resource_limits(cbm_index_worker_handle_t *handle) { + if (!handle || handle->resource_limit != CBM_INDEX_RESOURCE_LIMIT_NONE || + atomic_load_explicit(&handle->explicit_cancel_requested, memory_order_acquire)) { + return; + } + + cbm_index_resource_limit_t breached = CBM_INDEX_RESOURCE_LIMIT_NONE; + uint64_t observed = 0; + uint64_t configured = 0; + size_t memory_bytes = 0; + if (handle->memory_limit_bytes > 0 && + cbm_subprocess_memory_bytes(handle->process, &memory_bytes) && + memory_bytes > handle->memory_limit_bytes) { + breached = CBM_INDEX_RESOURCE_LIMIT_MEMORY; + observed = (uint64_t)memory_bytes; + configured = (uint64_t)handle->memory_limit_bytes; + } + if (breached == CBM_INDEX_RESOURCE_LIMIT_NONE && handle->task_temp_limit_bytes > 0) { + uint64_t task_temp_bytes = worker_task_temp_bytes(handle); + if (task_temp_bytes > handle->task_temp_limit_bytes) { + breached = CBM_INDEX_RESOURCE_LIMIT_TASK_TEMP; + observed = task_temp_bytes; + configured = handle->task_temp_limit_bytes; + } + } + if (breached == CBM_INDEX_RESOURCE_LIMIT_NONE && handle->max_duration_ms > 0) { + uint64_t elapsed_ms = cbm_now_ms() - handle->started_ms; + if (elapsed_ms > handle->max_duration_ms) { + breached = CBM_INDEX_RESOURCE_LIMIT_DURATION; + observed = elapsed_ms; + configured = handle->max_duration_ms; + } + } + if (breached != CBM_INDEX_RESOURCE_LIMIT_NONE && + cbm_subprocess_request_cancel(handle->process)) { + handle->resource_limit = breached; + handle->resource_observed = observed; + handle->resource_limit_value = configured; + } +} + cbm_index_worker_poll_t cbm_index_worker_poll(cbm_index_worker_handle_t *handle, const cbm_index_worker_result_t **result_out) { if (result_out) { @@ -700,6 +818,7 @@ cbm_index_worker_poll_t cbm_index_worker_poll(cbm_index_worker_handle_t *handle, } bool relay_caught_up = true; if (!handle->process_terminal) { + worker_enforce_resource_limits(handle); cbm_proc_result_t process_result; cbm_proc_poll_t state = cbm_subprocess_poll(handle->process, &process_result); relay_caught_up = worker_relay_log(handle); @@ -724,10 +843,14 @@ cbm_index_worker_poll_t cbm_index_worker_poll(cbm_index_worker_handle_t *handle, handle->result.outcome = process_result->outcome; handle->result.exit_code = process_result->exit_code; handle->result.term_signal = process_result->term_signal; - handle->result.cancellation_requested = process_result->cancellation_requested; + handle->result.cancellation_requested = + atomic_load_explicit(&handle->explicit_cancel_requested, memory_order_acquire); handle->result.forced = process_result->forced; handle->result.tree_quiesced = process_result->tree_quiesced; handle->result.supervision_failed = process_result->supervision_failed; + handle->result.resource_limit = handle->resource_limit; + handle->result.resource_observed = handle->resource_observed; + handle->result.resource_limit_value = handle->resource_limit_value; if (worker_result_succeeded(&handle->result)) { worker_response_read_status_t response_status; handle->result.response = slurp_worker_response(handle->response_path, &response_status); @@ -747,8 +870,14 @@ cbm_index_worker_poll_t cbm_index_worker_poll(cbm_index_worker_handle_t *handle, } bool cbm_index_worker_request_cancel(cbm_index_worker_handle_t *handle) { - return handle && !atomic_load_explicit(&handle->terminal, memory_order_acquire) && - cbm_subprocess_request_cancel(handle->process); + if (!handle || atomic_load_explicit(&handle->terminal, memory_order_acquire)) { + return false; + } + bool accepted = cbm_subprocess_request_cancel(handle->process); + if (accepted) { + atomic_store_explicit(&handle->explicit_cancel_requested, true, memory_order_release); + } + return accepted; } const char *cbm_index_worker_response_path(const cbm_index_worker_handle_t *handle) { @@ -773,11 +902,11 @@ void cbm_index_worker_destroy(cbm_index_worker_handle_t *handle) { free(handle); } -int cbm_index_spawn_worker_with_log_cancel(const char *args_json, bool single_thread, - const char *marker_file, const char *quarantine_file, - cbm_proc_log_cb log_callback, void *log_context, - const atomic_int *cancel_requested, - cbm_index_worker_result_t *result) { +int cbm_index_spawn_worker_with_limits_and_log_cancel( + const char *args_json, size_t memory_limit_bytes, uint64_t task_temp_limit_bytes, + uint64_t max_duration_ms, bool low_priority, bool single_thread, const char *marker_file, + const char *quarantine_file, cbm_proc_log_cb log_callback, void *log_context, + const atomic_int *cancel_requested, cbm_index_worker_result_t *result) { if (!result) { return -1; } @@ -787,6 +916,12 @@ int cbm_index_spawn_worker_with_log_cancel(const char *args_json, bool single_th log_callback, log_context, &handle) != 0) { return -1; } + cbm_index_worker_set_memory_limit(handle, memory_limit_bytes); + cbm_index_worker_set_task_temp_limit(handle, task_temp_limit_bytes); + cbm_index_worker_set_max_duration(handle, max_duration_ms); + if (!cbm_index_worker_set_low_priority(handle, low_priority)) { + cbm_log_warn("index.supervisor.priority_not_lowered", "action", "continue"); + } const cbm_index_worker_result_t *cached = NULL; bool cancellation_forwarded = false; for (;;) { @@ -810,6 +945,16 @@ int cbm_index_spawn_worker_with_log_cancel(const char *args_json, bool single_th return 0; } +int cbm_index_spawn_worker_with_log_cancel(const char *args_json, bool single_thread, + const char *marker_file, const char *quarantine_file, + cbm_proc_log_cb log_callback, void *log_context, + const atomic_int *cancel_requested, + cbm_index_worker_result_t *result) { + return cbm_index_spawn_worker_with_limits_and_log_cancel( + args_json, 0, 0, 0, false, single_thread, marker_file, quarantine_file, log_callback, + log_context, cancel_requested, result); +} + int cbm_index_spawn_worker_with_log(const char *args_json, bool single_thread, const char *marker_file, const char *quarantine_file, cbm_proc_log_cb log_callback, void *log_context, diff --git a/src/mcp/index_supervisor.h b/src/mcp/index_supervisor.h index 5f8f57018..c85079845 100644 --- a/src/mcp/index_supervisor.h +++ b/src/mcp/index_supervisor.h @@ -23,6 +23,7 @@ #include #include #include +#include #include "foundation/subprocess.h" /* cbm_proc_outcome_t */ @@ -108,6 +109,15 @@ int cbm_index_supervisor_spawn_count(void); * recovery is parallel-only; no sequential runs). */ int cbm_index_supervisor_spawn_st_count(void); +typedef enum { + CBM_INDEX_RESOURCE_LIMIT_NONE = 0, + CBM_INDEX_RESOURCE_LIMIT_MEMORY, + CBM_INDEX_RESOURCE_LIMIT_TASK_TEMP, + CBM_INDEX_RESOURCE_LIMIT_DURATION, +} cbm_index_resource_limit_t; + +const char *cbm_index_resource_limit_name(cbm_index_resource_limit_t limit); + typedef struct { cbm_proc_outcome_t outcome; /* how the worker ended */ int exit_code; /* worker exit code (-1 if signalled) */ @@ -117,8 +127,11 @@ typedef struct { bool tree_quiesced; bool supervision_failed; bool response_rejected; /* clean worker exceeded the bounded response protocol */ - char *response; /* worker result only after a contained, uncancelled CLEAN exit; - * borrowed for async polls, caller-owned from the sync wrapper */ + cbm_index_resource_limit_t resource_limit; + uint64_t resource_observed; + uint64_t resource_limit_value; + char *response; /* worker result only after a contained, uncancelled CLEAN exit; + * borrowed for async polls, caller-owned from the sync wrapper */ } cbm_index_worker_result_t; /* Daemon-owned, nonblocking supervisor for one contained worker process tree. */ @@ -147,6 +160,17 @@ int cbm_index_worker_start_with_log(const char *args_json, size_t memory_budget_ const char *quarantine_file, cbm_proc_log_cb log_callback, void *log_context, cbm_index_worker_handle_t **handle_out); +/* Configure process-tree resource ceilings for a newly started worker. These + * are distinct from memory_budget_bytes, which remains the worker allocator's + * internal budget. Zero disables a ceiling. The owner must call these before + * polling the handle. */ +void cbm_index_worker_set_memory_limit(cbm_index_worker_handle_t *handle, + size_t memory_limit_bytes); +void cbm_index_worker_set_task_temp_limit(cbm_index_worker_handle_t *handle, + uint64_t task_temp_limit_bytes); +void cbm_index_worker_set_max_duration(cbm_index_worker_handle_t *handle, uint64_t max_duration_ms); +bool cbm_index_worker_set_low_priority(cbm_index_worker_handle_t *handle, bool low_priority); + /* Strictly nonblocking and called by one owner thread/event loop. result_out is * set to NULL while running and to a borrowed immutable cached result only at * terminal; repeated terminal polls return the same result until destroy. */ @@ -200,6 +224,12 @@ int cbm_index_spawn_worker_with_log_cancel(const char *args_json, bool single_th const atomic_int *cancel_requested, cbm_index_worker_result_t *result); +int cbm_index_spawn_worker_with_limits_and_log_cancel( + const char *args_json, size_t memory_limit_bytes, uint64_t task_temp_limit_bytes, + uint64_t max_duration_ms, bool low_priority, bool single_thread, const char *marker_file, + const char *quarantine_file, cbm_proc_log_cb log_callback, void *log_context, + const atomic_int *cancel_requested, cbm_index_worker_result_t *result); + void cbm_index_worker_result_free(cbm_index_worker_result_t *result); #endif /* CBM_INDEX_SUPERVISOR_H */ diff --git a/src/mcp/mcp.c b/src/mcp/mcp.c index 78706924c..c231ad24e 100644 --- a/src/mcp/mcp.c +++ b/src/mcp/mcp.c @@ -59,6 +59,7 @@ enum { #include "foundation/compat_thread.h" #include "foundation/log.h" #include "foundation/limits.h" +#include "foundation/index_limits.h" #include "foundation/subprocess.h" #include "mcp/index_supervisor.h" #include "mcp/compact_out.h" @@ -7093,6 +7094,34 @@ static char *build_worker_failure_response(const char *args, cbm_proc_outcome_t return result; } +static char *build_worker_resource_limit_response(const char *args, + const cbm_index_worker_result_t *worker_result) { + char *repo_path = cbm_mcp_get_string_arg(args, "repo_path"); + yyjson_mut_doc *doc = yyjson_mut_doc_new(NULL); + yyjson_mut_val *root = yyjson_mut_obj(doc); + yyjson_mut_doc_set_root(doc, root); + yyjson_mut_obj_add_str(doc, root, "status", "error"); + yyjson_mut_obj_add_str(doc, root, "outcome", "resource_limit_exceeded"); + yyjson_mut_obj_add_str(doc, root, "resource", + cbm_index_resource_limit_name(worker_result->resource_limit)); + yyjson_mut_obj_add_uint(doc, root, "observed", worker_result->resource_observed); + yyjson_mut_obj_add_uint(doc, root, "limit", worker_result->resource_limit_value); + yyjson_mut_obj_add_str( + doc, root, "hint", + "Indexing exceeded its configured resource limit. The complete worker process tree " + "was terminated; increase the matching index resource setting only for a trusted " + "repository."); + if (repo_path) { + yyjson_mut_obj_add_strcpy(doc, root, "repo_path", repo_path); + } + char *json = yy_doc_to_str(doc); + yyjson_mut_doc_free(doc); + free(repo_path); + char *result = cbm_mcp_text_result(json, true); + free(json); + return result; +} + static char *build_worker_unsafe_terminal_response(const char *args, cbm_proc_outcome_t outcome, bool cancellation_requested) { char *repo_path = cbm_mcp_get_string_arg(args, "repo_path"); @@ -7257,6 +7286,10 @@ cbm_mcp_supervised_result_disposition_t cbm_mcp_supervised_result_disposition( if (spawn_result != 0 || !worker_result || worker_result->outcome == CBM_PROC_SPAWN_FAILED) { return CBM_MCP_SUPERVISED_RESULT_FALLBACK; } + if (worker_result->resource_limit != CBM_INDEX_RESOURCE_LIMIT_NONE && + worker_result->tree_quiesced && !worker_result->supervision_failed) { + return CBM_MCP_SUPERVISED_RESULT_RESOURCE_LIMIT; + } if (worker_result->cancellation_requested || !worker_result->tree_quiesced || worker_result->supervision_failed) { return CBM_MCP_SUPERVISED_RESULT_UNSAFE_TERMINAL; @@ -7279,14 +7312,27 @@ cbm_mcp_supervised_result_disposition_t cbm_mcp_supervised_result_disposition( * - a contained-failure response only if even that cannot produce a clean run. * A physical CBM host never falls back to its in-process pipeline: an initial * start/protocol failure is returned as an explicit error response. */ -static char *index_run_supervised(cbm_mcp_server_t *srv, const char *args) { +static size_t index_limit_size(uint64_t value) { + return value > SIZE_MAX ? SIZE_MAX : (size_t)value; +} + +static size_t index_effective_memory_limit(uint64_t configured_bytes) { + uint64_t effective = + cbm_index_effective_memory_limit(configured_bytes, (uint64_t)cbm_mem_budget()); + return index_limit_size(effective); +} + +static char *index_run_supervised(cbm_mcp_server_t *srv, const char *args, + const cbm_index_limits_t *limits) { invalidate_cached_store(srv); /* First attempt: normal parallel run. */ cbm_index_worker_result_t wr; - int rc = cbm_index_spawn_worker_with_log_cancel( - args, false, NULL, NULL, srv ? srv->index_log_callback : NULL, - srv ? srv->index_log_context : NULL, srv ? &srv->pipeline_cancel_requested : NULL, &wr); + int rc = cbm_index_spawn_worker_with_limits_and_log_cancel( + args, index_effective_memory_limit(limits->memory_limit_bytes), limits->max_task_temp_bytes, + limits->max_duration_ms, limits->low_priority, false, NULL, NULL, + srv ? srv->index_log_callback : NULL, srv ? srv->index_log_context : NULL, + srv ? &srv->pipeline_cancel_requested : NULL, &wr); cbm_mcp_supervised_result_disposition_t disposition = cbm_mcp_supervised_result_disposition(rc, &wr); @@ -7303,6 +7349,12 @@ static char *index_run_supervised(cbm_mcp_server_t *srv, const char *args) { invalidate_cached_store(srv); return failure; } + if (disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_LIMIT) { + char *failure = build_worker_resource_limit_response(args, &wr); + cbm_index_worker_result_free(&wr); + invalidate_cached_store(srv); + return failure; + } if (disposition == CBM_MCP_SUPERVISED_RESULT_SUCCESS) { /* Clean exit → transfer the worker's response (the common path). */ char *resp = wr.response; /* transfer ownership to caller (may be NULL) */ @@ -7359,8 +7411,10 @@ static char *index_run_supervised(cbm_mcp_server_t *srv, const char *args) { bool terminal_cancelled = false; for (int i = 0; i < cap; i++) { cbm_index_worker_result_t wr2; - int rc2 = cbm_index_spawn_worker_with_log_cancel( - args, /*single_thread=*/false, marker_path, quarantine_path, + int rc2 = cbm_index_spawn_worker_with_limits_and_log_cancel( + args, index_effective_memory_limit(limits->memory_limit_bytes), + limits->max_task_temp_bytes, limits->max_duration_ms, limits->low_priority, + /*single_thread=*/false, marker_path, quarantine_path, srv ? srv->index_log_callback : NULL, srv ? srv->index_log_context : NULL, srv ? &srv->pipeline_cancel_requested : NULL, &wr2); cbm_mcp_supervised_result_disposition_t recovery_disposition = @@ -7377,6 +7431,11 @@ static char *index_run_supervised(cbm_mcp_server_t *srv, const char *args) { cbm_index_worker_result_free(&wr2); break; } + if (recovery_disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_LIMIT) { + resp = build_worker_resource_limit_response(args, &wr2); + cbm_index_worker_result_free(&wr2); + break; + } if (recovery_disposition == CBM_MCP_SUPERVISED_RESULT_SUCCESS) { resp = wr2.response; /* transfer ownership to caller */ wr2.response = NULL; @@ -7450,10 +7509,12 @@ static char *index_run_supervised(cbm_mcp_server_t *srv, const char *args) { * so it cannot itself hang. Rare given monotonic progress. */ if (!resp && !unsafe_terminal && quarantined > 0) { cbm_index_worker_result_t wrp; - int rcp = cbm_index_spawn_worker_with_log_cancel( - args, /*single_thread=*/false, NULL, quarantine_path, - srv ? srv->index_log_callback : NULL, srv ? srv->index_log_context : NULL, - srv ? &srv->pipeline_cancel_requested : NULL, &wrp); + int rcp = cbm_index_spawn_worker_with_limits_and_log_cancel( + args, index_effective_memory_limit(limits->memory_limit_bytes), + limits->max_task_temp_bytes, limits->max_duration_ms, limits->low_priority, + /*single_thread=*/false, NULL, quarantine_path, srv ? srv->index_log_callback : NULL, + srv ? srv->index_log_context : NULL, srv ? &srv->pipeline_cancel_requested : NULL, + &wrp); cbm_mcp_supervised_result_disposition_t partial_disposition = cbm_mcp_supervised_result_disposition(rcp, &wrp); if (partial_disposition == CBM_MCP_SUPERVISED_RESULT_SUCCESS) { @@ -7467,6 +7528,8 @@ static char *index_run_supervised(cbm_mcp_server_t *srv, const char *args) { last_outcome = wrp.outcome; unsafe_terminal = true; terminal_cancelled = wrp.cancellation_requested; + } else if (partial_disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_LIMIT) { + resp = build_worker_resource_limit_response(args, &wrp); } cbm_index_worker_result_free(&wrp); } @@ -7487,6 +7550,9 @@ static char *index_run_supervised(cbm_mcp_server_t *srv, const char *args) { * run it through index_run_supervised. Shared by the session auto-index (srv * present → its cached store is invalidated) and the watcher re-index (srv NULL). * Returns the worker's response string (caller frees) or NULL to degrade. */ +static char *index_args_with_repo_path(const char *args, const char *canonical_repo_path, + const cbm_index_limits_t *limits); + static char *index_run_supervised_path(cbm_mcp_server_t *srv, const char *root_path) { if (!root_path || !root_path[0]) { return NULL; @@ -7500,8 +7566,20 @@ static char *index_run_supervised_path(cbm_mcp_server_t *srv, const char *root_p if (!args) { return NULL; } - char *resp = index_run_supervised(srv, args); + cbm_index_limits_t limits; + char limits_error[CBM_SZ_256] = {0}; + if (!cbm_config_load_index_limits(srv ? srv->config : NULL, &limits, limits_error, + sizeof(limits_error))) { + free(args); + return NULL; + } + char *worker_args = index_args_with_repo_path(args, root_path, &limits); free(args); + if (!worker_args) { + return NULL; + } + char *resp = index_run_supervised(srv, worker_args, &limits); + free(worker_args); return resp; } @@ -7543,8 +7621,9 @@ static bool resolve_session_repo_path(cbm_mcp_server_t *srv, char **repo_path) { /* Preserve every index option while replacing all caller-supplied repo_path * keys with the one canonical path that was actually authorized. */ -static char *index_args_with_repo_path(const char *args, const char *canonical_repo_path) { - if (!args || !canonical_repo_path) { +static char *index_args_with_repo_path(const char *args, const char *canonical_repo_path, + const cbm_index_limits_t *limits) { + if (!args || !canonical_repo_path || !limits) { return NULL; } yyjson_doc *source = yyjson_read(args, strlen(args), 0); @@ -7562,15 +7641,61 @@ static char *index_args_with_repo_path(const char *args, const char *canonical_r return NULL; } (void)yyjson_mut_obj_remove_key(copy_root, "repo_path"); + (void)yyjson_mut_obj_remove_key(copy_root, "_cbm_index_limits"); if (!yyjson_mut_obj_add_strcpy(copy, copy_root, "repo_path", canonical_repo_path)) { yyjson_mut_doc_free(copy); return NULL; } + yyjson_mut_val *policy = yyjson_mut_obj(copy); + bool policy_ok = policy != NULL; + for (size_t i = 0; policy_ok && i < cbm_index_limits_config_key_count(); i++) { + const char *key = cbm_index_limits_config_key_at(i); + char value[CBM_SZ_4K]; + policy_ok = cbm_index_limits_format_value(limits, key, value, sizeof(value)) && + yyjson_mut_obj_add_strcpy(copy, policy, key, value); + } + if (!policy_ok || !yyjson_mut_obj_add_val(copy, copy_root, "_cbm_index_limits", policy)) { + yyjson_mut_doc_free(copy); + return NULL; + } char *rewritten = yy_doc_to_str(copy); yyjson_mut_doc_free(copy); return rewritten; } +static bool load_index_limits(cbm_mcp_server_t *srv, const char *args, cbm_index_limits_t *limits, + char *error, size_t error_size) { + if (!cbm_index_worker_active()) { + return cbm_config_load_index_limits(srv->config, limits, error, error_size); + } + yyjson_doc *doc = yyjson_read(args, strlen(args), 0); + yyjson_val *root = doc ? yyjson_doc_get_root(doc) : NULL; + yyjson_val *policy = + root && yyjson_is_obj(root) ? yyjson_obj_get(root, "_cbm_index_limits") : NULL; + if (!policy || !yyjson_is_obj(policy) || + yyjson_obj_size(policy) != cbm_index_limits_config_key_count()) { + yyjson_doc_free(doc); + if (error && error_size > 0) { + (void)snprintf(error, error_size, "missing or incomplete trusted worker policy"); + } + return false; + } + cbm_index_limits_defaults(limits); + bool valid = true; + for (size_t i = 0; valid && i < cbm_index_limits_config_key_count(); i++) { + const char *key = cbm_index_limits_config_key_at(i); + yyjson_val *value = yyjson_obj_get(policy, key); + valid = value && yyjson_is_str(value) && + cbm_index_limits_set(limits, key, yyjson_get_str(value), error, error_size); + } + yyjson_doc_free(doc); + return valid; +} + +static const char *canonical_policy_root(const char *path, char *buffer, size_t buffer_size) { + return path && cbm_canonical_path(path, buffer, buffer_size) ? buffer : path; +} + static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) { char *repo_path = cbm_mcp_get_string_arg(args, "repo_path"); char *mode_str = cbm_mcp_get_string_arg(args, "mode"); @@ -7613,10 +7738,41 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) { return result; } + cbm_index_limits_t index_limits; + char policy_error[CBM_SZ_256] = {0}; + if (!load_index_limits(srv, args, &index_limits, policy_error, sizeof(policy_error))) { + char message[CBM_SZ_512]; + (void)snprintf(message, sizeof(message), "invalid index resource configuration: %s", + policy_error); + free(mode_str); + free(name_override); + free(repo_path); + return cbm_mcp_text_result(message, true); + } + + char home_canonical[CBM_SZ_4K]; + char cache_path[CBM_SZ_4K]; + char cache_canonical[CBM_SZ_4K]; + const char *home_root = + canonical_policy_root(cbm_get_home_dir(), home_canonical, sizeof(home_canonical)); + const char *cache_root = canonical_policy_root(cache_dir(cache_path, sizeof(cache_path)), + cache_canonical, sizeof(cache_canonical)); + char root_reason[64] = {0}; + if (!cbm_index_root_allowed(repo_path, home_root, cache_root, index_limits.denied_roots, + root_reason, sizeof(root_reason))) { + char message[CBM_SZ_256]; + (void)snprintf(message, sizeof(message), "repo_path is denied by index resource policy: %s", + root_reason); + free(mode_str); + free(name_override); + free(repo_path); + return cbm_mcp_text_result(message, true); + } + /* A daemon session delegates the one physical write to its shared job * registry only after path canonicalization and workspace authorization. */ if (srv->index_executor) { - char *worker_args = index_args_with_repo_path(args, repo_path); + char *worker_args = index_args_with_repo_path(args, repo_path, &index_limits); char *coordinated = worker_args ? srv->index_executor(srv->index_executor_context, repo_path, worker_args) : NULL; @@ -7648,7 +7804,7 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) { * installs the same guard before running the in-process pipeline. A marked * host fails closed if preparation or worker startup cannot complete. */ if (cbm_index_supervisor_should_wrap()) { - char *worker_args = index_args_with_repo_path(args, repo_path); + char *worker_args = index_args_with_repo_path(args, repo_path, &index_limits); if (!worker_args) { free(mutation_project); free(repo_path); @@ -7656,7 +7812,7 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) { free(name_override); return cbm_mcp_text_result("failed to prepare supervised index request", true); } - char *supervised = index_run_supervised(srv, worker_args); + char *supervised = index_run_supervised(srv, worker_args, &index_limits); free(worker_args); if (supervised) { free(mutation_project); @@ -7720,6 +7876,7 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) { } free(name_override); cbm_pipeline_set_persistence(p, persistence); + cbm_pipeline_set_index_limits(p, &index_limits); char *project_name = heap_strdup(cbm_pipeline_project_name(p)); @@ -7756,6 +7913,10 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) { cbm_file_error_t *file_errors = NULL; int file_error_count = 0; cbm_pipeline_get_file_errors(p, &file_errors, &file_error_count); + cbm_discover_report_t discovery_limit = {0}; + cbm_pipeline_limit_report_t pipeline_limit = {0}; + cbm_pipeline_get_resource_violation(p, &discovery_limit); + cbm_pipeline_get_limit_violation(p, &pipeline_limit); cbm_mem_collect(); /* return mimalloc pages to OS after large indexing */ @@ -7784,6 +7945,25 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) { srv, doc, root, project_name, repo_path, persistence, p, excluded_dirs, excluded_count, file_errors, file_error_count, has_logfile ? logfile_path : NULL); yyjson_mut_obj_add_str(doc, root, "status", degraded ? "degraded" : "indexed"); + } else if (rc == CBM_PIPELINE_RESOURCE_LIMIT) { + const char *limit_name = pipeline_limit.violation != CBM_PIPELINE_LIMIT_NONE + ? cbm_pipeline_limit_name(pipeline_limit.violation) + : cbm_discover_limit_name(discovery_limit.violation); + uint64_t observed = pipeline_limit.violation != CBM_PIPELINE_LIMIT_NONE + ? pipeline_limit.observed + : discovery_limit.observed; + uint64_t configured = pipeline_limit.violation != CBM_PIPELINE_LIMIT_NONE + ? pipeline_limit.limit + : discovery_limit.limit; + yyjson_mut_obj_add_str(doc, root, "status", "error"); + yyjson_mut_obj_add_str(doc, root, "outcome", "resource_limit_exceeded"); + yyjson_mut_obj_add_str(doc, root, "resource", limit_name); + yyjson_mut_obj_add_uint(doc, root, "observed", observed); + yyjson_mut_obj_add_uint(doc, root, "limit", configured); + yyjson_mut_obj_add_str( + doc, root, "hint", + "Indexing exceeded its configured resource policy. Narrow the repository root or " + "raise only the matching setting for a trusted repository."); } else { yyjson_mut_obj_add_str(doc, root, "status", "error"); yyjson_mut_obj_add_str(doc, root, "hint", diff --git a/src/mcp/mcp.h b/src/mcp/mcp.h index 58b01a32a..85d0a961a 100644 --- a/src/mcp/mcp.h +++ b/src/mcp/mcp.h @@ -204,6 +204,7 @@ typedef enum { CBM_MCP_SUPERVISED_RESULT_SUCCESS, CBM_MCP_SUPERVISED_RESULT_CONTAINED_FAILURE, CBM_MCP_SUPERVISED_RESULT_UNSAFE_TERMINAL, + CBM_MCP_SUPERVISED_RESULT_RESOURCE_LIMIT, } cbm_mcp_supervised_result_disposition_t; cbm_mcp_supervised_result_disposition_t cbm_mcp_supervised_result_disposition( diff --git a/src/pipeline/pipeline.c b/src/pipeline/pipeline.c index 0c3041959..f997b6a11 100644 --- a/src/pipeline/pipeline.c +++ b/src/pipeline/pipeline.c @@ -85,6 +85,9 @@ struct cbm_pipeline { atomic_int cancelled_storage; atomic_int *cancelled; bool persistence; /* write .codebase-memory/graph.db.zst after indexing */ + cbm_index_limits_t index_limits; + cbm_discover_report_t resource_violation; + cbm_pipeline_limit_report_t limit_violation; /* Indexing state (set during run) */ cbm_gbuf_t *gbuf; @@ -190,6 +193,7 @@ cbm_pipeline_t *cbm_pipeline_new(const char *repo_path, const char *db_path, p->branch_qn = cbm_git_context_branch_qn(p->project_name, &p->git_ctx); p->mode = mode; p->persistence = false; + cbm_index_limits_defaults(&p->index_limits); p->committed_nodes = -1; p->committed_edges = -1; atomic_init(&p->cancelled_storage, 0); @@ -198,6 +202,45 @@ cbm_pipeline_t *cbm_pipeline_new(const char *repo_path, const char *db_path, return p; } +void cbm_pipeline_set_index_limits(cbm_pipeline_t *p, const cbm_index_limits_t *limits) { + if (p && limits) { + p->index_limits = *limits; + } +} + +void cbm_pipeline_get_resource_violation(const cbm_pipeline_t *p, cbm_discover_report_t *report) { + if (!report) { + return; + } + *report = p ? p->resource_violation : (cbm_discover_report_t){0}; +} + +void cbm_pipeline_get_limit_violation(const cbm_pipeline_t *p, + cbm_pipeline_limit_report_t *report) { + if (!report) { + return; + } + *report = p ? p->limit_violation : (cbm_pipeline_limit_report_t){0}; +} + +const char *cbm_pipeline_limit_name(cbm_pipeline_limit_t limit) { + switch (limit) { + case CBM_PIPELINE_LIMIT_NONE: + return "none"; + case CBM_PIPELINE_LIMIT_STAGING_BYTES: + return "staging_bytes"; + case CBM_PIPELINE_LIMIT_DATABASE_BYTES: + return "database_bytes"; + case CBM_PIPELINE_LIMIT_FREE_DISK_BYTES: + return "free_disk_bytes"; + case CBM_PIPELINE_LIMIT_CACHE_BYTES: + return "cache_bytes"; + case CBM_PIPELINE_LIMIT_TASK_TEMP_BYTES: + return "task_temp_bytes"; + } + return "unknown"; +} + void cbm_pipeline_set_persistence(cbm_pipeline_t *p, bool enabled) { if (p) { p->persistence = enabled; @@ -1577,10 +1620,26 @@ static int cbm_pipeline_run_staged(cbm_pipeline_t *p, bool *was_incremental) { /* Phase 1: Discover files */ CBM_PROF_START(t_discover); + uint64_t now_ms = cbm_now_ms(); + cbm_discover_limits_t discover_limits = { + .max_files = p->index_limits.max_files, + .max_directories = p->index_limits.max_directories, + .max_entries = p->index_limits.max_entries, + .max_depth = p->index_limits.max_depth, + .max_source_bytes = p->index_limits.max_source_bytes, + .deadline_ms = p->index_limits.scan_timeout_ms > UINT64_MAX - now_ms + ? UINT64_MAX + : now_ms + p->index_limits.scan_timeout_ms, + }; + p->resource_violation = (cbm_discover_report_t){0}; cbm_discover_opts_t opts = { .mode = p->mode, .ignore_file = NULL, - .max_file_size = 0, + .max_file_size = p->index_limits.max_file_bytes > INT64_MAX + ? INT64_MAX + : (int64_t)p->index_limits.max_file_bytes, + .limits = &discover_limits, + .report = &p->resource_violation, }; cbm_file_info_t *files = NULL; int file_count = 0; @@ -1605,7 +1664,7 @@ static int cbm_pipeline_run_staged(cbm_pipeline_t *p, bool *was_incremental) { cbm_log_info("pipeline.discover", "files", itoa_buf(file_count), "elapsed_ms", itoa_buf((int)elapsed_ms(t0))); if (rc != 0 || check_cancel(p)) { - rc = CBM_NOT_FOUND; + rc = rc == CBM_DISCOVER_LIMIT_EXCEEDED ? CBM_PIPELINE_RESOURCE_LIMIT : CBM_NOT_FOUND; goto cleanup; } @@ -1681,6 +1740,96 @@ static void cleanup_staging_db(const char *path) { (void)cbm_remove_db_sidecars(path); } +static uint64_t db_footprint_bytes(const char *path) { + static const char *suffixes[] = {"", "-wal", "-shm", "-journal"}; + uint64_t total = 0; + for (size_t i = 0; i < sizeof(suffixes) / sizeof(suffixes[0]); i++) { + char artifact[CBM_SZ_4K]; + int written = snprintf(artifact, sizeof(artifact), "%s%s", path, suffixes[i]); + if (written <= 0 || (size_t)written >= sizeof(artifact)) { + return UINT64_MAX; + } + struct stat info; + if (stat(artifact, &info) != 0 || info.st_size <= 0) { + continue; + } + uint64_t size = (uint64_t)info.st_size; + total = size > UINT64_MAX - total ? UINT64_MAX : total + size; + } + return total; +} + +static int pipeline_limit_failed(cbm_pipeline_t *p, cbm_pipeline_limit_t violation, + uint64_t observed, uint64_t limit) { + p->limit_violation = (cbm_pipeline_limit_report_t){ + .violation = violation, + .observed = observed, + .limit = limit, + }; + char observed_text[32]; + char limit_text[32]; + (void)snprintf(observed_text, sizeof(observed_text), "%llu", (unsigned long long)observed); + (void)snprintf(limit_text, sizeof(limit_text), "%llu", (unsigned long long)limit); + cbm_log_warn("pipeline.resource_limit", "limit", cbm_pipeline_limit_name(violation), "observed", + observed_text, "configured", limit_text); + return CBM_PIPELINE_RESOURCE_LIMIT; +} + +static int check_staging_limits(cbm_pipeline_t *p, const char *staging_path, + bool check_database_limit) { + uint64_t footprint = db_footprint_bytes(staging_path); + if (footprint > p->index_limits.max_staging_bytes) { + return pipeline_limit_failed(p, CBM_PIPELINE_LIMIT_STAGING_BYTES, footprint, + p->index_limits.max_staging_bytes); + } + if (footprint > p->index_limits.max_task_temp_bytes) { + return pipeline_limit_failed(p, CBM_PIPELINE_LIMIT_TASK_TEMP_BYTES, footprint, + p->index_limits.max_task_temp_bytes); + } + if (check_database_limit && footprint > p->index_limits.max_db_bytes) { + return pipeline_limit_failed(p, CBM_PIPELINE_LIMIT_DATABASE_BYTES, footprint, + p->index_limits.max_db_bytes); + } + return CBM_PIPELINE_OK; +} + +static int check_cache_limit(cbm_pipeline_t *p, const char *cache_path) { + uint64_t cache_bytes = 0; + if (!cbm_directory_size_bounded(cache_path, p->index_limits.cache_max_bytes, &cache_bytes)) { + return CBM_PIPELINE_ERROR; + } + return cache_bytes > p->index_limits.cache_max_bytes + ? pipeline_limit_failed(p, CBM_PIPELINE_LIMIT_CACHE_BYTES, cache_bytes, + p->index_limits.cache_max_bytes) + : CBM_PIPELINE_OK; +} + +static bool db_parent_path(const char *path, char *parent, size_t parent_size) { + if (!path || !parent || parent_size == 0) { + return false; + } + int written = snprintf(parent, parent_size, "%s", path); + if (written <= 0 || (size_t)written >= parent_size) { + return false; + } + char *slash = strrchr(parent, '/'); +#ifdef _WIN32 + char *backslash = strrchr(parent, '\\'); + if (backslash && (!slash || backslash > slash)) { + slash = backslash; + } +#endif + if (!slash) { + return snprintf(parent, parent_size, ".") == 1; + } + if (slash == parent) { + slash[1] = '\0'; + } else { + *slash = '\0'; + } + return true; +} + static bool ensure_db_parent(const char *path) { if (!path) { return false; @@ -1826,15 +1975,37 @@ static int export_after_publish(cbm_pipeline_t *p, const char *final_path, bool return 0; } -int cbm_pipeline_run(cbm_pipeline_t *p) { +static int pipeline_run_with_limits(cbm_pipeline_t *p) { if (!p) { return CBM_NOT_FOUND; } + p->limit_violation = (cbm_pipeline_limit_report_t){0}; char *final_path = resolve_db_path(p); if (!final_path || !ensure_db_parent(final_path)) { free(final_path); return CBM_NOT_FOUND; } + char parent_path[CBM_SZ_4K]; + uint64_t free_disk_bytes = 0; + if (!db_parent_path(final_path, parent_path, sizeof(parent_path)) || + !cbm_disk_free_bytes(parent_path, &free_disk_bytes)) { + free(final_path); + return CBM_NOT_FOUND; + } + if (free_disk_bytes < p->index_limits.min_free_disk_bytes) { + int limit_rc = pipeline_limit_failed(p, CBM_PIPELINE_LIMIT_FREE_DISK_BYTES, free_disk_bytes, + p->index_limits.min_free_disk_bytes); + free(final_path); + return limit_rc; + } + bool enforce_cache_limit = p->db_path == NULL; + if (enforce_cache_limit) { + int cache_rc = check_cache_limit(p, parent_path); + if (cache_rc != 0) { + free(final_path); + return cache_rc; + } + } struct stat final_st; bool final_existed = stat(final_path, &final_st) == 0; char *staging_path = create_staging_path(final_path); @@ -1867,7 +2038,26 @@ int cbm_pipeline_run(cbm_pipeline_t *p) { free(p->db_path); p->db_path = configured_db_path; - if (rc != 0 || check_cancel(p) || seal_staging_db(staging_path) != 0) { + if (rc != 0) { + cleanup_staging_db(staging_path); + free(staging_path); + free(final_path); + return rc == CBM_PIPELINE_RESOURCE_LIMIT ? rc : CBM_NOT_FOUND; + } + rc = check_staging_limits(p, staging_path, false); + if (rc != 0) { + cleanup_staging_db(staging_path); + free(staging_path); + free(final_path); + return rc; + } + if (enforce_cache_limit && (rc = check_cache_limit(p, parent_path)) != 0) { + cleanup_staging_db(staging_path); + free(staging_path); + free(final_path); + return rc; + } + if (check_cancel(p) || seal_staging_db(staging_path) != 0) { cleanup_staging_db(staging_path); free(staging_path); free(final_path); @@ -1892,6 +2082,13 @@ int cbm_pipeline_run(cbm_pipeline_t *p) { free(final_path); return CBM_NOT_FOUND; } + rc = check_staging_limits(p, staging_path, true); + if (rc != 0) { + cleanup_staging_db(staging_path); + free(staging_path); + free(final_path); + return rc; + } if (!prepare_publish_destination(final_path, final_existed, backup_succeeded) || (p->rename_hook ? p->rename_hook(staging_path, final_path, p->rename_hook_ctx) @@ -1908,3 +2105,13 @@ int cbm_pipeline_run(cbm_pipeline_t *p) { free(final_path); return rc; } + +int cbm_pipeline_run(cbm_pipeline_t *p) { + if (!p) { + return CBM_NOT_FOUND; + } + int previous_worker_limit = cbm_worker_limit_set_for_thread(p->index_limits.cpu_cores); + int rc = pipeline_run_with_limits(p); + (void)cbm_worker_limit_set_for_thread(previous_worker_limit); + return rc; +} diff --git a/src/pipeline/pipeline.h b/src/pipeline/pipeline.h index 78cfd37fd..359d9a184 100644 --- a/src/pipeline/pipeline.h +++ b/src/pipeline/pipeline.h @@ -20,6 +20,7 @@ #include #include "discover/discover.h" /* cbm_ignored_file_t (#963) */ +#include "foundation/index_limits.h" /* Forward declarations */ typedef struct cbm_store cbm_store_t; @@ -29,6 +30,29 @@ typedef struct cbm_gbuf cbm_gbuf_t; typedef struct cbm_pipeline cbm_pipeline_t; +enum { + CBM_PIPELINE_OK = 0, + CBM_PIPELINE_ERROR = -1, + CBM_PIPELINE_RESOURCE_LIMIT = -3, +}; + +typedef enum { + CBM_PIPELINE_LIMIT_NONE = 0, + CBM_PIPELINE_LIMIT_STAGING_BYTES, + CBM_PIPELINE_LIMIT_DATABASE_BYTES, + CBM_PIPELINE_LIMIT_FREE_DISK_BYTES, + CBM_PIPELINE_LIMIT_CACHE_BYTES, + CBM_PIPELINE_LIMIT_TASK_TEMP_BYTES, +} cbm_pipeline_limit_t; + +typedef struct { + cbm_pipeline_limit_t violation; + uint64_t observed; + uint64_t limit; +} cbm_pipeline_limit_report_t; + +const char *cbm_pipeline_limit_name(cbm_pipeline_limit_t limit); + /* ── Index mode ─────────────────────────────────────────────────── */ #ifndef CBM_INDEX_MODE_T_DEFINED @@ -52,6 +76,15 @@ cbm_pipeline_t *cbm_pipeline_new(const char *repo_path, const char *db_path, cbm * When enabled, the pipeline writes a compressed artifact after indexing. */ void cbm_pipeline_set_persistence(cbm_pipeline_t *p, bool enabled); +/* Replace the default index resource policy with a previously validated + * policy. The value is copied and may be released by the caller. */ +void cbm_pipeline_set_index_limits(cbm_pipeline_t *p, const cbm_index_limits_t *limits); + +/* Copy the discovery resource violation from the most recent run. A successful + * run or non-resource error returns CBM_DISCOVER_LIMIT_NONE. */ +void cbm_pipeline_get_resource_violation(const cbm_pipeline_t *p, cbm_discover_report_t *report); +void cbm_pipeline_get_limit_violation(const cbm_pipeline_t *p, cbm_pipeline_limit_report_t *report); + /* Free a pipeline and all its internal state. NULL-safe. */ void cbm_pipeline_free(cbm_pipeline_t *p); diff --git a/tests/test_cli.c b/tests/test_cli.c index 3edb18c84..79497d36f 100644 --- a/tests/test_cli.c +++ b/tests/test_cli.c @@ -1598,8 +1598,10 @@ TEST(cli_uninstall_quiesces_active_cohort_before_removing_binary_and_index) { } char *old_home = NULL; char *old_cache = NULL; + char *old_path = save_test_env("PATH"); cli_activation_save_env(&old_home, &old_cache); cbm_setenv("HOME", tmpdir, 1); + cbm_setenv("PATH", tmpdir, 1); char cache_dir[512]; char index_path[640]; @@ -1636,6 +1638,7 @@ TEST(cli_uninstall_quiesces_active_cohort_before_removing_binary_and_index) { const char *installed = read_test_file(bin_target); bool binary_preserved = installed && strcmp(installed, "binary must survive active-daemon refusal") == 0; + restore_test_env("PATH", old_path); cli_activation_restore_env(old_home, old_cache); test_rmdir_r(tmpdir); diff --git a/tests/test_daemon_application.c b/tests/test_daemon_application.c index e81b766a3..b0e8826ab 100644 --- a/tests/test_daemon_application.c +++ b/tests/test_daemon_application.c @@ -3989,6 +3989,7 @@ TEST(daemon_application_worker_lock_serializes_external_mutation) { cbm_daemon_application_config_t config = { .worker_ops = &worker_ops, .project_locks = locks_ready ? locks.daemon_locks : NULL, + .physical_job_limit = 2, }; cbm_daemon_application_t *application = locks_ready ? cbm_daemon_application_new(&config) : NULL; @@ -4737,7 +4738,7 @@ TEST(daemon_application_queues_explicit_index_behind_physical_job_limit) { PASS(); } -TEST(daemon_application_default_limit_admits_four_and_rejects_fifth) { +TEST(daemon_application_explicit_limit_admits_four_and_rejects_fifth) { enum { DEFAULT_CAP_RUNNING = 4, DEFAULT_CAP_TOTAL = 5 }; const size_t aggregate_budget = 4099; app_fake_worker_context_t fake; @@ -4753,6 +4754,7 @@ TEST(daemon_application_default_limit_admits_four_and_rejects_fifth) { cbm_daemon_application_config_t config = { .worker_ops = &worker_ops, .aggregate_memory_budget_bytes = aggregate_budget, + .physical_job_limit = DEFAULT_CAP_RUNNING, }; cbm_daemon_application_t *application = cbm_daemon_application_new(&config); char roots[DEFAULT_CAP_TOTAL][APP_TEST_PATH_CAP]; @@ -4814,6 +4816,37 @@ TEST(daemon_application_default_limit_admits_four_and_rejects_fifth) { PASS(); } +TEST(daemon_application_resource_policy_defaults_to_one_job) { + cbm_daemon_application_t *application = cbm_daemon_application_new(NULL); + ASSERT_NOT_NULL(application); + ASSERT_EQ(cbm_daemon_application_physical_job_limit(application), 1); + cbm_daemon_application_free(application); + PASS(); +} + +TEST(daemon_application_resource_policy_loads_configured_job_limit) { + char cache[APP_TEST_PATH_CAP]; + (void)snprintf(cache, sizeof(cache), "%s/cbm-app-resource-config-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + cbm_config_t *stored_config = cbm_config_open(cache); + ASSERT_NOT_NULL(stored_config); + ASSERT_EQ(cbm_config_set(stored_config, "index_concurrent_jobs", "3"), 0); + + cbm_daemon_application_config_t config = { + .config = stored_config, + .aggregate_memory_budget_bytes = 9000, + }; + cbm_daemon_application_t *application = cbm_daemon_application_new(&config); + ASSERT_NOT_NULL(application); + ASSERT_EQ(cbm_daemon_application_physical_job_limit(application), 3); + ASSERT_EQ(cbm_daemon_application_worker_memory_budget_bytes(application), 3000); + + cbm_daemon_application_free(application); + cbm_config_close(stored_config); + th_rmtree(cache); + PASS(); +} + /* RED on the former void destruction contract: free could retain an * application with live owners, but the daemon host had no way to observe that * refusal and freed the application's borrowed dependencies anyway. */ @@ -4915,7 +4948,9 @@ SUITE(daemon_application) { RUN_TEST(daemon_application_cancellation_between_recovery_attempts_stops_retry); RUN_TEST(daemon_application_thread_start_failure_rolls_back_job_reservation); RUN_TEST(daemon_application_queues_explicit_index_behind_physical_job_limit); - RUN_TEST(daemon_application_default_limit_admits_four_and_rejects_fifth); + RUN_TEST(daemon_application_explicit_limit_admits_four_and_rejects_fifth); + RUN_TEST(daemon_application_resource_policy_defaults_to_one_job); + RUN_TEST(daemon_application_resource_policy_loads_configured_job_limit); RUN_TEST(daemon_application_free_reports_retained_live_ownership); RUN_TEST(daemon_application_rejects_clean_exit_when_process_tree_is_not_contained); } diff --git a/tests/test_discover.c b/tests/test_discover.c index 57cbefe75..d3e4c32bb 100644 --- a/tests/test_discover.c +++ b/tests/test_discover.c @@ -395,6 +395,108 @@ TEST(discover_bounded_count_fails_closed_after_deadline) { PASS(); } +TEST(discover_resource_limits_report_exact_boundary_and_discard_partial_results) { + char *base = th_mktempdir("cbm_disc_resource_files"); + ASSERT(base != NULL); + th_write_file(TH_PATH(base, "src/first.c"), "int first;\n"); + th_write_file(TH_PATH(base, "src/second.c"), "int second;\n"); + + cbm_discover_limits_t limits = { + .max_files = 1, + .max_directories = 100, + .max_entries = 100, + .max_depth = 10, + .max_source_bytes = 1024, + .deadline_ms = cbm_now_ms() + 2000, + }; + cbm_discover_report_t report = {0}; + cbm_discover_opts_t opts = { + .mode = CBM_MODE_FULL, + .limits = &limits, + .report = &report, + }; + cbm_file_info_t *files = (cbm_file_info_t *)1; + int count = 99; + + int rc = cbm_discover(base, &opts, &files, &count); + + th_cleanup(base); + ASSERT_EQ(rc, CBM_DISCOVER_LIMIT_EXCEEDED); + ASSERT(files == NULL); + ASSERT_EQ(count, 0); + ASSERT_EQ(report.violation, CBM_DISCOVER_LIMIT_FILES); + ASSERT_EQ(report.observed, 2); + ASSERT_EQ(report.limit, 1); + ASSERT_STR_EQ(cbm_discover_limit_name(report.violation), "files"); + PASS(); +} + +TEST(discover_resource_limits_cover_directories_entries_depth_and_source_bytes) { + char *base = th_mktempdir("cbm_disc_resource_dimensions"); + ASSERT(base != NULL); + th_write_file(TH_PATH(base, "a/b/c/deep.c"), "int deep;\n"); + th_write_file(TH_PATH(base, "root.c"), "int root_value;\n"); + + struct { + cbm_discover_limits_t limits; + cbm_discover_limit_t expected; + } cases[] = { + { + .limits = {.max_files = 100, + .max_directories = 1, + .max_entries = 100, + .max_depth = 10, + .max_source_bytes = 1024}, + .expected = CBM_DISCOVER_LIMIT_DIRECTORIES, + }, + { + .limits = {.max_files = 100, + .max_directories = 100, + .max_entries = 1, + .max_depth = 10, + .max_source_bytes = 1024}, + .expected = CBM_DISCOVER_LIMIT_ENTRIES, + }, + { + .limits = {.max_files = 100, + .max_directories = 100, + .max_entries = 100, + .max_depth = 1, + .max_source_bytes = 1024}, + .expected = CBM_DISCOVER_LIMIT_DEPTH, + }, + { + .limits = {.max_files = 100, + .max_directories = 100, + .max_entries = 100, + .max_depth = 10, + .max_source_bytes = 1}, + .expected = CBM_DISCOVER_LIMIT_SOURCE_BYTES, + }, + }; + + for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) { + cases[i].limits.deadline_ms = cbm_now_ms() + 2000; + cbm_discover_report_t report = {0}; + cbm_discover_opts_t opts = { + .mode = CBM_MODE_FULL, + .limits = &cases[i].limits, + .report = &report, + }; + cbm_file_info_t *files = NULL; + int count = 0; + int rc = cbm_discover(base, &opts, &files, &count); + ASSERT_EQ(rc, CBM_DISCOVER_LIMIT_EXCEEDED); + ASSERT(files == NULL); + ASSERT_EQ(count, 0); + ASSERT_EQ(report.violation, cases[i].expected); + ASSERT(report.observed > report.limit); + } + + th_cleanup(base); + PASS(); +} + TEST(discover_skips_git_dir) { char *base = th_mktempdir("cbm_disc_git"); ASSERT(base != NULL); @@ -1432,6 +1534,8 @@ SUITE(discover) { RUN_TEST(discover_wide_sibling_fanout_exceeds_initial_walk_stack); RUN_TEST(discover_bounded_count_is_allocation_free_and_limit_exact); RUN_TEST(discover_bounded_count_fails_closed_after_deadline); + RUN_TEST(discover_resource_limits_report_exact_boundary_and_discard_partial_results); + RUN_TEST(discover_resource_limits_cover_directories_entries_depth_and_source_bytes); RUN_TEST(discover_skips_git_dir); RUN_TEST(discover_with_gitignore); RUN_TEST(discover_with_global_xdg_ignore); diff --git a/tests/test_index_limits.c b/tests/test_index_limits.c new file mode 100644 index 000000000..ba026bb13 --- /dev/null +++ b/tests/test_index_limits.c @@ -0,0 +1,156 @@ +/* + * test_index_limits.c — Unified index resource policy tests. + */ +#include "foundation/index_limits.h" +#include "cli/cli.h" +#include "test_framework.h" +#include "test_helpers.h" + +#include +#include + +#define TEST_MIB (UINT64_C(1024) * UINT64_C(1024)) + +TEST(index_limits_defaults_are_bounded) { + cbm_index_limits_t limits; + cbm_index_limits_defaults(&limits); + + ASSERT_EQ(limits.max_files, 100000); + ASSERT_EQ(limits.max_directories, 20000); + ASSERT_EQ(limits.max_entries, 500000); + ASSERT_EQ(limits.max_depth, 64); + ASSERT_EQ(limits.max_source_bytes, UINT64_C(4096) * TEST_MIB); + ASSERT_EQ(limits.max_file_bytes, UINT64_C(64) * TEST_MIB); + ASSERT_EQ(limits.scan_timeout_ms, 30000); + ASSERT_EQ(limits.cpu_cores, 4); + ASSERT_EQ(limits.concurrent_jobs, 1); + ASSERT_EQ(limits.memory_limit_bytes, UINT64_C(8192) * TEST_MIB); + ASSERT_EQ(limits.max_db_bytes, UINT64_C(16384) * TEST_MIB); + ASSERT_EQ(limits.max_staging_bytes, UINT64_C(20480) * TEST_MIB); + ASSERT_EQ(limits.max_task_temp_bytes, UINT64_C(24576) * TEST_MIB); + ASSERT_EQ(limits.cache_max_bytes, UINT64_C(32768) * TEST_MIB); + ASSERT_EQ(limits.min_free_disk_bytes, UINT64_C(4096) * TEST_MIB); + ASSERT_EQ(limits.max_duration_ms, UINT64_C(3600000)); + ASSERT_TRUE(limits.low_priority); + PASS(); +} + +TEST(index_limits_config_set_converts_units_and_rejects_unsafe_values) { + cbm_index_limits_t limits; + cbm_index_limits_defaults(&limits); + char error[128] = {0}; + + ASSERT_TRUE(cbm_index_limits_set(&limits, "index_max_files", "75000", error, sizeof(error))); + ASSERT_EQ(limits.max_files, 75000); + ASSERT_TRUE( + cbm_index_limits_set(&limits, "index_memory_limit_mb", "4096", error, sizeof(error))); + ASSERT_EQ(limits.memory_limit_bytes, UINT64_C(4096) * TEST_MIB); + ASSERT_TRUE(cbm_index_limits_set(&limits, "index_low_priority", "false", error, sizeof(error))); + ASSERT_FALSE(limits.low_priority); + + cbm_index_limits_t unchanged = limits; + ASSERT_FALSE(cbm_index_limits_set(&limits, "index_max_files", "0", error, sizeof(error))); + ASSERT(strstr(error, "positive") != NULL); + ASSERT_EQ(memcmp(&limits, &unchanged, sizeof(limits)), 0); + + ASSERT_FALSE( + cbm_index_limits_set(&limits, "index_max_files", "not-a-number", error, sizeof(error))); + ASSERT_EQ(memcmp(&limits, &unchanged, sizeof(limits)), 0); + ASSERT_FALSE(cbm_index_limits_set(&limits, "index_memory_limit_mb", "999999999999999999999999", + error, sizeof(error))); + ASSERT_EQ(memcmp(&limits, &unchanged, sizeof(limits)), 0); + ASSERT_FALSE(cbm_index_limits_set(&limits, "unknown_index_limit", "1", error, sizeof(error))); + ASSERT(strstr(error, "unknown") != NULL); + ASSERT_EQ(memcmp(&limits, &unchanged, sizeof(limits)), 0); + PASS(); +} + +TEST(index_limits_config_registry_identifies_only_public_resource_keys) { + ASSERT_TRUE(cbm_index_limits_is_config_key("index_max_files")); + ASSERT_TRUE(cbm_index_limits_is_config_key("index_max_db_mb")); + ASSERT_TRUE(cbm_index_limits_is_config_key("index_low_priority")); + ASSERT_FALSE(cbm_index_limits_is_config_key("auto_index")); + ASSERT_FALSE(cbm_index_limits_is_config_key("_cbm_index_max_files")); + ASSERT_FALSE(cbm_index_limits_is_config_key(NULL)); + + size_t count = cbm_index_limits_config_key_count(); + ASSERT_GT(count, 0); + for (size_t i = 0; i < count; i++) { + const char *key = cbm_index_limits_config_key_at(i); + ASSERT_NOT_NULL(key); + ASSERT_TRUE(cbm_index_limits_is_config_key(key)); + } + ASSERT_NULL(cbm_index_limits_config_key_at(count)); + PASS(); +} + +TEST(index_limits_persisted_config_rejects_invalid_resource_values_atomically) { + char cache[512]; + (void)snprintf(cache, sizeof(cache), "%s/cbm-index-config-validation-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + cbm_config_t *config = cbm_config_open(cache); + ASSERT_NOT_NULL(config); + + ASSERT_EQ(cbm_config_set(config, "index_max_files", "75000"), 0); + ASSERT_STR_EQ(cbm_config_get(config, "index_max_files", ""), "75000"); + ASSERT_TRUE(cbm_config_set(config, "index_max_files", "0") != 0); + ASSERT_STR_EQ(cbm_config_get(config, "index_max_files", ""), "75000"); + ASSERT_TRUE(cbm_config_set(config, "index_unknown_limit", "1") != 0); + ASSERT_EQ(cbm_config_set(config, "extension_setting", "preserved"), 0); + + cbm_config_close(config); + th_rmtree(cache); + PASS(); +} + +TEST(index_limits_rejects_dangerous_builtin_roots) { + char reason[64] = {0}; + + ASSERT_FALSE( + cbm_index_root_allowed("/", "/home/example", "/var/cache/cbm", "", reason, sizeof(reason))); + ASSERT_STR_EQ(reason, "filesystem_root"); + + ASSERT_FALSE(cbm_index_root_allowed("/home/example/", "/home/example", "/var/cache/cbm", "", + reason, sizeof(reason))); + ASSERT_STR_EQ(reason, "home_directory"); + + ASSERT_FALSE(cbm_index_root_allowed("/var/cache/cbm", "/home/example", "/var/cache/cbm", "", + reason, sizeof(reason))); + ASSERT_STR_EQ(reason, "cache_directory"); + + ASSERT_TRUE(cbm_index_root_allowed("/workspace/repository", "/home/example", "/var/cache/cbm", + "", reason, sizeof(reason))); + ASSERT_STR_EQ(reason, ""); + PASS(); +} + +TEST(index_limits_rejects_exact_configured_roots_only) { + char reason[64] = {0}; + const char *denied = "/workspace;/srv/source"; + + ASSERT_FALSE(cbm_index_root_allowed("/workspace/", "/home/example", "/var/cache/cbm", denied, + reason, sizeof(reason))); + ASSERT_STR_EQ(reason, "configured_root"); + + ASSERT_TRUE(cbm_index_root_allowed("/workspace/repository", "/home/example", "/var/cache/cbm", + denied, reason, sizeof(reason))); + ASSERT_STR_EQ(reason, ""); + PASS(); +} + +TEST(index_memory_limit_is_clamped_to_detected_budget) { + ASSERT_EQ(cbm_index_effective_memory_limit(8U * 1024U, 4U * 1024U), 4U * 1024U); + ASSERT_EQ(cbm_index_effective_memory_limit(4U * 1024U, 8U * 1024U), 4U * 1024U); + ASSERT_EQ(cbm_index_effective_memory_limit(4U * 1024U, 0), 4U * 1024U); + PASS(); +} + +SUITE(index_limits) { + RUN_TEST(index_limits_defaults_are_bounded); + RUN_TEST(index_limits_config_set_converts_units_and_rejects_unsafe_values); + RUN_TEST(index_limits_config_registry_identifies_only_public_resource_keys); + RUN_TEST(index_limits_persisted_config_rejects_invalid_resource_values_atomically); + RUN_TEST(index_limits_rejects_dangerous_builtin_roots); + RUN_TEST(index_limits_rejects_exact_configured_roots_only); + RUN_TEST(index_memory_limit_is_clamped_to_detected_budget); +} diff --git a/tests/test_index_supervisor.c b/tests/test_index_supervisor.c index d94fe7329..4af351648 100644 --- a/tests/test_index_supervisor.c +++ b/tests/test_index_supervisor.c @@ -10,6 +10,7 @@ #include "foundation/profile.h" #include "mcp/index_supervisor.h" +#include #include #include #include @@ -762,6 +763,152 @@ TEST(index_supervisor_oversized_response_is_contained_and_log_is_retained) { PASS(); } +TEST(index_supervisor_memory_limit_terminates_contained_worker_tree) { + char cache[INDEX_SUPERVISOR_TEST_PATH_CAP]; + (void)snprintf(cache, sizeof(cache), "%s/cbm-index-memory-limit-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + const char *old_cache = getenv("CBM_CACHE_DIR"); + char *saved_cache = old_cache ? cbm_strdup(old_cache) : NULL; + (void)cbm_setenv("CBM_CACHE_DIR", cache, 1); + + cbm_index_worker_handle_t *handle = NULL; + int start_rc = cbm_index_worker_start("{\"__cbm_test_worker\":\"hang-tree\"}", 1, false, NULL, + NULL, &handle); + cbm_index_worker_set_memory_limit(handle, 1); + const cbm_index_worker_result_t *result = NULL; + bool terminal = handle && index_supervisor_test_poll_terminal( + handle, INDEX_SUPERVISOR_TEST_TERMINAL_MS, &result); + bool memory_limited = terminal && result && + result->resource_limit == CBM_INDEX_RESOURCE_LIMIT_MEMORY && + result->resource_observed > result->resource_limit_value && + result->resource_limit_value == 1 && !result->cancellation_requested && + result->tree_quiesced && !result->supervision_failed && !result->response; + + if (terminal) { + cbm_index_worker_destroy(handle); + } else { + index_supervisor_test_cleanup_handle(handle); + } + index_supervisor_test_restore_env("CBM_CACHE_DIR", saved_cache); + (void)th_rmtree(cache); + + ASSERT_EQ(start_rc, 0); + ASSERT_TRUE(terminal); + ASSERT_TRUE(memory_limited); + PASS(); +} + +TEST(index_supervisor_duration_limit_terminates_contained_worker_tree) { + char cache[INDEX_SUPERVISOR_TEST_PATH_CAP]; + (void)snprintf(cache, sizeof(cache), "%s/cbm-index-duration-limit-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + const char *old_cache = getenv("CBM_CACHE_DIR"); + char *saved_cache = old_cache ? cbm_strdup(old_cache) : NULL; + (void)cbm_setenv("CBM_CACHE_DIR", cache, 1); + + cbm_index_worker_handle_t *handle = NULL; + int start_rc = cbm_index_worker_start("{\"__cbm_test_worker\":\"hang-tree\"}", 0, false, NULL, + NULL, &handle); + cbm_index_worker_set_max_duration(handle, 1); + const cbm_index_worker_result_t *result = NULL; + bool terminal = handle && index_supervisor_test_poll_terminal( + handle, INDEX_SUPERVISOR_TEST_TERMINAL_MS, &result); + bool duration_limited = + terminal && result && result->resource_limit == CBM_INDEX_RESOURCE_LIMIT_DURATION && + result->resource_observed > result->resource_limit_value && + result->resource_limit_value == 1 && !result->cancellation_requested && + result->tree_quiesced && !result->supervision_failed && !result->response; + + if (terminal) { + cbm_index_worker_destroy(handle); + } else { + index_supervisor_test_cleanup_handle(handle); + } + index_supervisor_test_restore_env("CBM_CACHE_DIR", saved_cache); + (void)th_rmtree(cache); + + ASSERT_EQ(start_rc, 0); + ASSERT_TRUE(terminal); + ASSERT_TRUE(duration_limited); + PASS(); +} + +TEST(index_supervisor_task_temp_limit_contains_oversized_worker_output) { + char cache[INDEX_SUPERVISOR_TEST_PATH_CAP]; + (void)snprintf(cache, sizeof(cache), "%s/cbm-index-task-temp-limit-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + const char *old_cache = getenv("CBM_CACHE_DIR"); + char *saved_cache = old_cache ? cbm_strdup(old_cache) : NULL; + (void)cbm_setenv("CBM_CACHE_DIR", cache, 1); + + cbm_index_worker_handle_t *handle = NULL; + int start_rc = cbm_index_worker_start("{\"__cbm_test_worker\":\"oversize\"}", 0, false, NULL, + NULL, &handle); + cbm_index_worker_set_task_temp_limit(handle, 1); + const cbm_index_worker_result_t *result = NULL; + bool terminal = handle && index_supervisor_test_poll_terminal( + handle, INDEX_SUPERVISOR_TEST_TERMINAL_MS, &result); + bool task_temp_limited = terminal && result && + result->resource_limit == CBM_INDEX_RESOURCE_LIMIT_TASK_TEMP && + result->resource_observed > result->resource_limit_value && + result->resource_limit_value == 1 && !result->cancellation_requested && + result->tree_quiesced && !result->supervision_failed && + !result->response && !result->response_rejected; + + if (terminal) { + cbm_index_worker_destroy(handle); + } else { + index_supervisor_test_cleanup_handle(handle); + } + index_supervisor_test_restore_env("CBM_CACHE_DIR", saved_cache); + (void)th_rmtree(cache); + + ASSERT_EQ(start_rc, 0); + ASSERT_TRUE(terminal); + ASSERT_TRUE(task_temp_limited); + PASS(); +} + +TEST(index_supervisor_can_lower_worker_scheduling_priority) { + char cache[INDEX_SUPERVISOR_TEST_PATH_CAP]; + (void)snprintf(cache, sizeof(cache), "%s/cbm-index-low-priority-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + const char *old_cache = getenv("CBM_CACHE_DIR"); + char *saved_cache = old_cache ? cbm_strdup(old_cache) : NULL; + (void)cbm_setenv("CBM_CACHE_DIR", cache, 1); + + cbm_index_worker_handle_t *handle = NULL; + int start_rc = cbm_index_worker_start("{\"__cbm_test_worker\":\"hang-tree\"}", 0, false, NULL, + NULL, &handle); + bool priority_lowered = cbm_index_worker_set_low_priority(handle, true); +#ifndef _WIN32 + int priority_error = errno; +#endif + bool cancel_accepted = handle && cbm_index_worker_request_cancel(handle); + const cbm_index_worker_result_t *result = NULL; + bool terminal = handle && index_supervisor_test_poll_terminal( + handle, INDEX_SUPERVISOR_TEST_TERMINAL_MS, &result); + + if (terminal) { + cbm_index_worker_destroy(handle); + } else { + index_supervisor_test_cleanup_handle(handle); + } + index_supervisor_test_restore_env("CBM_CACHE_DIR", saved_cache); + (void)th_rmtree(cache); + + ASSERT_EQ(start_rc, 0); +#ifndef _WIN32 + if (!priority_lowered && (priority_error == EPERM || priority_error == EACCES)) { + SKIP_PLATFORM("process-priority changes are blocked by this test sandbox"); + } +#endif + ASSERT_TRUE(priority_lowered); + ASSERT_TRUE(cancel_accepted); + ASSERT_TRUE(terminal); + PASS(); +} + SUITE(index_supervisor) { RUN_TEST(index_supervisor_worker_argv_requires_exact_build_bound_grammar); RUN_TEST(index_supervisor_async_jobs_are_isolated_cancellable_and_terminal_cached); @@ -769,4 +916,8 @@ SUITE(index_supervisor) { RUN_TEST(index_supervisor_terminal_log_lifecycle_matches_outcome_and_profiling); RUN_TEST(index_supervisor_drains_terminal_backlog_into_request_progress_callback); RUN_TEST(index_supervisor_oversized_response_is_contained_and_log_is_retained); + RUN_TEST(index_supervisor_memory_limit_terminates_contained_worker_tree); + RUN_TEST(index_supervisor_duration_limit_terminates_contained_worker_tree); + RUN_TEST(index_supervisor_task_temp_limit_contains_oversized_worker_output); + RUN_TEST(index_supervisor_can_lower_worker_scheduling_priority); } diff --git a/tests/test_main.c b/tests/test_main.c index 6d6c3b3d1..729725034 100644 --- a/tests/test_main.c +++ b/tests/test_main.c @@ -605,6 +605,7 @@ extern void suite_log(void); extern void suite_str_util(void); extern void suite_platform(void); extern void suite_diagnostics(void); +extern void suite_index_limits(void); extern void suite_subprocess(void); extern void suite_private_file_lock(void); extern void suite_lock_registry(void); @@ -833,6 +834,7 @@ int main(int argc, char **argv) { RUN_SELECTED_SUITE(str_util); RUN_SELECTED_SUITE(platform); RUN_SELECTED_SUITE(diagnostics); + RUN_SELECTED_SUITE(index_limits); RUN_SELECTED_SUITE(subprocess); RUN_SELECTED_SUITE(private_file_lock); RUN_SELECTED_SUITE(lock_registry); diff --git a/tests/test_mcp.c b/tests/test_mcp.c index a56aebdf5..244d13b5c 100644 --- a/tests/test_mcp.c +++ b/tests/test_mcp.c @@ -7898,6 +7898,13 @@ TEST(index_supervisor_unsafe_clean_is_never_fallback_or_recovery) { CBM_MCP_SUPERVISED_RESULT_UNSAFE_TERMINAL); result.supervision_failed = false; + result.resource_limit = CBM_INDEX_RESOURCE_LIMIT_MEMORY; + result.resource_observed = 4097; + result.resource_limit_value = 4096; + ASSERT_EQ(cbm_mcp_supervised_result_disposition(0, &result), + CBM_MCP_SUPERVISED_RESULT_RESOURCE_LIMIT); + + result.resource_limit = CBM_INDEX_RESOURCE_LIMIT_NONE; result.outcome = CBM_PROC_CRASH; result.response = NULL; ASSERT_EQ(cbm_mcp_supervised_result_disposition(0, &result), @@ -9231,6 +9238,83 @@ TEST(index_repository_honors_allowed_root) { PASS(); } +TEST(index_repository_rejects_configured_aggregate_root) { + char root[512]; + snprintf(root, sizeof(root), "%s/cbm_denied_root_XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(root)); + + char canonical[CBM_SZ_4K]; + ASSERT_TRUE(cbm_canonical_path(root, canonical, sizeof(canonical))); + + cbm_config_t *config = cbm_config_open(root); + ASSERT_NOT_NULL(config); + ASSERT_EQ(cbm_config_set(config, "index_denied_roots", canonical), 0); + + cbm_mcp_server_t *srv = cbm_mcp_server_new(NULL); + ASSERT_NOT_NULL(srv); + cbm_mcp_server_set_config(srv, config); + + char request[CBM_SZ_8K]; + snprintf(request, sizeof(request), + "{\"jsonrpc\":\"2.0\",\"id\":89,\"method\":\"tools/call\"," + "\"params\":{\"name\":\"index_repository\"," + "\"arguments\":{\"repo_path\":\"%s\"}}}", + root); + char *response = cbm_mcp_server_handle(srv, request); + ASSERT_NOT_NULL(response); + ASSERT_NOT_NULL(strstr(response, "configured_root")); + ASSERT_NOT_NULL(strstr(response, "isError")); + + free(response); + cbm_mcp_server_free(srv); + cbm_config_close(config); + th_rmtree(root); + PASS(); +} + +TEST(index_repository_reports_structured_discovery_resource_limit) { + char repo[512]; + char cache[512]; + (void)snprintf(repo, sizeof(repo), "%s/cbm-index-limit-repo-XXXXXX", cbm_tmpdir()); + (void)snprintf(cache, sizeof(cache), "%s/cbm-index-limit-cache-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(repo)); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + char first[768]; + char second[768]; + (void)snprintf(first, sizeof(first), "%s/first.c", repo); + (void)snprintf(second, sizeof(second), "%s/second.c", repo); + ASSERT_EQ(th_write_file(first, "int first(void) { return 1; }\n"), 0); + ASSERT_EQ(th_write_file(second, "int second(void) { return 2; }\n"), 0); + + cbm_config_t *config = cbm_config_open(cache); + ASSERT_NOT_NULL(config); + ASSERT_EQ(cbm_config_set(config, "index_max_files", "1"), 0); + cbm_mcp_server_t *srv = cbm_mcp_server_new(NULL); + ASSERT_NOT_NULL(srv); + cbm_mcp_server_set_config(srv, config); + + char request[CBM_SZ_4K]; + (void)snprintf(request, sizeof(request), + "{\"jsonrpc\":\"2.0\",\"id\":90,\"method\":\"tools/call\"," + "\"params\":{\"name\":\"index_repository\"," + "\"arguments\":{\"repo_path\":\"%s\",\"mode\":\"fast\"}}}", + repo); + char *response = cbm_mcp_server_handle(srv, request); + ASSERT_NOT_NULL(response); + bool structured = strstr(response, "resource_limit_exceeded") != NULL && + strstr(response, "files") != NULL && strstr(response, "observed") != NULL && + strstr(response, "limit") != NULL && strstr(response, "isError") != NULL; + + free(response); + cbm_mcp_server_free(srv); + cbm_config_close(config); + th_rmtree(cache); + th_rmtree(repo); + + ASSERT_TRUE(structured); + PASS(); +} + TEST(index_repository_relative_path_uses_explicit_session_root) { char session_root[512]; char cache[512]; @@ -9482,6 +9566,103 @@ TEST(index_repository_supervisor_uses_canonical_session_path) { #endif } +#ifndef _WIN32 +enum { + IDXPOLICY_OK = 0, + IDXPOLICY_CONFIG_FAILED = 91, + IDXPOLICY_NO_SERVER = 92, + IDXPOLICY_NO_SPAWN = 93, + IDXPOLICY_NO_RESULT = 94, + IDXPOLICY_LIMIT_BYPASSED = 95, +}; + +static int idxpolicy_supervised_check(const char *repo, const char *cache) { + cbm_index_supervisor_mark_host(); + cbm_unsetenv("CBM_INDEX_SUPERVISOR"); + cbm_unsetenv("CBM_ALLOWED_ROOT"); + cbm_setenv("CBM_INDEX_MAX_RESTARTS", "1", 1); + cbm_setenv("CBM_INDEX_WORKER_TIMEOUT_S", "30", 1); + + cbm_config_t *config = cbm_config_open(cache); + if (!config || cbm_config_set(config, "index_max_files", "1") != 0) { + cbm_config_close(config); + return IDXPOLICY_CONFIG_FAILED; + } + cbm_mcp_server_t *srv = cbm_mcp_server_new(NULL); + if (!srv) { + cbm_config_close(config); + return IDXPOLICY_NO_SERVER; + } + cbm_mcp_server_set_config(srv, config); + + char args[CBM_SZ_8K]; + (void)snprintf(args, sizeof(args), + "{\"repo_path\":\"%s\",\"mode\":\"fast\",\"_cbm_index_limits\":" + "{\"index_max_files\":\"999999\"}}", + repo); + int spawns_before = cbm_index_supervisor_spawn_count(); + char *response = cbm_mcp_handle_tool(srv, "index_repository", args); + int spawns_after = cbm_index_supervisor_spawn_count(); + + int code = IDXPOLICY_OK; + if (spawns_after == spawns_before) { + code = IDXPOLICY_NO_SPAWN; + } else if (!response) { + code = IDXPOLICY_NO_RESULT; + } else if (!strstr(response, "resource_limit_exceeded") || !strstr(response, "files") || + !strstr(response, "\"limit\":1")) { + code = IDXPOLICY_LIMIT_BYPASSED; + } + free(response); + cbm_mcp_server_free(srv); + cbm_config_close(config); + return code; +} +#endif + +TEST(index_repository_supervisor_propagates_trusted_resource_policy) { +#ifdef _WIN32 + SKIP_PLATFORM("supervisor-host guard needs fork isolation (POSIX-only)"); +#else + char repo[512]; + char cache[512]; + (void)snprintf(repo, sizeof(repo), "%s/cbm-policy-repo-XXXXXX", cbm_tmpdir()); + (void)snprintf(cache, sizeof(cache), "%s/cbm-policy-cache-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(repo)); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + char first[768]; + char second[768]; + (void)snprintf(first, sizeof(first), "%s/first.c", repo); + (void)snprintf(second, sizeof(second), "%s/second.c", repo); + ASSERT_EQ(th_write_file(first, "int first(void) { return 1; }\n"), 0); + ASSERT_EQ(th_write_file(second, "int second(void) { return 2; }\n"), 0); + + const char *saved_cache = getenv("CBM_CACHE_DIR"); + char *saved_cache_copy = saved_cache ? strdup(saved_cache) : NULL; + cbm_setenv("CBM_CACHE_DIR", cache, 1); + + fflush(NULL); + pid_t pid = fork(); + if (pid == 0) { + alarm(60); + _exit(idxpolicy_supervised_check(repo, cache)); + } + ASSERT_TRUE(pid > 0); + int status = 0; + (void)waitpid(pid, &status, 0); + int code = WIFEXITED(status) ? WEXITSTATUS(status) : -1; + + restore_cache_dir(saved_cache_copy); + free(saved_cache_copy); + th_rmtree(repo); + th_rmtree(cache); + + ASSERT_TRUE(WIFEXITED(status)); + ASSERT_EQ(code, IDXPOLICY_OK); + PASS(); +#endif +} + /* ══════════════════════════════════════════════════════════════════ * SUITE * ══════════════════════════════════════════════════════════════════ */ @@ -9492,6 +9673,8 @@ SUITE(mcp) { RUN_TEST(detect_changes_rejects_windows_cmd_metacharacters_in_base_branch); RUN_TEST(detect_changes_rejects_windows_cmd_metacharacters_in_project_root); RUN_TEST(index_repository_honors_allowed_root); + RUN_TEST(index_repository_rejects_configured_aggregate_root); + RUN_TEST(index_repository_reports_structured_discovery_resource_limit); /* JSON-RPC parsing */ RUN_TEST(jsonrpc_parse_request); RUN_TEST(jsonrpc_parse_notification); @@ -9639,6 +9822,7 @@ SUITE(mcp) { RUN_TEST(tool_index_repository_dot_uses_absolute_project_key_and_preserves_adr); RUN_TEST(index_repository_relative_path_uses_explicit_session_root); RUN_TEST(index_repository_supervisor_uses_canonical_session_path); + RUN_TEST(index_repository_supervisor_propagates_trusted_resource_policy); RUN_TEST(index_repository_cli_name_override_issue823); RUN_TEST(index_supervisor_unsafe_clean_is_never_fallback_or_recovery); RUN_TEST(index_supervisor_gate_requires_marked_host_issue845); diff --git a/tests/test_pipeline.c b/tests/test_pipeline.c index 2685b425d..52191917e 100644 --- a/tests/test_pipeline.c +++ b/tests/test_pipeline.c @@ -5988,6 +5988,196 @@ TEST(cancelled_full_reindex_preserves_committed_db) { PASS(); } +TEST(discovery_resource_limit_preserves_committed_db) { + if (setup_incremental_repo() != 0) { + FAIL("setup failed"); + } + + cbm_pipeline_t *p = cbm_pipeline_new(g_incr_tmpdir, g_incr_dbpath, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + ASSERT_EQ(cbm_pipeline_run(p), 0); + char *project = strdup(cbm_pipeline_project_name(p)); + cbm_pipeline_free(p); + ASSERT_NOT_NULL(project); + + cbm_store_t *live = cbm_store_open_path(g_incr_dbpath); + ASSERT_NOT_NULL(live); + int baseline_nodes = cbm_store_count_nodes(live, project); + cbm_store_close(live); + ASSERT_GT(baseline_nodes, 0); + + cbm_index_limits_t limits; + cbm_index_limits_defaults(&limits); + limits.max_files = 1; + p = cbm_pipeline_new(g_incr_tmpdir, g_incr_dbpath, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + cbm_pipeline_set_index_limits(p, &limits); + int rc = cbm_pipeline_run(p); + cbm_discover_report_t report = {0}; + cbm_pipeline_get_resource_violation(p, &report); + cbm_pipeline_free(p); + + live = cbm_store_open_path(g_incr_dbpath); + ASSERT_NOT_NULL(live); + bool live_valid = cbm_store_check_integrity(live); + int nodes_after = cbm_store_count_nodes(live, project); + cbm_store_close(live); + + free(project); + cleanup_incremental_repo(); + + ASSERT_EQ(rc, CBM_PIPELINE_RESOURCE_LIMIT); + ASSERT_EQ(report.violation, CBM_DISCOVER_LIMIT_FILES); + ASSERT_EQ(report.limit, 1); + ASSERT_TRUE(live_valid); + ASSERT_EQ(nodes_after, baseline_nodes); + PASS(); +} + +TEST(storage_resource_limits_preserve_committed_db) { + if (setup_incremental_repo() != 0) { + FAIL("setup failed"); + } + + cbm_pipeline_t *p = cbm_pipeline_new(g_incr_tmpdir, g_incr_dbpath, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + ASSERT_EQ(cbm_pipeline_run(p), 0); + char *project = strdup(cbm_pipeline_project_name(p)); + cbm_pipeline_free(p); + ASSERT_NOT_NULL(project); + + cbm_store_t *live = cbm_store_open_path(g_incr_dbpath); + ASSERT_NOT_NULL(live); + int baseline_nodes = cbm_store_count_nodes(live, project); + cbm_store_close(live); + ASSERT_GT(baseline_nodes, 0); + + ASSERT_TRUE(write_go_file(g_incr_tmpdir, "resource_candidate.go", + "package main\n\nfunc ResourceCandidate() int { return 44 }\n")); + + cbm_index_limits_t limits; + cbm_pipeline_limit_report_t report = {0}; + cbm_index_limits_defaults(&limits); + limits.max_staging_bytes = 1; + p = cbm_pipeline_new(g_incr_tmpdir, g_incr_dbpath, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + cbm_pipeline_set_index_limits(p, &limits); + int staging_rc = cbm_pipeline_run(p); + cbm_pipeline_get_limit_violation(p, &report); + cbm_pipeline_free(p); + bool staging_identified = report.violation == CBM_PIPELINE_LIMIT_STAGING_BYTES && + report.observed > report.limit && report.limit == 1; + + cbm_index_limits_defaults(&limits); + limits.max_db_bytes = 1; + p = cbm_pipeline_new(g_incr_tmpdir, g_incr_dbpath, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + cbm_pipeline_set_index_limits(p, &limits); + int database_rc = cbm_pipeline_run(p); + cbm_pipeline_get_limit_violation(p, &report); + cbm_pipeline_free(p); + bool database_identified = report.violation == CBM_PIPELINE_LIMIT_DATABASE_BYTES && + report.observed > report.limit && report.limit == 1; + + cbm_index_limits_defaults(&limits); + limits.min_free_disk_bytes = UINT64_MAX; + p = cbm_pipeline_new(g_incr_tmpdir, g_incr_dbpath, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + cbm_pipeline_set_index_limits(p, &limits); + int disk_rc = cbm_pipeline_run(p); + cbm_pipeline_get_limit_violation(p, &report); + cbm_pipeline_free(p); + bool disk_identified = report.violation == CBM_PIPELINE_LIMIT_FREE_DISK_BYTES && + report.observed < report.limit && report.limit == UINT64_MAX; + + live = cbm_store_open_path(g_incr_dbpath); + ASSERT_NOT_NULL(live); + bool live_valid = cbm_store_check_integrity(live); + int nodes_after = cbm_store_count_nodes(live, project); + int unpublished = count_nodes_named(live, project, "ResourceCandidate"); + cbm_store_close(live); + + free(project); + cleanup_incremental_repo(); + + ASSERT_EQ(staging_rc, CBM_PIPELINE_RESOURCE_LIMIT); + ASSERT_TRUE(staging_identified); + ASSERT_EQ(database_rc, CBM_PIPELINE_RESOURCE_LIMIT); + ASSERT_TRUE(database_identified); + ASSERT_EQ(disk_rc, CBM_PIPELINE_RESOURCE_LIMIT); + ASSERT_TRUE(disk_identified); + ASSERT_TRUE(live_valid); + ASSERT_EQ(nodes_after, baseline_nodes); + ASSERT_EQ(unpublished, 0); + PASS(); +} + +TEST(cache_resource_limit_blocks_new_staging_without_eviction) { + if (setup_incremental_repo() != 0) { + FAIL("setup failed"); + } + char cache[512]; + (void)snprintf(cache, sizeof(cache), "%s/cbm-pipeline-cache-limit-XXXXXX", cbm_tmpdir()); + ASSERT_NOT_NULL(cbm_mkdtemp(cache)); + const char *old_cache = getenv("CBM_CACHE_DIR"); + char *saved_cache = old_cache ? strdup(old_cache) : NULL; + (void)cbm_setenv("CBM_CACHE_DIR", cache, 1); + + cbm_pipeline_t *p = cbm_pipeline_new(g_incr_tmpdir, NULL, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + char *project = strdup(cbm_pipeline_project_name(p)); + ASSERT_NOT_NULL(project); + ASSERT_EQ(cbm_pipeline_run(p), 0); + cbm_pipeline_free(p); + + char db_path[CBM_SZ_4K]; + (void)snprintf(db_path, sizeof(db_path), "%s/%s.db", cache, project); + cbm_store_t *live = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(live); + int baseline_nodes = cbm_store_count_nodes(live, project); + cbm_store_close(live); + ASSERT_GT(baseline_nodes, 0); + + ASSERT_TRUE(write_go_file(g_incr_tmpdir, "cache_candidate.go", + "package main\n\nfunc CacheCandidate() int { return 45 }\n")); + cbm_index_limits_t limits; + cbm_index_limits_defaults(&limits); + limits.cache_max_bytes = 1; + p = cbm_pipeline_new(g_incr_tmpdir, NULL, CBM_MODE_FULL); + ASSERT_NOT_NULL(p); + cbm_pipeline_set_index_limits(p, &limits); + int rc = cbm_pipeline_run(p); + cbm_pipeline_limit_report_t report = {0}; + cbm_pipeline_get_limit_violation(p, &report); + cbm_pipeline_free(p); + + live = cbm_store_open_path(db_path); + ASSERT_NOT_NULL(live); + bool live_valid = cbm_store_check_integrity(live); + int nodes_after = cbm_store_count_nodes(live, project); + int unpublished = count_nodes_named(live, project, "CacheCandidate"); + cbm_store_close(live); + + if (saved_cache) { + (void)cbm_setenv("CBM_CACHE_DIR", saved_cache, 1); + } else { + (void)cbm_unsetenv("CBM_CACHE_DIR"); + } + free(saved_cache); + free(project); + cleanup_incremental_repo(); + th_rmtree(cache); + + ASSERT_EQ(rc, CBM_PIPELINE_RESOURCE_LIMIT); + ASSERT_EQ(report.violation, CBM_PIPELINE_LIMIT_CACHE_BYTES); + ASSERT_GT(report.observed, report.limit); + ASSERT_EQ(report.limit, 1); + ASSERT_TRUE(live_valid); + ASSERT_EQ(nodes_after, baseline_nodes); + ASSERT_EQ(unpublished, 0); + PASS(); +} + /* The same contract applies to the incremental path: the modified file is * present in a complete staging DB at the hook, but cancellation leaves the * prior committed snapshot queryable and removes every staging artifact. */ @@ -7540,6 +7730,9 @@ SUITE(pipeline) { RUN_TEST(incremental_detects_deleted_file); RUN_TEST(incremental_new_file_added); RUN_TEST(cancelled_full_reindex_preserves_committed_db); + RUN_TEST(discovery_resource_limit_preserves_committed_db); + RUN_TEST(storage_resource_limits_preserve_committed_db); + RUN_TEST(cache_resource_limit_blocks_new_staging_without_eviction); RUN_TEST(cancelled_incremental_reindex_preserves_committed_db); RUN_TEST(backup_failed_publish_failure_preserves_final_sidecars); RUN_TEST(backup_failed_rename_failure_preserves_corrupt_main); diff --git a/tests/test_platform.c b/tests/test_platform.c index d705c672f..4a2ebff3b 100644 --- a/tests/test_platform.c +++ b/tests/test_platform.c @@ -42,8 +42,7 @@ TEST(platform_file_apis_survive_max_path_overflow) { ASSERT_NOT_NULL(cbm_mkdtemp(base)); enum { LONG_SEGMENTS = 5 }; - static const char segment[] = - "segment-abcdefghijklmnopqrstuvwxyz0123456789-abcdefghijklmnop"; + static const char segment[] = "segment-abcdefghijklmnopqrstuvwxyz0123456789-abcdefghijklmnop"; char deep[CBM_SZ_1K]; written = snprintf(deep, sizeof(deep), "%s", base); ASSERT_TRUE(written > 0 && written < (int)sizeof(deep)); @@ -431,6 +430,61 @@ TEST(platform_default_workers_env_unset) { PASS(); } +TEST(platform_worker_limit_caps_environment_override_and_restores) { + cbm_setenv("CBM_WORKERS", "32", 1); + + int previous = cbm_worker_limit_set_for_thread(3); + ASSERT_EQ(cbm_default_worker_count(true), 3); + ASSERT_EQ(cbm_default_worker_count(false), 3); + + ASSERT_EQ(cbm_worker_limit_set_for_thread(previous), 3); + ASSERT_EQ(cbm_default_worker_count(true), 32); + cbm_unsetenv("CBM_WORKERS"); + PASS(); +} + +#ifndef _WIN32 +TEST(platform_directory_size_is_bounded_and_does_not_follow_symlinks) { + char base[CBM_SZ_512]; + int written = snprintf(base, sizeof(base), "/tmp/cbm-dir-size-XXXXXX"); + ASSERT_TRUE(written > 0 && written < (int)sizeof(base)); + ASSERT_NOT_NULL(cbm_mkdtemp(base)); + + char first[CBM_SZ_1K]; + char second[CBM_SZ_1K]; + char loop[CBM_SZ_1K]; + written = snprintf(first, sizeof(first), "%s/first", base); + ASSERT_TRUE(written > 0 && written < (int)sizeof(first)); + written = snprintf(second, sizeof(second), "%s/second", base); + ASSERT_TRUE(written > 0 && written < (int)sizeof(second)); + written = snprintf(loop, sizeof(loop), "%s/loop", base); + ASSERT_TRUE(written > 0 && written < (int)sizeof(loop)); + + FILE *file = cbm_fopen(first, "wb"); + ASSERT_NOT_NULL(file); + ASSERT_EQ(fwrite("1234", 1, 4, file), 4); + ASSERT_EQ(fclose(file), 0); + file = cbm_fopen(second, "wb"); + ASSERT_NOT_NULL(file); + ASSERT_EQ(fwrite("123456", 1, 6, file), 6); + ASSERT_EQ(fclose(file), 0); + ASSERT_EQ(symlink(base, loop), 0); + + uint64_t complete = 0; + uint64_t bounded = 0; + ASSERT_TRUE(cbm_directory_size_bounded(base, UINT64_MAX, &complete)); + ASSERT_TRUE(cbm_directory_size_bounded(base, 3, &bounded)); + + ASSERT_EQ(cbm_unlink(loop), 0); + ASSERT_EQ(cbm_unlink(first), 0); + ASSERT_EQ(cbm_unlink(second), 0); + ASSERT_EQ(cbm_rmdir(base), 0); + ASSERT_EQ(complete, 10); + ASSERT_GT(bounded, 3); + PASS(); +} +#endif + TEST(platform_system_info) { cbm_system_info_t info = cbm_system_info(); ASSERT_GT(info.total_cores, 0); @@ -633,6 +687,10 @@ SUITE(platform) { RUN_TEST(platform_default_workers_env_override); RUN_TEST(platform_default_workers_env_invalid); RUN_TEST(platform_default_workers_env_unset); + RUN_TEST(platform_worker_limit_caps_environment_override_and_restores); +#ifndef _WIN32 + RUN_TEST(platform_directory_size_is_bounded_and_does_not_follow_symlinks); +#endif RUN_TEST(platform_system_info); #ifdef __linux__ RUN_TEST(cgroup_v2_cpu_quota); diff --git a/tests/test_subprocess.c b/tests/test_subprocess.c index 39a4b1249..1b78cccc8 100644 --- a/tests/test_subprocess.c +++ b/tests/test_subprocess.c @@ -368,6 +368,8 @@ TEST(subprocess_spawn_returns_while_child_is_running) { uint64_t before_poll = cbm_now_ms(); cbm_proc_poll_t first_poll = cbm_subprocess_poll(process, &result); uint64_t poll_elapsed = cbm_now_ms() - before_poll; + size_t tree_memory_bytes = 0; + bool memory_observed = cbm_subprocess_memory_bytes(process, &tree_memory_bytes); bool cancel_accepted = cbm_subprocess_request_cancel(process); bool terminal = poll_until_terminal(process, 2000, &result); if (terminal) { @@ -383,6 +385,8 @@ TEST(subprocess_spawn_returns_while_child_is_running) { ASSERT_EQ(first_poll, CBM_PROC_POLL_RUNNING); ASSERT_LT(spawn_elapsed, 3500); ASSERT_LT(poll_elapsed, 3500); + ASSERT_TRUE(memory_observed); + ASSERT_GT(tree_memory_bytes, 0); ASSERT_TRUE(cancel_accepted); ASSERT_TRUE(terminal); ASSERT_TRUE(result.cancellation_requested); diff --git a/tests/test_worker_watchdog.sh b/tests/test_worker_watchdog.sh index 01ce360a6..6392a0608 100755 --- a/tests/test_worker_watchdog.sh +++ b/tests/test_worker_watchdog.sh @@ -83,11 +83,22 @@ cleanup() { } trap cleanup EXIT -# Fixture: one good file + one the injector busy-spins on. -mkdir -p "${tmpdir}/repo" +# Fixture: one good file + one the injector busy-spins on. Keep the test cache +# isolated and private so the worker's exact-build admission never depends on +# the invoking user's real cache permissions. +mkdir -p "${tmpdir}/repo" "${tmpdir}/cache" +chmod 700 "${tmpdir}/cache" printf 'def good():\n return 1\n' > "${tmpdir}/repo/good.py" printf 'def slow():\n return 2\n' > "${tmpdir}/repo/hang_me.py" +# A physical worker rejects missing or partial resource policy. Production +# supervisors serialize this complete object after removing caller-supplied +# values; this direct watchdog fixture must model that trusted boundary too. +# Keeping the full registry here is intentional: adding a new policy key must +# fail this integration guard until the physical-worker contract is updated. +TRUSTED_INDEX_POLICY='{"index_max_files":"100000","index_max_directories":"20000","index_max_entries":"500000","index_max_depth":"64","index_max_source_mb":"4096","index_max_file_mb":"64","index_scan_timeout_seconds":"30","index_cpu_cores":"4","index_concurrent_jobs":"1","index_memory_limit_mb":"8192","index_max_db_mb":"16384","index_max_staging_mb":"20480","index_max_task_temp_mb":"24576","index_cache_max_mb":"32768","index_min_free_disk_mb":"4096","index_max_duration_seconds":"3600","index_low_priority":"true","index_denied_roots":""}' +WORKER_ARGS="{\"repo_path\":\"${tmpdir}/repo\",\"_cbm_index_limits\":${TRUSTED_INDEX_POLICY}}" + # Wrapper "parent": launches the worker exactly as the supervisor would # (cli --index-worker … --response-out), records the child PID, then waits. cat >"${tmpdir}/wrapper.sh" <<'SH' @@ -106,7 +117,8 @@ SH chmod +x "${tmpdir}/wrapper.sh" CBM_BINARY="${BINARY}" BUILD_FINGERPRINT="${BUILD_FINGERPRINT}" TMPDIR_PATH="${tmpdir}" \ - ARGS_JSON="{\"repo_path\":\"${tmpdir}/repo\"}" \ + ARGS_JSON="${WORKER_ARGS}" \ + CBM_CACHE_DIR="${tmpdir}/cache" \ CBM_TEST_HANG_ON=hang_me \ CBM_TEST_WORKER_DESCENDANT_PID_FILE="${tmpdir}/descendant.pid" \ "${tmpdir}/wrapper.sh" & @@ -141,11 +153,12 @@ if ! kill -0 "${descendant_pid}" 2>/dev/null; then echo "worker descendant exited before supervisor death" >&2 exit 3 fi -child_pgid="$(ps -p "${child_pid}" -o pgid= 2>/dev/null | tr -d '[:space:]')" -descendant_pgid="$(ps -p "${descendant_pid}" -o pgid= 2>/dev/null | tr -d '[:space:]')" +child_pgid="$(ps -p "${child_pid}" -o pgid= 2>/dev/null | tr -d '[:space:]' || true)" +descendant_pgid="$(ps -p "${descendant_pid}" -o pgid= 2>/dev/null | tr -d '[:space:]' || true)" if [[ -z "${child_pgid}" || "${child_pgid}" != "${child_pid}" || "${descendant_pgid}" != "${child_pgid}" ]]; then echo "worker tree is not in its expected isolated process group" >&2 + [[ -s "${tmpdir}/child.err" ]] && cat "${tmpdir}/child.err" >&2 exit 3 fi