Skip to content

fix: search hidden files and keep partial repo_search results - #90

Merged
youdie006 merged 3 commits into
mainfrom
fix/repo-search-gaps
Nov 9, 2026
Merged

youdie006 merged 3 commits into
mainfrom
fix/repo-search-gaps

Conversation

@youdie006

Copy link
Copy Markdown
Owner

Problems

Found 2026-11-09 by calling the MCP repo tool handlers against throwaway repos:

  • repo_search never searched hidden files. .github/workflows/ci.yml, .eslintrc.cjs and .husky/ returned no matches, even with the glob .github/**, although repo_read_file reads them. ripgrep skips hidden files by default.
  • A match on a line that is not UTF-8 was dropped. ripgrep reports such a line as base64 bytes, and the parser required text. The same applied to file names that are not UTF-8.
  • One directory ripgrep could not read (mode 000, such as a database volume) made it exit 2. The whole search then failed with the raw ripgrep command line, and the matches it had printed were lost.
  • A broad query (6,000 matching lines) outgrew the 1 MB output buffer and failed as "too many matches" instead of returning the first 100 with truncated: true.
  • A query containing a line break failed with the raw command line.
  • ..notes.md was refused as escaping the repository, because the check was relative.startsWith("..").

Changes (src/repo.ts)

  • ripgrep runs with --hidden. The exclusion globs gain !.git and !**/.git for a worktree's .git file. Every result still passes isAllowedRepoSearchResult, the same path policy as reads.
  • bytes paths and lines are decoded lossily.
  • Exit code 2 with matches returns those matches. A regex error is still reported as before.
  • On a buffer overflow, the partial output is parsed and returned marked truncated. The old error remains when the partial output holds no complete match, which keeps the existing "oversized output" test valid.
  • A query containing CR or LF is refused up front with an explanation.
  • escapesRoot() treats only .. or ../... as leaving the root. It is used by resolveRepoPath and by the realpath check.

Verification

  • tests/repo-search-gaps.test.ts: 7 cases. 6 fail against main; the seventh is a guard that sensitive dot-files stay out of the results. All 7 pass here. tests/repo.test.ts passes unchanged.
  • The original repro scripts now return the expected results: hidden files found, the Latin-1 line found, matches returned despite the unreadable directory, 100 matches with truncation, and ..notes.md read.
  • npx tsc --noEmit passes. Full suite: 1872 passed, 3 skipped.
  • Not changed: a file name over about 192 bytes still fails repo_write_file_apply with ENAMETOOLONG on the temp file name (src/safe-file.ts).

Found 2026-11-09 driving the MCP repo tools against throwaway repos:

- ripgrep skips hidden files unless asked, so repo_search never found
  anything in .github/, .eslintrc.cjs or .husky/, even with a glob such as
  .github/**, while repo_read_file reads them. Search now passes --hidden.
  Every result still goes through isAllowedRepoSearchResult, the same path
  policy as reads, and the exclusion globs (now also the .git file of a
  worktree) keep .env, .git, .bridge and key material unopened.
- ripgrep reports a line or path that is not UTF-8 as base64 "bytes", and
  such matches were dropped. They are decoded lossily now.
- A path ripgrep could not read made it exit 2; the whole search failed
  with the raw command line and the other matches were lost. Exit 2 with
  matches now returns them (a regex error still gets its own message).
- Output over the 1 MB buffer failed as "too many matches" instead of
  returning the first 100 marked truncated. The partial output is parsed
  now; the error remains when it holds no complete match.
- A query with a line break failed with the raw command line; it now
  says a search matches one line at a time.
- resolveRepoPath and the realpath check refused "..notes.md" as
  escaping the root (relative.startsWith("..")).

Verified: tests/repo-search-gaps.test.ts fails 6 of 7 against main and
passes here; tests/repo.test.ts unchanged and passing; the original repro
scripts now return the expected results; npx tsc --noEmit; full suite
1872 passed, 3 skipped.

Not changed: a file name over about 192 bytes still fails apply with
ENAMETOOLONG on the temp file name (safe-file.ts).

Claude-Session: https://claude.ai/code/session_01EMqNwnnhHXznd2P2PDsaDV
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T01:21:43.539142Z adabbd0 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

What this touches

  • bridge store
  • tests

3 file(s), 177 line(s) changed. Tests changed.

This is an automated first pass, not a review. CI is the gate; a person decides.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: adabbd0324

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/repo.ts
if (!isRipgrepJsonMatch(event)) return undefined;
return {
path: normalizeRipgrepMatchPath(event.data.path.text),
path: normalizeRipgrepMatchPath(ripgrepText(event.data.path)),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Validate the original path bytes before lossy decoding

On Unix, lossily decoding a non-UTF-8 filename before passing it to isAllowedRepoSearchResult makes the security checks inspect a different file. I reproduced this with an invalid-byte filename bad-\xff.txt hard-linked to an outside secret and an ordinary, single-link file named bad-�.txt: searching for TOKEN returns the outside secret under the ordinary file's name because the realpath and hard-link checks validate that ordinary file instead. Previously these byte-path events were discarded. Preserve the original path bytes for filesystem validation, or reject paths that cannot round-trip through UTF-8, rather than using the replacement-character path for authorization.

Useful? React with 👍 / 👎.

@youdie006
youdie006 merged commit d514236 into main Nov 9, 2026
7 checks passed
youdie006 added a commit that referenced this pull request Nov 10, 2026
…ch results) to the container image

Same change as main PR #90 (merge d514236), released in 0.40.51, applied to
src, tests and docs without changes. The CHANGELOG lines are main's with
the PR noted.

Claude-Session: https://claude.ai/code/session_01EMqNwnnhHXznd2P2PDsaDV
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant