From 687a0659b900275b6e85e329167d980101cd87b8 Mon Sep 17 00:00:00 2001 From: tomiir Date: Tue, 20 Jan 2026 11:41:28 +0100 Subject: [PATCH] feat: add context-drift skill for detecting documentation drift Adds a new skill that analyzes .claude/ context files (skills, commands, agents) for drift from the codebase. Features include: - Broken reference detection (missing files, invalid scripts) - Semantic mismatch detection (documented vs actual behavior) - Staleness flags for human review - Orphaned code detection (significant code lacking docs) Co-Authored-By: Claude Opus 4.5 --- .claude/skills/context-drift/REFERENCE.md | 243 +++++++++++++++++ .claude/skills/context-drift/SKILL.md | 304 ++++++++++++++++++++++ 2 files changed, 547 insertions(+) create mode 100644 .claude/skills/context-drift/REFERENCE.md create mode 100644 .claude/skills/context-drift/SKILL.md diff --git a/.claude/skills/context-drift/REFERENCE.md b/.claude/skills/context-drift/REFERENCE.md new file mode 100644 index 0000000..ba1962d --- /dev/null +++ b/.claude/skills/context-drift/REFERENCE.md @@ -0,0 +1,243 @@ +# Context Drift Reference + +Detailed patterns, heuristics, and thresholds for drift detection. + +--- + +## Context File Types + +### Skills (`.claude/skills/*/SKILL.md`) + +**Frontmatter fields**: +```yaml +--- +name: skill-name # Required +description: ... # Required +user-invocable: true # Optional, defaults to true +--- +``` + +**Common references to check**: +- File paths in code blocks +- Bash commands with `!` prefix +- Tool names (Read, Write, Bash, Grep, etc.) +- Referenced `REFERENCE.md`, `WORKFLOWS.md` files + +### Commands (`.claude/commands/*.md`) + +**Frontmatter fields**: +```yaml +--- +description: ... # Optional +argument-hint: # Optional +--- +``` + +**Common references to check**: +- Variable substitutions (`${VAR}`, `$1`, `$ARGUMENTS`) +- Bash scripts in code blocks +- File path patterns +- CLI tool invocations (`npm`, `git`, `gh`, etc.) + +### Agents (`.claude/agents/*.md`) + +**Frontmatter fields**: +```yaml +--- +name: agent-name # Required +description: ... # Required +tools: Tool1, Tool2 # Required (comma-separated) +model: sonnet|opus|haiku # Optional +permissionMode: ... # Optional +--- +``` + +**Common references to check**: +- Tool list validity +- Model name validity +- Permission mode validity + +--- + +## Reference Extraction Patterns + +### File Paths + +Extract paths from these patterns: + +| Pattern | Example | Regex | +|---------|---------|-------| +| Code block paths | `` `src/utils.ts` `` | `` `([^`]+\.(ts|js|py|go|rs|md))` `` | +| At-mentions | `@src/utils.ts` | `@([\w/./-]+\.\w+)` | +| Quoted paths | `"src/utils.ts"` | `"([\w/./-]+\.\w+)"` | +| In bash commands | `cat src/file.ts` | `(cat\|head\|tail\|read) ([\w/./-]+)` | + +### Bash Commands + +Extract and validate these command types: + +| Type | Pattern | Validation | +|------|---------|------------| +| Script execution | `./scripts/*.sh` | File exists + executable | +| npm/bun/pnpm | `npm run X` | Script exists in package.json | +| Git commands | `git worktree add` | Valid git subcommand | +| CLI tools | `gh pr create` | Tool installed (optional) | + +### Tool References + +Valid Claude Code tools to check against: + +``` +Read, Write, Edit, Bash, Glob, Grep, Task, TodoWrite, +WebFetch, WebSearch, AskUserQuestion, NotebookEdit +``` + +MCP tools follow pattern: `mcp__server__tool_name` + +--- + +## Detection Heuristics + +### Broken Reference Detection + +**Critical severity** (always flag): +- Referenced file does not exist +- Referenced script not found +- Directory path invalid + +**High severity** (likely broken): +- Referenced npm script not in package.json +- Tool name not in valid tool list +- MCP tool format invalid + +**Validation commands**: +```bash +# Check file exists +test -f "$path" && echo "exists" || echo "MISSING" + +# Check script exists +test -x "$script" && echo "executable" || echo "NOT EXECUTABLE" + +# Check npm script +jq -e ".scripts[\"$script_name\"]" package.json >/dev/null 2>&1 +``` + +### Semantic Mismatch Detection + +**Workflow step count**: +1. Count numbered steps in context file (regex: `^\d+\.\s`) +2. Compare to actual implementation steps +3. Flag if difference > 1 step + +**Parameter mismatch**: +1. Extract documented parameters from context +2. Find actual function/command signature +3. Flag if count differs or names don't match + +**Tool usage claims**: +1. Extract "uses X tool" claims from context +2. Search codebase for actual tool usage +3. Flag if claimed tool not found in referenced code + +**Command substitution**: +1. Extract package manager commands (`npm`, `yarn`, `pnpm`, `bun`) +2. Check actual package.json for which is used +3. Flag if documented != actual + +### Staleness Detection + +**Thresholds**: + +| Signal | Threshold | Severity | +|--------|-----------|----------| +| Context age | > 90 days unchanged | Low | +| Code commits since context | > 10 commits | Medium | +| Significant code changes | > 50 lines changed | Medium | + +**Git commands for staleness**: +```bash +# Last context file modification +git log -1 --format="%ci" -- .claude/skills/foo/SKILL.md + +# Commits to related code since context update +git log --since="2024-01-01" --oneline -- src/related/ + +# Lines changed in related code +git diff --stat $(git log -1 --format="%H" -- .claude/skills/foo/SKILL.md) -- src/ +``` + +--- + +## Orphan Detection + +### Significance Filters + +Code is "significant" and needs documentation if ANY of: + +| Criterion | Threshold | Rationale | +|-----------|-----------|-----------| +| File size | > 200 lines | Complex enough to document | +| Export count | > 5 exports | Public API surface | +| Entry point | `index.ts`, `main.ts` | Module boundary | +| Handler pattern | `*Handler.ts`, `*Controller.ts` | API surface | +| Core naming | `core/*.ts`, `engine/*.ts` | Business logic | + +### Exclusion Filters + +Skip these from orphan detection: + +| Pattern | Reason | +|---------|--------| +| `*.test.ts`, `*.spec.ts` | Test files | +| `*.d.ts` | Type definitions | +| `node_modules/**` | Dependencies | +| `dist/**`, `build/**` | Build output | +| `.git/**` | Git internals | +| `*.config.ts` | Config files | +| `*.generated.ts` | Generated code | + +### Detection Algorithm + +``` +1. referenced_paths = extract_all_paths(context_files) +2. all_code_files = glob("**/*.{ts,js,py,go,rs}") +3. orphans = all_code_files - referenced_paths +4. significant_orphans = filter(orphans, is_significant) +5. report(significant_orphans) +``` + +--- + +## Severity Classification Matrix + +| Issue Type | Severity | Action | +|------------|----------|--------| +| File not found | Critical | Must fix before merge | +| Script missing | Critical | Must fix before merge | +| Invalid tool name | High | Should fix soon | +| Workflow step mismatch | High | Review and update | +| Parameter mismatch | High | Review and update | +| Command substitution | Medium | Update when convenient | +| Stale context (90+ days) | Low | Review for relevance | +| Orphaned significant code | Advisory | Consider documenting | + +--- + +## Report Generation + +### Section Order + +1. **Summary table** - Quick overview +2. **Critical issues** - Broken references (must fix) +3. **Semantic mismatches** - Behavior drift (should fix) +4. **Review flags** - Potential issues (investigate) +5. **Orphaned code** - Missing documentation (consider) +6. **Recommendations** - Prioritized actions + +### Formatting Guidelines + +- Use tables for structured data +- Include line numbers for issues +- Show both "documented" and "actual" values for mismatches +- Group issues by context file +- Sort by severity within groups diff --git a/.claude/skills/context-drift/SKILL.md b/.claude/skills/context-drift/SKILL.md new file mode 100644 index 0000000..ef98145 --- /dev/null +++ b/.claude/skills/context-drift/SKILL.md @@ -0,0 +1,304 @@ +--- +name: context-drift +description: Analyzes .claude/ context files for drift from codebase. Detects broken references, semantic mismatches, and orphaned code. Use when auditing documentation freshness, checking context alignment, or before releases. +user-invocable: true +--- + +# Context Drift Detector + +Analyzes `.claude/` context files (skills, commands, agents) for drift from the actual codebase. Detects broken references, semantic mismatches, and identifies code that lacks documentation. + +## When to use + +- Auditing documentation freshness before a release +- After major refactoring to catch stale context +- Periodic maintenance to ensure docs match reality +- When onboarding to verify context files are current +- Before merging PRs that touch documented areas + +## When not to use + +- Writing new documentation (use `/update-docs`) +- General code quality checks (use `/tech-debt-hunt`) +- Understanding codebase structure (use `/understand-codebase`) + +--- + +## Detection Methods + +This skill uses three complementary detection approaches: + +| Method | Severity | What it detects | +|--------|----------|-----------------| +| **Broken References** | Critical/High | File paths, commands, scripts that no longer exist | +| **Semantic Mismatch** | High/Medium | Documented behavior that doesn't match code | +| **Review Flags** | Low/Advisory | Potential drift needing human review | + +See REFERENCE.md for detailed detection patterns. + +--- + +## Default Workflow + +### Phase 1: Discovery + +1. **Find context files**: + ```bash + find .claude/skills -name "*.md" -type f + find .claude/commands -name "*.md" -type f + find .claude/agents -name "*.md" -type f + ``` + +2. **Parse each file** to extract: + - File path references (code blocks, `@path` mentions) + - Bash commands and scripts + - Tool/MCP references + - Documented workflows and parameters + +3. **Build reference map**: context file → referenced code paths + +### Phase 2: Analysis + +For each context file, run all detection methods: + +1. **Validate references**: + - Check all file paths exist + - Verify scripts/commands are runnable + - Confirm tool names are valid + +2. **Semantic analysis** (for skills/commands): + - Compare documented workflow steps vs actual + - Check parameter counts match + - Verify tool usage claims + +3. **Staleness check**: + - Get last modified date of context file + - Check if referenced code has significant changes since + - Flag if context untouched for 90+ days + +4. **Classify by severity**: Critical → High → Medium → Low + +### Phase 3: Orphan Detection + +1. **Collect all referenced paths** from context files +2. **Find significant code** not referenced: + - Files > 200 lines + - Public exports/entry points + - API handlers +3. **Report as "Needs Documentation"** + +### Phase 4: Reporting + +Generate structured report with: +- Summary counts by severity +- Critical issues (must fix) +- Semantic mismatches (should fix) +- Review flags (investigate) +- Orphaned code table + +--- + +## Output Format + +```markdown +# Context Drift Report + +**Generated**: {timestamp} +**Repository**: {repo_name} + +## Summary + +| Category | Count | +|----------|-------| +| Context files scanned | N | +| Critical issues | X | +| Semantic mismatches | Y | +| Review flags | Z | +| Orphaned code files | O | + +## Critical Issues (Broken References) + +### .claude/skills/foo/SKILL.md +- **Line 45**: References `src/utils/helper.ts` - FILE NOT FOUND +- **Line 78**: Command `./scripts/validate.sh` - SCRIPT MISSING + +## Semantic Mismatches + +### .claude/commands/bar.md +- **Workflow drift**: Documents 3 steps, implementation has 5 +- **Command drift**: Mentions `npm test`, package.json uses `bun test` + +## Review Flags + +### .claude/agents/baz.md +- Related code changed 15 commits since last context update +- Context file unchanged for 120 days + +## Orphaned Code (Needs Documentation) + +| File | Lines | Significance | +|------|-------|--------------| +| src/api/handlers.ts | 450 | Public API handler | +| src/core/engine.ts | 380 | Core module, exported | + +## Recommendations + +1. Fix critical issues immediately (broken references) +2. Review semantic mismatches for accuracy +3. Update or remove flagged stale documentation +4. Consider adding context for orphaned code +``` + +--- + +## Validation Checklist + +- [ ] All `.claude/skills/`, `.claude/commands/`, `.claude/agents/` files discovered +- [ ] File path references extracted from code blocks and inline mentions +- [ ] Bash commands extracted and validated +- [ ] Broken references flagged as Critical +- [ ] Semantic analysis completed without false positives +- [ ] Orphaned code filtered by significance threshold +- [ ] Report sections organized by severity +- [ ] Recommendations are actionable + +--- + +## Examples + +### Example 1: Full repository scan + +``` +User: /context-drift + +Claude: Running context drift analysis... + +# Context Drift Report + +**Generated**: 2024-01-15 14:30:00 +**Repository**: my-project + +## Summary + +| Category | Count | +|----------|-------| +| Context files scanned | 12 | +| Critical issues | 2 | +| Semantic mismatches | 3 | +| Review flags | 1 | +| Orphaned code files | 4 | + +## Critical Issues + +### .claude/skills/worktree/SKILL.md +- **Line 58**: References `git worktree add -b` but worktree command deprecated +- **Line 120**: Script `./scripts/cleanup.sh` does not exist + +[...continues with full report...] +``` + +### Example 2: No drift found + +``` +User: /context-drift + +Claude: Running context drift analysis... + +# Context Drift Report + +**Generated**: 2024-01-15 14:30:00 +**Repository**: my-project + +## Summary + +| Category | Count | +|----------|-------| +| Context files scanned | 8 | +| Critical issues | 0 | +| Semantic mismatches | 0 | +| Review flags | 0 | +| Orphaned code files | 0 | + +All context files are current. No drift detected. +``` + +### Example 3: Repository without .claude/ directory + +``` +User: /context-drift + +Claude: No `.claude/` directory found in this repository. + +To use context drift detection, create context files: +- `.claude/skills/` - For complex workflows with subcommands +- `.claude/commands/` - For simple prompt templates +- `.claude/agents/` - For specialized sub-agent definitions + +Run `/skill-writing` or `/command-creator` to get started. +``` + +--- + +## Notes + +- This skill is resource-intensive; run manually, not automatically +- Semantic analysis is heuristic-based and may need human judgment +- Orphan detection uses conservative thresholds to reduce noise +- For large repos, consider scoping to specific directories +- See REFERENCE.md for customizing detection patterns + +--- + +## Evaluation Prompts + +### Test 1: Activation - should trigger +``` +User: /context-drift +Expected: Skill activates, scans .claude/ files, produces drift report +``` + +### Test 2: Activation - should trigger +``` +User: Check if my claude context files are up to date +Expected: Skill activates, runs full analysis +``` + +### Test 3: Non-activation - should NOT trigger +``` +User: How do I write good documentation? +Expected: Normal response about documentation practices, NOT drift scan +``` + +### Test 4: Non-activation - should NOT trigger +``` +User: Update the README with the latest changes +Expected: Normal documentation update (like /update-docs), NOT drift audit +``` + +### Test 5: Edge case - no .claude directory +``` +User: /context-drift +Context: Repository has no .claude/ folder +Expected: Helpful message explaining how to create context files +``` + +### Test 6: Edge case - all current +``` +User: /context-drift +Context: All context files recently updated, all references valid +Expected: Report stating "No drift detected" with green summary +``` + +### Test 7: Edge case - broken reference +``` +User: /context-drift +Context: A SKILL.md references a file that was deleted +Expected: Critical severity flag with exact line number and file path +``` + +### Test 8: Edge case - semantic mismatch +``` +User: /context-drift +Context: Command documents "npm test" but package.json uses "bun test" +Expected: Semantic mismatch flagged with both documented and actual values +```