Skip to content

feat: built-in bash skill (bash -c, shell parity with a native tool) - #145

Merged
snimu merged 5 commits into
mainfrom
feat/bash-skill
Aug 26, 2026
Merged

feat: built-in bash skill (bash -c, shell parity with a native tool)#145
snimu merged 5 commits into
mainfrom
feat/bash-skill

Conversation

@samsja

@samsja samsja commented Aug 25, 2026

Copy link
Copy Markdown
Member

Summary

Adds a built-in bash skill (enabled via RLM_SKILLS): await bash(command=...) runs a shell command from the REPL and returns combined stdout/stderr as a string (exit code appended when nonzero, output clipped at 30k chars).

Why: lets the agent mix shell with Python in one cell — capture output as a value, parse it programmatically, or sidestep shell-quoting by building commands as triple-quoted Python strings.

Implementation note: executes via ["bash", "-c", command] explicitly rather than create_subprocess_shell, which resolves to /bin/sh (dash on debian images) and rejects bashisms — we hit sh: Syntax error: "(" unexpected on process substitution in ablation runs before pinning this down. A parity test (cat <(echo ...)) guards it.

Tests: 4 new (output capture, bashism parity, exit-code reporting, stderr merging); suite green except 6 pre-existing test_acp.py failures that also fail on clean main in this environment.

Extracted from #142 so it can ship independently.


Note

Medium Risk
Introduces arbitrary shell execution from the REPL; mitigated by shared git-history blocking and timeouts, but still expands the agent’s command surface.

Overview
Adds a bash built-in skill (enable with RLM_SKILLS) so agents can await bash(...) from IPython and get shell output as a string instead of only using separate shell tools.

The new rlm.skills.bash module runs one command per call via bash -c (not /bin/sh), merges stdout and stderr, appends [exit code N] on failure, honors RLM_EXEC_TIMEOUT (default 300s), and applies the same find_blocked_command / git-history guard as other shell paths. Registration in _BUILTIN_SKILLS wires it into the existing session stub / pre-import flow like edit.

Tests cover echo output, bash-specific syntax (process substitution), nonzero exits, stderr, git guard behavior, timeouts, and that enable_builtin_skills(["bash"], ...) writes the expected re-export stub.

Reviewed by Cursor Bugbot for commit bab8cf6. Bugbot is set up for automated code reviews on this repo. Configure here.

A ``bash`` builtin skill for RLM_SKILLS: run a shell command from the REPL and
get combined stdout/stderr back as a string (exit code appended when nonzero,
output clipped at 30k chars). Useful for mixing shell with Python in one cell
(parse output programmatically) or avoiding shell-quoting via triple-quoted
Python strings.

Executes via ["bash", "-c", command] explicitly — not create_subprocess_shell,
which resolves to /bin/sh (dash) and rejects bashisms like process
substitution. Covered by tests including a bashism-parity check.
hallerite
hallerite previously approved these changes Aug 25, 2026
…ault

The skill now runs the same policy as shell tool paths: find_blocked_command
refusal before execution (an agent could previously launder blocked git-history
commands through `await bash(...)`), and the default timeout follows the
harness exec timeout (RLM_EXEC_TIMEOUT, fallback 300) instead of a hardcoded 120.
Sync subprocess.run in a thread, same output contract. Tests for both.
@snimu
snimu marked this pull request as ready for review August 25, 2026 21:23

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 03f601b. Configure here.

Comment thread src/rlm/skills/bash.py
The registration edit targeted the pre-rename dict shape and silently never
applied — RLM_SKILLS=bash raised unknown-skill and the kernel never
pre-imported it. Register it and cover the registration path with a
stub-writing test (the direct-import tests missed this).

Reported by cursor bugbot on #145.
@samsja

samsja commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

Fixed in bab8cf6 — the registration edit had targeted the pre-rename dict shape and silently never applied. bash is now in _BUILTIN_SKILLS and the registration path is covered by a stub-writing test (test_enable_bash_skill_writes_stub). Good catch.

@snimu
snimu merged commit 7ca4573 into main Aug 26, 2026
9 checks passed
@snimu
snimu deleted the feat/bash-skill branch August 26, 2026 08:01
snimu pushed a commit that referenced this pull request Aug 26, 2026
…mpaction handoff (#142)

## Summary

The rlm harness, rebuilt on evidence from a k3-focused ablation campaign
(9 models, up to 128 tasks/arm, SWE-rebench-V2):

1. **Terse coding-agent persona** replaces the verbose default system
prompt (~3.2kB -> ~1.4kB)
2. **`RLM_TOOLING` preset — `dual` (default) | `tools` | `skills`** —
one knob for how shell/edit are exposed
3. **Kernel-isolation prompt line** (stops project imports into the
kernel venv)
4. **Actionable compaction handoff** (fenced command blocks + numbered
next steps)
5. bash tool/skill share one guarded runner (`bash -c`, git-history
guard, harness timeout); output-clipping removed

## The starting point: something was wrong

kimi-k3 on stock rlm vs a plain bash+edit agent, same tasks:

| setup | solve (easy 32) | reasoning tok/turn | solve (SWE-bench Pro
32) |
|---|---|---|---|
| stock rlm (ipython-only, stock prompt) | 0.53 | 327 | 0.75 |
| bash+edit agent (verifiers harness) | 0.69 | 268 | 0.94 |

Ablations isolated the cause: **the verbose prompt** (not the ipython
interface, not `%%bash` syntax). Same tools, persona prompt: 209-232
tok/turn, solve 0.62-0.75. Frontier models are insensitive either way;
kimi-k3 and opus-5 are prompt-sensitive (~2x reasoning inflation under
the stock prompt).

## Final ablation: the three `RLM_TOOLING` presets (128 tasks, kimi-k3,
same seeded set)

| preset | n | solve | turns | reason/turn | error turns |
|---|---|---|---|---|---|
| `tools` | 128 | 0.711 | 36.1 | 216 | **0.8%** |
| `skills` (with triple-quote guidance) | 125 | 0.704 | **33.5** | 256 |
2.5% |
| `dual` | 128 | 0.695 | 35.4 | 239 | 2.1% |

Converged on solve and total reasoning (~8k tok). `tools` is cleanest,
`skills` fewest turns (REPL cells batch multiple actions), `dual` within
noise of both.

The skills preset needed two fixes, each isolated on the same 32 tasks:

| skills variant | turns | error turns |
|---|---|---|
| buggy skill (ran /bin/sh) + no guidance | 53.5 | 4.2% |
| + shell-parity fix (`bash -c`) | 35.9 | 5.0% |
| + triple-quote prompt line | **33.5** | **2.5%** |

The bugfix bought the turns; the prompt line bought the errors. The
`skills` preset auto-renders the validated line: *"Run shell with `out =
await bash(\'\'\'command here\'\'\')` — always triple-quote the command
so shell quotes and multi-line scripts never need escaping..."*

Cross-model check (GLM-5.3 @32): `tools` 0.61/1.3% err, `skills` 0.63
but 2.4x the reasoning tokens and 3.7% err, `dual` 0.59/0.6% err —
skills-only is kimi-validated, not universal; hence **`dual` as
default** (never the outlier on any model; both alternatives one env var
away).

## Kernel-isolation line

kimi imported project modules into the rlm kernel (~25 attempts/32
rollouts, 1 in 5 hitting ModuleNotFoundError). The line "The ipython
kernel is an isolated venv without the project's packages... everything
that executes project code goes through bash" drops kernel
project-imports 5 -> 0, ImportErrors -> 0, at *lower* reasoning (161 vs
182 tok/turn on the probe set) — informational phrasing, no prohibition
cost.

## Compaction

`CHECKPOINT_COMPACTION_PROMPT` now demands runnable handoffs (exact
repro/test commands + pending edits as fenced code blocks, numbered next
steps): validated 0 -> ~4 command blocks per handoff, compacted-rollout
solve 0.75-0.88 vs 0.53 under the stock prompt. Also checked: advance
compaction notice in the system prompt is unnecessary (models stash
state naturally; the at-compaction REPL note suffices; an explicit stash
line doubles state-keeping but buys no solve).

## Notes

- Merged with main (absorbs #145 bash skill and #146 knob removal; keeps
#134 supervisor semantics — presets respected via `_selected()` in the
tool registry, `RLM_SKILLS` default now preset-aware in
`RuntimeConfig.from_env`).
- Tests: 125 passed; the 6 `test_acp.py` failures are pre-existing on
clean main in this environment. Ruff clean.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes default tooling, prompts, and compaction handoffs that
directly steer agent behavior; bash/edit run on the host cwd with file
writes, though git guards and timeouts are centralized.
> 
> **Overview**
> **Default agent surface** moves from IPython-only to native
**`bash`**, **`edit`**, and **`ipython`** tools, with **`RLM_TOOLING`**
(`dual` | `tools` | `skills`) and **`RLM_BUILTIN_TOOLS`** to override.
When **`RLM_SKILLS`** is unset, skills now follow the preset via
**`preset_skills()`** instead of an empty list.
> 
> **`rlm.tools.bash`** adds shared **`run_bash`** (guarded `bash -c`,
cwd, `allow_git`) used by **`BashTool`**, **`EditTool`**, and the
**`bash`** skill; the IPython kernel gets **`RLM_EXEC_TIMEOUT`** /
**`RLM_ALLOW_GIT`** at startup from engine policy.
> 
> **System prompt** is shorter and tool-aware: coding-agent role lines
per active tool, kernel-isolation guidance (**`PROJECT_ENV_PROMPT`**),
conditional bash-skill triple-quote hints, and trimmed IPython/edit
copy. **Compaction** **`CHECKPOINT_COMPACTION_PROMPT`** now requires
fenced repro commands, pending **`edit`** calls, and numbered next
steps.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
41e2698. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants