Add Devin CLI provider - #23
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
![amazon-q-developer[bot]](https://avatars.githubusercontent.com/in/1220912?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR successfully adds Devin CLI provider support with comprehensive implementation and test coverage. The provider follows established patterns in the codebase and is properly registered in all required locations:
✅ Complete provider implementation with status detection logic
✅ Comprehensive unit test coverage (initialization, status detection, response extraction, tool restrictions, registration)
✅ Proper integration with provider manager, tool mapping, and CLI commands
✅ Temporary file handling with cleanup
✅ MCP server configuration support
The implementation is well-structured and ready to merge.
You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.
codeant-ai
Bot
added
the
size:XL
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughAdds Devin CLI provider support across provider registration, input delivery, status handling, web UI exposure, and test coverage. ChangesDevin CLI Provider Integration
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the DevinCliProvider to support the Devin CLI, along with its registration across the orchestrator and a suite of unit tests. The code review highlights several critical integration issues where the new provider violates the BaseProvider interface: initialize should be asynchronous (and its unit tests updated accordingly), get_status must accept a buffer parameter to avoid runtime TypeErrors, and the constructor and manager should support skill_prompt to enable custom skills. Additionally, temporary files should be cleaned up in _build_command to prevent leaks.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (2 files)
Previous Review Summaries (8 snapshots, latest commit 325cd33)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 325cd33)Status: No Issues Found | Recommendation: Merge Files Reviewed (1 file)
Previous review (commit f94d97d)Status: No Issues Found | Recommendation: Merge Files Reviewed (1 file)
Previous review (commit 0032a82)Status: No Issues Found | Recommendation: Merge Files Reviewed (1 file)
Previous review (commit 2427398)Status: No Issues Found | Recommendation: Merge Files Reviewed (1 file)
Previous review (commit f990d4e)Status: No Issues Found | Recommendation: Merge Files Reviewed (1 file)
Previous review (commit 2b1b42d)Status: No Issues Found | Recommendation: Merge Files Reviewed (2 files)
Previous review (commit b8c121b)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (4 files)
Fix these issues in Kilo Cloud Previous review (commit a1b5197)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (3 files)
Reviewed by step-3.7-flash-20260528 · Input: 269.5K · Output: 3.5K · Cached: 605.2K |
![codeant-ai[bot]](https://avatars.githubusercontent.com/in/646884?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
test/providers/test_devin_cli_unit.py (1)
1-362: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winApply Black formatting to pass CI checks.
The pipeline reports Black formatting failures (would reformat). Run:
uv run black test/providers/test_devin_cli_unit.py🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/providers/test_devin_cli_unit.py` around lines 1 - 362, The test file test_devin_cli_unit.py does not conform to Black formatting standards, causing CI pipeline failures. Run the Black code formatter on this file by executing the command provided in the comment to automatically reformat the file according to Black's standards. This will fix all formatting issues reported by the pipeline such as line length, spacing, and other style conventions that Black enforces.Source: Pipeline failures
src/cli_agent_orchestrator/providers/devin_cli.py (1)
1-353: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winRun Black on this file before merge.
CI is currently failing because this file would be reformatted by Black. Please format it to unblock the quality gate.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cli_agent_orchestrator/providers/devin_cli.py` around lines 1 - 353, The file devin_cli.py needs to be formatted according to Black's code style standards to pass the CI quality gate. Run the Black formatter on this file by executing the command: black src/cli_agent_orchestrator/providers/devin_cli.py. This will automatically reformat the code to comply with Black's standards without requiring manual changes to the logic or structure.Source: Pipeline failures
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/cli_agent_orchestrator/providers/devin_cli.py`:
- Line 228: The list comprehension on line 228 uses the ambiguous loop variable
`l` which triggers linting errors and reduces readability. Rename the loop
variable `l` to a more descriptive name such as `line` throughout the
comprehension `[l for l in tail[:idx] if l.strip()]` to improve code clarity and
comply with linting standards. Replace all occurrences of `l` in this
comprehension with the new variable name.
In `@src/cli_agent_orchestrator/utils/tool_mapping.py`:
- Around line 51-56: The "fs_*" mapping in the "devin_cli" configuration
includes "Bash" as an allowed permission, which grants arbitrary shell execution
to profiles that only authorize wildcard filesystem operations. Remove "Bash"
from the "fs_*" list in the mapping so that filesystem wildcard access only
includes "Read" and "Write" permissions, without implicitly granting
execute_bash capabilities.
---
Outside diff comments:
In `@src/cli_agent_orchestrator/providers/devin_cli.py`:
- Around line 1-353: The file devin_cli.py needs to be formatted according to
Black's code style standards to pass the CI quality gate. Run the Black
formatter on this file by executing the command: black
src/cli_agent_orchestrator/providers/devin_cli.py. This will automatically
reformat the code to comply with Black's standards without requiring manual
changes to the logic or structure.
In `@test/providers/test_devin_cli_unit.py`:
- Around line 1-362: The test file test_devin_cli_unit.py does not conform to
Black formatting standards, causing CI pipeline failures. Run the Black code
formatter on this file by executing the command provided in the comment to
automatically reformat the file according to Black's standards. This will fix
all formatting issues reported by the pipeline such as line length, spacing, and
other style conventions that Black enforces.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: a274b730-1d69-4dc7-b425-1aed3d0223c6
📒 Files selected for processing (16)
src/cli_agent_orchestrator/api/main.pysrc/cli_agent_orchestrator/cli/commands/launch.pysrc/cli_agent_orchestrator/models/provider.pysrc/cli_agent_orchestrator/providers/devin_cli.pysrc/cli_agent_orchestrator/providers/manager.pysrc/cli_agent_orchestrator/services/settings_service.pysrc/cli_agent_orchestrator/utils/agent_profiles.pysrc/cli_agent_orchestrator/utils/tool_mapping.pytest/api/test_api_endpoints.pytest/providers/fixtures/devin_cli_completed_output.txttest/providers/fixtures/devin_cli_complex_response.txttest/providers/fixtures/devin_cli_error_output.txttest/providers/fixtures/devin_cli_heading_response.txttest/providers/fixtures/devin_cli_idle_output.txttest/providers/fixtures/devin_cli_processing_output.txttest/providers/test_devin_cli_unit.py
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
4 issues found across 16 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…ocations Agent-Logs-Url: https://github.com/ThePlenkov/cli-agent-orchestrator/sessions/ec6174e8-a19e-406a-87c1-fa0063d109bb Co-authored-by: ThePlenkov <6381507+ThePlenkov@users.noreply.github.com>
Agent-Logs-Url: https://github.com/ThePlenkov/cli-agent-orchestrator/sessions/ec6174e8-a19e-406a-87c1-fa0063d109bb Co-authored-by: ThePlenkov <6381507+ThePlenkov@users.noreply.github.com>
…tools via security prompt Agent-Logs-Url: https://github.com/ThePlenkov/cli-agent-orchestrator/sessions/1a11fb12-9327-45e4-a3bd-6eb94a07f9eb Co-authored-by: ThePlenkov <6381507+ThePlenkov@users.noreply.github.com>
This commit addresses feedback from PR #10 to align with the build-cao-provider skill requirements: **MCP Configuration Enhancement:** - Merge MCP servers from agent profile with user's existing Devin config - Ensure CAO_TERMINAL_ID is set in env for all MCP servers (critical for orchestration) - Preserve user's existing MCP settings while adding cao-mcp-server - Support agent profiles with custom MCP server configurations **E2E Testing Infrastructure:** - Add require_devin fixture to test/e2e/conftest.py - Add TestDevinCliSupervisorOrchestration class with 3 test methods: - test_supervisor_handoff: validates handoff MCP tool delegation - test_supervisor_assign_and_handoff: validates multi-agent workflow - test_supervisor_assign_three_analysts: canonical examples/assign smoke test **Agent Profile Installation:** - Install analysis_supervisor, data_analyst, and report_generator profiles for devin_cli - Enables supervisor orchestration patterns (assign + handoff + send_message) These changes implement critical requirements from the build-cao-provider skill, particularly lesson #1 (CAO_TERMINAL_ID forwarding) and the E2E supervisor orchestration validation pattern. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Add devin_cli to FALLBACK_PROVIDERS in AgentPanel.tsx - Add 'Devin' to SOURCE_LABELS - Add Playwright E2E test scripts to package.json Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
These tests verify that Devin CLI is properly integrated with the CAO web interface: - Web interface loads correctly - Devin CLI appears in providers list - Spawn Agent modal shows Devin CLI option - Agent profiles are available for Devin CLI - Provider registration is functional Compared to existing unit/API tests, these E2E tests: - Test the full user journey through the web UI - Catch integration issues between frontend and backend - Verify UI rendering and user interaction flows - Provide confidence that web features work end-to-end Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…paste-buffer control Root cause: The event-driven architecture (commit 75e98ac) changed wait_until_status to rely on FIFO-based status monitoring. In WSL2, tmux pipe-pane cannot write to named pipes, causing the buffer to stay empty and status to remain UNKNOWN forever. cao-poc worked because it used the old API (provider.get_status() with backend.get_history()). Changes: - status_monitor: Add fallback to backend.get_history() for tmux backends when FIFO buffer is empty (e.g., WSL limitation). Reads pane history directly and runs provider detection on it. Caches result to avoid repeated history reads. Provides WSL compatibility without affecting the normal FIFO path. - base.py: Add use_paste_buffer property to allow providers to opt out of paste-buffer (Devin CLI doesn't support it for user input) - tmux_client: Add use_paste_buffer parameter to send_keys; when False, uses send-keys instead of paste-buffer for user input - devin_cli: Refactor to align with cao-poc implementation; add use_paste_buffer_for_input=False; improve _clean() with OSC pattern removal; add allowed_tools security constraint support; fix get_status() signature to match base class (buffer parameter) - constants: Move FIFO_DIR to /tmp to avoid WSL2 Windows mount limitations (already documented, now enforced) - tests: Update devin_cli unit tests for new signature; add @pytest.mark.asyncio to async test; fix error message assertions; update API test provider count from 11 to 12; enable E2E tests (removed WSL skip decorator) Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fc6df1c to
2dd285e
Compare
|
CodeAnt AI is running Incremental review |
codeant-ai
Bot
added
size:XXL
|
CodeAnt AI Incremental review completed. |
|
Hi ThePlenkov, thank you for the updates. It's great to see these fixes addressing the critical issues identified in the review. Since you've confirmed that the |
|
Hi ThePlenkov, thank you for these additional fixes. It's great to see the loop variable issue, the security constraint regarding Bash execution, and the formatting issues addressed. These improvements, combined with your previous fixes, should resolve the critical issues and ensure the |
|
Tip For best results, initiate chat on the files or code changes.
If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! |
|
Tip For best results, initiate chat on the files or code changes.
If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! |
|
Hi ThePlenkov, thank you for the comprehensive updates and for addressing the feedback regarding the loop variable, security constraints, and formatting. It's great to see the |
Configure Web UI Build to use public npmjs.org registry and clear npm cache to avoid JFrog artifacts when offline. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
CodeAnt AI is running Incremental review |
codeant-ai
Bot
added
size:XXL
|
CodeAnt AI Incremental review completed. |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/cli_agent_orchestrator/providers/devin_cli.py (1)
292-298: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winKeep prompt detection horizontal-rule-aware.
Line 297 still matches any line starting with
#when the status bar is present, so Markdown headings can be treated as Devin prompts despite the safer_has_input_prompt()helper above. Use the same prompt-context check here before returningIDLE/COMPLETED.Suggested fix
- # 2. Check for the # prompt with fallback - has_prompt = self._detect_prompt_with_fallback(clean_output) + # 2. Check for the # prompt with horizontal-rule context to avoid + # Markdown heading collisions in agent responses. + has_prompt = self._has_input_prompt(lines)Also applies to: 324-331
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cli_agent_orchestrator/providers/devin_cli.py` around lines 292 - 298, The prompt fallback in _detect_prompt_with_fallback is too permissive and can misread Markdown headings as Devin prompts when the status bar is visible. Update this logic to use the same prompt-context check as _has_input_prompt before treating a trailing “#” as a prompt, and apply the same safeguard in the related IDLE/COMPLETED detection path. Keep the relaxed fallback, but only return true when the hash line is actually in prompt context rather than any heading-like line.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/cli_agent_orchestrator/providers/devin_cli.py`:
- Around line 292-298: The prompt fallback in _detect_prompt_with_fallback is
too permissive and can misread Markdown headings as Devin prompts when the
status bar is visible. Update this logic to use the same prompt-context check as
_has_input_prompt before treating a trailing “#” as a prompt, and apply the same
safeguard in the related IDLE/COMPLETED detection path. Keep the relaxed
fallback, but only return true when the hash line is actually in prompt context
rather than any heading-like line.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 6c9d8881-a985-48a3-9789-06c37c5acb22
📒 Files selected for processing (6)
.github/workflows/ci.ymlsrc/cli_agent_orchestrator/constants.pysrc/cli_agent_orchestrator/providers/devin_cli.pysrc/cli_agent_orchestrator/providers/manager.pyweb/e2e/devin-provider.spec.tsweb/vite.config.ts
✅ Files skipped from review due to trivial changes (1)
- src/cli_agent_orchestrator/constants.py
🚧 Files skipped from review as they are similar to previous changes (3)
- web/vite.config.ts
- .github/workflows/ci.yml
- web/e2e/devin-provider.spec.ts
Remove unnecessary --registry flag and cache clean since npm uses public registry by default. Only need to remove package-lock.json. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Remove rm -f package-lock.json so CI generates fresh lock file with public registry URLs. This fixes SonarCloud warning about unpredictable dependencies. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Local npm install fails with Invalid Version error. Keeping rm -f package-lock.json in workflow for now. Will generate proper lock file when npm is fixed or when online. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Remove rm -f package-lock.json so CI generates fresh lock file with public registry URLs. This will fix SonarCloud warning. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Add package-lock.json generated with public npm registry to fix SonarCloud warning about unpredictable dependencies. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Restore rm -f package-lock.json to workflow as originally designed. Now that package-lock.json is committed with public registry URLs, CI can regenerate it fresh each run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
95ca53c to
325cd33
Compare
|
CodeAnt AI is running Incremental review |
codeant-ai
Bot
added
size:XXL
|
CodeAnt AI Incremental review completed. |
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Reformat 6 files to comply with Black code style requirements: - Split long lines to meet 88 character limit - Remove trailing whitespace - Fix docstring indentation - Remove extra blank lines Fixes CI failure in Code Quality job (PR #23) Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The test_send_input_allows_manual_answer_when_provider_waits_for_user_answer test was failing because it didn't expect the use_paste_buffer parameter that was added to send_keys in the WSL compatibility fix. This commit updates the test to set the use_paste_buffer_for_input attribute on the mock provider and include use_paste_buffer in the assertion. Fixes Python 3.11 unit test failure in PR #23.
Upstream PR: awslabs#26
User description
Add Devin CLI provider implementation with unit tests and registration in all required locations.
What Changed
DevinCliProviderwith prompt/status parsing and/exithandling--config, launchingcao-mcp-serverand passingCAO_TERMINAL_IDagent_profilesystem prompts and softallowed_toolsenforcement via a prepended security promptdevin_cliacross the app:ProviderType, provider manager factory,api/main.pyproviders list,launchworkspace-access list, settings agent dirs, agent profile listing, andtool_mapping(Bash/Read/Write)devin_cliTrue End-to-End Testing with Devcontainer
The Problem
WSL has tmux limitations (
[Errno 95] Operation not supported) that prevent real Devin CLI spawns. Unit and API tests pass, but actual agent spawning cannot be tested in WSL.The Solution: Devcontainer on Host Machine
A
.devcontainer/configuration that:~/.config/devin/How to Use for True E2E Testing
Prerequisites:
~/.config/devin/Steps:
Why This Works:
Alternative: CI Testing
If you don't have Docker on your host machine, the CI pipeline (which runs on real Linux) will perform the true end-to-end testing of Devin CLI spawns.
Web UI E2E Tests - Why They Matter
What They Test
These E2E tests verify the complete user journey through the CAO web interface:
How They Help vs Existing Tests
Existing Unit Tests:
Existing API Tests:
NEW E2E Tests:
Example Issues E2E Tests Catch
Trade-offs
Overall, E2E tests complement unit and API tests by providing confidence that the complete user experience works correctly, not just individual components in isolation.
Bug Fixes
#) as input prompts; terminate on horizontal rules/status bar onlySummary by CodeRabbit
New Features
devin_clias installed when thedevinbinary is available.Bug Fixes
Chores
CodeAnt-AI Description
Add Devin CLI as a supported provider with web UI and status detection
What Changed
Impact
✅ Use Devin CLI from the agent picker✅ Fewer input failures when sending tasks to Devin CLI✅ More reliable agent status updates in WSL💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas of improvement in your code repository, ensuring your repository maintains high standards of code health.