Add Devin CLI provider by ThePlenkov · Pull Request #336 · awslabs/cli-agent-orchestrator

ThePlenkov · 2026-06-28T15:27:04Z

Add Devin CLI provider implementation with unit tests and registration in all required locations.

What Changed

Implemented DevinCliProvider with prompt/status parsing and /exit handling
MCP integration via temp --config, launching cao-mcp-server and passing CAO_TERMINAL_ID
Supports agent_profile system prompts and soft allowed_tools enforcement via a prepended security prompt
Registered devin_cli across the app: ProviderType, provider manager factory, api/main.py providers list, launch workspace-access list, settings agent dirs, agent profile listing, and tool_mapping (Bash/Read/Write)
Added comprehensive unit tests and fixtures; updated API test to include devin_cli
NEW: Added Playwright E2E tests for web UI integration

Web UI E2E Tests - Why They Matter

What They Test

These E2E tests verify the complete user journey through the CAO web interface:

Web interface loads and renders correctly
Devin CLI appears in the providers list (both API and UI)
Spawn Agent modal opens and shows Devin CLI as an option
Agent profiles are available for Devin CLI selection
Provider registration works end-to-end

How They Help vs Existing Tests

Existing Unit Tests:

✅ Test individual functions in isolation
✅ Fast and focused
❌ Don't test integration between components
❌ Don't catch UI rendering issues
❌ Don't verify user interaction flows

Existing API Tests:

✅ Test HTTP endpoints
✅ Verify backend logic
❌ Don't test frontend rendering
❌ Don't catch JavaScript errors
❌ Don't verify UI state management

NEW E2E Tests:

✅ Test the full stack: frontend → API → backend
✅ Verify UI rendering and user interactions
✅ Catch integration issues between React components and backend
✅ Provide confidence that web features work for real users
✅ Prevent regressions in web UI functionality

Example Issues E2E Tests Catch

Frontend shows wrong provider list (API works, UI broken)
Modal doesn't open due to JavaScript error
Provider dropdown missing options (CSS/React state issue)
Button clicks don't trigger API calls (event handler bug)

Trade-offs

Pros: Catches integration bugs, validates user experience, prevents UI regressions
Cons: Slower than unit tests, can be flaky in CI, requires browser setup

Overall, E2E tests complement unit and API tests by providing confidence that the complete user experience works correctly, not just individual components in isolation.

ThePlenkov · 2026-06-28T15:40:39Z

Replacing with cleaner version without unnecessary CI/quality fixes

codecov-commenter · 2026-06-29T01:12:58Z

Codecov Report

❌ Patch coverage is 78.41191% with 87 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@dcdc04c). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/cli_agent_orchestrator/providers/devin_cli.py	79.43%	44 Missing ⚠️
src/cli_agent_orchestrator/constants.py	56.25%	21 Missing ⚠️
src/cli_agent_orchestrator/services/fifo_reader.py	30.00%	7 Missing ⚠️
...c/cli_agent_orchestrator/backends/herdr_backend.py	58.33%	5 Missing ⚠️
.../cli_agent_orchestrator/services/status_monitor.py	92.06%	5 Missing ⚠️
src/cli_agent_orchestrator/providers/manager.py	93.54%	2 Missing ⚠️
src/cli_agent_orchestrator/clients/tmux.py	92.85%	1 Missing ⚠️
src/cli_agent_orchestrator/providers/base.py	66.66%	1 Missing ⚠️
...li_agent_orchestrator/services/terminal_service.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #336   +/-   ##
=======================================
  Coverage        ?   88.08%           
=======================================
  Files           ?      135           
  Lines           ?    16477           
  Branches        ?        0           
=======================================
  Hits            ?    14513           
  Misses          ?     1964           
  Partials        ?        0

Flag	Coverage Δ
unittests	`88.08% <78.41%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR adds a new devin_cli provider to the CLI Agent Orchestrator, wiring it through provider registration points, tool mapping, agent profile directories, and adding both unit- and end-to-end tests (including new Playwright UI E2E coverage for the web frontend).

Changes:

Implemented DevinCliProvider (prompt/status parsing, /exit, MCP config injection) and registered devin_cli across backend/API/CLI.
Extended input delivery to support providers that can’t use tmux paste-buffer for interactive input (use_paste_buffer / use_paste_buffer_for_input).
Added test coverage: provider unit tests + Python E2E supervisor orchestration + web Playwright E2E scaffolding and scripts.

Reviewed changes

Copilot reviewed 34 out of 35 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
web/vite.config.ts	Ensures Vitest excludes the new Playwright `e2e/` suite.
web/src/components/AgentPanel.tsx	Adds `devin_cli` to UI fallback providers and label mapping.
web/playwright.config.ts	Introduces Playwright configuration for web UI E2E testing.
web/package.json	Adds Playwright dependency and E2E test scripts.
web/package-lock.json	Locks Playwright and related dependency updates.
web/e2e/devin-provider.spec.ts	Adds web UI E2E tests validating `devin_cli` registration and basic UI flows.
test/services/test_terminal_service_full.py	Updates tests to assert the new `use_paste_buffer` parameter behavior.
test/providers/test_devin_cli_unit.py	Adds unit tests for Devin provider status detection, extraction, and registration.
test/providers/fixtures/devin_cli_processing_output.txt	Fixture for Devin “processing” output parsing.
test/providers/fixtures/devin_cli_idle_output.txt	Fixture for Devin “idle” output parsing.
test/providers/fixtures/devin_cli_heading_response.txt	Fixture for markdown-heading response regression coverage.
test/providers/fixtures/devin_cli_error_output.txt	Fixture for Devin error output parsing.
test/providers/fixtures/devin_cli_complex_response.txt	Fixture for multi-line response extraction.
test/providers/fixtures/devin_cli_completed_output.txt	Fixture for completed output parsing.
test/e2e/test_supervisor_orchestration.py	Adds `devin_cli` coverage to the supervisor orchestration E2E suite.
test/e2e/conftest.py	Adds `require_devin` fixture to skip E2E tests when `devin` isn’t installed.
test/backends/test_tmux_backend.py	Updates backend tests for the `use_paste_buffer` parameter.
test/api/test_api_endpoints.py	Updates provider listing API test expectations to include `devin_cli`.
src/cli_agent_orchestrator/utils/tool_mapping.py	Adds tool mapping entries for `devin_cli`.
src/cli_agent_orchestrator/utils/agent_profiles.py	Registers `devin_cli` agent-profile directory key (`devin`).
src/cli_agent_orchestrator/services/terminal_service.py	Adds provider-aware selection of paste-buffer vs send-keys for input.
src/cli_agent_orchestrator/services/status_monitor.py	Adds event-inbox status path and tmux-history fallback logic (WSL-related).
src/cli_agent_orchestrator/services/settings_service.py	Ensures `devin_cli` uses the agent-store path for settings.
src/cli_agent_orchestrator/providers/manager.py	Registers and factory-wires `DevinCliProvider`; refactors provider creation.
src/cli_agent_orchestrator/providers/devin_cli.py	New provider implementation: prompt/status parsing, MCP config merging, tool restriction prompt.
src/cli_agent_orchestrator/providers/base.py	Adds `use_paste_buffer` default property for provider input delivery.
src/cli_agent_orchestrator/models/provider.py	Adds `DEVIN_CLI` enum entry.
src/cli_agent_orchestrator/constants.py	Moves FIFO directory under `/tmp` to avoid WSL FIFO limitations.
src/cli_agent_orchestrator/clients/tmux.py	Adds `use_paste_buffer` option to `send_keys`; hardens `pipe-pane` command quoting.
src/cli_agent_orchestrator/cli/commands/launch.py	Adds `devin_cli` to workspace-access provider allowlist.
src/cli_agent_orchestrator/backends/tmux_backend.py	Plumbs `use_paste_buffer` through tmux backend.
src/cli_agent_orchestrator/backends/herdr_backend.py	Adds `use_paste_buffer` arg (ignored) for backend interface consistency.
src/cli_agent_orchestrator/backends/base.py	Extends backend `send_keys` signature with `use_paste_buffer`.
src/cli_agent_orchestrator/api/main.py	Adds `devin_cli` to the API provider list/binary mapping.
.github/workflows/ci.yml	Removes Node npm cache configuration.

Files not reviewed (1)

web/package-lock.json: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

gutosantos82

Superseded — this review was a duplicate produced by a dashboard tooling bug and referenced an outdated revision. See the current review at HEAD: #336 (review)

The single "activity since review" flag was noisy — it keyed off the PR's updatedAt, which bumps on anything (CI, labels, bot comments), so it fired on Codecov-only activity (awslabs#350/awslabs#351) and conflated "code changed" with "someone commented." Now two precise signals in metadata + dashboard pills: - code_changed (🔁 re-review): a review exists at an older sha but not the current head → the author pushed, so it genuinely needs re-review. - human_activity (💬 discussion since review): the newest NON-BOT comment or review is later than our review file. Bots (codecov/dependabot/ github-actions/copilot) are excluded so their automated comments don't masquerade as human engagement. Verified: awslabs#231/awslabs#336/awslabs#115 flag human_activity (real maintainer comments); awslabs#350/awslabs#351 (Codecov-only) no longer flag; none flag code_changed (no pushes).

The human_activity flag was firing on the dashboard user's OWN comments (awslabs#115/awslabs#231/awslabs#336 were all gutosantos82) — no signal, you know what you said. Now exclude $ME (gh api user login) alongside bots, so the flag means 'someone OTHER than you engaged after the review' (author reply, another maintainer) — the actually-actionable case.

…ocations Agent-Logs-Url: https://github.com/ThePlenkov/cli-agent-orchestrator/sessions/ec6174e8-a19e-406a-87c1-fa0063d109bb Co-authored-by: ThePlenkov <6381507+ThePlenkov@users.noreply.github.com>

Agent-Logs-Url: https://github.com/ThePlenkov/cli-agent-orchestrator/sessions/ec6174e8-a19e-406a-87c1-fa0063d109bb Co-authored-by: ThePlenkov <6381507+ThePlenkov@users.noreply.github.com>

…tools via security prompt Agent-Logs-Url: https://github.com/ThePlenkov/cli-agent-orchestrator/sessions/1a11fb12-9327-45e4-a3bd-6eb94a07f9eb Co-authored-by: ThePlenkov <6381507+ThePlenkov@users.noreply.github.com>

This commit addresses feedback from PR #10 to align with the build-cao-provider skill requirements: **MCP Configuration Enhancement:** - Merge MCP servers from agent profile with user's existing Devin config - Ensure CAO_TERMINAL_ID is set in env for all MCP servers (critical for orchestration) - Preserve user's existing MCP settings while adding cao-mcp-server - Support agent profiles with custom MCP server configurations **E2E Testing Infrastructure:** - Add require_devin fixture to test/e2e/conftest.py - Add TestDevinCliSupervisorOrchestration class with 3 test methods: - test_supervisor_handoff: validates handoff MCP tool delegation - test_supervisor_assign_and_handoff: validates multi-agent workflow - test_supervisor_assign_three_analysts: canonical examples/assign smoke test **Agent Profile Installation:** - Install analysis_supervisor, data_analyst, and report_generator profiles for devin_cli - Enables supervisor orchestration patterns (assign + handoff + send_message) These changes implement critical requirements from the build-cao-provider skill, particularly lesson #1 (CAO_TERMINAL_ID forwarding) and the E2E supervisor orchestration validation pattern. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

- Add devin_cli to FALLBACK_PROVIDERS in AgentPanel.tsx - Add 'Devin' to SOURCE_LABELS - Add Playwright E2E test scripts to package.json Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

These tests verify that Devin CLI is properly integrated with the CAO web interface: - Web interface loads correctly - Devin CLI appears in providers list - Spawn Agent modal shows Devin CLI option - Agent profiles are available for Devin CLI - Provider registration is functional Compared to existing unit/API tests, these E2E tests: - Test the full user journey through the web UI - Catch integration issues between frontend and backend - Verify UI rendering and user interaction flows - Provide confidence that web features work end-to-end Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…paste-buffer control Root cause: The event-driven architecture (commit 75e98ac) changed wait_until_status to rely on FIFO-based status monitoring. In WSL2, tmux pipe-pane cannot write to named pipes, causing the buffer to stay empty and status to remain UNKNOWN forever. cao-poc worked because it used the old API (provider.get_status() with backend.get_history()). Changes: - status_monitor: Add fallback to backend.get_history() for tmux backends when FIFO buffer is empty (e.g., WSL limitation). Reads pane history directly and runs provider detection on it. Caches result to avoid repeated history reads. Provides WSL compatibility without affecting the normal FIFO path. - base.py: Add use_paste_buffer property to allow providers to opt out of paste-buffer (Devin CLI doesn't support it for user input) - tmux_client: Add use_paste_buffer parameter to send_keys; when False, uses send-keys instead of paste-buffer for user input - devin_cli: Refactor to align with cao-poc implementation; add use_paste_buffer_for_input=False; improve _clean() with OSC pattern removal; add allowed_tools security constraint support; fix get_status() signature to match base class (buffer parameter) - constants: Move FIFO_DIR to /tmp to avoid WSL2 Windows mount limitations (already documented, now enforced) - tests: Update devin_cli unit tests for new signature; add @pytest.mark.asyncio to async test; fix error message assertions; update API test provider count from 11 to 12; enable E2E tests (removed WSL skip decorator) Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…ty fixes - Add temp file cleanup in _build_command to prevent leaks on retries - Apply skill prompt via self._apply_skill_prompt in _build_command - Rename loop variable 'l' to 'line' for better readability - Remove Bash from fs_* mapping in tool_mapping.py (security fix) - Apply Black formatting to all modified files Addresses gemini-code-assist and coderabbitai review comments. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

The WSL compatibility fix added use_paste_buffer parameter to send_keys, so update existing tests to expect this parameter. - test/backends/test_tmux_backend.py::test_send_keys_delegates - test/services/test_terminal_service_full.py::test_send_input_success Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Fixes CodeQL security vulnerabilities (3 high-severity alerts): - Replaced tempfile.mktemp() with tempfile.NamedTemporaryFile() in 3 locations - Eliminates race condition vulnerability from deprecated mktemp function - Maintains same functionality with secure tempfile handling Changes: - Line 140-147: Temp prompt file for security constraints - Line 166-173: Temp prompt file for agent profiles - Line 206-213: Temp config file for MCP servers All 27 unit tests pass after the fix.

Reformat 6 files to comply with Black code style requirements: - Split long lines to meet 88 character limit - Remove trailing whitespace - Fix docstring indentation - Remove extra blank lines Fixes CI failure in Code Quality job (PR #23) Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

The test_send_input_allows_manual_answer_when_provider_waits_for_user_answer test was failing because it didn't expect the use_paste_buffer parameter that was added to send_keys in the WSL compatibility fix. This commit updates the test to set the use_paste_buffer_for_input attribute on the mock provider and include use_paste_buffer in the assertion. Fixes Python 3.11 unit test failure in PR #23.

Exclude web/package-lock.json using git skip-worktree to prevent JFrog registry URLs from being committed to the public repository. The file remains locally for development but is not tracked in git. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Pull deterministic GitHub signals into per-PR metadata and show them as dashboard badges so the operator sees the human context before acting: - mergeable state -> '⚠️ conflicts' badge (e.g. awslabs#336 now conflicts with main) - reviewDecision + deduped non-bot reviewers -> '👤 human-approved' / '👤 human: changes requested' / '👤 reviewing: <who>' (e.g. awslabs#451 already human-approved by anilkmr-a2z, awslabs#336 changes requested incl. you) - verdict-vs-human mismatch -> '⚠️ verdict vs human' when our verdict disagrees with the human review decision - latest non-bot comment/review snippet -> a 'last human note' line Adds merge/human/mismatch filter dropdowns. Signals are pulled in build_meta by extending the existing gh call (no extra API round-trips) and are refreshed by --refresh-meta, so they stay accurate without spending review agents.

…text Introduce a sixth static reviewer that owns the human-conversation angle the diff reviewers ignore: it reads the existing human comments/reviews, the GitHub reviewDecision, and the mergeable state, then judges whether each human maintainer condition is UNADDRESSED at the current head and returns a gate: MUST-NOT-APPROVE / HUMAN-ALREADY-APPROVED / CONFLICTS / NO-HUMAN-BLOCKER. Wire it into pr_review_supervisor: fetch reviewDecision+mergeable, fan out to six reviewers (pass the human conversation + decision + mergeable to the new one), and make its gate authoritative in the HARD RULE — never approve over an unaddressed maintainer condition, escalate verdict-vs-human disagreement to needs_human, and flag CONFLICTING branches. Validated on awslabs#336: the conversation reviewer flagged CHANGES_REQUESTED + CONFLICTING + two unaddressed conditions, and the verdict correctly came back 'Request changes' (was 'Approve with nits').