Add openhands-enterprise-troubleshooting skill for OHE self-hosted debugging

README.md

@@ -89,7 +89,7 @@ The JS and Python versions are kept in lock-step by `release-please` and guarded
 ## Extensions Catalog
 
 <!-- BEGIN AUTO-GENERATED CATALOG -->
-This repository contains **2 marketplace(s)** with **64 extensions** (54 skills, 10 plugins).
+This repository contains **2 marketplace(s)** with **65 extensions** (55 skills, 10 plugins).
 
 ### large-codebase
 
@@ -108,7 +108,7 @@ OpenHands skills for interacting, improving, and refactoring large codebases
 
 Official skills and plugins for OpenHands — the open-source AI software engineer.
 
-**60 extensions** (52 skills, 8 plugins)
+**61 extensions** (53 skills, 8 plugins)
 
 | Name | Type | Description | Commands |
 |------|------|-------------|----------|
@@ -152,6 +152,7 @@ Official skills and plugins for OpenHands — the open-source AI software engine
 | openhands | plugin | Unified OpenHands plugin — bundles Cloud CLI, REST API (openhands-api), and Automations (openhands-automation) into a... | `/openhands-cloud` |
 | openhands-api | skill | Use the OpenHands Cloud REST API (V1) and agent-server APIs to create and manage Cloud or local backend conversations... | — |
 | openhands-automation | skill | Create and manage OpenHands automations - scheduled tasks that run in sandboxes. Use the prompt preset to create auto... | `/automation:create` |
+| openhands-enterprise-troubleshooting | skill | Diagnose and resolve common issues on OpenHands Enterprise self-hosted installations. Covers sandbox startup failures... | — |
 | openhands-sdk | skill | Reference skill for the OpenHands Software Agent SDK - build AI agents with custom tools, LLM configuration, conversa... | `/sdk` |
 | pdflatex | skill | Install and use pdflatex to compile LaTeX documents into PDFs on Linux. Use when generating academic papers, research... | — |
 | plain-english-content | skill | Write and edit clear, accessible prose in a plain English content style: active voice, front-loaded content, sentence... | — |

marketplaces/openhands-extensions.json

@@ -141,7 +141,7 @@
     {
       "name": "bitbucket-cloud",
       "source": "./skills/bitbucket-cloud",
-      "description": "Bitbucket Cloud (bitbucket.org) specifics — authenticate with BITBUCKET_TOKEN, use the REST API v2, workspace/repo_slug repositories, and the create_bitbucket_pr tool. Loaded on demand by the bitbucket skill once a Cloud environment is detected.",
+      "description": "Bitbucket Cloud (bitbucket.org) specifics \u2014 authenticate with BITBUCKET_TOKEN, use the REST API v2, workspace/repo_slug repositories, and the create_bitbucket_pr tool. Loaded on demand by the bitbucket skill once a Cloud environment is detected.",
       "category": "code-hosting",
       "keywords": [
         "bitbucket",
@@ -153,7 +153,7 @@
     {
       "name": "bitbucket-data-center",
       "source": "./skills/bitbucket-data-center",
-      "description": "Bitbucket Data Center (self-hosted Bitbucket Server) specifics — authenticate with BITBUCKET_DATA_CENTER_TOKEN, use the REST API 1.0, PROJECT/repo_slug repositories, scm/ git remotes, and the create_bitbucket_data_center_pr tool. Loaded on demand by the bitbucket skill once a Data Center environment is detected.",
+      "description": "Bitbucket Data Center (self-hosted Bitbucket Server) specifics \u2014 authenticate with BITBUCKET_DATA_CENTER_TOKEN, use the REST API 1.0, PROJECT/repo_slug repositories, scm/ git remotes, and the create_bitbucket_data_center_pr tool. Loaded on demand by the bitbucket skill once a Data Center environment is detected.",
       "category": "code-hosting",
       "keywords": [
         "bitbucket",
@@ -770,6 +770,21 @@
         "jira",
         "atlassian"
       ]
+    },
+    {
+      "name": "openhands-enterprise-troubleshooting",
+      "source": "./skills/openhands-enterprise-troubleshooting",
+      "description": "Diagnose and resolve common issues on OpenHands Enterprise self-hosted installations. Covers sandbox startup failures, auth issues, certificate errors, LLM connectivity, Keycloak login, Replicated Admin Console access, upgrade failures, and resource exhaustion. Includes support bundle generation and escalation handoffs.",
+      "category": "integrations",
+      "keywords": [
+        "openhands",
+        "enterprise",
+        "troubleshooting",
+        "debugging",
+        "replicated",
+        "support",
+        "diagnostics"
+      ]
     }
   ]
-}
+}

skills/openhands-enterprise-troubleshooting/README.md

@@ -0,0 +1,61 @@
+# OpenHands Enterprise Troubleshooting
+
+An agent-runnable skill for diagnosing and resolving common issues on **OpenHands Enterprise (OHE)** - self-hosted installations using Replicated on VM-based infrastructure.
+
+## What This Skill Does
+
+### 1. Triage and Diagnosis
+- Detects failure modes from symptoms or log output
+- Checks common problem areas: sandbox startup, auth, certificates, LLM connectivity, Keycloak, Replicated Admin Console, upgrades, resource exhaustion
+- Runs targeted diagnostic commands against the live environment
+
+### 2. Guided Recovery
+- Walks through resolution steps for identified issues
+- Validates each step before proceeding
+- Covers the most common failures seen across real OHE installations
+
+### 3. Support Bundle Generation
+- Guides customers through generating and sending support bundles
+- Parses and summarizes bundle output to highlight likely root cause
+- Reduces back-and-forth with the platform team
+
+### 4. Escalation Handoff
+- Produces a clear summary when issues cannot be resolved
+- Documents what was tried, what logs show, and likely root cause
+- Ready to paste into a support ticket
+
+## Common Issues Covered
+
+- Sandbox fails to start / 120s timeout
+- Git provider auth broken (GitHub App, GitLab token)
+- Certificate errors (self-signed, expired, chain issues)
+- LLM connectivity failures (endpoint unreachable, bad credentials)
+- Keycloak login issues
+- Replicated Admin Console unreachable
+- Upgrade stuck or failed
+- OOM / resource exhaustion on the VM
+
+## Usage
+
+This skill is automatically triggered when users describe OHE issues such as:
+- "OpenHands is not working"
+- "Sandbox failed to start"
+- "Can't access admin console"
+- "Certificate error"
+- "LLM connection failed"
+- "Upgrade failed"
+
+## Files
+
+- `SKILL.md` - Main skill with diagnostic workflow and quick reference
+- `references/diagnostics.md` - Detailed diagnostic commands and log interpretation for each failure mode
+
+## For Contributors
+
+When new failure modes are discovered in the field, update `references/diagnostics.md` with:
+1. New symptoms and error patterns
+2. Diagnostic commands to run
+3. Resolution steps that worked
+4. Log excerpts showing the error
+
+This skill should grow with each support issue resolved.

skills/openhands-enterprise-troubleshooting/SKILL.md

@@ -0,0 +1,217 @@
+---
+name: openhands-enterprise-troubleshooting
+description: This skill should be used when a user reports an issue with OpenHands Enterprise (OHE) on a self-hosted (Replicated VM-based) installation. Use for diagnosing sandbox startup failures, auth issues, certificate errors, LLM connectivity problems, Keycloak login issues, Replicated Admin Console access, upgrade failures, or resource exhaustion. Helps triage symptoms, run diagnostic commands, guide through recovery steps, generate support bundles, and produce escalation handoffs.
+triggers:
+- openhands enterprise
+- OHE troubleshooting
+- openhands not working
+- sandbox failed
+- replicated admin console
+- keycloak login
+- certificate error
+- LLM connectivity
+- upgrade failed
+- support bundle
+- openhands install
+---
+
+# OpenHands Enterprise Troubleshooting
+
+This skill helps diagnose and resolve common issues on OpenHands Enterprise (OHE) self-hosted installations using Replicated. It covers triage, guided recovery, support bundle generation, and escalation handoffs.
+
+## Diagnostic Workflow
+
+When a user reports an OHE issue:
+
+1. **Collect symptoms** - Ask user to describe what they see, error messages, when it started
+2. **Identify failure mode** - Match symptoms to one of the common issues below
+3. **Run targeted diagnostics** - Use commands in `references/diagnostics.md`
+4. **Guide recovery** - Follow resolution steps for the identified issue
+5. **Verify fix** - Confirm the issue is resolved
+6. **Generate handoff** - If unresolved, produce a clear summary for the platform team
+
+## Common Failure Modes
+
+### 1. Sandbox Fails to Start / 120s Timeout
+
+**Symptoms:**
+- Conversation hangs then times out
+- "Sandbox failed to start" error
+- 120-second timeout in logs
+
+**Diagnosis:** Check sandbox service status, podman/docker runtime, resource availability
+
+**Reference:** See `references/diagnostics.md` - Section "Sandbox Startup"
+
+### 2. Git Provider Auth Broken
+
+**Symptoms:**
+- "Authentication failed" for GitHub/GitLab
+- Can't clone or push repos
+- GitHub App shows as disconnected
+
+**Diagnosis:** Check gitProvider secrets in kubernetes, GitHub App installation status
+
+**Reference:** See `references/diagnostics.md` - Section "Git Provider Auth"
+
+### 3. Certificate Errors
+
+**Symptoms:**
+- "certificate expired" or "self-signed certificate" errors
+- TLS handshake failures
+- Browser shows insecure connection warning
+
+**Diagnosis:** Check cert expiry, certificate chain, ingress configuration
+
+**Reference:** See `references/diagnostics.md` - Section "Certificate Issues"
+
+### 4. LLM Connectivity Failures
+
+**Symptoms:**
+- "LLM endpoint unreachable"
+- "Authentication failed" for LLM API
+- Conversations fail to start
+
+**Diagnosis:** Check LLM endpoint URL, API key secrets, network policies
+
+**Reference:** See `references/diagnostics.md` - Section "LLM Connectivity"
+
+### 5. Keycloak Login Issues
+
+**Symptoms:**
+- Can't access admin console
+- Login loop or "invalid credentials"
+- Keycloak pod showing errors
+
+**Diagnosis:** Check Keycloak pod status, database connectivity, realm configuration
+
+**Reference:** See `references/diagnostics.md` - Section "Keycloak"
+
+### 6. Replicated Admin Console Unreachable
+
+**Symptoms:**
+- Can't access admin console URL
+- Connection refused or timeout
+- Browser shows "site cannot be reached"
+
+**Diagnosis:** Check Replicated operator pod, ingress, service endpoints
+
+**Reference:** See `references/diagnostics.md` - Section "Replicated Admin Console"
+
+### 7. Upgrade Stuck or Failed
+
+**Symptoms:**
+- Replicated shows upgrade as "failed"
+- Pods in crash loop after upgrade
+- Migration jobs failing
+
+**Diagnosis:** Check failed job logs, resource availability, pre-flight failures
+
+**Reference:** See `references/diagnostics.md` - Section "Upgrade Issues"
+
+### 8. OOM / Resource Exhaustion
+
+**Symptoms:**
+- Pods being OOMKilled
+- "Too many open files" errors
+- Services becoming unresponsive
+
+**Diagnosis:** Check node resources (memory, disk, file descriptors)
+
+**Reference:** See `references/diagnostics.md` - Section "Resource Exhaustion"
+
+## Diagnostic Commands Quick Reference
+
+Access the VM and run these common commands:
+
+```bash
+# Check overall pod status
+kubectl get pods -n openhands
+
+# View pod logs (replace POD_NAME)
+kubectl logs -n openhands POD_NAME
+kubectl logs -n openhands POD_NAME --previous
+
+# Describe a pod for events
+kubectl describe pod -n openhands POD_NAME
+
+# Check resource usage
+kubectl top nodes
+kubectl top pods -n openhands
+
+# Check certificate expiry
+echo | openssl s_client -connect HOST:443 2>/dev/null | openssl x509 -noout -dates
+
+# Check Replicated operator
+kubectl get pods -n replicated
+kubectl logs -n replicated -l app=replicated-operator
+```
+
+## Support Bundle Generation
+
+When the issue requires deeper investigation, guide the user to generate a support bundle.
+
+### Generating the Support Bundle
+
+1. Access the VM via SSH
+2. Run the Replicated support bundle command:
+
+```bash
+replicated admin support-bundle --kubecontext=KUBE_CONTEXT --namespace=openhands
+```
+
+3. The bundle will be saved locally, then upload/share with the platform team
+
+### Parsing the Support Bundle
+
+After obtaining a support bundle:
+
+1. Extract the archive
+2. Focus on these key files:
+   - `pod-status.json` - Current pod states
+   - `pod-logs/*.log` - Container logs
+   - `events.json` - Kubernetes events
+   - `nodes.json` - Node resource info
+
+3. Look for patterns in `references/diagnostics.md`
+
+## Escalation Handoff Template
+
+When an issue cannot be resolved, produce this summary:
+
+```
+## Issue Summary
+**Problem:** [One-line description]
+**Duration:** [When it started]
+**Impact:** [Who is affected]
+
+## Symptoms Observed
+- [Symptom 1]
+- [Symptom 2]
+
+## Diagnostic Steps Taken
+1. [Step 1]
+2. [Step 2]
+
+## Logs / Evidence
+```
+[Relevant log excerpts]
+```
+
+## Resolution Attempts
+- [Attempt 1] - [Result]
+- [Attempt 2] - [Result]
+
+## Likely Root Cause
+[Analysis]
+```
+
+## Additional Resources
+
+- **Diagnostic Reference:** `references/diagnostics.md` - Detailed commands and log interpretation for each failure mode
+- **Replicated Docs:** https://docs.replicated.com/vendor/support-bundle-generating
+- **OHE Architecture:** Internal docs on OHE components and their relationships
+
+## Maintenance
+
+As new failure modes are discovered in the field, add them to this skill. Update `references/diagnostics.md` with new patterns and resolution steps.