Skip to content

[ci-maintainer] [ci] matrix integration jobs stuck in QUEUED state across PRs #680

Closed

Description

@hanthor-hive-agent

CI Issue

Multiple PRs show the same pattern: after the primary check job completes (compile, clippy, unit tests), matrix integration jobs (journey, app-content, gtk-shell, scroll-proof, drm-smoke, arch-package) get stuck in QUEUED state for 1–6+ minutes instead of proceeding.

Evidence

Root Cause Analysis

Likely one of:

  1. Concurrency group configuration — the CI workflow may have a concurrency group limiting how many matrix shards can run in parallel, blocking all downstream jobs
  2. Runner capacity — insufficient runners for the matrix size (6 integration jobs), causing scheduler backlog
  3. Workflow dispatch throttle — GitHub Actions may be rate-limiting job dispatch on this repository

Recommendation

  1. Check .github/workflows/*.yml for concurrency groups with cancel-in-progress: true on matrix jobs — this can cause unnecessary serialization
  2. If runners are self-hosted, verify fleet size and whether runners are busy waiting for capacity
  3. If using GitHub-hosted runners, confirm matrix job count isn't exceeding available parallelism on the account

This needs a maintainer to inspect workflow configuration (ci-maintainer cannot modify workflows in this token tier).


Filed by ci-maintainer agent (ACMM L4/L5 — hold-gated mode). Needs-decision: are integration jobs intentionally serialized, or is this an unintended queue bottleneck?


🐝 Hive Agent: ci-maintainer | Instance: hive-slim-hare | SHA: 325e7b2

— hive: agent=ci-maintainer backend=pi model=kiro-api-key/claude-haiku-4-5:low pi=1.2.0

Activity

  1. hanthor commented on Oct 16, 2026

    @hanthor
    Member

    Fixed. #681 (15b2b00a, merged 2026-10-15) added workflow concurrency groups that cancel superseded PR runs, and #683 runs only the proof lanes a pull request can affect, which cuts the matrix queued per PR. The queued-matrix runs cited here were on #674 and #675, which have both since merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions