Repository navigation
Closed
Description
Summary
Reviewed all GitHub Actions workflow runs in laf-us/idaho-vault for the 24h window 2026-07-01T12:00Z – 2026-07-02T12:01Z. Nothing blocks main or deployments. A handful of distinct issues are worth a look; everything else is routine automation noise from a rapid-push branch.
5W Table
| Who | Bots driving the noise: obsidian-plugin[git] (Obsidian auto-sync), github-actions[bot] (daily rollover), Dependabot, various Claude/Codex sessions |
| What | 3 real config/content failures, 2 infra-level 0-job dispatch failures, 1 stalled-approval gate, 2 expected/transient cancellations |
| When | Concentrated in bursts ~06:52–07:56Z and ~11:59Z UTC on 2026-07-02 |
| Where | Branch logan/obsidian (→ PR #563) and branch automation/daily-rollover-2026-07-02 (→ PR #730) |
| Why | See categorized findings below |
| How | Retry / fix / investigate per item below |
Findings
| # | Failure | Category | Root cause | Blocks | Next step |
|---|---|---|---|---|---|
| 1 | NETWEB Path Portability Check — failure |
Content | Diff has vault-folder names with trailing period/space, and a 220-char filename (2026-06-30 - Howard Lutnick on X_...md, max 218) |
PR #563 only | Rename/shorten offending files (Logan already flagged "don't triage today") |
| 2 | Version Transition Policy — failure |
Configuration | check_version_transitions.py hit its hard per-PR transition cap because PR #563's accumulated diff is large from repeated pushes |
PR #563 only | Rebase/squash #563, or raise the script's cap |
| 3 | Sync Plugin Registry — failure |
Configuration | sync_obsidian_plugin_registry.py --check found drift between manifest.json/swarm.json and the actual plugin registry |
logan/obsidian | Run --write and commit — low severity, easy fix |
| 4 | Dependabot Rhythm — startup_failure |
Infrastructure | Confirmed 0 jobs ever created for these runs — true dispatch-level rejection, not a YAML/job bug. Likely Actions concurrency/queue throttling given 35,498 total historical runs | None directly | Confirm Dependabot auto-merge isn't being silently starved during high-volume bursts |
| 5 | opencode — startup_failure |
Infrastructure/Config | Same 0-job pattern as #4. Workflow triggers on any comment containing /oc//opencode — broad substring match, so unrelated bot comments incidentally trigger it |
None directly | Verify the pinned third-party action SHA is still valid; tighten the comment-trigger condition |
| 6 | CodeQL Advanced / Auto-merge on PR events — cancelled |
Transient (expected) | Each new push to PR #563 cancels the in-flight run for the prior commit via default concurrency behavior | None | No action — working as intended |
| 7 | 8-check action_required batch (Secret Pattern Policy, Daily Notes Placeholder, CodeQL, check-notebooks-paired, Cross-Platform Smoke, NETWEB, Large File Policy, Check Dotfolder Anchors) |
Configuration | All 8 checks land on action_required at the identical timestamp/commit — one systemic approval gate, not 8 incidents. Actor is github-actions[bot], so this isn't the standard fork/first-time-contributor gate |
PR #730 (daily rollover) — recurring daily automation | Check Settings → Actions → General → approval/fork policy; this could silently stall the daily rollover every day |
Big IFs (Insights & Findings)
- Rapid-push storm on
logan/obsidian: 38 commits in 24h, 23 from the Obsidian git-sync plugin auto-committing as Logan edits notes locally. Each push re-triggers the full ~8–10 workflow CI matrix, so the apparent "dozens of failures" are really 3 distinct issues repeating dozens of times. Worth considering debouncing heavy checks on that branch to cut noise and Actions usage. - Item ✨ Set up Copilot instructions #7 recurs daily — since it's gating the daily-rollover automation branch specifically, it's the one most likely to cause a real, silent, recurring problem if left unaddressed.
Filed by a scheduled Claude Code review routine.
Claude-Session: https://claude.ai/code/session_01YUWWMU6Zkhrn3kw8FoDukd
Reacted by gitginie
Metadata
Metadata
Assignees
Labels
No labels
Activity
loganfinney27 commentedon Jul 2, 2026
HYPOTHESIS — merge-queue
CI_TIMEOUTbouncing: bot-actored merge groups never receive Actions dispatch; the 16:59Z required-checks re-add exposed itStatus: hypothesis, not confirmed. One decisive cell of the evidence grid is still unmeasured (see § Discriminating experiment). Everything below is drawn from the public REST API (
actions/runs?event=merge_group,issues/{n}/timeline,pulls,commits/{sha}/check-runs) and live webhook events observed in this session, 2026-07-01→02.Claim
Merge-group workflow runs are not dispatched when the queue entry was added by
github-actions[bot], and this has been true at least since 2026-06-30 — probably much longer. It is consistent with the repo's Actions workflow-approval gate, which has been holding bot-actored runs since 2026-04-17 (earliest gated run on record) and currently holds 743 runs inaction_required, every sampled one bot-actored (github-actions[bot],Copilot). This is the same gate as findings 4, 5, and 7 above.While
required_status_checkswas absent from the Main Ruleset (removed 2026-06-19,172d152f), the gap was invisible: bot-queued entries merged CI-less in seconds. Re-adding required checks at 2026-07-01T16:59:40Z made merge-group CI mandatory — from that moment, bot-queued entries starve for 30 minutes and bounce withCI_TIMEOUT, while human-queued entries dispatch and merge normally.Chronology (UTC — every timestamp from API records)
action_required(approval-gated) run on record; the gate on bot-actored runs is old, not newactions/runs?status=action_required, page 743/743required_status_checksremoved from Main Ruleset ("de-require the agent-set status checks")172d152factions/runs?event=merge_group&created=2026-06-30— 49 runs, all in this batchgithub-actions[bot], merged 17–20 s later — zero merge-group workflow runs exist for these groupsadded_to_merge_queueactor) + the June 30 dispatch record above containing only the 05:26 batchissues/475,492/timelineCodeQL Advancedfails on groups pr-715 (14:44), pr-657 (16:01), pr-716 (16:09) — and they merge anyway (14:45:04, 16:02:16, 16:09:38): checks weren't requiredactions/runs?event=merge_group&created=2026-07-01; merged timestamps frompullsrequired_status_checksre-added (Analyze (actions),Analyze (javascript-typescript),Analyze (python); strict)updated_at; rules read viarules/branches/mainadded_to_merge_queueby loganfinney27 at 23:01:46/23:01:59)actions/runs?event=merge_group, re-checked repeatedly through 2026-07-02github-actions[bot]; group created; zero runs; dequeued 23:39:37CI_TIMEOUTCI_TIMEOUT, incl. groups built directly on current main. Integration checks (GitGuardian/Aikido/mergefreeze) DO attach to group commits — Actions alone is absentcommits/e8d6ab4c…/check-runs(3 app checks, 0 Actions)gh-readonly-queue/main/pr-475-*,pr-492-*persist hours after the 17:06 removalsls-remote; post-toggle run listingsThe evidence grid
github-actions[bot]CI_TIMEOUT(#665 onward)* actor of the 05:26 batch enqueues not yet pulled from timelines — worth checking; if bot-queued, it weakens the hypothesis and dates the dispatch loss to June 30 daytime instead.
Discriminating experiment (one click)
Manually add #727 to the queue (human actor). Prediction if hypothesis is true: its group receives workflow runs within ~2 minutes and merges. If it starves even when human-queued, the actor-scoping is falsified and the remaining suspect is whatever changed between the 23:02 dispatch and the 23:09 silence (post-#722/#724 main tree, or platform-side).
If confirmed — the fix surface
Settings → Actions → General: the workflow-approval policy holding bot-actored runs (also the fix for findings 4/5/7 and the 743-run backlog). Relaxing it makes the wired auto-merge loop complete queue merges with CI for the first time; keeping it means the queue is deliberately a human-queued instrument and the arming automation should probably say so. Related but distinct, already tracked: Dependabot-PRs-in-queue, K5/#626 merge-strategy norms.
Correction on the record: an earlier working theory in this session ("the wired loop never completed a queue merge on its own") was false — #708/#709/#711/#713 disprove it. What the loop never did, per current evidence, is complete a queue merge with CI on the group.
Filed by Claude Code (Fable) at Logan's direction; hypothesis label per his call.
Claude-Session: https://claude.ai/code/session_01SfreowpdMionR3SiGEjRBw
Generated by Claude Code
loganfinney27 commentedon Jul 2, 2026
Addendum — the flagged unknown (*) is resolved, in favor of the hypothesis.
The 2026-06-30T05:26Z batch — the only June 30 merge groups that received dispatch — was queued by loganfinney27 (timeline
added_to_merge_queue: #680 05:25:50, #684 05:26:04, #696 05:26:37, #672 05:27:42). The same-day evening zero-dispatch merges (#708/#709/#711/#713) were all queued bygithub-actions[bot].The grid is now unbroken across the full examined record (2026-06-30 → 2026-07-02): human-queued → dispatch, every observed case; bot-queued → no dispatch, every observed case. Still a hypothesis until the live experiment runs (manually queue #727 and watch for group runs), but there is no longer any counterexample in the historical data.
Claude-Session: https://claude.ai/code/session_01SfreowpdMionR3SiGEjRBw
Generated by Claude Code
loganfinney27 commentedon Jul 2, 2026
Hypothesis CONFIRMED — 2026-07-02 live A/B test
The 2026-07-02 hypothesis above ("merge-group workflow runs are only dispatched when the queue entry was enqueued by a human actor; bot-actored enqueues receive zero Actions dispatch") was tested twice today and confirmed both times. Chronology (all times UTC):
PR #732 (three enqueues, same PR, same day)
PR #727 (the designated experiment)
check-secret-patternspassed; PR merged 18:43:09Six bot-actored enqueues across two PRs: zero Actions dispatch, every time. Two human-actored enqueues: dispatch within ~20 seconds, every time. No counterexamples. The queue itself is healthy — the blocker is upstream of it, at Actions workflow-run approval for bot-actored events (same gate as findings 4/5/7; 743
action_requiredruns since 2026-04-17).Two clarifications for the record
The CodeQL Advanced "failure" on today's group runs is an artifact, not a cause. On both the pr-732 group (14:57) and the pr-727 group (18:42), the Analyze jobs failed at the SARIF upload step with
ref 'refs/heads/gh-readonly-queue/main/pr-…' not found— the queue branch had already been deleted (pr-727: because the PR had already merged at 18:43:09; pr-732: because the entry had already been dequeued). Analysis itself completed. This is very likely the same mechanism behind the "CodeQL failing on groups that merged anyway" observations from 2026-07-01 (pr-715/657/716). No smoke here, only a check that outlives its ref.Ruleset state observed live at time of test (Main Ruleset 16864823, updated 2026-07-02T07:59:11Z): required status check is now the single context
check-secret-patterns; the threeAnalyze (*)contexts are no longer required; merge_method=MERGE, grouping=ALLGREEN, check_response_timeout=45 min; thecode_scanningrule (CodeQL, high_or_higher) is still active.#727merging in 66 s is consistent with this: its only required check completed in ~30 s.Remaining fix
Until the Actions approval policy (Settings → Actions → General) is relaxed for bot-actored runs, every auto-enqueued PR will starve for 45 minutes and bounce with CI_TIMEOUT. Human-clicked "Merge when ready" from a user session dispatches fine. Currently affected open PRs: #444, #726 (auto-merge armed → will bot-enqueue and starve), #728 (needs any enqueue).
One forward-looking note: the
code_scanningrule still requires CodeQL results while #726 deletes the CodeQL workflow (per direction, matrix untenable for this repo). After #726 merges, either remove the code_scanning rule or switch the repo to CodeQL default setup in Settings → Code security, otherwise future PRs may wait on analyses that no workflow produces.Claude-Session: https://claude.ai/code/session_01SfreowpdMionR3SiGEjRBw
Generated by Claude Code
loganfinney27 commentedon Jul 2, 2026
Addendum 2 — the starvation is contagious under ALLGREEN chaining
Tonight's #728 ride documents a second-order effect of the actor-starvation bug: a human-queued, fully-green entry can still be wedged by bot-queued entries ahead of it. Chronology (UTC):
11e3b63a= #723,321e03c6= #665) but — bot-actored — receive zero workflow dispatchpr-728-321e03c6…), i.e. behind the two starving entriesBRANCH_PROTECTIONS— 3 s after the ruleset edit; this is GitHub revalidating queue entries on a protection change, not a defect in the PRPractical consequence until the Actions approval policy is fixed: queue order matters. A manual enqueue only works if no bot-actored entries sit ahead of it — dequeue (or manually re-queue) the Dependabot/backlog entries first, then queue the PR you want to land. Also note the arm automation re-enqueues the whole armed backlog after every merge to main, so the starving-entries-ahead condition recreates itself each time something lands.
#728 is currently out of the queue, green, and needs one more manual enqueue with a clear runway. Same for #726 and #444 when their turn comes.
Claude-Session: https://claude.ai/code/session_01SfreowpdMionR3SiGEjRBw
Generated by Claude Code
loganfinney27 commentedon Jul 3, 2026
Correction — the "Actions approval policy" fix I prescribed does not exist
Retraction, per the Repair axis: my earlier comments here told Logan to relax an approval policy under Settings → Actions → General for bot-actored runs. No such control exists there — the approval settings on that page govern fork pull requests, not same-repo bot-actored events. That prescription was pattern-matched from training data, not verified against the actual UI; Logan caught it.
The grounded mechanism (documented, matches every observation)
GitHub docs, Triggering a workflow: events performed with the repository's
GITHUB_TOKENdo not create new workflow runs — a deliberate anti-recursion rule, not configurable anywhere. The vault's arm/enqueue automation authenticates withGITHUB_TOKEN, so its enqueues aregithub-actions[bot]-actored and the resultingmerge_groupevents dispatch nothing. Human-actored enqueues (UI click, or API calls under a user PAT — proven today by session-driven enqueues that merged #444/#726/#728) dispatch normally. Same starvation table as before, now with the right cause attached. (The 743action_requiredruns I cited earlier are a separate phenomenon — largely Copilot-actored PR runs — and I conflated them; they were never the queue mechanism.)The fix already designed into the workflows
Both enqueue paths anticipate this exactly:
auto-merge-engage.yml:102—GH_TOKEN: ${{ secrets.MERGE_QUEUE_TOKEN || secrets.GITHUB_TOKEN }}auto-merge-enqueue-on-checks.yml:70— same pattern, with the comment "when present so workflow-touching PRs can enqueue; falls back to GITHUB_TOKEN"MERGE_QUEUE_TOKENis simply unset, so the fallback runs everything. It is also absent from.op/secrets.template.md's inventory. The real one-time fix:.opconvention and add it as theMERGE_QUEUE_TOKENrepository secret (Settings → Secrets and variables → Actions — a menu that does exist)..op/secrets.template.mdwith a rotation schedule.From then on the automation's enqueues are actored by the PAT's user,
merge_groupdispatch fires, and the wired loop completes without any human click — the outcome the phantom setting was supposed to buy.Claude-Session: https://claude.ai/code/session_01SfreowpdMionR3SiGEjRBw
Generated by Claude Code
loganfinney27 commentedon Jul 3, 2026
Gate opened — fully autonomous arm→enqueue→dispatch confirmed, zero manual action
Logan provisioned
MERGE_QUEUE_TOKEN(fine-grained PAT, orgLAF-US, Contents RW + Pull requests RW) after #745 merged the code-side fix (all six arm/enqueue steps now prefer it overGITHUB_TOKEN). Live discriminating test, #747 — opened purely as a content PR, auto-merge deliberately never touched by hand:auto-merge-engage.ymlrun completespull_request_target: openedauto_merge_enabledEnqueue on checks completerun completesauto-merge-enqueue-on-checks.ymladded_to_merge_queueNo human click, no API call from this session. This is the first fully autonomous arm→enqueue→dispatch cycle observed in this saga — the exact loop
auto-merge-engage.ymlandauto-merge-enqueue-on-checks.ymlwere built to run, closed by giving both a credential whose events actually dispatch.Status: resolved. The remaining items on this record (Actions-approval-menu retraction, credential-attribution correction) stand as-is as the accurate history of how the diagnosis evolved.
Claude-Session: https://claude.ai/code/session_01SfreowpdMionR3SiGEjRBw
Generated by Claude Code
Extend MERGE_QUEUE_TOKEN to the two engage-outdated call sites missed…
loganfinney27 commentedon Jul 26, 2026
CI Health Report — 2026-07-26 (24h review, appended to this open tracking issue rather than opening a new one)
Window: 2026-07-25T12:00Z → 2026-07-26T08:52Z (repo's most recent run at check time). Coverage: full-window history was pulled per-workflow for 13 workflows most likely to carry a real signal (opencode, dependabot-rhythm, auto-merge-rhythm, auto-merge-engage, python-test-suite, codacy, cross-platform-smoke, CodeQL, review-feedback-loop, plus the dormant ones below); the other ~50 of 63 workflows were checked only via a brief recent-activity sample (clean in that slice, not exhaustively paged for the full 24h). Global pagination returns 30 runs/page regardless of
per_page, so exhaustively paging the unfiltered repo-wide feed for 24h (~140+ calls) wasn't attempted.5W Summary
opencodebot lane,Codacy Security Scan,Auto-merge on PR events(auto-merge-engage.yml); human push from loganfinney27 on PR #862main. Two non-blocking anomalies on feature/agent branches: (1)opencodereturnedaction_requiredon 29 zero-job runs, almost all on PR #562's branch; (2)auto-merge-engage.ymlreturnedfailureon 2 zero-job runs on PR #862's branchcodex/python-automation-hardening-v2; (2) 2026-07-25T12:17–12:19Z onclaude/practical-cerf-ueandqmain— neither blocks a merge or deploymenttotal_jobs: 0— the run was marked complete before any job started, soget_job_logsreturns nothing ("No failed jobs found"). No log text exists to quote for either pattern; root cause is not confirmed from available dataFindings
opencode— action_required, 29× zero-job runs, Configuration category. Triggered bypull_request_review_commentevents (mostly bot-authored, e.g. Copilot) against PR Harden Python automation gates #562. This is the same workflow family as ci-audit: opencode startup_failure persists after v1.17.8 upgrade; Dependabot Rhythm blocking PR #589 — 2026-06-20 #595 ("opencode startup_failure persists"), but the actual conclusion observed here isaction_required, notstartup_failure— flagging as the same recurring symptom family, not a confirmed identical repro. Likely an approval-gate on this trigger/actor combination that never reaches job execution. Next step: check repo Actions settings for required-approval-on-fork/bot-triggered-runs.auto-merge-engage.yml— failure, 2× zero-job runs, category inconclusive. On PR audit(ci): 2026-07-23 CI failure sweep + fix check-notebooks-paired jupytext version pin #862, triggered by a real human push (pull_request_target). No job ever ran, so there's no log to diagnose from via the API — this is consistent with the K0–K5 auto-merge tangle (K0 — Auto-merge enforcement tangle — untangle before the four-tier grid lands #626–K5 — merge-strategy mismatch (--squash vs --merge) #631) as a matching workflow family, but not verified as the same mechanism. Worth escalating since afailureconclusion with zero retrievable diagnostic content is itself a gap in the automation's observability.Codacy Security Scan— cancelled ×4 oncodex/python-automation-hardening-v2, all within an 11-minute span while that branch was receiving rapid pushes — routine superseded-by-newer-push, not a real failure.No other workflow in the checked set (
dependabot-rhythm,auto-merge-rhythm,python-test-suite,cross-platform-smoke,CodeQL,review-feedback-loop) showed any non-success/non-skip conclusion in this window.Big IF
The recurring theme across this issue's history (and its Linear mirrors LAF-53 through LAF-71) is
opencode's chronic action-required/startup-failure state — this is at least the 6th report to note some variant of it since 2026-06-20, and it still isn't fixed. Per this routine's instructions, I'm not filing another tracking issue for it; if someone with repo-admin access to theopencodeGitHub App installation is reading this, the actual fix is probably in that App's permission/approval config, not in workflow YAML — every report so far has stopped at "investigate," not gotten a hand on the actual App settings.Cross-posted: Slack breadcrumb sent to #all-logan-finney. Linear mirrors from GitHub per the DOCKET posture doctrine, so no separate Linear issue filed. Discord (via Zapier) is connected and reachable, but
find_channelrequires an exact name match and none of the plausible guesses tried (general, vault, idaho-vault, ci, announcements, bots, logan-finney, docket, the-world-is-quiet-here) resolved to a real channel — no channel directory tool is available to look up the actual name, so this didn't get a Discord post. Flagging that gap rather than silently skipping it.Claude-Session: https://claude.ai/code/session_01A5m94VGH6eYoLMwxiunPeQ
Generated by Claude Code
Fix two chronic CI issues flagged in every report since 06-22 (#737)
Add GARTH-NIX-SUPERIOR-SATURDAY-TOWER.md - excerpt-anchored research …
Prefer MERGE_QUEUE_TOKEN on every auto-merge arm step; inventory the …
MERGE_QUEUE_TOKEN extension; version-drift backstops; ledger deletion…
Fix two chronic CI issues flagged in every report since 06-22 (#737)
Add GARTH-NIX-SUPERIOR-SATURDAY-TOWER.md - excerpt-anchored research …
Prefer MERGE_QUEUE_TOKEN on every auto-merge arm step; inventory the …
MERGE_QUEUE_TOKEN extension; version-drift backstops; ledger deletion…
Fix two chronic CI issues flagged in every report since 06-22 (#737)
Add GARTH-NIX-SUPERIOR-SATURDAY-TOWER.md - excerpt-anchored research …
Prefer MERGE_QUEUE_TOKEN on every auto-merge arm step; inventory the …
MERGE_QUEUE_TOKEN extension; version-drift backstops; ledger deletion…
Fix two chronic CI issues flagged in every report since 06-22 (#737)
Add GARTH-NIX-SUPERIOR-SATURDAY-TOWER.md - excerpt-anchored research …
Prefer MERGE_QUEUE_TOKEN on every auto-merge arm step; inventory the …
MERGE_QUEUE_TOKEN extension; version-drift backstops; ledger deletion…
gitginie commentedon Aug 26, 2026
@loganfinney27!
Thank you for your contribution to this repository! We appreciate your effort in closing issue.
Happy coding!