RebindGoMethodReceivers hangs for 2+ hours on the gortex repo itself (holds global writeMu the whole time)

[timkjr]

Summary

Store.RebindGoMethodReceivers("") (internal/graph/store_sqlite/method_receiver_rebind.go:85), the SQLite-native fast path for the resolver's rebind_go_method_receivers attribution sub-pass, hung for 2+ hours reindexing the gortex repository itself, and never completed before I gave up and restarted the daemon.

Evidence (confirmed from logs)

Every other occurrence of this exact sub-pass in the same daemon's log history completed in well under a second:

{"ts":..., "msg":"resolver: attribution sub-pass starting","pass":"rebind_go_method_receivers"}
{"ts":..., "msg":"resolver: attribution sub-passes","rebind_go_method_receivers":0.000966664, ...}

(4 separate occurrences across different sessions, all sub-second, one example above)

This occurrence, on a fresh gortex track --wait reindex of the gortex repo itself:

09:07:02  "resolver: attribution sub-pass starting","pass":"rebind_go_method_receivers"

— never logged a completion line. From 09:08:48 through 11:33:48 (2h26m), the log shows a wal checkpoint deferred mode=PASSIVE reason=writer_gate message every 5 minutes without interruption, consistent with something continuously holding the store's write role. I restarted the daemon at 11:35:02 without it ever finishing.

Repo being indexed at the time: gortex itself — 1,997 files, 162,498 nodes, 726,118 edges, 6,251 methods (per gortex status / the repo's own generated CLAUDE.md stats). Every prior fast completion was presumably on a smaller/less method-dense repo — I don't have a log line proving which repo those ran against, so I can't rule out repo-size as the sole variable, but the ~10,000x+ slowdown on this specific run stood out enough to investigate.

Hypothesis (not confirmed — I did not attach a debugger or get an EXPLAIN QUERY PLAN before restarting)

RebindGoMethodReceivers holds s.writeMu for its entire duration (method_receiver_rebind.go:86-87), across:

1. The candidate-collection query — a 3-way JOIN (edges × nodes × nodes, using nodes_go_receiver_type and edges_by_kind indexes) with GROUP BY e.id HAVING COUNT(*) = 1 AND MIN(c.id) <> e.to_id (goMethodReceiverCandidatesGlobalSQL, lines 20-40).
2. A 3-statement mutating transaction (conflict-delete, dedup-delete, update).

On a package/dir layout with many same-named Go types clustering under nodes_go_receiver_type (repo_prefix, file_dir, name) — plausible in a codebase this large with common receiver/type names repeated across many packages — the pre-GROUP BY intermediate row set from the join could blow up combinatorially before collapsing, turning what's normally a cheap indexed lookup into a very expensive scan. I have not verified this against EXPLAIN QUERY PLAN — it's the most plausible mechanism I could identify from reading the query, not a confirmed root cause.

Whatever the actual query-plan behavior, holding the single global writeMu for the full duration means every other write-dependent operation (and, per the daemon's own log, hits/subsequent analysis passes queued behind the same store) stalls right along with it — worth decoupling the long-running candidate SELECT from the writer lock if it can safely run under a read/shared-read scope before the short mutating transaction.

Environment

• gortex v0.63.8, linux/amd64
• Repo under index: gortex's own source tree (self-hosted/dogfooding case)
• Store: SQLite backend, default config

What I did

Restarted the daemon rather than continue waiting or attempt to intervene in the live process (didn't want to risk corrupting a stuck write transaction). No repro script yet — this was observed once, on this specific reindex. Happy to help gather EXPLAIN QUERY PLAN output or attach pprof if it recurs and someone wants a live capture before restart.

[zzet]

Hey @timkjr, thanks for the detailed report.

I ran a cold index of 20+ repositories, including the Gortex repository, which took about 15 minutes. Still, after reviewing the relevant code and logs, I don’t consider that an equivalent reproduction of your environment.

What the logs confirm:

• rebind_go_method_receivers started and did not return for approximately 2h28m.
• The SQLite writer gate was unavailable at each five-minute checkpoint attempt.
• The checkpoint process was only observing the blockage; it was not causing it.

However, reason=writer_gate does not prove that receiver rebinding owned the gate. It could have been waiting to acquire it behind another writer. If it had acquired the gate, it could have been blocked while obtaining the writer connection, populating the temporary candidate table, running the update transaction, committing, or cleaning up. These operations currently use context.Background(), so they have no resolver-level timeout.

The strongest code-level hypothesis currently concerns SQLite planner statistics. The v12 migration recreates the nodes_go_receiver_type index without running ANALYZE. The open-time statistics healer can then exit after checking two unrelated indexes and does not verify statistics for nodes_go_receiver_type or edges_by_kind.

This could leave an upgraded database without useful receiver-index statistics. INDEXED BY forces index selection but does not force join order, so SQLite could still choose a pathological loop order that repeatedly searches the member_of edge range and builds a temporary GROUP BY tree.

The current query-plan test would not detect every such plan: it checks that the expected indexes appear and that edges is not fully scanned, but it does not require edge-first ordering, forbid a method-node scan, or forbid a temporary grouping tree.

The proposed common-name Cartesian explosion is less likely in the intended plan because receiver candidates are matched by repository, directory, and name. Common type names in different packages should not multiply each other. A planner-selected bad join order remains plausible, though.

If this happens again, the most useful evidence would be:

• Preserve or copy the database before reindexing or running ANALYZE. Please include the -wal and -shm files if copying it while relevant.
• Exact daemon version/commit and the preceding shadow-swap decision log.
• Database user_version, all tracked repositories, store-wide node/edge counts, and DB/WAL sizes.
• A goroutine or pprof dump during the stall. This will distinguish writer-gate wait, connection-pool wait, and active SQLite execution.
• From an unmodified database copy:

SELECT sqlite_version();

SELECT idx, stat
FROM sqlite_stat1
WHERE idx IN ('edges_by_kind', 'nodes_go_receiver_type');

An EXPLAIN QUERY PLAN of the exact v0.63.8 candidate query before and after ANALYZE would be especially useful.

I’m treating this as a valid unresolved issue rather than ordinary machine slowness. The leading hypothesis is a query-plan problem tied to an upgraded/shared database, with contention on another store-global writer as the main alternative. I’ll also add finer-grained timing around writer-gate acquisition, connection acquisition, candidate collection, transaction statements, and cleanup so a recurrence identifies the exact blocking phase.