fix: link email-only maintainers on incremental runs (DE-980)

[joanagmaia]

Summary

Email-only maintainers in large MAINTAINERS files — most visibly the Linux kernel repo (torvalds/linux) — were not being linked in maintainersInternal.identityId on incremental runs, even when the corresponding identity already existed in memberIdentities. Concrete example that motivated the ticket: Joshua Crofts <joshua.crofts1@gmail.com> (VEML3328 driver) — his identity was created 2026-05-30, but the kernel repo's incremental sync has been running daily since 2026-01-13 and never linked him.

Measured impact on the kernel repo: only 1980 of 2303 maintainers whose identities already exist in memberIdentities are actually linked in maintainersInternal — i.e. 323 resolvable identities (~14%) are silently dropped on every incremental run.

Root cause

Two independent bugs in the incremental code path. Both were invisible on first-run repos because insert_new_maintainers doesn't share any of this logic.

Bug 1 — compare_and_update_maintainers keyed by github_username

compare_and_update_maintainers builds dicts of current vs. new maintainers using github_username as the key:

current_maintainers_dict = {m[\"github_username\"]: m for m in current_maintainers}
new_maintainers_dict     = {m.github_username:     m for m in maintainers}

The AI extractor frequently emits github_username=\"unknown\" when the source only carries an email — for the kernel MAINTAINERS file that's 3959 of 4216 entries. All 3959 collapse into a single \"unknown\" slot in each dict, so at most one of them survives per run; the other 3958 are silently dropped. (The remaining 257 entries with real github usernames are processed normally.)

Bug 2 — get_maintainers_for_repo filtered out email-linked identities

The current-state query restricted the join with:

mem.platform = 'github' AND mem.type = 'username' AND mem.verified = TRUE

But the write path uses find_maintainer_identity_by_email which matches platform IN ('github','git','gitlab'). The kernel email identities resolve to platform='git' rows, so they were excluded from the comparison set entirely — leading to spurious re-inserts (idempotent via ON CONFLICT, so harmless on its own) and a broken view of "what's currently linked".

The general rule: the read filter must not be stricter than the write filter, or the diff loses sight of rows that were legitimately persisted.

Why incremental-only

insert_new_maintainers was already iterating maintainers directly without keying by username, and used find_github_identity → email fallback per maintainer. So on first run every email-only maintainer whose identity existed at that moment was linked. The bug only manifested on subsequent runs — any identity created after a repo's first run (Joshua's case: identity created 2026-05-30, repo's first run was 2026-01-13) could never be picked up, and any incremental run on a repo with many "unknown" extractions silently collapsed them down to one.

Fix

• Centralised identity resolution — _resolve_identity (github → email fallback) and _resolve_maintainers (concurrent batch resolver capped at MAX_CONCURRENT_CHUNKS). Both first-run (insert_new_maintainers) and incremental (compare_and_update_maintainers) paths share identical lookup semantics.
• Identity-based diff key — compare_and_update_maintainers keys current and new maintainer sets by (identityId, role) instead of github_username. Matches the unique index on maintainersInternal (repoId, identityId, role) and stops \"unknown\" entries from collapsing into one slot.
• get_maintainers_for_repo aligned with the write path — replaced platform='github' AND type='username' AND verified=TRUE with endDate IS NULL. The write path (find_github_identity / find_maintainer_identity_by_email) is the real gatekeeper for what gets persisted. endDate IS NULL restricts the comparison set to active rows so a reappearing maintainer falls into the "new" branch and gets reactivated via upsert_maintainer's ON CONFLICT … SET \"endDate\" = NULL clause (this restores the reactivation behaviour the old platform='github' filter incidentally provided for email-linked maintainers).
• End-date safety guard — under the new identity-based key, a transient resolution failure could wrongly end-date a maintainer still listed in the source. Added a guard that skips end-dating when the row's identifying value still appears in the extracted file. Restores the invariant the old github_username-keyed new_maintainers_dict provided for free (every extracted entry was unconditionally in the "new" set regardless of lookup outcome).
• Concurrent upserts preserved in insert_new_maintainers — restored asyncio.gather + Semaphore(MAX_CONCURRENT_CHUNKS) so first-run processing of large MAINTAINERS files doesn't serialise the thousands of writes.

Expected outcome

After the next scheduled run of the kernel repo, the gap between resolvable identities (memberIdentities rows that match a maintainer entry) and linked rows in maintainersInternal should close — currently 1980/2303, expected to approach 2303/2303 (modulo any new identities created between this PR landing and the next sync).

Test plan

• Local extraction sanity-checked end-to-end against the full kernel MAINTAINERS file (4216 maintainers extracted; Joshua Crofts <joshua.crofts1@gmail.com> correctly returned as github_username=\"unknown\", email=\"joshua.crofts1@gmail.com\").
• Ruff lint + format clean on both modified files.
• After deploy, verify on the next scheduled run of the kernel repo that the 1980/2303 ratio moves toward parity and that Joshua Crofts specifically appears in maintainersInternal.identityId.
• Verify no regressions on a typical github-username-driven repo (the most common case): existing rows remain linked, no spurious end-dates.

JIRA

DE-980

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes maintainer persistence and end-dating logic on recurring syncs; incorrect guards could leave stale rows or skip legitimate end-dates, though the PR adds explicit safety around unresolved identities.

Overview
Fixes incremental maintainer sync so email-only entries (e.g. Linux MAINTAINERS with github_username="unknown") are linked instead of collapsed or ignored.

Shared resolution — _resolve_identity / _resolve_maintainers (GitHub username then email, concurrent) are used by both first-run insert_new_maintainers and incremental compare_and_update_maintainers. find_github_identity now requires verified = TRUE.

Incremental diff — Comparison keys (identityId, role) instead of github_username, so thousands of "unknown" rows no longer overwrite each other. get_maintainers_for_repo returns active rows (endDate IS NULL) with identity value / platform / type (not GitHub-username-only), aligned with how rows are written.

End-dating — Rows missing from the resolved set are end-dated unless a kind-aware guard sees the same GitHub username or email still present in the file but identity lookup failed this run (avoids false removals on transient lookup failure).

Fork filtering — Parent-repo dedup uses new get_github_maintainer_usernames_for_repo because the broadened maintainer query includes email-linked identities.

^{Reviewed by Cursor Bugbot for commit 9c35de2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR fixes incremental maintainer syncing in services/apps/git_integration so email-only maintainers (common in large MAINTAINERS files like the Linux kernel’s) can be linked to existing identities on subsequent runs, instead of being silently skipped due to username-based key collisions and overly strict read-side filtering.

Changes:

• Centralizes identity lookup with a GitHub-username-first, email-fallback resolver and reuses it across first-run and incremental paths.
• Reworks incremental diffing to compare maintainers by (identityId, role) rather than github_username to avoid collapsing "unknown" entries.
• Relaxes the current-state query to include email-linked identities in the read set for incremental comparisons.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
services/apps/git_integration/src/crowdgit/services/maintainer/maintainer_service.py	Adds centralized identity resolution + switches incremental comparison to identity-based keys; refactors first-run insertion to share the same resolution semantics.
services/apps/git_integration/src/crowdgit/database/crud.py	Relaxes get_maintainers_for_repo filtering so incremental reads can “see” maintainers stored via email-linked identities.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4299076. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.