Epic: TableBase Coverage Dashboard & Enrichment Queue — Implementation Plan

[OAGr]

TableBase Coverage Dashboard & Enrichment Queue — Implementation Plan

TL;DR

TableBase completeness data is CLI-only and ephemeral — there is no way to see coverage trends, prioritize enrichment work, or dispatch overnight jobs without re-running the scanner manually each time. This plan adds PG persistence for nightly scan results, a read-only Pattern A dashboard that computes a ranked task queue on-the-fly (using entities joined to wiki_pages for parent-org importance), and a Phase 3 job-dispatch layer. Phase 1 persists scan results + adds a groundskeeper cron task. Phase 2 is the dashboard (coverage bars, trends, task queue ordered by org importance). Phase 3 adds dispatch and completion tracking. Top open question: whether to use the existing jobs queue or a standalone GitHub Actions workflow for overnight enrichment dispatch.

Problem

The TableBase enrichment pipeline (crux tb enrich) has no visibility layer: every time an engineer wants to know what's missing, they must re-run the CLI scanner (~90 seconds), and results are immediately discarded. There is no history of coverage over time, no ranked queue of high-impact tasks, no way to dispatch enrichment jobs from a dashboard, and no mechanism to avoid re-running already-completed tasks without manually checking .tablebase-completed.txt. Current coverage is approximately: personnel 6%, funding_rounds 4%, investments 5%, benchmarks 45%, grants 99%.

Current State

Asset	Status	Gap
crux/tablebase/scanner.ts	CLI only, ephemeral results	Needs --persist flag + wiki-server POST endpoint
crux/tablebase/task-ranker.ts	Impact formula: (100-completeness) × typeWeight × importanceMultiplier	Runs client-side only; importance data is hard-coded, not real page importance
.tablebase-completed.txt	196 completed task IDs, git-committed	Stays authoritative for CLI; PG tracking added in Phase 3
tablebase_scan_results table	Does not exist	New table needed (Phase 1)
tablebase_enrichment_tasks table	Does not exist	New table needed (Phase 3 only)
things table	13 domain types indexed; source_id = record's own ID (not entity stableId)	Cannot be used to JOIN entity importance — use entities table instead
wiki_pages.reader_importance	Exists for entities with wiki pages (orgs, people with wiki pages)	Not set for personnel/grant records directly; must inherit from parent org
tablebase-enrichment job type	NOT registered in groundskeeper or jobs table	Must add handler before any dispatching (Phase 3)
Groundskeeper cron tasks	8 existing tasks; requires config.ts + index.ts changes	No nightly TableBase scan task
Pattern A dashboards	26 exist, all working	None for TableBase coverage
apps/wiki-server/src/routes/tablebase/	Code organization directory; NOT a URL prefix	New routes mount as /api/tablebase-coverage in app.ts

Proposed Approach

Three incremental phases, each independently shippable:

Phase 1 — Persist scan results: Add tablebase_scan_results PG table, wiki-server POST /api/tablebase-coverage/scan-results Hono RPC route, modify scanner.ts to POST per-table results incrementally (not in a single batch at the end), and add a nightly groundskeeper cron task.

Phase 2 — Read-only coverage dashboard: Build a Pattern A dashboard that reads scan history from PG and shows: per-table coverage % with trend sparklines, and a ranked task queue computed on-the-fly via SELECT … FROM entities e LEFT JOIN wiki_pages wp ON wp.wiki_id = e.wiki_id for importance data. Importance for personnel/funding_rounds is inherited from the parent org's reader_importance. No dispatch in Phase 2 — the dashboard is read-only.

Phase 3 — Job dispatch + completion tracking: Add tablebase_enrichment_tasks table, register tablebase-enrichment job handler in groundskeeper, add a "Dispatch" button with table-type filter, job count input, and estimated cost, wire inline job status in the task queue (JOIN with jobs table, client-side polling).

Importance inheritance approach (Phase 2 key decision): Most personnel and funding_round records don't have wiki pages, so wiki_pages.reader_importance is NULL for them directly. The task queue inherits importance from the parent org: for a personnel record, join to the org entity via entities.id = tet.entity_id, then get wp.reader_importance from the org's wiki page. An Anthropic employee inherits Anthropic's importance (90) even without their own wiki page. This separates "key researcher at top-tier org" from "unknown at obscure lab" — fallback 50 only applies when even the parent org has no wiki page.

ASCII diagram — Coverage Dashboard (Phase 2):

┌─────────────────────────────────────────────────────────────┐
│  TableBase Coverage                 Last scan: 2h ago       │
│                                                             │
│  Table          Coverage   Trend (30d)     Records          │
│  ─────────────────────────────────────────────────────      │
│  personnel      ██░░░░  6%  ↑ +2%          43/718          │
│  funding_rounds █░░░░░  4%  → 0%           8/198           │
│  investments    █░░░░░  5%  ↑ +1%          12/240          │
│  benchmarks     ████░░ 45%  ↑ +5%          180/400         │
│  grants         ██████ 99%  ✓              412/416         │
│                                                             │
│  Overall: 23%   [Run scan now]                              │
├─────────────────────────────────────────────────────────────┤
│  High-Impact Task Queue — 847 pending tasks                 │
│  [Filter: All · Personnel · Funding Rounds · Investments]   │
│                                                             │
│  # Entity               Type          Org         Impact    │
│  1 Sam Altman           personnel     OpenAI (90)   89     │
│  2 Dario Amodei         personnel     Anthropic(90) 87     │
│  3 Series B - Cohere    funding-round Cohere (75)   71     │
│  4 ...                                                      │
│                                                             │
│  [→ Dispatch jobs (Phase 3)]                                │
└─────────────────────────────────────────────────────────────┘

Key Decisions

• Use entities table for importance JOIN, not things.source_id

  • Alternatives: things.source_id (wrong — it's the record's own ID, not entity stableId), new entity registry table
  • Rationale: things.source_id for personnel/grant records is the record's own PK, not the parent entity's stableId. The correct JOIN is entities e ON e.stable_id = tet.entity_id LEFT JOIN wiki_pages wp ON wp.wiki_id = e.wiki_id.

• Inherit parent org importance for personnel/funding_rounds

  • Alternatives: fallback 50 for all no-wiki-page records (arbitrary ordering within tables); no importance for non-wiki records
  • Rationale: Personnel and funding_rounds have entity_id pointing to their parent org. Org importance (via wiki_pages.reader_importance) meaningfully ranks child records. Fallback 50 only applies when even the parent org has no wiki page.

• Task queue computed on-the-fly in Phase 2, not persisted

  • Alternatives: tablebase_enrichment_tasks table in Phase 2; materialized view
  • Rationale: The profiles JSONB in tablebase_scan_results already has per-entity completeness. A ranked queue is just a JOIN of that data with entities + wiki_pages. Persisting it in Phase 2 adds a table, migration, and backfill complexity with zero Phase 2 benefit — status tracking only matters once dispatch exists (Phase 3).

• Scanner POSTs incrementally (per-table), not in a single batch

  • Alternatives: POST entire scan in one request at the end
  • Rationale: Scanner takes 90s+. A groundskeeper timeout mid-scan would produce zero persisted data if posting happens only at the end. Per-table posting ensures partial runs still record what completed.

• Phase 2 is read-only; dispatch button deferred to Phase 3

  • Alternatives: feature-flagged dispatch button in Phase 2
  • Rationale: An inert button with no handler is a maintenance burden and a UX confusion point. Phase 2 delivers complete value as a read-only coverage and prioritization tool.

• .tablebase-completed.txt stays authoritative for CLI through Phase 3

  • Alternatives: immediate deprecation; migrate to PG in Phase 2
  • Rationale: The 196 IDs are SHA-256 hashes of ${taskType}:${entityId} — the original inputs are not stored, so backfill requires enumerating all entity × taskType combinations and hash-matching. This complexity belongs in Phase 3 when tablebase_enrichment_tasks is created, not Phase 2.

Architecture

File	Action	Purpose
apps/wiki-server/src/schema.ts	modify	Add tablebase_scan_results table (Phase 1); tablebase_enrichment_tasks (Phase 3)
apps/wiki-server/drizzle/0<N>_tablebase_scan_results.sql	create	Migration for tablebase_scan_results (Phase 1)
apps/wiki-server/drizzle/0<M>_tablebase_enrichment_tasks.sql	create	Migration for tablebase_enrichment_tasks + backfill script (Phase 3)
apps/wiki-server/src/routes/tablebase/coverage.ts	create	Hono RPC: POST /scan-results, GET /scan-results, GET /task-queue, POST /task-queue/dispatch
apps/wiki-server/src/app.ts	modify	app.route("/api/tablebase-coverage", coverageRoute)
crux/tablebase/scanner.ts	modify	Add --persist flag; POST per-table results to /api/tablebase-coverage/scan-results incrementally
crux/commands/tablebase/scan.ts	modify	Pass --persist flag to scanner
apps/groundskeeper/src/tasks/tablebase-scan.ts	create	Nightly scan + persist task (cron: 02:00 UTC)
apps/groundskeeper/src/config.ts	modify	Add tablebaseScan: TaskConfig interface + TASK_TABLEBASE_SCAN_ENABLED env var
apps/groundskeeper/src/index.ts	modify	Register tablebase-scan cron task + log block entry
apps/groundskeeper/src/tasks/tablebase-enrichment-handler.ts	create	Job handler for tablebase-enrichment job type (Phase 3)
apps/web/src/app/internal/tablebase-coverage/	create	Dashboard directory
apps/web/src/app/internal/tablebase-coverage/page.tsx	create	Redirect to /wiki/E<N>
apps/web/src/app/internal/tablebase-coverage/tablebase-coverage-content.tsx	create	Server component: stat cards + coverage table (from groundskeeper-runs pattern)
apps/web/src/app/internal/tablebase-coverage/task-queue.tsx	create	Client component with polling (from entity-source-checks/action-queue pattern)
content/docs/internal/tablebase-coverage.mdx	create	MDX stub with allocated E-number
apps/web/src/components/mdx-components.tsx	modify	Register TablebaseCoverageContent
apps/web/src/lib/wiki-nav.ts	modify	Add sidebar entry via internalHref("tablebase-coverage")

New PG table — tablebase_scan_results (Phase 1):

CREATE TABLE tablebase_scan_results (
  id BIGSERIAL PRIMARY KEY,
  scan_timestamp TIMESTAMPTZ NOT NULL,
  triggered_by TEXT NOT NULL DEFAULT 'manual',  -- 'manual' | 'groundskeeper' | 'cli'
  duration_ms INTEGER,
  table_name TEXT NOT NULL,
  total_entities INTEGER NOT NULL,
  entities_with_records INTEGER NOT NULL,
  total_records INTEGER NOT NULL,
  avg_completeness_pct NUMERIC(5,2) NOT NULL,  -- float, not INTEGER
  profiles JSONB NOT NULL,                      -- EntityProfile[] for drill-down
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  UNIQUE (scan_timestamp, table_name)           -- idempotent upsert
);
CREATE INDEX idx_tsr_table_time ON tablebase_scan_results(table_name, scan_timestamp DESC);

New PG table — tablebase_enrichment_tasks (Phase 3):

CREATE TABLE tablebase_enrichment_tasks (
  id TEXT PRIMARY KEY,               -- SHA256(taskType:entityId), 12 chars
  task_type TEXT NOT NULL,
  entity_id TEXT NOT NULL,           -- stableId of parent entity (org/person)
  entity_name TEXT NOT NULL,
  entity_type TEXT,
  table_name TEXT NOT NULL,
  impact_score INTEGER NOT NULL,
  completeness_pct NUMERIC(5,2),
  status TEXT NOT NULL DEFAULT 'pending',  -- 'pending' | 'dispatched' | 'completed' | 'failed'
  job_id BIGINT REFERENCES jobs(id),
  dispatched_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ,
  entity_importance INTEGER,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_tet_status_impact ON tablebase_enrichment_tasks(status, impact_score DESC);

Task queue query (Phase 2 — on-the-fly, no tablebase_enrichment_tasks):

-- Compute ranked task queue from latest scan results + entity importance
WITH latest_scan AS (
  SELECT DISTINCT ON (table_name)
    table_name, profiles, scan_timestamp
  FROM tablebase_scan_results
  ORDER BY table_name, scan_timestamp DESC
),
entity_profiles AS (
  SELECT
    table_name,
    jsonb_array_elements(profiles) AS profile
  FROM latest_scan
),
ranked AS (
  SELECT
    (profile->>'entityId')::text AS entity_id,
    (profile->>'entityName')::text AS entity_name,
    table_name,
    (profile->>'completenessPercent')::numeric AS completeness_pct,
    COALESCE(wp.reader_importance, 50) AS importance,
    -- Inherit parent org importance via entities → wiki_pages
    ((100 - (profile->>'completenessPercent')::numeric)
      * CASE table_name
          WHEN 'personnel' THEN 1.5
          WHEN 'grants' THEN 1.2
          ELSE 1.0
        END
      * (0.5 + COALESCE(wp.reader_importance, 50) / 100.0)
    ) AS impact_score
  FROM entity_profiles ep
  JOIN entities e ON e.stable_id = (profile->>'entityId')::text
  LEFT JOIN wiki_pages wp ON wp.wiki_id = e.wiki_id
)
SELECT * FROM ranked
ORDER BY impact_score DESC, entity_name ASC
LIMIT 50;

New Hono RPC route (apps/wiki-server/src/routes/tablebase/coverage.ts):

const coverageApp = new Hono()
  .post("/scan-results", zv("json", ScanResultSchema), async (c) => {
    // upsert: ON CONFLICT (scan_timestamp, table_name) DO UPDATE
    return c.json({ ok: true }, 201);
  })
  .get("/scan-results", zv("query", z.object({ limit: z.coerce.number().default(30) })), async (c) => {
    // returns last N scan results grouped by table (for trend charts + coverage bars)
    return c.json({ results });
  })
  .get("/task-queue", zv("query", TaskQueueQuerySchema), async (c) => {
    // computed on-the-fly via the SQL above; supports ?table_name= filter
    return c.json({ tasks, total });
  })
  // Phase 3 only:
  .post("/task-queue/dispatch", zv("json", DispatchSchema), async (c) => {
    // creates N tablebase-enrichment jobs, updates task status to 'dispatched'
    return c.json({ dispatched: N, jobs });
  });

export const coverageRoute = coverageApp;
export type CoverageRoute = typeof coverageApp;

Implementation Phases

Phase 1: PG persistence + nightly scan — S/M

Goal: After this ships, SELECT * FROM tablebase_scan_results ORDER BY scan_timestamp DESC LIMIT 5 shows real data; nightly scan runs automatically.

• Add tablebase_scan_results table to schema.ts (with UNIQUE(scan_timestamp, table_name) and NUMERIC(5,2))
• Write Drizzle migration SQL
• Add Hono RPC POST /api/tablebase-coverage/scan-results (upsert semantics)
• Add GET /api/tablebase-coverage/scan-results?limit=N route
• Modify scanner.ts: add --persist flag; POST per-table incrementally after each table completes (not batch at end)
• Add groundskeeper task apps/groundskeeper/src/tasks/tablebase-scan.ts (02:00 UTC)
• Modify apps/groundskeeper/src/config.ts: add tablebaseScan: TaskConfig + TASK_TABLEBASE_SCAN_ENABLED env var
• Modify apps/groundskeeper/src/index.ts: register task + add to startup log block
• Write tests: POST scan-result roundtrip, upsert idempotency (same timestamp+table → UPDATE not INSERT), partial scan (3 tables → 3 rows, not 0), --persist flag triggers POST

Quality gates: pnpm test, pnpm build, pnpm crux tb scan --persist --tables=grants --max=5 → verify row in DB
Exit criteria: SELECT table_name, avg_completeness_pct FROM tablebase_scan_results ORDER BY created_at DESC LIMIT 5 returns real data after manual scan with --persist

Phase 2: Read-only coverage dashboard — M

Goal: After this ships, the dashboard at /wiki/E<N> shows per-table coverage bars with 30-day trends, an overall coverage KPI, and a ranked task queue with parent-org importance inheritance.

• Allocate entity ID: pnpm crux tb ids allocate tablebase-coverage-dashboard
• Add GET /api/tablebase-coverage/task-queue?table_name=&limit=50 route (on-the-fly SQL with entities JOIN)
• Build tablebase-coverage-content.tsx (server component following groundskeeper-runs-content.tsx pattern): stat cards (overall coverage %, total entities, scan age), per-table coverage bars (using coverage-bars.tsx from entity-source-checks), sparkline trend chart
• Build task-queue.tsx (client component with polling, following entity-source-checks/action-queue.tsx pattern): ranked table with org-inherited importance, table-type filter pills, "Showing N of M" count, stable secondary sort by entity_name
• Create MDX stub content/docs/internal/tablebase-coverage.mdx with allocated E-number
• Register TablebaseCoverageContent in apps/web/src/components/mdx-components.tsx
• Add sidebar entry in apps/web/src/lib/wiki-nav.ts
• Add dashboard URL to crux tb scan CLI output ("View dashboard: https://www.longtermwiki.com/wiki/E")
• Write tests: task-queue endpoint ordering (higher org importance → higher rank), table-name filter, empty result (no scans yet → 200 with empty array)

Quality gates: pnpm build, cd apps/web && PLAYWRIGHT_BASE_URL=https://www.longtermwiki.com npx playwright test e2e/render-audit.spec.ts (add /wiki/E<N> to render-audit list), verify coverage bars render with real data
Exit criteria: Dashboard loads showing coverage bars with % values matching tablebase_scan_results; task queue shows Sam Altman / Dario Amodei near top (org importance 90); no console errors

Phase 3: Job dispatch + completion tracking — L

Goal: After this ships, clicking "Dispatch 10 jobs" creates tablebase-enrichment jobs that groundskeeper executes; task queue shows inline dispatched/completed/failed status with live refresh.

• Add tablebase_enrichment_tasks table + migration
• Write backfill script crux/scripts/migrate-completed-tasks.ts: enumerate all current entity × taskType combinations, hash-match against .tablebase-completed.txt, insert matched rows with status='completed'; log orphaned IDs as warnings
• Add POST /api/tablebase-coverage/task-queue/dispatch route (table-type filter, count input, creates jobs, returns cost estimate based on taskType weights)
• Add groundskeeper job handler tablebase-enrichment-handler.ts: calls crux tb enrich --entity-id=X --task-type=Y, marks task completed or failed
• Register tablebase-enrichment in groundskeeper (analogous to existing job types)
• Update task queue endpoint: JOIN tablebase_enrichment_tasks + jobs table for inline status (color pills: pending/dispatched/running/completed/failed); link job_id to /internal/jobs
• Add dispatch UI to task-queue.tsx: table-type filter pills, numeric job count input (default 10, max 50), cost estimate display ("~$X estimated"), confirmation dialog
• Update task-queue.tsx polling to refresh status every 30s when any tasks are dispatched/running
• Deprecate .tablebase-completed.txt once backfill is verified: remove from CLI reads, keep file for 1 release cycle then delete
• Write tests: dispatch creates correct job count with right types, handler marks task completed on success and failed on error, backfill script matches known IDs and logs orphaned ones

Quality gates: pnpm test, integration test dispatching 1 job with --table=grants → verify job in DB with type='tablebase-enrichment', verify task_queue shows status='dispatched'
Exit criteria: End-to-end: dispatch button → job created → groundskeeper picks up → task shows completed in dashboard

Scope Cuts

The following are explicitly NOT included:

• Auto-enrichment without human review: No auto-dispatch on schedule. The dashboard shows the queue; humans decide when to dispatch. This prevents runaway API costs.
• Per-field completeness drill-down UI: The profiles JSONB is stored in tablebase_scan_results but not surfaced in the Phase 2/3 UI. Expose via GET /scan-results/profiles?table=X in a future phase.
• Cross-table importance inference beyond direct parent: Personnel inherits parent org importance. Grants inheriting program importance, or benchmark-results inheriting benchmark importance, deferred.
• FactBase/WikiBase quality queues: TableBase only per user directive.
• Automated source checking of newly enriched records: Handled by the existing pnpm crux fb source-check pipeline.
• Scan results retention policy: Table will grow at ~100KB/day. Retention cron task deferred — low urgency for first year.

Quality & Verification Infrastructure

Area	Specification
Tests	apps/wiki-server/src/__tests__/tablebase-coverage.test.ts: POST scan-result roundtrip, upsert idempotency, partial scan (3 tables → 3 rows), task-queue ordering by impact score, table-name filter, dispatch creates correct jobs. crux/__tests__/scanner-persist.test.ts: --persist flag calls wiki-server POST; incremental posting (per-table, not single batch at end).
Migration verification	After Phase 3: grep -r "tablebase-completed.txt" crux/ --include="*.ts" → 0 results confirming deprecation is complete.
Integration smoke test	Phase 1: pnpm crux tb scan --persist --tables=grants && psql $DATABASE_URL -c "SELECT table_name, avg_completeness_pct, created_at FROM tablebase_scan_results ORDER BY created_at DESC LIMIT 3;" Phase 3: pnpm crux gh issues start 0 (noop) and verify one job dispatched + picked up within 10 min by groundskeeper.
UI verification	Phase 2: Add /wiki/E<N> to apps/web/e2e/render-audit.spec.ts page list. Run cd apps/web && PLAYWRIGHT_BASE_URL=https://www.longtermwiki.com npx playwright test e2e/render-audit.spec.ts after deploy.
Deploy steps	Phase 1: Auto-migrates on deploy. Set TASK_TABLEBASE_SCAN_ENABLED=true in groundskeeper env. Phase 2: No manual steps. Phase 3: Run backfill script after deploy: pnpm crux scripts migrate-completed-tasks. Verify task count with SELECT COUNT(*) FROM tablebase_enrichment_tasks WHERE status='completed' matches .tablebase-completed.txt line count (~196).
Documentation	Update content/docs/internal/data-architecture.mdx to mention new tables. Add crux tb scan --persist to CLAUDE.md Quick Reference.
Monitoring	Add to daily data quality snapshot: tablebaseLastScanAge (hours since most recent tablebase_scan_results row). Alert if >26h (missed nightly run).

Risks & Mitigations

Risk	Severity	Mitigation
.tablebase-completed.txt / PG divergence during transition	blocking	Phase 3 backfill uses hash-matching enumeration (not direct reconstruction). File stays authoritative for CLI until backfill is verified.
tablebase-enrichment job type dispatched before handler exists	blocking	Phase 3 adds handler and dispatch button in same PR. No dispatch UI ships until handler is live.
Backfill misses orphaned IDs (entities deleted after task completion)	significant	Backfill script logs orphaned IDs as warnings; they're excluded from PG backfill. ~5-10 orphans expected — acceptable data loss.
Scanner groundskeeper timeout mid-run	significant	Per-table incremental posting means completed tables are persisted even if the task times out. Failed tables simply have no new row for that scan timestamp.
Phase 2 task queue SQL is slow (JSONB array expansion + 2 JOINs)	significant	profiles JSONB is per-table (not per-entity in one giant blob). Latest-scan CTE uses DISTINCT ON. Add idx_tet_status_impact in Phase 3. Benchmark query against ~700 entities in staging before Phase 2 deploy.
things.wiki_id was assumed correct for the JOIN (was broken in earlier plan draft)	resolved	Corrected: use entities.stable_id for JOIN, not things.source_id. things table NOT used for entity importance lookup.
Parent org importance is 0 for orgs with very new wiki pages	minor	Fallback 50 applies when reader_importance IS NULL. This is acceptable — zero-importance entities will appear near the bottom of the queue either way.

Open Questions

1. Dispatch mechanism for Phase 3: Use the existing PG jobs queue (groundskeeper SELECT FOR UPDATE SKIP LOCKED) or trigger a GitHub Actions workflow per batch? Jobs queue is consistent with all other job types but requires Phase 3 handler work. GHA workflow is simpler to implement but harder to monitor inline.
2. Cost estimate calculation: The dispatch UI shows "~$X estimated." What's the per-task cost model? Based on TASK_TYPE_WEIGHTS in task-ranker.ts (personnel 1.5x = more expensive), but actual $ cost depends on model tier used by the enrichment command. Needs a cost-per-task table or a fixed approximate.
3. Scan coverage tables: Should the Phase 1 nightly scan cover all 7 TableBase types (~90s) or only low-coverage ones — personnel, funding_rounds, investments (~40s)? Benchmarks (45%) and grants (99%) change slowly; daily scan of them is low-value.

Rejected Approaches

• Use things.source_id for entity importance JOIN: Rejected — things.source_id is the record's own PK for personnel/grant records, not the parent entity stableId. Must use entities.stable_id instead.
• Compute task queue as a materialized view: Rejected — requires scheduled REFRESH and adds a new DB object type. On-the-fly SQL with indexed CTEs is sufficient at 700 entities.
• tablebase_enrichment_tasks table in Phase 2: Rejected — status tracking only matters once dispatch exists (Phase 3). Phase 2 computes the queue from scan results + entities on-the-fly.
• Feature-flagged dispatch button in Phase 2: Rejected — an inert button before the handler exists is a UX confusion point and maintenance burden.
• Migrate .tablebase-completed.txt in Phase 2: Rejected — backfill requires rainbow-table enumeration over entity × taskType combinations (hashes are not reversible). This complexity belongs in Phase 3 with the tablebase_enrichment_tasks table.
• Full approach B (all features at once): Rejected per user directive to focus on TableBase coverage with incremental phases.

Red Team Log

Round 1 — Technical critic:

• BLOCKING: things.source_id is record's own ID, not entity stableId — JOIN for importance was broken. Fixed: use entities table.
• BLOCKING: .tablebase-completed.txt hash IDs not reconstructable standalone — backfill requires enumeration. Moved entirely to Phase 3.
• Significant: /api/tablebase/ is code directory, not URL prefix. Fixed: mount as /api/tablebase-coverage.
• Significant: Groundskeeper config.ts not in architecture table. Fixed: added with note about 4 required changes.
• Significant: Missing UNIQUE constraint on (scan_timestamp, table_name), avg_completeness_pct should be NUMERIC(5,2), scanner should POST per-table incrementally. All fixed.
• Minor: VARCHAR(12) for task ID → changed to TEXT. JSONB growth / retention deferred to Scope Cuts.

Round 1 — Scope critic:

• tablebase_enrichment_tasks not needed in Phase 2. Accepted: removed, deferred to Phase 3.
• Dispatch button not needed in Phase 2. Accepted: removed.
• Merge history/latest endpoints → single GET /scan-results?limit=N. Accepted.
• .tablebase-completed.txt backfill cut from Phase 2. Accepted.
• Copy groundskeeper-runs pattern more directly. Accepted: explicit in Phase 2 implementation.

Round 1 — UX critic:

• BLOCKING: fallback-50 makes task queue arbitrary within large tables. Fixed: parent-org importance inheritance.
• High: No table-type filter on dispatch. Added in Phase 3 UI spec.
• High: No cost estimate / count control. Added in Phase 3 UI spec.
• High: No inline job status after dispatch. Added in Phase 3 (JOIN jobs table, client polling).
• Medium: ISR staleness for task queue. Fixed: task-queue.tsx is a client component with polling.
• Medium: No CLI-to-dashboard link. Added: print URL after --persist scan.
• Medium: Coverage definition ambiguous — tooltip with top missing fields. Deferred to future phase; profiles JSONB is stored for this purpose.

Tasks

• Add nightly groundskeeper task for TableBase scan with --persist #4007 — Add nightly groundskeeper task for TableBase scan with --persist

[OAGr]

Branch: claude/deploy-tasks-verify-cmd

Cross-link: sibling of #3945 (Things UI Universal Record Inspector — detail) and #3998 (Things Listing Enhancements — listing). #4005 covers the coverage dashboard / enrichment queue side of the same overall Things UI initiative. Consider treating these as siblings of one Things UI epic. — Discussion review 2026-04-08.