fix(superset): API datamodel fields + SQL lineage, cache datasource/database fetches

[akashverma0786]

Fixes #30407

What

Two related improvements to the Superset API source (plus a perf fix uncovered while doing them):

1. Datamodel fields + real column names — API-mode datamodels now carry sql, description and sourceUrl from the dataset payload, and columns use the real column_name instead of the numeric dataset-column id.
2. SQL (virtual dataset) lineage for API mode — virtual datasets defined by SQL now resolve lineage to their real upstream source tables (with column-level lineage), matching what the DB source already did. The dataset SQL is fetched, parsed via the shared _parse_lineage_from_dataset_sql, and each source table's FQN is resolved (_get_input_tables → _get_source_table_fqn), with a shared _resolve_lineage_database_name helper to avoid duplication.
3. Cache fetch_datasource / fetch_database — a bounded LRUCache (via cachetools.cachedmethod) on the Superset client. The same dataset/database was being re-fetched over REST many times per run — once per chart, again per lineage pass / dbServicePrefix, and again per SQL-parsed source table. These responses are immutable within a run.

Why

API-mode datamodels were missing sql/description/sourceUrl and displayed numeric column ids; virtual (SQL) datasets produced no source-table lineage. Resolving datamodels and lineage also amplified redundant /dataset/{id} and /database/{id} REST calls, which the cache removes.

Performance — same ingestion, byte-identical output

Local benchmark ingesting the same dashboards/datasets with dbServicePrefixes: [retail_mysql, analytics_pg]:

Metric	Before	After
GET /dataset/{id} calls	860	26
GET /database/{id} calls	260	3
Network GET time	592s	31s
Workflow wall-clock	11m 10s	2m 07s (~5.3× faster)

Ingested output verified identical before/after — datamodels, columns, SQL, dashboards, charts and lineage edges all diff-clean, 0 errors. The cache changes how many times Superset is called, not what is ingested.

Tests

Added to test_superset.py: cache dedup (same id → 1 network call, distinct ids → N), dataset-SQL parsing to source tables, and source-table FQN resolution. Existing tests already cover the datamodel fields and real column names.

Greptile Summary

This PR improves Superset API metadata ingestion and lineage. The main changes are:

• Adds SQL, descriptions, source URLs, and real column names to API datamodels.
• Resolves virtual-dataset SQL lineage to upstream source tables.
• Adds bounded caches for datasource and database responses.
• Keeps empty and failed fetches retryable.

Confidence Score: 5/5

This looks safe to merge.

• Empty responses and exceptions are not cached.
• Later calls can retry after a temporary fetch failure.
• No blocking issue remains in the updated code.

Important Files Changed

Filename	Overview
ingestion/src/metadata/ingestion/source/dashboard/superset/api_source.py	Adds richer API datamodel fields and resolves virtual-dataset lineage through parsed SQL.
ingestion/src/metadata/ingestion/source/dashboard/superset/client.py	Adds bounded datasource and database caches while leaving failed fetches uncached.
ingestion/src/metadata/ingestion/source/dashboard/superset/mixin.py	Propagates datasource identifiers for lineage and uses actual Superset column names.
ingestion/tests/integration/superset/test_superset.py	Adds coverage for API datamodel fields, SQL lineage, cache reuse, and retry behavior.

_{Reviews (10): Last reviewed commit: "Merge branch 'main' into fix/superset-da..." | Re-trigger Greptile}

Context used (3)

• Context used - openmetadata-ui-core-components/CLAUDE.md (source)
• Context used - CLAUDE.md (source)
• Context used - AGENTS.md (source)

[github-actions]

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit de7a80f85487a9204b0496558b082272f5305e9a in Playwright run 30990646970, attempt 1.

✅ 107 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 46m 48s

⏱️ Max setup 3m 7s · max shard execution 11m 25s · max shard-job elapsed before upload 17m 44s · reporting 3s

🌐 211.04 requests/attempt · 1.76 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

• Browser traffic was 211.04 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 1.76 per UI scenario (208 boots / 118 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	46	0	0	0	0	0
✅ Shard ingestion-01	32	0	0	0	0	0
✅ Shard ingestion-02	29	0	0	0	0	0

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[github-actions]

🚦 Removed from the merge queue — merge_conflict (2026-08-04T14:08:26Z)

The entry left the queue before it was built, so no checks ran against it.

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
70.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

🚦 Removed from the merge queue — failed_checks (2026-08-07T10:22:45Z)

Blocked the queue: py-tests-status

• PostgreSQL PR RDF E2E Tests — Playwright RDF (Knowledge Graph + Ontology), RDF Playwright execution
• py-tests — py-tests-status, python / Verify Expected Jobs, python / Build Backend Distribution

[github-actions]

Changes have been cherry-picked to the 2.0 branch.

[github-actions]

Failed to cherry-pick changes to the 1.13 branch.
Please cherry-pick the changes manually.
You can find more details here.

[gitar-bot]

Code Review ✅ Approved 2 resolved / 2 findings

Adds SQL virtual dataset lineage, enriched datamodel metadata, and success-only caching for Superset API fetches to significantly reduce REST overhead. No issues found.

✅ 2 resolved

✅ Edge Case: Cached datasource/database error fallbacks poison cache for the run

📄 ingestion/src/metadata/ingestion/source/dashboard/superset/client.py:222-236
fetch_datasource/fetch_database are now wrapped with @cachedmethod, but on a transient REST failure (or falsy response) they return the empty fallback (SupersetDatasource() / ListDatabaseResult()), and that empty object is stored in the LRU cache. Since these ids are re-requested many times per run (chart, lineage pass, SQL source-table resolution), a single temporary failure permanently yields empty results for that id for the rest of the run — dropping datamodel sql/description/sourceUrl and lineage that would previously have been recovered on retry. Only cache successful responses, e.g. raise on failure (cachedmethod does not cache exceptions) or populate the cache manually only when a real response is returned.

✅ Edge Case: Missing database triggers spurious /database/None REST call

📄 ingestion/src/metadata/ingestion/source/dashboard/superset/api_source.py:156-158
In _resolve_lineage_database_name, when datasource_json.result.database is None the new guard passes None into self.client.fetch_database(None), which issues an actual GET /database/None request and then hits database_json.result.parameters on an empty ListDatabaseResult (result is None) — an AttributeError that only the outer try/except in _get_source_table_fqn swallows. Short-circuit before calling the client: return None (or skip the fetch) when database is None so no wasted/failing REST call is made.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}