Repository navigation
fix(catalog-ui): wait for the episodes file before binding an append - #775
kstonekuan merged 2 commits into
Conversation
An append stores its dependent tables before its episodes file, which is its commit marker. The poll loop rebound the open UI as soon as any empty table gained Parquet, so a poll between those writes bound check_runs, measurements and observations while episodes was still the empty shell, and printed that the views were available. Dependent Parquet now counts only once an episodes file exists; ingest_failures still binds on its own.
[Medium impact] The PR appears safe to merge; the remaining test timing gap is non-blocking. Findings
|
| time.sleep(0.2) | ||
| assert session.refresh_count == 0 |
There was a problem hiding this comment.
The new test sleeps for 0.2 seconds without checking that the UI polled after the dependent files were copied. If the background thread is delayed, refresh_count == 0 passes even with the old bug. Wait for a completed poll against the dependents-only catalog before checking that no refresh happened, then copy the episodes file.
The fixed sleep could pass without the UI ever polling the dependents-only catalog. Wait until a poll that started after the copy has finished, then check that it did not refresh.
kstonekuan
left a comment
There was a problem hiding this comment.
Thanks! Ruff, format, and ty pass; 2629 tests passed, 47 skipped. LGTM.
Summary
The open catalog UI no longer binds a half written first append. Before,
hflow catalog uion an empty catalog could print "Catalog Parquet detected. Views are now available in the open UI." whileepisodeswas still empty, because the dependent tables of the append had landed and its episodes file had not. Now the first refresh waits for the episodes file, so it always shows the episode row, which is whathflow catalog ui --helpalready promises ("refreshes its views after the first completed append").Why
Catalog.append_episodewrites check_runs, measurements and observations first and the episodes file last, and_refresh_local_catalog_connection's docstring relies on that ("the first episodes Parquet is enough to rebind every table that append wrote"). The poll trigger,_empty_table_relations_have_landed_parquet, did not: it returned true for Parquet in any empty table. A poll that ran between the two writes bound the dependents, keptepisodesas the empty shell, and spent the "first Parquet" message on it. The next poll repaired it, so the wrong state lasts one interval, but it is the window an operator watching the first ingest sees, and it is wider on a bucket catalog where each dependent is its own put plus a mirror sync.The trigger now skips Parquet in tables from the append's write set (
TABLE_COLUMN_DDL) until the episodes directory has a file.ingest_failuresis outside that set, so a failure only catalog (#537) still binds without any episode.Repro, as in the new test: start the UI on an empty catalog, then copy every table of a finished append except
episodesinto it. On main the UI refreshes within one poll withSELECT count(*) FROM episodesreturning 0. With this change it waits, and refreshes once the episodes file is copied in.Validation
test_catalog_ui_waits_for_the_episodes_file_before_binding_an_append: on main it fails 5 of 5 runs (assert session.refresh_count == 0, got 1); with the fix it passes 20 of 20.uv run pytest -q tests/test_catalog_ui.py tests/test_catalog_curation.py: 216 passed.uv run pytest -q -n 4: 2627 passed, 49 skipped (macOS, Python 3.11 env fromuv sync --locked).uv run ruff check,uv run ruff format --check,uv run ty check: clean.Checklist
uv run ruff check --fix,uv run ruff format, anduv run ty check.Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.