fix(search): don't fail the reindex when the embedding provider is unreachable - #31131
Conversation
…reachable #30364 un-gated the staged chunk recreate, so every job-driven full recreate now calls beginStagedChunkRecreate(), which opens with a pre-flight embed. The pre-flight rethrows, and that exception fails the whole SearchIndexApp run — so a reindex that needs no embeddings at all is now blocked by an optional AI provider being unreachable. That is not a narrow case. Collate ships semanticSearchEnabled=true by default while llmConfiguration.embeddings.provider defaults to bedrock, and BedrockEmbeddingClient's constructor never calls AWS — it validates the model id, dimension and region, then builds the SDK client. So on any deployment where a region resolves (anything on AWS) but bedrock:InvokeModel was never granted, the client constructs happily, initializeVectorSearchService reports success, and the first full reindex dies: User: arn:aws:sts::...:assumed-role/... is not authorized to perform: bedrock:InvokeModel on resource: .../amazon.titan-embed-text-v2:0 (Status Code: 403) surfacing as status='failed' within seconds with an empty failureContext. The misconfiguration is otherwise invisible: live indexing logs embedding errors and carries on, so the deployment looks healthy right up until someone reindexes. This is what has been failing the nightly Java IT suites on main and 2.0. Treat an unavailable provider as "do not stage" rather than "fail": return null, which the caller already handles as the partial-recreate outcome — existing chunks stay live and are swept by the next recreate that runs with a working provider. markEntityTypeReindexed already ignores marks from a run without staging, so this reuses a supported state rather than inventing one. Genuine staging failures — an indeterminate live-target probe, a failed index create — still throw, because those mean continuing could destroy live chunks.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adjusts staged chunk recreation to skip staging (instead of failing the whole reindex) when the embedding provider is unreachable.
Changes:
beginStagedChunkRecreate()now returnsnullwhen embedding pre-flight fails, allowing the caller to proceed with an in-place reindex.- Replaces throwing pre-flight method with
isEmbeddingAvailable()that logs and returns a boolean. - Adds a regression test covering the “embedding unavailable ⇒ do not touch cluster and return null” behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| openmetadata-service/src/main/java/org/openmetadata/service/search/vector/OpenSearchVectorService.java | Implements “skip staging when embedding provider is unavailable” via a boolean pre-flight and a nullable return. |
| openmetadata-service/src/test/java/org/openmetadata/service/search/vector/OpenSearchVectorServiceChunkStagingTest.java | Adds test asserting beginStagedChunkRecreate() returns null and makes no OpenSearch calls when embeddings fail. |
| public String beginStagedChunkRecreate() { | ||
| preflightEmbedding(); | ||
| if (!isEmbeddingAvailable()) { | ||
| LOG.warn( | ||
| "Embedding pre-flight failed — skipping the staged chunk recreate. The entity reindex " | ||
| + "continues and existing chunks stay live; orphaned chunks, if any, are swept by the " | ||
| + "next recreate that runs with a working embedding provider."); | ||
| return null; | ||
| } |
| private boolean isEmbeddingAvailable() { | ||
| try { | ||
| embeddingClient.embedQuery("chunk index recreate pre-flight"); | ||
| return true; | ||
| } catch (Exception e) { | ||
| throw new RuntimeException( | ||
| "Refusing to start a staged chunk-index recreate: embedding client pre-flight failed", e); | ||
| LOG.warn("Embedding client pre-flight failed: {}", e.getMessage(), e); | ||
| return false; | ||
| } | ||
| } |
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
github-actions
Bot
added
backend
safe to test
✅ Playwright Results — workflow succeededValidated commit ✅ 1026 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 55m 36s ⏱️ Max setup 3m 2s · max shard execution 19m 17s · max shard-job elapsed before upload 24m 13s · reporting 6s 🌐 189.20 requests/attempt · 2.21 app boots/UI scenario · 26.88% common-shard skew Optimization targets still in progress:
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
🚦 Removed from the merge queue —
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
openmetadata-service/src/main/java/org/openmetadata/service/search/vector/OpenSearchVectorService.java:521
isEmbeddingAvailable()currently swallows all exceptions fromembedQuery()and turns them into "not staged". If the thread is interrupted (e.g., shutdown/cancellation while waiting for a permit), we should not continue the reindex; treat that as a genuine failure and rethrow, and narrow the catch toRuntimeException(the only typeembedQuerycan throw).
LOG.warn("Embedding client pre-flight failed: {}", e.getMessage(), e);
return false;
}
openmetadata-service/src/main/java/org/openmetadata/service/search/vector/OpenSearchVectorService.java:517
isEmbeddingAvailable()always performs anembedQuery()call, which can block on provider timeouts even when the embedding client circuit breaker is already open. Consider short-circuiting viaembeddingClient.isAvailable()first to skip staging without making a provider call during an outage (you’ll need to update unit tests to stubisAvailable()where appropriate).
private boolean isEmbeddingAvailable() {
try {
embeddingClient.embedQuery("chunk index recreate pre-flight");
return true;
Code Review ✅ ApprovedMakes embedding provider failures non-fatal during search reindexing by treating unreachable providers as a skip condition rather than failing the run. No issues found. OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
Follows up #30364, which is on
mainand the2.0release branch.What happens today
#30364 un-gated the staged chunk recreate, so every job-driven full recreate now calls
beginStagedChunkRecreate(), which opens with a pre-flight embed. That pre-flight rethrows, and the exception fails the entireSearchIndexApprun — so a reindex that needs no embeddings at all is blocked by an optional AI provider being unreachable.Why this is not a narrow case
Collate ships
naturalLanguageSearch.semanticSearchEnabled=trueby default, whilellmConfiguration.embeddings.providerdefaults tobedrock. AndBedrockEmbeddingClient's constructor never calls AWS — it validates model id, dimension and region, then builds the SDK client.So on any deployment where a region resolves (anything running on AWS) but
bedrock:InvokeModelwas never granted:initializeVectorSearchServicereports success,surfacing as
status='failed'within seconds with an emptyfailureContext.The misconfiguration is otherwise invisible. Live indexing logs embedding errors and carries on, so the deployment looks healthy right up until someone reindexes. OSS is unaffected —
semanticSearchEnableddefaults tofalsethere, so the vector service never initialises and the gate returns early.This is what has been failing the nightly Java IT reindex suites on
mainand2.0since 2026-08-04.The change
Treat an unavailable provider as "do not stage" rather than "fail": return
null, which the caller already handles as the partial-recreate outcome — existing chunks stay live and get swept by the next recreate that runs with a working provider.markEntityTypeReindexedalready ignores marks from a run without staging (the "unbound chunk-type mark" branch), so this reuses a supported state rather than inventing one.The pre-flight's intent is preserved: it still refuses to stage a generation it could never finish. It just no longer takes the entity reindex down with it.
Genuine staging failures still throw — an indeterminate live-target probe, a failed index create — because those mean continuing could destroy live chunks. The existing
beginStagedChunkRecreate_abortsWhenTheLiveTargetProbeIsIndeterminatetest still passes unchanged.Test plan
OpenSearchVectorServiceChunkStagingTest— 8/8 pass, including a new case asserting an unavailable provider returnsnulland touches nothing in the cluster (verify(client, never()).indices()/.generic()).RecreateWithEmbeddingsTest— 7/7 pass.Unit tests were executed on
main; the2.0change is character-identical (verified by diffing the two patches) but its module could not be built locally offline.Related
🤖 Generated with Claude Code