Skip to content

HIVE-29584: Invalidate RawStore after Direct SQL connection failures - #6664

Draft
gaoxq wants to merge 1 commit into
apache:masterfrom
gaoxq:agent/hive-29584-broken-connection-recovery
Draft

HIVE-29584: Invalidate RawStore after Direct SQL connection failures#6664
gaoxq wants to merge 1 commit into
apache:masterfrom
gaoxq:agent/hive-29584-broken-connection-recovery

Conversation

@gaoxq

@gaoxq gaoxq commented Jul 31, 2026

Copy link
Copy Markdown

What changes were proposed in this pull request?

When Direct SQL fails with a JDBC connection-level error, invalidate and shut
down the RawStore cached in HMSHandlerContext instead of falling back to ORM
with the same broken PersistenceManager/connection.

The change:

  • preserves the original SQLException from MetaStoreDirectSql.prepareTxn();
  • recognizes SQLState class 08 and JDBC connection exception subclasses;
  • removes the thread-local RawStore before shutdown;
  • wraps the failure as a JDO datastore error so RetryingHMSHandler can retry;
  • makes ObjectStore cleanup safe when an outer handler subsequently rolls back;
  • keeps the existing ORM fallback for non-connection Direct SQL failures.

Why are the changes needed?

After a transient metastore database communication failure, a handler thread
can retain a closed connection through its cached PersistenceManager. HikariCP
cannot replace that connection while the wrapper remains in use, so later RPCs
on the same handler thread can repeatedly fail with Connection is closed
until HMS is restarted.

JIRA: https://issues.apache.org/jira/browse/HIVE-29584

How was this patch tested?

  • Added a regression test for SQLState 08S01 that verifies no ORM fallback,
    RawStore invalidation, preservation of the JDO-backed error, and safe outer
    transaction rollback. The test failed with an NPE before the cleanup fix and
    passes afterward.
  • TestObjectStore#testDirectSqlConnectionErrorInvalidatesRawStore
  • TestObjectStore#testDirectSqlErrorMetrics
  • TestObjectStore#testNoJdoForUnrecoverableException
  • TestRetriesInRetryingHMSHandler#testWrappedMetaExceptionRetry
  • mvn -pl standalone-metastore/metastore-server -DskipTests checkstyle:check

Preserve JDBC connection errors and remove the cached RawStore before
shutdown, allowing RetryingHMSHandler to retry with a fresh
PersistenceManager. Make post-shutdown rollback idempotent so outer handler
cleanup cannot mask the original failure.

Co-Authored-By: GPT-5 Codex <noreply@openai.com>
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants