-
Notifications
You must be signed in to change notification settings - Fork 33
fix: transition exporter to AVAILABLE after afterLease hook failure #871
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -825,9 +825,8 @@ async def run_after_lease_hook( | |||||||||||||||||||||||||||
| shutdown_called = True | ||||||||||||||||||||||||||||
| else: | ||||||||||||||||||||||||||||
| # on_failure='endLease' - report failure to the client, then release the lease. | ||||||||||||||||||||||||||||
| # AFTER_LEASE_HOOK_FAILED is a transient status: the client sees the failure, | ||||||||||||||||||||||||||||
| # the lease is released in the finally block, and the exporter's main loop | ||||||||||||||||||||||||||||
| # clears the lease context and accepts new leases. | ||||||||||||||||||||||||||||
| # AFTER_LEASE_HOOK_FAILED is transient: the finally block transitions to AVAILABLE | ||||||||||||||||||||||||||||
| # and releases the lease, allowing the exporter to accept new leases. | ||||||||||||||||||||||||||||
| logger.error("afterLease hook failed with on_failure='endLease': %s", e) | ||||||||||||||||||||||||||||
| await report_status( | ||||||||||||||||||||||||||||
| ExporterStatus.AFTER_LEASE_HOOK_FAILED, | ||||||||||||||||||||||||||||
|
|
@@ -836,8 +835,8 @@ async def run_after_lease_hook( | |||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| except Exception as e: | ||||||||||||||||||||||||||||
| # Unexpected errors: report failure but do not shut down. | ||||||||||||||||||||||||||||
| # Same transient status - the lease is released and the exporter | ||||||||||||||||||||||||||||
| # accepts new leases after the finally block completes. | ||||||||||||||||||||||||||||
| # AFTER_LEASE_HOOK_FAILED is transient: the finally block transitions to AVAILABLE | ||||||||||||||||||||||||||||
| # and releases the lease, allowing the exporter to accept new leases. | ||||||||||||||||||||||||||||
| logger.error("afterLease hook failed with unexpected error: %s", e, exc_info=True) | ||||||||||||||||||||||||||||
| await report_status( | ||||||||||||||||||||||||||||
| ExporterStatus.AFTER_LEASE_HOOK_FAILED, | ||||||||||||||||||||||||||||
|
|
@@ -851,6 +850,9 @@ async def run_after_lease_hook( | |||||||||||||||||||||||||||
| # Don't release lease when exporter is shutting down - unregistration handles cleanup. | ||||||||||||||||||||||||||||
| # Releasing here would report AVAILABLE to the controller right before shutdown. | ||||||||||||||||||||||||||||
| if request_lease_release and not shutdown_called: | ||||||||||||||||||||||||||||
| # Transition to AVAILABLE to clear AFTER_LEASE_HOOK_FAILED. | ||||||||||||||||||||||||||||
| # Idempotent if already AVAILABLE from the happy/warn paths. | ||||||||||||||||||||||||||||
| await report_status(ExporterStatus.AVAILABLE, "Available for new lease") | ||||||||||||||||||||||||||||
| try: | ||||||||||||||||||||||||||||
| await request_lease_release() | ||||||||||||||||||||||||||||
|
Comment on lines
+853
to
857
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win Wrap If 🔒 Proposed fix: protect report_status call if request_lease_release and not shutdown_called:
# Transition to AVAILABLE to clear AFTER_LEASE_HOOK_FAILED.
# Idempotent if already AVAILABLE from the happy/warn paths.
- await report_status(ExporterStatus.AVAILABLE, "Available for new lease")
+ try:
+ await report_status(ExporterStatus.AVAILABLE, "Available for new lease")
+ except Exception as e:
+ logger.error("Failed to report AVAILABLE status: %s", e, exc_info=True)
try:
await request_lease_release()
except Exception as e:
logger.error("Failed to request lease release: %s", e, exc_info=True)📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||
| except Exception as e: | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry to be so late back to this PR.
I have another question. Why does not the exporter return "AVAILABLE" once it's shutdown/restarted, which would be a more natural place?, why do we need to do it here?
I feels to me like there is some bug in the sequence that makes the exporter exit and register the ready status once it's ready again.
Here we would be declaring ourselves ready even though we have not gone through exporter init on the restarted instance.