Skip to content

A3 conformance probe can wedge relay boot without a deadline or early health listener #7292

Description

@wpfleger96

Problem

At 7a9a523, relay startup runs the A3 git object-store conformance probe as a hard boot gate. The default probe uses 32 writers across 3 rounds against blob storage. Individual requests have timeouts, but the probe has no overall deadline.

On a homelab MinIO deployment, the probe wedged silently for more than an hour during if_none_match_race round 0. The container remained running with zero restarts and emitted no further logs. Restarting the container fixed the issue; a normal probe completes in about five seconds.

The health listener on port 8080, including /_liveness and /_readiness, starts only after the boot gates complete. During the wedge, the Compose healthcheck received connection refusals until its full budget expired and marked the container unhealthy. Orchestrators cannot distinguish a relay that is still booting from one stuck in the probe.

Suggested fixes

  • Add an overall deadline to the A3 probe and fail startup with an explicit error when it expires.
  • Bind the health listener before long-running boot gates so liveness and readiness remain observable during startup.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions