Skip to content

/health returns HTTP 200 even when the service is DOWN #297

Description

@nangelovv

Bug

The /health endpoint returns HTTP 200 even when the health check fails. Liveness/readiness probes (Docker HEALTHCHECK, Kubernetes, load balancers) only inspect the status code, so they treat a downed service as healthy.

Cause

In app/routes/document_routes.py the handler returns a tuple:

@router.get("/health")
async def health_check():
    try:
        if await is_health_ok():
            return {"status": "UP"}
        else:
            logger.error("Health check failed")
            return {"status": "DOWN"}, 503        # Flask idiom
    except Exception as e:
        ...
        return {"status": "DOWN", "error": str(e)}, 503

return body, 503 sets the status code in Flask, but FastAPI does not interpret a returned tuple that way. FastAPI serializes the whole tuple through jsonable_encoder, so the body becomes the JSON array [{"status": "DOWN"}, 503] and the HTTP status stays at the default 200.

Reproduction

docker compose up -d
docker compose stop db   # make the health check fail
curl -s -o - -w '\nHTTP %{http_code}\n' http://localhost:8000/health

Observed:

[{"status":"DOWN"},503]
HTTP 200

Expected: HTTP 503 with body {"status":"DOWN"}.

Impact

With a constant 200, a failed /health never marks the container unhealthy, never triggers a restart, and never removes the instance from a load-balancer pool — the outage is masked.

Fix

Return a JSONResponse with an explicit status_code=503 for both the unhealthy and exception branches. PR incoming.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions