Skip to content

[Bug]: Airflow health check rejects deployments without dag_processor #779

Closed

Description

@harsh21234i

Problem

AirflowHealth.healthy requires dag_processor == "healthy", even though the implementation comment says dag_processor may legitimately be absent (null) in minimal deployments.

The Airflow health endpoint can return healthy metadatabase and scheduler components while omitting or returning null for optional components. HFlow then reports the runtime as unhealthy and wait_until_healthy() can time out unnecessarily.

Reproduction

from hflow.runtime._client import AirflowHealth

health = AirflowHealth(
    components={
        "metadatabase": "healthy",
        "scheduler": "healthy",
        "triggerer": None,
    }
)
assert health.healthy  # currently False

The current implementation checks:

required = ("metadatabase", "scheduler", "dag_processor")
return all(self.components.get(name) == "healthy" for name in required)

Expected behavior

Missing or null optional components should not make the runtime unhealthy. Only components that are required for the configured deployment should be required to report healthy.

Additional context

The source comment at src/hflow/runtime/_client.py:77-80 explicitly says dag_processor may be absent, but the code still treats it as required. I found no existing issue covering this behavior.

Activity

  1. kstonekuan commented on Nov 13, 2026

    @kstonekuan
    Contributor

    Which Airflow deployment are you using, and does it parse DAGs without a separate DAG processor? HFlow’s bundled runtime starts one and needs it to register the ingest DAG.

  2. harsh21234i commented on Nov 13, 2026

    @harsh21234i
    ContributorAuthor

    Thanks for checking. I was not testing a separate production deployment; the report came from reading the REST client contract and reproducing its behavior with a health response that contains healthy metadatabase/scheduler but omits dag_processor.

    I confirmed that HFlow’s bundled Compose runtime always starts airflow-dag-processor and relies on it to parse/register the ingest DAG. So the issue does not affect the bundled runtime. The remaining question is whether HFlow intends to support external/minimal Airflow deployments where DAG parsing is handled without a separate dag_processor component. If not, the comment in AirflowHealth.healthy should be clarified to describe the bundled/runtime requirement rather than treating the component as optional.

  3. kstonekuan commented on Nov 16, 2026

    @kstonekuan
    Contributor

    Not supported. HFlow needs the DAG processor to register the ingest DAG, so the check's right and the comment's stale. Want to send a PR fixing the comment?

  4. harsh21234i commented on Nov 20, 2026

    @harsh21234i
    ContributorAuthor

    Thanks for the clarification. You’re right: HFlow requires the DAG processor, and deployments without it are not supported. I created this issue based on the stale comment without first reproducing it in a supported deployment. I’ll close it and leave the runtime behavior unchanged.

  5. harsh21234i commented on Nov 20, 2026

    @harsh21234i
    ContributorAuthor

    Before proceeding, is this distinct case worth fixing separately, or is it covered by the existing numeric-overflow work from #739? I found  ManifestPartition("train", 10**400)  still leaks  OverflowError .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions