You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Airflow health check rejects deployments without dag_processor #779
AirflowHealth.healthy requires dag_processor == "healthy", even though the implementation comment says dag_processor may legitimately be absent (null) in minimal deployments.
The Airflow health endpoint can return healthy metadatabase and scheduler components while omitting or returning null for optional components. HFlow then reports the runtime as unhealthy and wait_until_healthy() can time out unnecessarily.
Missing or null optional components should not make the runtime unhealthy. Only components that are required for the configured deployment should be required to report healthy.
Additional context
The source comment at src/hflow/runtime/_client.py:77-80 explicitly says dag_processor may be absent, but the code still treats it as required. I found no existing issue covering this behavior.
Which Airflow deployment are you using, and does it parse DAGs without a separate DAG processor? HFlow’s bundled runtime starts one and needs it to register the ingest DAG.
Thanks for checking. I was not testing a separate production deployment; the report came from reading the REST client contract and reproducing its behavior with a health response that contains healthy metadatabase/scheduler but omits dag_processor.
I confirmed that HFlow’s bundled Compose runtime always starts airflow-dag-processor and relies on it to parse/register the ingest DAG. So the issue does not affect the bundled runtime. The remaining question is whether HFlow intends to support external/minimal Airflow deployments where DAG parsing is handled without a separate dag_processor component. If not, the comment in AirflowHealth.healthy should be clarified to describe the bundled/runtime requirement rather than treating the component as optional.
Not supported. HFlow needs the DAG processor to register the ingest DAG, so the check's right and the comment's stale. Want to send a PR fixing the comment?
Thanks for the clarification. You’re right: HFlow requires the DAG processor, and deployments without it are not supported. I created this issue based on the stale comment without first reproducing it in a supported deployment. I’ll close it and leave the runtime behavior unchanged.
Before proceeding, is this distinct case worth fixing separately, or is it covered by the existing numeric-overflow work from #739? I found ManifestPartition("train", 10**400) still leaks OverflowError .
Activity
kstonekuan commentedon Nov 13, 2026
Which Airflow deployment are you using, and does it parse DAGs without a separate DAG processor? HFlow’s bundled runtime starts one and needs it to register the ingest DAG.
harsh21234i commentedon Nov 13, 2026
Thanks for checking. I was not testing a separate production deployment; the report came from reading the REST client contract and reproducing its behavior with a health response that contains healthy
metadatabase/schedulerbut omitsdag_processor.I confirmed that HFlow’s bundled Compose runtime always starts
airflow-dag-processorand relies on it to parse/register the ingest DAG. So the issue does not affect the bundled runtime. The remaining question is whether HFlow intends to support external/minimal Airflow deployments where DAG parsing is handled without a separatedag_processorcomponent. If not, the comment inAirflowHealth.healthyshould be clarified to describe the bundled/runtime requirement rather than treating the component as optional.kstonekuan commentedon Nov 16, 2026
Not supported. HFlow needs the DAG processor to register the ingest DAG, so the check's right and the comment's stale. Want to send a PR fixing the comment?
harsh21234i commentedon Nov 20, 2026
Thanks for the clarification. You’re right: HFlow requires the DAG processor, and deployments without it are not supported. I created this issue based on the stale comment without first reproducing it in a supported deployment. I’ll close it and leave the runtime behavior unchanged.
harsh21234i commentedon Nov 20, 2026
Before proceeding, is this distinct case worth fixing separately, or is it covered by the existing numeric-overflow work from #739? I found ManifestPartition("train", 10**400) still leaks OverflowError .