Fail wait_for_job if a Pathways worker pod failed

[MarcosAsh]

wait_for_job returned "success" as soon as the leader pod reached Succeeded (or container exit code 0), even when a worker pod in the LWS was in Failed state, so real multi-host failures surfaced as successful jobs. Before returning success on any of the three success paths (phase Succeeded, current container exit 0, restarted last-state exit 0), wait_for_job now scans the LWS pods and raises via the existing failure-details collector if any non-leader pod is Failed. A list_namespaced_pod ApiException falls back to the leader status so a transient API blip doesn't manufacture a false failure.

Fixes #239

[gemini-code-assist]

Code Review

This pull request enhances the wait_for_job logic in the Pathways client to ensure that worker pod failures are not masked by a successful leader pod status. It introduces helper functions to identify failed worker pods and raise detailed errors, accompanied by comprehensive regression tests. Review feedback suggests extending this logic to get_job_status for consistency across the API and implementing a "fail-fast" mechanism by checking worker status during each polling iteration rather than only upon leader completion.

[gemini-code-assist]

This check correctly identifies failed workers when the leader pod completes. However, for better resource efficiency, consider performing this worker failure check in every iteration of the main polling loop (e.g., by using list_namespaced_pod to fetch all pods at once). This would enable "fail-fast" behavior, terminating the wait immediately if a worker crashes, rather than waiting for the leader to finish or the timeout to expire.

References

1. Your goal is to critically test the logic. Actively search for and point out failing edge cases, race conditions, or unhandled exceptions in the implementation. ^(link)

[MarcosAsh]

I think this is out of scope for this pr

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@6cfd0fb). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #240   +/-   ##
=======================================
  Coverage        ?   91.64%           
=======================================
  Files           ?       82           
  Lines           ?    12020           
  Branches        ?        0           
=======================================
  Hits            ?    11016           
  Misses          ?     1004           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[divyashreepathihalli]

@MarcosAsh have you verified the bug and the fix on your cloud setup?

[MarcosAsh]

Yeah, verified it on a real GKE cluster. I recreated the #239 state directly (leader pod Succeeded, worker pod Failed, both with the job-name label LWS sets) and ran the actual wait_for_job against it. On main it returns "success". With this change it raises RuntimeError: ... worker pod(s) keras-pathways-j1-1 failed. Healthy case (all workers Succeeded) still returns success, so no false alarms.