Poll: Should we poll succeeded, failed, expired tasks?

[Scott-Owen-James]

Links to #7123, further discussion, see initial discussion there.

[oliver-sanders]

Should we be able to poll succeeded, failed, submit-failed and expired tasks?

Copying this message from #7123

submit-failed

• Can be caused by submission failure.
• But otherwise, this would have to be detected by a poll, so probably not a lot of point in allowing the job to be re-polled.

failed & succeeded

• Final output has been received either via direct message or poll, so subsequent poll shouldn't be necessary.
• Except that we do have an issue because our message system does not provide guaranteed receipt, if the "final-status" event arrived via message (rather than poll) we could be missing prior events if their messages failed.
• So there is a legit use case for polling these jobs until
  message: network glitches can cause custom outputs to be missed #7257 is resolved.

expired

• We could, potentially, be missing events from expired tasks?
• But we probably wouldn't expect/want these to be received/processed after the task has expired?

Then there's the awkward situation where the task status has been manually set...

[hjoliver]

expired ...

Expiry is a scheduler- (or user, via the scheduler-) defined thing. So if the scheduler thinks that a task has expired, it has by definition expired, and there's no need to poll it to see if really is not expired and hence change its status. And of course polling a task that expired before running (which is the usual way to expire) can't be done at all. But are are we also worried about detecting missed messages from a task that was running before it expired?? 🤯

failed & succeeded ...

What about the original referenced use case (which I confess to being a bit hazy on a that point):

tests that detect a change of state by polling but do not rely on jobs quickly disappearing from the batch system listing require the ability to poll non-active tasks.

[oliver-sanders]

But are are we also worried about detecting missed messages from a task that was running before it expired?? 🤯

That is the question!

This can happen with manual expiry, if we don't poll, the task state remains correct, but the job, less so. This is the same situation as orphaning a job by resetting the task to succeeded/failed.

[hjoliver]

Seems to me we should in fact allow polling of any task that has a job ID - but not succeeded tasks unless specifically targeted (because polling ALL succeeded tasks would be a bad idea, obviously).