fix: Cancel HTTP inference requests when the client disconnects

[LiRunGuo]

What does the PR do?

Cancels an in-flight HTTP inference request when the client disconnects, instead of letting it run to completion.

InferRequestClass pauses the evhtp request while the inference runs, and evhtp_connection_pause() disables EV_READ on the connection's bufferevent. Nothing reads the socket during that time, so the peer's FIN/RST is not seen and RequestFiniHook (the only place the HTTP frontend calls TRITONSERVER_InferenceRequestCancel) runs only after the response has been written.

This PR registers a small EV_READ | EV_PERSIST libevent watcher on the connection fd for the lifetime of the request, on the evhtp thread that owns the connection. The callback does a non-consuming recv(..., MSG_PEEK):

• 0 or a socket error: cancel the Triton request and drop the watcher.
• data available (e.g. a pipelined request): drop the watcher, since a disconnect can't be detected without consuming that data. The bytes stay in the socket for evhtp.
• EAGAIN / EWOULDBLOCK / EINTR: keep watching.

The watcher is freed in the request fini hook and in the destructor. evhtp frees the request (and runs the fini hook) before it frees the bufferevent that closes the fd. No libevhtp change is needed. Since the logic lives in InferRequestClass, it covers /infer, /generate, /generate_stream before the first chunk, and the SageMaker / Vertex AI request classes.

Checklist

• I have read the Contribution guidelines and signed the Contributor License
  Agreement (CLA is being submitted)
• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• I ran pre-commit locally (pre-commit install, pre-commit run --all)
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

• fix

Related PRs:

None.

Where should the reviewer start?

DisconnectWatchCallback and StartDisconnectWatch / StopDisconnectWatch in src/http_server.cc, and where they are called from (the InferRequestClass constructor, RequestFiniHook, the destructor in src/http_server.h).

Test plan:

Added qa/L0_request_cancellation/http_cancellation_test.py and a section in that suite's test.sh, using the existing execute_cancel Python model (it polls is_cancelled() every second):

• test_http_infer_client_disconnect: close the socket 2 s into a 10 s /infer request and expect the model to log the cancellation.
• test_http_generate_client_disconnect: same through /generate.
• test_http_infer_client_connected: two requests on one keep-alive connection complete normally and nothing is cancelled.

I built the tritonserver executable from main and from this branch and ran both against the 26.08 container's libtritonserver.so and backends:

Test	main	this PR
test_http_infer_client_disconnect	FAILED (model ran the full 10 s)	OK (cancelled at 2.0 s)
test_http_generate_client_disconnect	FAILED	OK
test_http_infer_client_connected	OK	OK

Server log with the fix:

model.py:83] "[execute_cancel] Request not cancelled at 1.0 s"
http_server.cc:4081] "HTTP client disconnected, cancelling inference request"
model.py:74] "[execute_cancel] Request cancelled at 2.0 s"

I also ran an ad-hoc stress script against the patched build (not part of the PR): 500 connections closed at random points around the response, half with RST; 32 threads x 25 keep-alive requests; two pipelined requests on one socket (neither cancelled, both answered). No crash, all responses correct, clean server exit.

Caveats:

• A client that half-closes (shutdown(SHUT_WR)) after sending the request and then waits for the response is treated as disconnected. This is rare for HTTP/1.1 clients and matches nginx's default (proxy_ignore_client_abort off). I can put the detection behind a server option if you'd prefer to keep the old behavior available.
• My verification was a frontend-only build (gRPC and GPU off) plus the new test section. I did not run the full build.py build, the rest of the L0 suites, or an ASAN build, and the _WIN32 branch has not been compiled. Opening as a draft so CI can cover that.
• Cancellation remains best effort: it depends on the backend checking the cancellation flag.

Background

Long-running requests (LLMs in particular) keep occupying the GPU after an HTTP client times out or is interrupted. gRPC already propagates cancellation; HTTP did not for non-streaming requests. Root cause analysis is in #7077 (comment).

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: tritonserver HTTPServer does not detect EOF/connection close unless response is streaming #7077