[CI - FAILURE]: / Misc.StatelessResetKey (msquictest.exe)

[github-actions]

https://github.com/microsoft/msquic/actions/runs/33412446003

[guhetier]

AI Generated analysis

This is a flaky kernel-test race, not a product failure in stateless-reset generation. In BVT run 33412446003, job 99557758986, Misc.StatelessResetKey rotates the global reset key and assumes the connected client will continue using a CID whose reset token was advertised before rotation.

The trace shows server-side partition migration concurrently generating replacement CIDs after the key update. Packet 195 advertises CID 61d3588808777fad84 with token 0b814163565b28d3808c797b34371102; the client installs and uses that CID. After silent server shutdown, the closed endpoint generates a stateless reset containing the identical token. The client correctly recognizes it and reports QUIC_STATUS_ABORTED, while the test expected QUIC_STATUS_CONNECTION_TIMEOUT.

The causal chain is: arbitrary sleep completes -> reset key rotates -> partition migration regenerates CIDs -> a replacement CID receives a token derived from the new key -> the client selects it -> shutdown generates a matching reset -> the client correctly accepts the reset -> the test assertion fails.

Alternative explanations were eliminated: SET_PARAM succeeded; the token was byte-for-byte identical across advertisement and reset, ruling out a stale partition key or lookup collision; queue delays were only 0-14 microseconds with no overload rejection; and no watchdog or certificate stall occurred. The test passed in the immediately preceding and subsequent comparable runs, confirming intermittency.

Root cause: the test races key rotation/shutdown against asynchronous CID advertisement and partition migration, while assuming every usable client CID retains a pre-rotation token. The core reset behavior is correct.

Confidence: High.

Design proposal

1. In src/test/lib/DataTest.cpp::QuicTestStatelessResetKey, create the registration with QUIC_EXECUTION_PROFILE_TYPE_SCAVENGER so unrelated partition migration cannot alter the test invariant.
2. Replace CxPlatSleep(15) with Client->ForceCidUpdate(). Record GetDestCidUpdateCount() first and assert that it advances, proving the client selected a CID/token advertised before rotation.
3. Rotate the key only after that synchronization point, then silently shut down the server and require the pre-rotation CID case to time out.
4. Add a complementary case that establishes a connection/token after rotation and verifies that its reset is accepted as QUIC_STATUS_ABORTED.
5. Clarify the QUIC_PARAM_GLOBAL_STATELESS_RESET_KEY documentation: rotation affects all future token generation, including later NEW_CONNECTION_ID frames on existing connections.
6. Validate with high-repeat kernel and user-mode runs, including multi-core/RSS configurations. Increasing the sleep or adding workflow retries would only mask the race.