feat: add lambda-runtime-invocation-id header - #1159
Conversation
b60f865 to
048c1e8
Compare
4c28d43 to
afb6708
Compare
afb6708 to
de1f2e4
Compare
c403f80 to
31c22d2
Compare
31c22d2 to
2cabdac
Compare
fc33a71 to
1152b7c
Compare
jlizen
left a comment
jlizen
left a comment
There was a problem hiding this comment.
Core approach is good, but some small tweaks.
Also: currently we log Lambda function timeout! for any 410, but we now will have a 410 if a stale response is rejected. We should tweak the message.
| let mut req = build_request().method(Method::POST).uri(uri).body(body)?; | ||
|
|
||
| if let Some(id) = self.invocation_id { | ||
| req.headers_mut().insert(LAMBDA_RUNTIME_INVOCATION_ID, id.parse()?); |
There was a problem hiding this comment.
This will error out and crash the runtime if a malformed invocation id header is sent.
We originally decode with String::from_utf8_lossy(), but that replaces non-utf8 bytes with U+FFFD which is anyway not ascii. So then this parse will fail.
I did a quick check and didn't find any other round trips, this new code is the only place impacted.
I know we control the sender but we should be defensive against malformed inputs anyway. I would suggest a rate-limited log warning if we have bad bytes.
| struct Request { | ||
| #[serde(rename = "command")] | ||
| _command: String, | ||
| sleep: u32, |
There was a problem hiding this comment.
This should have #[serde(default)], otherwise the example command in the header will actually have a deserialization failure.
| @@ -1,2 +1,2 @@ | |||
| """ | |||
| Multi-concurrency test scenarios for basic-lambda-concurrent. | |||
There was a problem hiding this comment.
IE, there are multiple lambda handlers now?
| TIMEOUT = 5 | ||
|
|
||
|
|
||
| def _make_env(concurrency: int = DEFAULT_CONCURRENCY) -> dict: |
There was a problem hiding this comment.
this should accept a handler input too, right now it makes it seem like everything is the basic concurrent
| let mut req = build_request().method(Method::POST).uri(uri).body(body)?; | ||
|
|
||
| if let Some(id) = self.invocation_id { | ||
| req.headers_mut().insert(LAMBDA_RUNTIME_INVOCATION_ID, id.parse()?); |
There was a problem hiding this comment.
nit: we use insert() here but append() on the streaming branch, I would make them consistent
| use tower::{service_fn, Service}; | ||
|
|
||
| #[tokio::test] | ||
| async fn forwards_invocation_id_from_next_response_headers() { |
There was a problem hiding this comment.
It would be good to have a test covering the deserialization error path as well
3 participants
Summary
Add
Lambda-Runtime-Invocation-Idheader support for cross-wiring protection.The RIC now echoes the invocation ID received from RAPID on
/nextback on/responseand/error, enabling RAPID to detect and reject stale responses from timed-out invocations.Problem
On Lambda Managed Instances (LMI) and On-Demand (OD), when an invoke times out, the runtime process continues running in the background. If a new invoke arrives with the same
requestId, RAPID accepts it. The still-running old invocation eventually posts its response, and RAPID matches it to the new invoke — delivering the wrong response (cross-wiring).Solution
RAPID sends a unique per-invoke nonce via
Lambda-Runtime-Invocation-Idheader on/next. The runtime echoes it back on/responseand/error. RAPID validates the match before accepting the response.Backward Compatibility
Fully backward compatible in both directions:
Testing