[larry-dalmeida]

Summary of changes

Preserve eligible stale responses without extending SIE expiry, isolate Vary variants, prevent standalone 304 responses from overwriting cached bodies, and skip superseded queued refreshes.

These changes affect refresh, failure and variant-selection behavior.

How to Verify

Shared setup

mkdir skipper-review && cd skipper-review
git clone https://github.com/zalando/skipper.git skipper && cd skipper
for PR in 4369 4370 4371 4372 4384 4385 4386 4387; do
  git fetch origin pull/$PR/head:pr-$PR
  git worktree add ../skipper-pr-$PR pr-$PR
done
cd ..
git clone https://github.com/larry-dalmeida/skipper-cache-e2e-tests.git
cd skipper-cache-e2e-tests
make verify SKIPPER_CHECKOUT=../skipper-pr-4387

make verify starts or reuses Minikube, the mock origin and the stack; runs the smoke and coalescing checks; and validates supported dashboard queries. It prints a report path and exits non-zero on failure. For outage evidence, run make chaos SKIPPER_CHECKOUT=../skipper-pr-4387 NAME=origin-stale; it includes its own timed load. Requirements: Python 3.9+, Make, Bash, Git, curl, kubectl, Minikube, Docker and Go compatible with Skipper's go.mod. Free ports: 8080, 8081, 6379, 4318, 9091, 9999, 9911 (8089 for endless load). Dynamic URLs: .runtime/urls.txt; stop with make stop.

Validated on #4387 (d767f636): smoke: 6,398 requests, 0 failures; origin-stale: 155 failed revalidations across 77 routes. Smoke metric checks: 42 PASS, 2 fault-only NOT EXERCISED, 0 FAIL.

Focused tests

go -C ../skipper-pr-4371 test -race -count=1 ./filters/cache -run '^(TestCacheFilter_SIE_RetentionAfterProcessAndVolatileL2Restart|TestProxy_CacheVary304PreservesBodyMetadata|TestCacheFilter_QueuedRefresh|TestCacheFilter_SIE_WindowExpiresDuringFetch|TestCacheFilter_SIE_FailureClassesPreserveStoredResponse|TestCacheFilter_SIE_EligibilityAndFailureBoundaries|TestCacheFilter_Vary_RevalidationPreservesVariantKey|TestProxy_CacheableVary_ConcurrentColdAndWarmIsolation)$'

Expected: 8 focused tests pass, covering SIE retention, queued refresh, Vary isolation and 304 metadata. origin-stale shows 503s and eventual recovery. hard/* and swr/* share cache("30s","10s","15s") with SIE disabled; 503s may replace good entries with 10-second cached errors, so these are not an SIE comparison.

Checklist

• Added regression tests and documentation.
• Confirm CI passes on the current head.
• Confirm passess on manual run of e2e tests for all known scenarios including past bugs (results: larry-dalmeida/skipper-cache-e2e-tests/results/README.md)

Rollback Plan

Revert #4372 first if merged, then this PR. No cache data-format migration is required.

References & Additional Context

Depends on #4370; followed by #4372. Real 24-hour SIE exhaustion remains unverified.

E2E Testing Results

E2E runs exercising refresh and origin failure. Full results: larry-dalmeida/skipper-cache-e2e-tests/results

Test Scenarios:

Scenario	Build	Requests	Failures	Status
Origin Stale	bda674b (top of stack)	50,870	33.3% (503s while origin was failing)	✓ Pass
Smoke	8a80d62e + local changes	6,270	0%	✓ Pass
Origin Stale	8a80d62e + local changes	44,288	26.7% (503s while origin was failing)	✓ Pass
Valkey Kill	8a80d62e + local changes	47,653	0%	✓ Pass

Validation Panels:

• Hit/Miss/Stale Rate — cache behavior while the origin returned 503 (bda674b origin-stale): hits stay high (up to ~184 req/s), misses rise to ~28 req/s, and stale serves drop to ~0 during the failure, then recover
  

• Origin Responses by Status — the same window at the mock origin: 503s peak at ~25 req/s, 200s fall to ~2 req/s and 304s to 0, then both resume once the origin recovers
  

Not shown by these runs: the run recorded no refresh errors (all 1,554 background refreshes succeeded), so the Refresh Errors panel stays empty. No panel shows a stale entry being kept when a refresh fails, or which variant was selected.

[zalando-robot]

Docker image "registry-write.opensource.zalan.do/teapot/skipper:e003f9a39de77e79e64be13be264d1099a2555e7" is not based on an approved base image. Any production deployment relying on this image will be blocked.

To create a compliant Docker image of your application, you must reference an allowed Docker image as its base image in your Dockerfile. This base image must come from the Zalando Container Registry namespace library and use a recommended version as listed in the documentation.

[larry-dalmeida]

Superseded by #4394. Changes consolidated into five focused commits with published e2e evidence.