What happens
The test suite has three backends, in tests/mock_vws/fixtures/vuforia_backends.py:
class VuforiaBackend(Enum):
REAL = "Real Vuforia"
MOCK = "In Memory Mock Vuforia"
DOCKER_IN_MEMORY = "In Memory version of Docker application"DOCKER_IN_MEMORY mounts all three Flask apps into a single responses mock, in one process:
add_flask_app_to_mock(mock_obj=mock, flask_app=VWS_FLASK_APP, ...)
add_flask_app_to_mock(mock_obj=mock, flask_app=CLOUDRECO_FLASK_APP, ...)
add_flask_app_to_mock(mock_obj=mock, flask_app=TARGET_MANAGER_FLASK_APP, ...)
That verifies the Flask handlers. It cannot verify the Flask topology, because in one process TARGET_MANAGER as imported by vws.py is the same object as the one the target manager app serves. In the real Docker deployment they are different objects in different containers.
Real multi-container behaviour is covered by exactly one test, test_build_and_run in tests/mock_vws/test_docker.py. It builds the three images, creates a database, adds one target, waits for processing, and runs one query. That is a build-and-smoke test, and a good one, but it is the whole of the coverage.
Why it matters
Anything which depends on state crossing the container boundary is untested. #3373 is exactly this: Model Target datasets and reco counts reports are stored in the VWS container's own TargetManager rather than in the target manager service, and the rate limiter there is never pruned. Every test passes, because in the single-process backend that distinction does not exist.
The gap is not visible from the test names either. A test parametrised over DOCKER_IN_MEMORY reads as covering the Docker deployment, and reviewers reasonably take it that way.
Endpoints with no real-topology coverage at all: the Model Target Web API, the reco counts report endpoints, VuMark instance generation, rate limiting, and database deletion.
Suggested resolution
Two parts, and the first is cheap.
Rename DOCKER_IN_MEMORY and its description so the name states what it covers. Something like "Flask handlers, single process" rather than "In Memory version of Docker application" would stop the backend being read as a Docker deployment test.
Then extend test_docker.py to cover the endpoints whose behaviour actually depends on the process split. Creating a Model Target dataset in one request and polling its status in another, against real containers, is the case which would have caught #3373; requesting a reco counts report and then downloading it is the same shape. These do not need to be exhaustive — one round trip per stateful feature is enough to pin the invariant that state lives in the target manager service.
test_docker.py is already its own CI matrix entry with a disk space cleanup step, so extra tests there cost wall-clock time in a job which is already paying the image build cost once.
What happens
The test suite has three backends, in
tests/mock_vws/fixtures/vuforia_backends.py:DOCKER_IN_MEMORYmounts all three Flask apps into a singleresponsesmock, in one process:That verifies the Flask handlers. It cannot verify the Flask topology, because in one process
TARGET_MANAGERas imported byvws.pyis the same object as the one the target manager app serves. In the real Docker deployment they are different objects in different containers.Real multi-container behaviour is covered by exactly one test,
test_build_and_runintests/mock_vws/test_docker.py. It builds the three images, creates a database, adds one target, waits for processing, and runs one query. That is a build-and-smoke test, and a good one, but it is the whole of the coverage.Why it matters
Anything which depends on state crossing the container boundary is untested. #3373 is exactly this: Model Target datasets and reco counts reports are stored in the VWS container's own
TargetManagerrather than in the target manager service, and the rate limiter there is never pruned. Every test passes, because in the single-process backend that distinction does not exist.The gap is not visible from the test names either. A test parametrised over
DOCKER_IN_MEMORYreads as covering the Docker deployment, and reviewers reasonably take it that way.Endpoints with no real-topology coverage at all: the Model Target Web API, the reco counts report endpoints, VuMark instance generation, rate limiting, and database deletion.
Suggested resolution
Two parts, and the first is cheap.
Rename
DOCKER_IN_MEMORYand its description so the name states what it covers. Something like "Flask handlers, single process" rather than "In Memory version of Docker application" would stop the backend being read as a Docker deployment test.Then extend
test_docker.pyto cover the endpoints whose behaviour actually depends on the process split. Creating a Model Target dataset in one request and polling its status in another, against real containers, is the case which would have caught #3373; requesting a reco counts report and then downloading it is the same shape. These do not need to be exhaustive — one round trip per stateful feature is enough to pin the invariant that state lives in the target manager service.test_docker.pyis already its own CI matrix entry with a disk space cleanup step, so extra tests there cost wall-clock time in a job which is already paying the image build cost once.