From de975846fae62d70190e14dd1580fb02e5bebc95 Mon Sep 17 00:00:00 2001 From: Joe Lockwood Date: Tue, 12 May 2026 19:04:34 -0400 Subject: [PATCH] fix(libcamera): publish mid-exposure timestamp SensorTimestamp marks start-of-exposure. The scene a detector integrates over spans [SOE, SOE+exposure], so mid-exposure is the closest single-instant approximation. Add exposure/2 to the published timestamp using the new getFrameExposureTimeUs JNI call; falls back to SOE when the driver returns 0. Depends on PhotonVision/photon-libcamera-gl-driver#33. --- .../contributing/design-descriptions/e2e-latency.md | 2 ++ .../frame/provider/LibcameraGpuFrameProvider.java | 10 +++++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/source/docs/contributing/design-descriptions/e2e-latency.md b/docs/source/docs/contributing/design-descriptions/e2e-latency.md index 7d81b59b44..e1b51cf9fd 100644 --- a/docs/source/docs/contributing/design-descriptions/e2e-latency.md +++ b/docs/source/docs/contributing/design-descriptions/e2e-latency.md @@ -37,6 +37,8 @@ I'm sure that we'll find a camera that doesn't play nice, because we can't have Other things to note: This gets us an estimate at when the camera *started* collecting photons. The camera's sensor will remain collecting light for up to the total integration time, plus readout time for rolling shutter cameras. +For the libcamera capture path, PhotonVision now applies a mid-exposure correction using the per-frame `ExposureTime` metadata from libcamera. The published timestamp is the midpoint of the integration window for global-shutter sensors. Rolling-shutter sensors have a residual row-dependent bias not yet corrected. If libcamera does not populate `ExposureTime` for a given frame, the driver returns 0 and the correction is skipped (behaviour matches the pre-correction code path). + ## Latency Testing Here, I've got a RoboRIO with an LED, an Orange Pi 5, and a network switch on a test bench. The LED is assumed to turn on basically instantly once we apply current, and based on DMA testing, the total time to switch a digital output on is on the order of 10uS. The RoboRIO is running a TimeSync Server, and the Orange Pi is running a TimeSync Client. diff --git a/photon-core/src/main/java/org/photonvision/vision/frame/provider/LibcameraGpuFrameProvider.java b/photon-core/src/main/java/org/photonvision/vision/frame/provider/LibcameraGpuFrameProvider.java index fe7d48e6f1..511eefebbc 100644 --- a/photon-core/src/main/java/org/photonvision/vision/frame/provider/LibcameraGpuFrameProvider.java +++ b/photon-core/src/main/java/org/photonvision/vision/frame/provider/LibcameraGpuFrameProvider.java @@ -85,6 +85,7 @@ public Frame get() { var now = LibCameraJNI.getLibcameraTimestamp(); var capture = LibCameraJNI.getFrameCaptureTime(p_ptr); + var exposureUs = LibCameraJNI.getFrameExposureTimeUs(p_ptr); var latency = (now - capture); LibCameraJNI.releasePair(p_ptr); @@ -92,12 +93,19 @@ public Frame get() { // Know frame is good -- increment sequence ++sequenceID; + // libcamera's SensorTimestamp marks start-of-exposure. The scene the + // detector integrates over spans [SOE, SOE+exposure], so mid-exposure + // is the closest single-instant approximation. When the driver + // returns 0, ExposureTime metadata was unavailable for this frame + // and we leave the timestamp uncorrected. + long midExposureCorrectionNs = exposureUs > 0 ? (exposureUs * 1000L) / 2L : 0L; + return new Frame( sequenceID, colorMat, processedMat, type, - MathUtils.wpiNanoTime() - latency, + MathUtils.wpiNanoTime() - latency + midExposureCorrectionNs, settables.getFrameStaticProperties().rotate(settables.getRotation())); } }