diff --git a/.github/skills/testing.md b/.github/skills/testing.md index a7a46bf646..985471bddb 100644 --- a/.github/skills/testing.md +++ b/.github/skills/testing.md @@ -13,9 +13,14 @@ ## Test Framework -- librealsense uses a **custom Python-based test framework** -- The test orchestrator is `unit-tests/run-unit-tests.py` -- Tests must be run **from the `unit-tests/` directory** +Two frameworks, split by test language: + +- **C++ tests** (`test-*.cpp`) are orchestrated by the legacy runner `unit-tests/run-unit-tests.py`. + Keeping that runner for C++ testing is its only remaining purpose. +- **Python tests** (`pytest-*.py`) are pytest only. See `.github/skills/pytest-infra.md`. + No legacy `test-*.py` scripts remain, and `run-unit-tests.py` never collects `pytest-*.py`. + +Either way, run **from the `unit-tests/` directory**. ## Test Categories @@ -31,7 +36,21 @@ | `unit-tests/log/` | Logging | Logging infrastructure tests | | `unit-tests/types/` | Types | Type system tests | -## Running All Tests +## Running the pytest Tests + +```bash +cd unit-tests +python3 -m pytest -v # all collectable tests +python3 -m pytest -v live/frames # one directory +python3 -m pytest -v pytest-fw-update.py # one file +python3 -m pytest -v -k "hdr and not preset" # filter by test name +``` + +Flags are registered in `conftest.py`: `--live`, `--not-live`, `--device`, `--exclude-device`, +`--context`, `--tag`, `--repeat`, `--reruns`, `--debug`, `--rslog`, `--test-dir`. Note that `-s` +disables the per-test log files. See `.github/skills/pytest-infra.md` for fixtures and markers. + +## Running the C++ Tests (legacy runner) Navigate to the `unit-tests/` directory and run: @@ -57,21 +76,23 @@ For full usage and all available flags: python3 run-unit-tests.py --help ``` -## Running Specific Tests +## Running Specific C++ Tests + +The flags below belong to `run-unit-tests.py`, so they select among the C++ tests. +For pytest use `-k`, `-m` and the flags listed above. ### By Name (Regex) Use `-r` / `--regex` to run tests whose names match a regular expression: ```bash -python3 run-unit-tests.py -s -r "test-hdr" -python3 run-unit-tests.py -s -r "test-metadata" -python3 run-unit-tests.py -s --regex "test-stream.*" +python3 run-unit-tests.py -s -r "test-log-vs-LOG" +python3 run-unit-tests.py -s --regex "rsutils-string.*" ``` **Test name derivation**: the orchestrator builds a test's name from its path relative to `unit-tests/`, replacing directory separators with `-` and stripping the leading `test-` from the filename. For example: -- `live/hw-reset/test-stress.py` → `test-live-hw-reset-stress` -- `func/test-hdr.py` → `test-func-hdr` +- `rsutils/string/test-hexarray.cpp` becomes `test-rsutils-string-hexarray` +- `log/test-vs-LOG-shared.cpp` becomes `test-log-vs-LOG-shared` So when using `-r`, omit the `test-` filename prefix and join subdirectories with `-`. @@ -80,18 +101,18 @@ So when using `-r`, omit the `test-` filename prefix and join subdirectories wit Use `--skip-regex` to exclude tests whose names match: ```bash -python3 run-unit-tests.py -s --skip-regex "test-fw-update" +python3 run-unit-tests.py -s --skip-regex "test-rsutils-string-hexarray" ``` ### By Tag -Use `-t` / `--tag` to run tests with a specific tag. Tags are assigned automatically based on: -- File type: `exe` (C++ binaries) or `py` (Python scripts) +Use `-t` / `--tag` to run tests with a specific tag. Tags come from: +- File type: `exe` (C++ binaries). The `py` tag matches nothing now that no legacy `test-*.py` remain. - Directory location: e.g., tests in `unit-tests/live/` get the `live` tag +- An explicit `//#test:tag ` directive in the source file ```bash python3 run-unit-tests.py -s -t live # run only live tests -python3 run-unit-tests.py -s -t py # run only Python tests python3 run-unit-tests.py -s -t exe # run only compiled C++ tests python3 run-unit-tests.py -s -t live -t exe # run tests that have BOTH tags ``` @@ -136,31 +157,34 @@ python3 run-unit-tests.py --rslog # enable LibRS debug logging in tests python3 run-unit-tests.py --debug # enable framework debug output; also prints "test took X seconds" per test ``` -## Running Nightly-Only Tests +## Running Nightly-Only and Weekly Tests -Some tests are marked `# test:donotrun:!nightly` and are **skipped by default**. Pass `--context nightly` to enable them: +Both runners take `--context`, which accepts a **space-separated list**. + +pytest tests gated with `@pytest.mark.context("nightly")` are skipped unless the context is passed: ```bash -python3 run-unit-tests.py --context nightly -r hw-reset-stress ../build/Release +python3 -m pytest -v live/hw-reset/pytest-stress.py --context nightly ``` -## Running Weekly Tests - -Weekly tests use a higher iteration count / longer timeout (controlled by `'weekly' in test.context` inside the test). The `--context` flag accepts a **space-separated list**, so to run a nightly-guarded test with weekly behaviour pass **both** contexts: +A test that also scales its iteration count or timeout for weekly runs reads the context through the +`test_context_var` fixture, so pass both words to get nightly collection plus weekly behaviour: ```bash -# 'nightly' satisfies the test:donotrun:!nightly guard -# 'weekly' activates higher iteration counts and longer timeouts inside the test -python3 run-unit-tests.py --context "nightly weekly" -r hw-reset-stress ../build/Release +python3 -m pytest -v live/hw-reset/pytest-stress.py --context "nightly weekly" ``` -Passing `--context weekly` alone is **not sufficient** — the `test:donotrun:!nightly` directive will still filter the test out. +C++ tests use a `//#test:donotrun:` directive in the source file, evaluated by +`run-unit-tests.py` against the same `--context` list. ## Repeating and Retrying ```bash python3 run-unit-tests.py --repeat 3 # repeat each test 3 times python3 run-unit-tests.py --retry 2 # retry failed tests up to 2 times + +python3 -m pytest --repeat 3 # pytest: repeat each file's tests 3 times +python3 -m pytest --reruns 2 # pytest: retry a failed test up to 2 times ``` ## Recording and Playback (Mock Hardware) @@ -185,7 +209,7 @@ This is useful for: The `UNIT_TESTS_ARGS` CMake variable passes arguments to `unit-test-config.py` during configuration: ```bash -cmake .. -DBUILD_UNIT_TESTS=ON -DUNIT_TESTS_ARGS="-t live -r test-streaming" +cmake .. -DBUILD_UNIT_TESTS=ON -DUNIT_TESTS_ARGS="-t live -r rsutils-string.*" ``` ## Using a Custom Test Directory @@ -196,17 +220,16 @@ python3 run-unit-tests.py --test-dir /path/to/custom/tests ## Custom Firmware for Testing -The SDK no longer ships a bundled firmware blob, so `test-fw-update` **requires** a custom firmware path for the device under test. Without one it logs a warning and skips. Download a signed `.bin` from , then: +The SDK no longer ships a bundled firmware blob, so `pytest-fw-update` **requires** a custom firmware path for the device under test. Without one it skips. Download a signed `.bin` from , then: ```bash -python3 run-unit-tests.py --custom-fw-d400 /path/to/firmware.bin -python3 run-unit-tests.py --custom-fw-d555 /path/to/firmware.bin -python3 run-unit-tests.py --custom-fw-d585 /path/to/firmware.bin +python3 -m pytest pytest-fw-update.py --custom-fw-d400 /path/to/firmware.bin +python3 -m pytest pytest-fw-update.py --custom-fw-d555 /path/to/firmware.bin +python3 -m pytest pytest-fw-update.py --custom-fw-d585 /path/to/firmware.bin ``` -`--custom-fw-d585` is only ever flashed onto a device whose name contains "D585" but not "D585S" -(e.g. "D585 Prototype") -- the safety SKU D585S is never updated with it, even though the -`#test:device each(D585)` directive dispatches the test for both (D585S just logs a skip). +`--custom-fw-d585` targets the non-safety D585 only (e.g. "D585 Prototype"). The safety SKU is kept +out of the test by `pytest.mark.device_exclude("D585S")`, so it is never collected and never flashed. ## Troubleshooting diff --git a/.github/workflows/buildsCI.yaml b/.github/workflows/buildsCI.yaml index 5bd7a89dbd..e76cbb7eb1 100644 --- a/.github/workflows/buildsCI.yaml +++ b/.github/workflows/buildsCI.yaml @@ -36,7 +36,7 @@ jobs: #-------------------------------------------------------------------------------- - Win_SH_EX_CfU: # Windows, shared, with Examples & Tools, and Check for Updates + Win_SH_EX_CfU_Stats: # Windows, shared, with Examples & Tools, Check for Updates, and RUM stats (ENABLE_STATS) runs-on: windows-2025 timeout-minutes: 60 steps: @@ -68,7 +68,7 @@ jobs: cd ${{env.WIN_BUILD_DIR}} pwd ls - cmake ${LRS_SRC_DIR} -A x64 -DBUILD_SHARED_LIBS=true -DBUILD_EXAMPLES=true -DBUILD_TOOLS=true -DCHECK_FOR_UPDATES=true + cmake ${LRS_SRC_DIR} -A x64 -DBUILD_SHARED_LIBS=true -DBUILD_EXAMPLES=true -DBUILD_TOOLS=true -DCHECK_FOR_UPDATES=true -DENABLE_STATS=ON - name: Build # Build your program with the given configuration @@ -368,7 +368,7 @@ jobs: #-------------------------------------------------------------------------------- - U22_SH_Py_CI: # Ubuntu 2020, Shared, Python, LibCI with executables + U22_SH_Py_CI: # Ubuntu 2020, Shared, Python, LibCI with executables, and RUM stats (ENABLE_STATS) runs-on: ubuntu-22.04 timeout-minutes: 60 steps: @@ -397,7 +397,7 @@ jobs: shell: bash run: | cd build - cmake .. -DCMAKE_BUILD_TYPE=${{env.LRS_RUN_CONFIG}} -DBUILD_SHARED_LIBS=true -DBUILD_EXAMPLES=false -DBUILD_TOOLS=true -DBUILD_UNIT_TESTS=true -DUNIT_TESTS_ARGS="--not-live --context=linux" -DCHECK_FOR_UPDATES=false -DBUILD_WITH_DDS=false -DBUILD_PYTHON_BINDINGS=true -DPYTHON_EXECUTABLE=$(which python3) + cmake .. -DCMAKE_BUILD_TYPE=${{env.LRS_RUN_CONFIG}} -DBUILD_SHARED_LIBS=true -DBUILD_EXAMPLES=false -DBUILD_TOOLS=true -DBUILD_UNIT_TESTS=true -DUNIT_TESTS_ARGS="--not-live --context=linux" -DCHECK_FOR_UPDATES=false -DBUILD_WITH_DDS=false -DBUILD_PYTHON_BINDINGS=true -DPYTHON_EXECUTABLE=$(which python3) -DENABLE_STATS=ON cmake --build . -- -j4 - name: LibCI @@ -413,8 +413,8 @@ jobs: python3 -m pytest unit-tests/ --color=no --debug -s --not-live --context "linux" #-------------------------------------------------------------------------------- - U24_SH_Py_CI_SYS_JSON: # Ubuntu 24.04, Shared, Python, LibCI with executables and system provided nlohmann_json library - runs-on: ubuntu-24.04 + U26_SH_Py_CI_SYS_JSON: # Ubuntu 26.04, Shared, Python, LibCI with executables and system provided nlohmann_json library + runs-on: ubuntu-26.04 timeout-minutes: 60 steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd #v6.0.2 diff --git a/.github/workflows/pre-release.yml b/.github/workflows/pre-release.yml index 95fc90715c..1fdcae793f 100644 --- a/.github/workflows/pre-release.yml +++ b/.github/workflows/pre-release.yml @@ -58,6 +58,10 @@ jobs: ROS_DISTRO: ${{ matrix.ros_distro }} PRERELEASE: true BASEDIR: ${{ github.workspace }}/.work + # industrial_ci defaults the prerelease host container to ros:noetic-ros-core + # (focal, EOL). Must stay a ros:* image — setting DOCKER_IMAGE disables + # industrial_ci's ROS apt setup, so colcon must be installable already. + DOCKER_IMAGE: ros:jazzy-ros-core steps: - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 #v4 diff --git a/CMake/external_libcurl.cmake b/CMake/external_libcurl.cmake index cfc66383b5..10c8ca126c 100644 --- a/CMake/external_libcurl.cmake +++ b/CMake/external_libcurl.cmake @@ -1,4 +1,4 @@ -if(CHECK_FOR_UPDATES) +if(CHECK_FOR_UPDATES OR ENABLE_STATS) pop_security_flags() # remove security flags @@ -65,4 +65,4 @@ if(CHECK_FOR_UPDATES) endif() push_security_flags() -endif() #CHECK_FOR_UPDATES +endif() #CHECK_FOR_UPDATES OR ENABLE_STATS diff --git a/CMake/global_config.cmake b/CMake/global_config.cmake index 35abedbdd1..f309717f7f 100644 --- a/CMake/global_config.cmake +++ b/CMake/global_config.cmake @@ -70,6 +70,10 @@ macro(global_set_flags) add_definitions(-DBUILD_SHARED_LIBS) endif() + if (ENABLE_STATS) + add_definitions(-DENABLE_STATS) + endif() + if (BUILD_WITH_CUDA) include(CMake/cuda_config.cmake) endif() @@ -87,10 +91,17 @@ macro(global_set_flags) message(STATUS "CHECK_FOR_UPDATES depends on BUILD_GRAPHICAL_EXAMPLES flag, turning it off..") set(CHECK_FOR_UPDATES false) else() - include(CMake/external_libcurl.cmake) add_definitions(-DCHECK_FOR_UPDATES) endif() endif() + + # libcurl is needed by sw-update (CHECK_FOR_UPDATES) and RUM cloud upload (ENABLE_STATS). + # BUILD_WITH_LIBCURL is the derived "curl is linked" guard - gates the shared "Online Services" + # viewer tab that hosts both features. + if(CHECK_FOR_UPDATES OR ENABLE_STATS) + include(CMake/external_libcurl.cmake) + add_definitions(-DBUILD_WITH_LIBCURL) + endif() add_definitions(-D${BACKEND} -DUNICODE) endmacro() diff --git a/CMake/lrs_options.cmake b/CMake/lrs_options.cmake index a45a0deccb..7cfbe724bb 100644 --- a/CMake/lrs_options.cmake +++ b/CMake/lrs_options.cmake @@ -68,4 +68,5 @@ option(USE_EXTERNAL_LZ4 "Use externally build LZ4 library instead of building an option(USE_EXTERNAL_NLOHMANN_JSON "Use an externally built nlohmann-json development package instead of downloading it as part of this build" OFF) option(BUILD_ASAN "Enable AddressSanitizer" OFF) option(BUILD_ROSBAG2 "Build and use rosbag2 recording system" ON) # temporary flag, should be removed when deprecated ROSBAG1 recording system is removed +option(ENABLE_STATS "Enable RUM (Real User Monitoring) usage-statistics collection" OFF) mark_as_advanced(BUILD_ASAN) diff --git a/CMake/unix_config.cmake b/CMake/unix_config.cmake index f85c1f5080..086809b1af 100644 --- a/CMake/unix_config.cmake +++ b/CMake/unix_config.cmake @@ -18,6 +18,11 @@ macro(os_set_flags) set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pedantic -Wno-missing-field-initializers") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-switch -Wno-multichar -Wsequence-point -Wformat -Wformat-security") + # Default -ffp-contract=fast fuses a*b+c*d into an FMA wherever the ISA has one (x86-64-v3 + # on Ubuntu 26.04, always on aarch64), dropping a rounding and shifting filter output 1 LSB. + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -ffp-contract=off") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -ffp-contract=off") + execute_process(COMMAND ${CMAKE_C_COMPILER} -dumpmachine OUTPUT_VARIABLE MACHINE) if(${MACHINE} MATCHES "arm64-*" OR ${MACHINE} MATCHES "aarch64-*") if(BUILD_WITH_NEON) diff --git a/common/CMakeLists.txt b/common/CMakeLists.txt index b34fddedc1..2dc4339375 100644 --- a/common/CMakeLists.txt +++ b/common/CMakeLists.txt @@ -79,9 +79,15 @@ set(COMMON_SRC "${CMAKE_CURRENT_LIST_DIR}/textual-icons.h" ) +set(HTTP_FILES + "${CMAKE_CURRENT_LIST_DIR}/http/curl-wrapper.h" + "${CMAKE_CURRENT_LIST_DIR}/http/curl-wrapper.cpp" + "${CMAKE_CURRENT_LIST_DIR}/http/http-downloader.h" + "${CMAKE_CURRENT_LIST_DIR}/http/http-downloader.cpp" + "${CMAKE_CURRENT_LIST_DIR}/http/http-uploader.h" + ) + set(SW_UPDATE_FILES - "${CMAKE_CURRENT_LIST_DIR}/sw-update/http-downloader.h" - "${CMAKE_CURRENT_LIST_DIR}/sw-update/http-downloader.cpp" "${CMAKE_CURRENT_LIST_DIR}/sw-update/dev-updates-profile.h" "${CMAKE_CURRENT_LIST_DIR}/sw-update/dev-updates-profile.cpp" "${CMAKE_CURRENT_LIST_DIR}/sw-update/versions-db-manager.h" @@ -98,10 +104,17 @@ set(UTILITIES_FILES "${CMAKE_CURRENT_LIST_DIR}/utilities/imgui/wrap.cpp" ) +set(RUM_UPLOADER_FILES + "${CMAKE_CURRENT_LIST_DIR}/rum-uploader/rum-uploader.h" + "${CMAKE_CURRENT_LIST_DIR}/rum-uploader/rum-uploader.cpp" + ) + set(COMMON_SRC ${COMMON_SRC} + ${HTTP_FILES} ${SW_UPDATE_FILES} ${REFLECTIVITY_FILES} ${UTILITIES_FILES} + ${RUM_UPLOADER_FILES} ) diff --git a/common/d500-on-chip-calib.cpp b/common/d500-on-chip-calib.cpp index bb12873573..353042f30e 100644 --- a/common/d500-on-chip-calib.cpp +++ b/common/d500-on-chip-calib.cpp @@ -170,8 +170,8 @@ namespace rs2 bool d500_on_chip_calib_manager::uses_interactive_triggered_calibration() const { - // Mirrors ds::d5x5_interactive_triggered_calibration_pids in src/ds/d500/d500-private.h — - // the viewer cannot include SDK-internal headers. Keep the two lists in sync when adding new PIDs. + // Mirrors ds::d5x5_family_pids in src/ds/d500/d500-private.h — the viewer cannot include + // SDK-internal headers. Keep the two lists in sync when adding new PIDs. static const std::set< std::string > interactive_triggered_calibration_pids = { "0C01", "0C02", "0C03", "0C04", "0C05", "0C06", "0C07", "0C08" }; diff --git a/common/device-model.cpp b/common/device-model.cpp index b4f6f8fea3..9eefc774f4 100644 --- a/common/device-model.cpp +++ b/common/device-model.cpp @@ -1337,22 +1337,35 @@ namespace rs2 } } - // PID toggle between Dual-RGB (2C) and Dedicated-RGB (3C) variants: - // D535: 0x0C01 <-> 0x0C02 - // D585: 0x0C04 <-> 0x0C05 - // D585 Proto: 0x0C07 <-> 0x0C08 - if (dev.supports(RS2_CAMERA_INFO_PRODUCT_ID) && dev.is()) - { - static constexpr uint32_t MWD_OPCODE = 0x02U; - static constexpr uint32_t MODE_REG_START_ADDR = 0x80000064U; - static constexpr uint32_t MODE_REG_END_ADDR = 0x80000068U; - static constexpr uint32_t MODE_DEDICATED_RGB = 0U; - static constexpr uint32_t MODE_DUAL_RGB = 1U; - - std::string current_pid = dev.get_info(RS2_CAMERA_INFO_PRODUCT_ID); - const bool is_dual_rgb = (current_pid == "0C01") || (current_pid == "0C04") || (current_pid == "0C07"); - const bool is_dedicated_rgb = (current_pid == "0C02") || (current_pid == "0C05") || (current_pid == "0C08"); - if (is_dual_rgb || is_dedicated_rgb) + // Dual-RGB (2C) / Dedicated-RGB (3C) toggle for D5x5 SKUs whose FW exposes + // depth_xu 0x12 (DUAL_RGB_MODE). Backed by RS2_OPTION_SENSORS_CONFIG_MODE on + // the depth sensor; the option's set() writes the XU and triggers + // hardware_reset internally, so the device re-enumerates under the new PID. + std::shared_ptr depth_sub; + for (auto& sub : subdevices) + { + if (sub->s->is()) + { + depth_sub = sub; + break; + } + } + if (depth_sub) + { + // Read the CACHED option value populated by subdevice_model's periodic + // option-value poll, not a fresh FW round-trip: the "more" popup redraws + // every ImGui frame while open, and calling get_option here would spam + // the FW with XU reads at the render rate. + bool is_dual_rgb = false; + bool can_query = false; + auto opt_it = depth_sub->options_metadata.find(RS2_OPTION_SENSORS_CONFIG_MODE); + if (opt_it != depth_sub->options_metadata.end()) + { + is_dual_rgb = opt_it->second.value_as_float() != 0.f; + can_query = true; + } + + if (can_query) { const std::string toggle_label = is_dual_rgb ? "Switch to Dedicated-RGB Mode" @@ -1363,18 +1376,8 @@ namespace rs2 { try { - const uint32_t value = is_dual_rgb ? MODE_DEDICATED_RGB : MODE_DUAL_RGB; - const std::vector data = { - static_cast( value & 0xFF), - static_cast((value >> 8 ) & 0xFF), - static_cast((value >> 16 ) & 0xFF), - static_cast((value >> 24 ) & 0xFF) }; - - auto dp = dev.as(); - auto cmd = dp.build_command(MWD_OPCODE, MODE_REG_START_ADDR, MODE_REG_END_ADDR, 0, 0, data); - - dp.send_and_receive_raw_data(cmd); - restarting_device_info = get_device_info(dev, false); + depth_sub->s->set_option(RS2_OPTION_SENSORS_CONFIG_MODE, is_dual_rgb ? 0.f : 1.f); + // XU write only takes effect on the next enumeration. dev.hardware_reset(); } catch (const error& e) @@ -2235,6 +2238,62 @@ namespace rs2 return false; } + namespace + { + // Fits text (plus an optional trailing_width, e.g. a badge drawn alongside it at the same + // scale) into max_width pixels: shrinks the window's font scale down to min_font_scale, + // then truncates with an ellipsis if it still doesn't fit. Font scale resets to 1.0 when + // this object goes out of scope, so callers control its lifetime by choosing that scope. + class fitted_string + { + public: + fitted_string(const std::string& text, float max_width, float min_font_scale, float trailing_width = 0.f) + : _full(text), _display(" " + text) + { + float text_width = ImGui::CalcTextSize(_display.c_str()).x; + if (max_width <= 0 || text_width + trailing_width <= max_width) + return; + + float scale = std::max(min_font_scale, max_width / (text_width + trailing_width)); + ImGui::SetWindowFontScale(scale); + _condensed = true; + + float text_budget = max_width - scale * trailing_width; + if (ImGui::CalcTextSize(_display.c_str()).x > text_budget) + _display = truncate(text, text_budget); + } + + ~fitted_string() { ImGui::SetWindowFontScale(1.0f); } + + const char* text() const { return _display.c_str(); } + const char* full_text() const { return _full.c_str(); } + bool condensed() const { return _condensed; } // full text is available via tooltip + + private: + // Truncates text with a trailing ellipsis so " text" fits within max_width pixels + // (current font). Uses a binary search on the character count rather than trimming one + // character at a time, since this runs every frame the name doesn't fit. + static std::string truncate(const std::string& text, float max_width) + { + const std::string ellipsis = "..."; + size_t lo = 0, hi = text.size(); + while (lo < hi) + { + size_t mid = (lo + hi + 1) / 2; + if (ImGui::CalcTextSize((" " + text.substr(0, mid) + ellipsis).c_str()).x <= max_width) + lo = mid; + else + hi = mid - 1; + } + return " " + text.substr(0, lo) + ellipsis; + } + + std::string _full; + std::string _display; + bool _condensed = false; + }; + } + void device_model::draw_controls(float panel_width, float panel_height, ux_window& window, std::string& error_message, @@ -2279,13 +2338,21 @@ namespace rs2 // Draw device name //////////////////////////////////////// const ImVec2 name_pos = { pos.x + 9, pos.y + 17 }; + const float name_area_right_margin = 55.f; // leave room for the remove (X) button + const float min_name_font_scale = 0.9f; // below this the name shrinks to illegibility - truncate instead ImGui::SetCursorPos(name_pos); std::stringstream ss; if (dev.supports(RS2_CAMERA_INFO_NAME)) ss << dev.get_info(RS2_CAMERA_INFO_NAME); if (is_ip_device) { - ImGui::Text(" %s", ss.str().substr(0, ss.str().find("\n IP Device")).c_str()); + std::string full_name = ss.str().substr(0, ss.str().find("\n IP Device")); + { + fitted_string name(full_name, panel_width - name_pos.x - name_area_right_margin, min_name_font_scale); + ImGui::Text("%s", name.text()); + if (name.condensed() && ImGui::IsItemHovered()) + RsImGui::CustomTooltip(" %s", name.full_text()); + } // name's destructor restores the font scale before the network-device line below ImGui::PushFont(window.get_font()); ImGui::Text("\tNetwork Device at %s", dev.get_info(RS2_CAMERA_INFO_IP_ADDRESS)); @@ -2293,42 +2360,64 @@ namespace rs2 } else { - ImGui::Text(" %s", ss.str().c_str()); + std::string full_name = ss.str(); + std::string badge_text; // includes the same leading spaces the old inline "% s" formatting produced + std::string usb_desc; + bool is_usb_badge = false; if (dev.supports(RS2_CAMERA_INFO_CONNECTION_TYPE)) { std::string connection_type = dev.get_info(RS2_CAMERA_INFO_CONNECTION_TYPE); if (connection_type == "USB" && dev.supports(RS2_CAMERA_INFO_USB_TYPE_DESCRIPTOR)) { - std::string desc = dev.get_info(RS2_CAMERA_INFO_USB_TYPE_DESCRIPTOR); - ss.str(""); - ss << " " << textual_icons::usb << " " << desc; - ImGui::SameLine(); - if (!starts_with(desc, "3.")) ImGui::PushStyleColor(ImGuiCol_Text, yellow); - else ImGui::PushStyleColor(ImGuiCol_Text, light_grey); - ImGui::Text(" %s", ss.str().c_str()); - ImGui::PopStyleColor(); - ss.str(""); - ss << "The camera was detected by the OS as connected to a USB " << desc << " port"; - ImGui::PushFont(window.get_font()); - ImGui::PushStyleColor(ImGuiCol_Text, light_grey); - if (ImGui::IsItemHovered()) - RsImGui::CustomTooltip(" %s", ss.str().c_str()); - ImGui::PopStyleColor(); - ImGui::PopFont(); + usb_desc = dev.get_info(RS2_CAMERA_INFO_USB_TYPE_DESCRIPTOR); + is_usb_badge = true; + badge_text = rsutils::string::from() << " " << textual_icons::usb << " " << usb_desc; } else { - ss.str(""); - ss << " " << connection_type; + badge_text = rsutils::string::from() << " " << connection_type; + } + } + + { // Dedicated scope: name and badge share the font scale fitted_string applies, and + // its destructor restores scale to 1.0 right after the badge - any code added below + // this scope, still inside the outer else, is guaranteed to run at normal scale. + fitted_string name(full_name, + panel_width - name_pos.x - name_area_right_margin, + min_name_font_scale, + ImGui::CalcTextSize(badge_text.c_str()).x); + ImGui::Text("%s", name.text()); + if (name.condensed() && ImGui::IsItemHovered()) + RsImGui::CustomTooltip(" %s", name.full_text()); + + if (!badge_text.empty()) + { ImGui::SameLine(); - ImGui::PushStyleColor(ImGuiCol_Text, white); - ImGui::Text(" %s", ss.str().c_str()); - ImGui::PopStyleColor(); + if (is_usb_badge) + { + if (!starts_with(usb_desc, "3.")) ImGui::PushStyleColor(ImGuiCol_Text, yellow); + else ImGui::PushStyleColor(ImGuiCol_Text, light_grey); + ImGui::Text("%s", badge_text.c_str()); + ImGui::PopStyleColor(); + ss.str(""); + ss << "The camera was detected by the OS as connected to a USB " << usb_desc << " port"; + ImGui::PushFont(window.get_font()); + ImGui::PushStyleColor(ImGuiCol_Text, light_grey); + if (ImGui::IsItemHovered()) + RsImGui::CustomTooltip(" %s", ss.str().c_str()); + ImGui::PopStyleColor(); + ImGui::PopFont(); + } + else + { + ImGui::PushStyleColor(ImGuiCol_Text, white); + ImGui::Text("%s", badge_text.c_str()); + ImGui::PopStyleColor(); + } } } } - //ImGui::Text(" %s", dev.get_info(RS2_CAMERA_INFO_NAME)); ImGui::PopFont(); //////////////////////////////////////// @@ -2689,6 +2778,13 @@ namespace rs2 label = rsutils::string::from() << "Controls ##" << sub->s->get_info(RS2_CAMERA_INFO_NAME) << "," << id; if (ImGui::TreeNode(label.c_str())) { + char filter_buf[TEXT_BUFF_SIZE]; + std::snprintf(filter_buf, sizeof(filter_buf), "%s", sub->options_filter.c_str()); + ImGui::PushItemWidth(295 - ImGui::GetCursorPosX()); // align with the sliders' right edge + if (ImGui::InputTextWithHint("##options_filter", "Search controls...", filter_buf, sizeof(filter_buf))) + sub->options_filter = filter_buf; + ImGui::PopItemWidth(); + auto const & supported_options = sub->options_metadata; // moving the color dedicated options to the end of the vector @@ -2722,10 +2818,16 @@ namespace rs2 so_ordered.push_back( opt ); } ); + const std::string filter_lc = rsutils::string::to_lower( sub->options_filter ); for (auto opt : so_ordered) { if( viewer.is_option_skipped( opt ) ) continue; + auto it = supported_options.find( opt ); + if( ! filter_lc.empty() && it != supported_options.end() + && rsutils::string::to_lower( it->second.label.substr( 0, it->second.label.find( "##" ) ) ) + .find( filter_lc ) == std::string::npos ) + continue; if (std::find(drawing_order.begin(), drawing_order.end(), opt) == drawing_order.end()) { if (serialize && opt == RS2_OPTION_VISUAL_PRESET) diff --git a/common/device-model.h b/common/device-model.h index d96b332509..b300d2da48 100644 --- a/common/device-model.h +++ b/common/device-model.h @@ -184,6 +184,15 @@ namespace rs2 static const char* show_skybox{ "performance.show_skybox" }; static const char* occlusion_invalidation{ "performance.occlusion_invalidation" }; } + namespace stats + { + // Same key the SDK's RUM config (src/rum/rum-config) uses in realsense-config.json. + static const char* rum_cloud_enabled{ "rum_cloud_enabled" }; + // Boot-upload throttle (config-only, no UI): min hours between uploads (default 24, 0 disables) + // and the last successful upload time in unix seconds. + static const char* rum_upload_interval_hours{ "rum_upload_interval_hours" }; + static const char* rum_last_upload{ "rum_last_upload" }; + } namespace ply { static const char* mesh{ "ply.mesh" }; diff --git a/common/embedded-filter-model.cpp b/common/embedded-filter-model.cpp index d712aed8b0..c7e294b537 100644 --- a/common/embedded-filter-model.cpp +++ b/common/embedded-filter-model.cpp @@ -69,12 +69,24 @@ namespace rs2 { for (option_value option : _embedded_filter->get_supported_option_values()) { - _options_id_to_model[option->id] = create_option_model( option, - opt_base_label, - model, - _embedded_filter, - model ? &model->_options_invalidated : nullptr, - error_message ); + // Build the model first and insert only on success: an option whose range cannot be read + // throws, and map::operator[] would leave a default-constructed (null-endpoint) entry + // behind. Isolate per option so one bad control does not drop the rest. + try + { + auto om = create_option_model( option, + opt_base_label, + model, + _embedded_filter, + model ? &model->_options_invalidated : nullptr, + error_message ); + _options_id_to_model[option->id] = std::move( om ); + } + catch( const std::exception & e ) + { + if( _viewer.not_model ) + _viewer.not_model->add_log( e.what(), RS2_LOG_SEVERITY_WARN ); + } } _enabled = _embedded_filter->get_option(RS2_OPTION_EMBEDDED_FILTER_ENABLED); diff --git a/common/http/curl-wrapper.cpp b/common/http/curl-wrapper.cpp new file mode 100644 index 0000000000..7ca94787d7 --- /dev/null +++ b/common/http/curl-wrapper.cpp @@ -0,0 +1,157 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#ifdef BUILD_WITH_LIBCURL +#include +#include +#endif + +#include "curl-wrapper.h" +#include + + +namespace rs2 +{ + namespace http + { + +#ifndef BUILD_WITH_LIBCURL + + // Dummy - built without libcurl. + curl_wrapper::curl_wrapper() : _curl( nullptr ) {} + curl_wrapper::~curl_wrapper() {} + bool curl_wrapper::get( const std::string &, const write_func &, const progress_func &, bool ) { return false; } + bool curl_wrapper::post_json( const std::string &, const std::string & ) { return false; } + +#else + + static const long CONNECT_TIMEOUT_SEC = 5; // connect phase cap + static const long UPLOAD_TIMEOUT_SEC = 15; // overall upload cap so a stalled transfer can't hang + static const curl_off_t PROGRESS_MIN_INTERVAL = 500000; // 0.5s between progress callbacks (microseconds) + + // Forward a body chunk to the write_func; return bytes accepted (curl treats a short + // count as a write error and aborts). + static size_t write_callback( void * data, size_t size, size_t nmemb, void * user ) + { + if( ! user ) + return 0; // no sink -> signal a write error, aborting the transfer + auto const & fn = *static_cast< curl_wrapper::write_func * >( user ); + size_t len = size * nmemb; + return ( fn && fn( static_cast< const char * >( data ), len ) ) ? len : 0; + } + + // Discard sink for uploads (we don't need the response body). + static size_t discard_callback( void *, size_t size, size_t nmemb, void * ) { return size * nmemb; } + + struct progress_state { curl_wrapper::progress_func fn; CURL * curl; curl_off_t last_time; }; + + // Throttled to one call per PROGRESS_MIN_INTERVAL; return non-zero to abort (curl convention). + static int progress_callback( void * p, curl_off_t dltotal, curl_off_t dlnow, curl_off_t, curl_off_t ) + { + if( ! p ) + return 0; // no progress state -> keep the transfer going + auto * st = static_cast< progress_state * >( p ); + curl_off_t curtime = 0; + if( curl_easy_getinfo( st->curl, CURLINFO_TOTAL_TIME_T, &curtime ) == CURLE_OK ) + if( dltotal != 0 && curtime - st->last_time > PROGRESS_MIN_INTERVAL ) + { + st->last_time = curtime; + return ( st->fn && st->fn( static_cast< uint64_t >( dlnow ), + static_cast< uint64_t >( dltotal ) ) ) ? 0 : 1; + } + return 0; + } + + curl_wrapper::curl_wrapper() + { + // One-time libcurl init before the first curl_easy_init anywhere. Thread-safe on the + // libcurl we build (>=7.84), so callers never touch curl global state themselves. + static std::once_flag curl_global_once; + std::call_once( curl_global_once, []() { curl_global_init( CURL_GLOBAL_DEFAULT ); } ); + _curl = curl_easy_init(); + } + + curl_wrapper::~curl_wrapper() + { + if( _curl ) + curl_easy_cleanup( static_cast< CURL * >( _curl ) ); + } + + bool curl_wrapper::get( const std::string & url, const write_func & on_data, + const progress_func & on_progress, bool insecure ) + { + if( ! _curl ) + return false; + CURL * curl = static_cast< CURL * >( _curl ); + + curl_easy_setopt( curl, CURLOPT_URL, url.c_str() ); + curl_easy_setopt( curl, CURLOPT_CONNECTTIMEOUT, CONNECT_TIMEOUT_SEC ); + curl_easy_setopt( curl, CURLOPT_FOLLOWLOCATION, 1L ); // follow HTTP 3xx redirects + curl_easy_setopt( curl, CURLOPT_NOSIGNAL, 1L ); + curl_easy_setopt( curl, CURLOPT_FAILONERROR, 1L ); // fail on HTTP >= 400 + curl_easy_setopt( curl, CURLOPT_WRITEFUNCTION, write_callback ); + curl_easy_setopt( curl, CURLOPT_WRITEDATA, (void *)&on_data ); + if( insecure ) + { + curl_easy_setopt( curl, CURLOPT_SSL_VERIFYPEER, 0L ); + curl_easy_setopt( curl, CURLOPT_SSL_VERIFYHOST, 0L ); + } + + progress_state st{ on_progress, curl, 0 }; + if( on_progress ) + { + curl_easy_setopt( curl, CURLOPT_XFERINFOFUNCTION, progress_callback ); + curl_easy_setopt( curl, CURLOPT_XFERINFODATA, &st ); + curl_easy_setopt( curl, CURLOPT_NOPROGRESS, 0L ); + } + else + { + curl_easy_setopt( curl, CURLOPT_NOPROGRESS, 1L ); + } + + auto res = curl_easy_perform( curl ); + if( res != CURLE_OK ) + { + LOG_ERROR( "HTTP GET from " << url << " failed: " << curl_easy_strerror( res ) ); + return false; + } + return true; + } + + bool curl_wrapper::post_json( const std::string & url, const std::string & body ) + { + if( ! _curl ) + return false; + CURL * curl = static_cast< CURL * >( _curl ); + + curl_slist * headers = curl_slist_append( nullptr, "Content-Type: application/json" ); + if( ! headers ) + { + LOG_ERROR( "Failed to allocate curl headers" ); + return false; + } + + curl_easy_setopt( curl, CURLOPT_URL, url.c_str() ); + curl_easy_setopt( curl, CURLOPT_POST, 1L ); + curl_easy_setopt( curl, CURLOPT_POSTFIELDS, body.c_str() ); + curl_easy_setopt( curl, CURLOPT_POSTFIELDSIZE, (long)body.size() ); + curl_easy_setopt( curl, CURLOPT_HTTPHEADER, headers ); + curl_easy_setopt( curl, CURLOPT_CONNECTTIMEOUT, CONNECT_TIMEOUT_SEC ); + curl_easy_setopt( curl, CURLOPT_TIMEOUT, UPLOAD_TIMEOUT_SEC ); + curl_easy_setopt( curl, CURLOPT_NOSIGNAL, 1L ); + curl_easy_setopt( curl, CURLOPT_FAILONERROR, 1L ); + curl_easy_setopt( curl, CURLOPT_WRITEFUNCTION, discard_callback ); + + auto res = curl_easy_perform( curl ); + bool ok = ( res == CURLE_OK ); + if( ! ok ) + LOG_ERROR( "HTTP POST to " << url << " failed: " << curl_easy_strerror( res ) ); + + curl_slist_free_all( headers ); + return ok; + } + +#endif // BUILD_WITH_LIBCURL + + } +} diff --git a/common/http/curl-wrapper.h b/common/http/curl-wrapper.h new file mode 100644 index 0000000000..acea230f95 --- /dev/null +++ b/common/http/curl-wrapper.h @@ -0,0 +1,45 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#pragma once + +#include +#include +#include + +namespace rs2 +{ + namespace http + { + // The one and only place that includes libcurl. Owns a single easy handle and runs the + // one-time global init. http_uploader / http_downloader build their requests through this + // curl-agnostic API and never see a curl type, so a libcurl API change stays in one file. + // Compiles to a no-op when curl isn't linked (BUILD_WITH_LIBCURL). + class curl_wrapper + { + public: + // Receives a chunk of the response/download body; return false to abort the transfer. + typedef std::function< bool( const char * data, size_t len ) > write_func; + // Throttled progress: (bytes so far, total); return false to abort. total may be 0 if unknown. + typedef std::function< bool( uint64_t now, uint64_t total ) > progress_func; + + curl_wrapper(); + ~curl_wrapper(); + curl_wrapper( const curl_wrapper & ) = delete; + curl_wrapper & operator=( const curl_wrapper & ) = delete; + + bool valid() const { return _curl != nullptr; } + + // GET `url`, streaming the body to `on_data`. Optional progress callback. `insecure` skips + // SSL peer/host verification. Follows redirects; fails on HTTP >= 400. true on success. + bool get( const std::string & url, const write_func & on_data, + const progress_func & on_progress = progress_func(), bool insecure = false ); + + // POST `body` to `url` as application/json (response body discarded). true on success. + bool post_json( const std::string & url, const std::string & body ); + + private: + void * _curl; + }; + } +} diff --git a/common/http/http-downloader.cpp b/common/http/http-downloader.cpp new file mode 100644 index 0000000000..0e4e874282 --- /dev/null +++ b/common/http/http-downloader.cpp @@ -0,0 +1,55 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "http-downloader.h" +#include +#include + + +namespace rs2 +{ + namespace http + { + + // Adapt the caller's callback_result progress callback to curl_wrapper's bool convention + // (true = keep going). Empty in -> empty out (no progress meter). + static curl_wrapper::progress_func adapt_progress( const user_callback_func_type & user_callback_func ) + { + if( ! user_callback_func ) + return curl_wrapper::progress_func(); + return [user_callback_func]( uint64_t now, uint64_t total ) + { + return user_callback_func( now, total ) == callback_result::CONTINUE_DOWNLOAD; + }; + } + + bool http_downloader::download_to_stream( const std::string & url, std::stringstream & output, user_callback_func_type user_callback_func ) + { + auto sink = [&output]( const char * data, size_t len ) { output.write( data, len ); return true; }; + // SSL verification disabled here to preserve this path's original behavior. + return _curl.get( url, sink, adapt_progress( user_callback_func ), true /*insecure*/ ); + } + + bool http_downloader::download_to_bytes_vector( const std::string & url, std::vector & output, user_callback_func_type user_callback_func ) + { + auto sink = [&output]( const char * data, size_t len ) + { + output.insert( output.end(), data, data + len ); + return true; + }; + return _curl.get( url, sink, adapt_progress( user_callback_func ) ); + } + + bool http_downloader::download_to_file( const std::string & url, const std::string & file_name, user_callback_func_type user_callback_func ) + { + std::ofstream out_file( file_name, std::ios::out | std::ios::binary ); + if( ! out_file.good() ) + { + LOG_ERROR( "Download error - Cannot open local file: " + file_name ); + return false; + } + auto sink = [&out_file]( const char * data, size_t len ) { out_file.write( data, len ); return true; }; + return _curl.get( url, sink, adapt_progress( user_callback_func ) ); + } + } +} diff --git a/common/sw-update/http-downloader.h b/common/http/http-downloader.h similarity index 64% rename from common/sw-update/http-downloader.h rename to common/http/http-downloader.h index 53879316d3..662f7595d3 100644 --- a/common/sw-update/http-downloader.h +++ b/common/http/http-downloader.h @@ -1,11 +1,13 @@ // License: Apache 2.0. See LICENSE file in root directory. -// Copyright(c) 2020 RealSense, Inc. All Rights Reserved. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. #pragma once +#include "curl-wrapper.h" #include #include #include +#include #include namespace rs2 @@ -16,28 +18,21 @@ namespace rs2 enum class callback_result { CONTINUE_DOWNLOAD, STOP_DOWNLOAD }; typedef std::function user_callback_func_type; - struct progress_data; // Forward Declaration - - // This class is a service class for downloading a file from an HTTP URL. - // The class use libcurl as a client-side URL transfer library. + // Service class for downloading a file from an HTTP URL. The transfer runs through + // curl_wrapper, which confines all libcurl usage; this class only shapes the request and + // routes the body to a stream / vector / file. class http_downloader { public: - http_downloader(); - ~http_downloader(); - // The optional callback function provides 2 major capabilities: // - Current status about the download progress - // - Control the download process (stop/continue) using the return value of the callback function (true = stop download) + // - Control the download process (stop/continue) using the return value of the callback function bool download_to_stream(const std::string& url, std::stringstream &output, user_callback_func_type user_callback_func = user_callback_func_type()); bool download_to_bytes_vector(const std::string& url, std::vector &output, user_callback_func_type user_callback_func = user_callback_func_type()); bool download_to_file(const std::string& url, const std::string &file_name, user_callback_func_type user_callback_func = user_callback_func_type()); private: - void register_progress_call_back(progress_data &progress_record, user_callback_func_type user_callback_func); - void set_common_options(const std::string &url); - - void* _curl; + curl_wrapper _curl; }; } } diff --git a/common/http/http-uploader.h b/common/http/http-uploader.h new file mode 100644 index 0000000000..0510e03f40 --- /dev/null +++ b/common/http/http-uploader.h @@ -0,0 +1,28 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#pragma once + +#include "curl-wrapper.h" +#include + +namespace rs2 +{ + namespace http + { + // POSTs a body to an HTTP(S) URL. A thin façade over curl_wrapper (which owns all the + // libcurl usage), mirroring http_downloader. Compiles to a no-op when curl isn't linked. + class http_uploader + { + public: + // POST json_body to url as "application/json"; true on success. + bool upload( const std::string & url, const std::string & json_body ) + { + return _curl.post_json( url, json_body ); + } + + private: + curl_wrapper _curl; + }; + } +} diff --git a/common/model-views.cpp b/common/model-views.cpp index 3e158855ca..f53c19f253 100644 --- a/common/model-views.cpp +++ b/common/model-views.cpp @@ -32,7 +32,7 @@ #include "metadata-helper.h" #include "calibration-model.h" -#include "sw-update/http-downloader.h" +#include "http/http-downloader.h" #include #include diff --git a/common/rum-uploader/rum-uploader.cpp b/common/rum-uploader/rum-uploader.cpp new file mode 100644 index 0000000000..53e138d611 --- /dev/null +++ b/common/rum-uploader/rum-uploader.cpp @@ -0,0 +1,211 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#ifdef ENABLE_STATS +#include "../rs-config.h" // config_file, configurations::stats +#include "../device-model.h" // configurations, device_model +#include "../ux-window.h" // ux_window (consent popup font) +#include "../subdevice-model.h" // subdevice_model::wait_for_stop +#include // rs2::rum::is_cloud_enabled +#include +#include +#include +#include +#include +#include +#endif // ENABLE_STATS + +#include "rum-uploader.h" +#include "../http/http-uploader.h" +#include + + +namespace rs2 { + + +#ifndef ENABLE_STATS + +// Dummy functions - built without RUM collection/upload; the whole feature compiles to no-ops. +std::string rum_uploader::saved_report() { return std::string(); } +bool rum_uploader::upload( std::string const & ) { LOG_WARNING( "RUM upload unavailable: built without ENABLE_STATS" ); return false; } +void rum_uploader::start() {} +void rum_uploader::upload_async( std::string, std::function< void( bool ) > ) {} +rum_uploader::~rum_uploader() {} +void rum_uploader::upload_data( ux_window & ) {} +void rum_uploader::join_pending_stops( std::shared_ptr< std::vector< std::unique_ptr< device_model > > > ) {} + +#else // ENABLE_STATS + + +// ---- tunables ---- +// No production endpoint yet; upload to the local dev-server stub (see dev-server/) for now. +// TODO: use the real cloud endpoint once it exists. +static char const * RUM_ENDPOINT = "http://127.0.0.1:8080/v1/rum"; +static char const * CONSENT_POPUP_ID = "Help improve RealSense"; +static const int DEFAULT_UPLOAD_INTERVAL_HOURS = 24; // 0 disables the throttle +static const int SECONDS_PER_HOUR = 3600; + + +std::string rum_uploader::saved_report() +{ + auto path = rsutils::os::get_special_folder( rsutils::os::special_folder::app_data ) + "rum/rum.json"; + // Read the file exactly as saved (binary = no newline translation). + std::ifstream f( path, std::ios::binary ); + if( ! f ) + return std::string(); + std::ostringstream ss; + ss << f.rdbuf(); + return ss.str(); +} + + +bool rum_uploader::upload( std::string const & json_report ) +{ + // Refuse to send without consent, so no caller can leak data by forgetting to check. + if( ! rs2::rum::is_cloud_enabled() ) + { + LOG_WARNING( "RUM upload refused: cloud upload not consented" ); + return false; + } + + // All HTTP/curl lives in http_uploader; we just hand it the endpoint and body. + http::http_uploader uploader; + return uploader.upload( RUM_ENDPOINT, json_report ); +} + + +void rum_uploader::start() +{ + if( _thread.joinable() ) + _thread.join(); // don't overwrite a running worker (a second start / start-after-upload would std::terminate) + _thread = std::thread( []() + { + try + { + if( ! rs2::rum::is_cloud_enabled() ) + return; + + // Throttle boot uploads to at most once per interval, read from config (no UI; default + // 24h, 0 disables). The last-upload time persists in the same config file. + auto & cfg = config_file::instance(); + int interval_hours = cfg.get_or_default( configurations::stats::rum_upload_interval_hours, DEFAULT_UPLOAD_INTERVAL_HOURS ); + auto now = std::chrono::duration_cast< std::chrono::seconds >( + std::chrono::system_clock::now().time_since_epoch() ).count(); + long long last = cfg.get_or_default< long long >( configurations::stats::rum_last_upload, 0 ); + if( last > now ) + last = now; // future timestamp (clock skew / corrupt config) - invalidate to now + if( interval_hours > 0 && now - last < (long long)interval_hours * SECONDS_PER_HOUR ) + return; // uploaded recently + + auto report = saved_report(); // prior session (nothing live yet at boot) + if( report.empty() ) + return; // nothing saved yet + if( upload( report ) ) + { + cfg.set( configurations::stats::rum_last_upload, now ); + LOG_INFO( "RUM report uploaded to " << RUM_ENDPOINT ); + } + } + catch( std::exception const & e ) { LOG_ERROR( "RUM upload error: " << e.what() ); } + } ); +} + + +void rum_uploader::upload_async( std::string report, std::function< void( bool ) > on_done ) +{ + if( _uploading.exchange( true ) ) + return; // an upload is already running; don't block the caller + if( _thread.joinable() ) + _thread.join(); // previous upload finished; join is instant + _thread = std::thread( [this, report = std::move( report ), on_done = std::move( on_done )]() + { + bool ok = false; + try + { + ok = upload( report ); + if( ok ) + LOG_INFO( "RUM report uploaded to " << RUM_ENDPOINT ); + else + LOG_ERROR( "RUM upload failed" ); + } + catch( std::exception const & e ) { LOG_ERROR( "RUM upload error: " << e.what() ); } + if( on_done ) + on_done( ok ); + _uploading = false; + } ); +} + + +rum_uploader::~rum_uploader() +{ + if( _thread.joinable() ) + _thread.join(); +} + + +static void draw_consent_popup( rum_uploader & uploader, ux_window & window ) +{ + ImGui::SetNextWindowSize( { 460.f, 0.f } ); + if( ! ImGui::BeginPopupModal( CONSENT_POPUP_ID, nullptr, + ImGuiWindowFlags_AlwaysAutoResize | ImGuiWindowFlags_NoMove ) ) + return; + + ImGui::PushFont( window.get_large_font() ); + ImGui::Text( "%s", CONSENT_POPUP_ID ); + ImGui::PopFont(); + ImGui::Separator(); + ImGui::Spacing(); + ImGui::TextWrapped( "Share anonymous usage statistics (devices, stream configs, options, " + "and errors) to help us prioritize fixes and features." ); + ImGui::Spacing(); + ImGui::TextWrapped( "No personal data, serial numbers, or image content is ever collected. " + "You can change this any time in Settings > Online Services." ); + ImGui::Spacing(); + ImGui::Separator(); + ImGui::Spacing(); + if( ImGui::Button( "Yes, enable", ImVec2( 150, 30 ) ) ) + { + config_file::instance().set( configurations::stats::rum_cloud_enabled, true ); + uploader.start(); // upload any saved report now (e.g. re-consent after a reset); no-op on a true first run + ImGui::CloseCurrentPopup(); + } + ImGui::SameLine(); + if( ImGui::Button( "No thanks", ImVec2( 150, 30 ) ) ) + { + config_file::instance().set( configurations::stats::rum_cloud_enabled, false ); + ImGui::CloseCurrentPopup(); + } + ImGui::EndPopup(); +} + + +void rum_uploader::upload_data( ux_window & window ) +{ + // First-run consent: decided once (missing -> ask; true/false -> silent). Persists immediately. + static bool startup_done = false; + if( ! startup_done ) + { + if( ! config_file::instance().contains( configurations::stats::rum_cloud_enabled ) ) + ImGui::OpenPopup( CONSENT_POPUP_ID ); // first run: ask; nothing saved yet, so no upload this session + else + start(); // background-upload the previous session's saved report + startup_done = true; + } + draw_consent_popup( *this, window ); +} + + +void rum_uploader::join_pending_stops( std::shared_ptr< std::vector< std::unique_ptr< device_model > > > device_models ) +{ + // stop() runs asynchronously and is where streamed-duration is recorded, so join any pending + // stop here; the session is persisted automatically when the SDK context is destroyed. + for( auto && dm : *device_models ) + for( auto && sub : dm->subdevices ) + try { sub->wait_for_stop(); } catch( ... ) {} +} + + +#endif // ENABLE_STATS + + +} // namespace rs2 diff --git a/common/rum-uploader/rum-uploader.h b/common/rum-uploader/rum-uploader.h new file mode 100644 index 0000000000..6f3ba1a6c2 --- /dev/null +++ b/common/rum-uploader/rum-uploader.h @@ -0,0 +1,59 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once + +#include +#include +#include +#include +#include +#include + + +namespace rs2 { + + +class ux_window; +class device_model; + + +// RUM uploader, linked into the viewer. Owns the background upload thread and joins it in the +// destructor, so a stray exception can't leave a thread running at shutdown. +class rum_uploader +{ +public: + rum_uploader() = default; + ~rum_uploader(); + rum_uploader( rum_uploader const & ) = delete; + rum_uploader & operator=( rum_uploader const & ) = delete; + + // The last report saved to disk (/rum/rum.json), or "" if none — the prior session + // the boot upload ships. (The live session is via rs2::rum::get_report().) + static std::string saved_report(); + + // POST the report over HTTP(S); true on success. No-op returning false when built without HTTP. + // This is the call that actually sends data off the machine. + static bool upload( std::string const & json_report ); + + // Returns immediately. If consented and a saved report exists, uploads it on a background + // thread. The server dedups repeats via the report's session_id. + void start(); + + // Returns immediately. Uploads `report` on a background thread; skips if one is already running + // so a UI button never blocks on a slow transfer. `on_done( ok )`, if given, runs on that thread + // when the upload finishes (not called on the skip). + void upload_async( std::string report, std::function< void( bool ) > on_done = {} ); + + // Show consent popup if consent not set, upload if consent granted, no-op if rejected. + void upload_data( ux_window & window ); + + // On shutdown, join any pending async sensor stops so streamed-duration is recorded before teardown. + static void join_pending_stops( std::shared_ptr< std::vector< std::unique_ptr< device_model > > > device_models ); + +private: + std::thread _thread; + std::atomic< bool > _uploading{ false }; +}; + + +} // namespace rs2 diff --git a/common/subdevice-model.cpp b/common/subdevice-model.cpp index 6d28d94386..44e50d1a11 100644 --- a/common/subdevice-model.cpp +++ b/common/subdevice-model.cpp @@ -813,6 +813,11 @@ namespace rs2 if (stream_enabled[f.first]) { + // The two imagers stream mono IR (Y8) OR Bayer color (BA81), not both, + // so enabling a color stream disables IR and vice versa (depth is free). + if( is_dual_color_subdevice() ) + enforce_dual_color_ir_exclusion(f.first); + // Find the stream type for this unique_id rs2_stream stream_type = RS2_STREAM_ANY; for (auto& p : profiles) @@ -1555,6 +1560,64 @@ namespace rs2 return is_cal_format; } + bool subdevice_model::is_dual_color_subdevice() const + { + // The color<->IR imager conflict is specific to the D401 GMSL dual-RGB, where the two OV9782 + // imagers each stream mono IR OR Bayer color (never both). Gate strictly on that product id + // (0xABCC == RS401_GMSL_PID, the same gate d400-device.cpp uses for the whole feature) so + // this stays a no-op on EVERY other camera -- standard D4xx (color on a separate sensor / + // single color) never reach the color>=2 check anyway, but the D500 dual-RGB (separate color + // sensors, 2 colors + stereo on one sensor) would, and it has no such imager conflict. + if (!dev.supports(RS2_CAMERA_INFO_PRODUCT_ID) + || std::string(dev.get_info(RS2_CAMERA_INFO_PRODUCT_ID)) != "ABCC") // RS401_GMSL_PID + return false; + + // Structural sanity: this subdevice actually exposes the dual-RGB config (two color streams + // alongside the stereo streams) rather than, say, the plain depth sensor of the same device. + int color_streams = 0; + bool has_stereo = false; + for (auto&& p : profiles) + { + if (p.stream_type() == RS2_STREAM_COLOR) + ++color_streams; + else if (p.stream_type() == RS2_STREAM_INFRARED || p.stream_type() == RS2_STREAM_DEPTH) + has_stereo = true; + } + return color_streams >= 2 && has_stereo; + } + + void subdevice_model::enforce_dual_color_ir_exclusion(int just_enabled_unique_id) + { + // Caller gates this on is_dual_color_subdevice(). + auto stream_type_of = [this](int unique_id) -> rs2_stream + { + for (auto&& p : profiles) + if (p.unique_id() == unique_id) + return p.stream_type(); + return RS2_STREAM_ANY; + }; + + auto is_color = [](rs2_stream st) { return st == RS2_STREAM_COLOR; }; + auto is_ir = [](rs2_stream st) { return st == RS2_STREAM_INFRARED; }; + + rs2_stream enabled_type = stream_type_of(just_enabled_unique_id); + // Only color<->IR conflict (the two imagers stream mono IR OR Bayer color, not both). Depth + // is a separate node - enabling it clears nothing, and it survives enabling color or IR. + if (!is_color(enabled_type) && !is_ir(enabled_type)) + return; + + for (auto& other : stream_enabled) + { + if (other.first == just_enabled_unique_id || !other.second) + continue; + + rs2_stream other_type = stream_type_of(other.first); + if ((is_color(enabled_type) && is_ir(other_type)) || + (is_ir(enabled_type) && is_color(other_type))) + other.second = false; // color and IR share the imagers -> mutually exclusive + } + } + bool subdevice_model::is_depth_calibration_profile() const { // Check if D555 at depth resolution of 1280x800 diff --git a/common/subdevice-model.h b/common/subdevice-model.h index aa28c2fcaa..e343d75828 100644 --- a/common/subdevice-model.h +++ b/common/subdevice-model.h @@ -164,6 +164,7 @@ namespace rs2 std::shared_ptr< atomic_objects_in_frame > detected_objects; std::map< rs2_option, option_model > options_metadata; + std::string options_filter; // live search text filtering the Controls option list by name std::vector resolutions; std::map> fpses_per_stream; std::vector shared_fpses; @@ -267,6 +268,15 @@ namespace rs2 std::pair get_max_resolution(rs2_stream stream) const; void sort_resolutions(std::vector>& resolutions) const; bool is_ir_calibration_profile() const; + // True when this subdevice exposes the dual-RGB configuration (two color streams alongside + // the stereo IR streams). On the D401 GMSL the two imagers each stream mono IR (Y8) OR Bayer + // color (BA81) - not both - so color and infrared are mutually exclusive on the imager nodes; + // depth is a separate node and coexists with either group. + bool is_dual_color_subdevice() const; + // Enforce the color/IR mutual exclusion on the dual-RGB subdevice: enabling a color stream + // disables the infrared streams and vice versa. Depth is left untouched (it can coexist with + // either dual-RGB or stereo-IR). + void enforce_dual_color_ir_exclusion(int just_enabled_unique_id); void set_extrinsics_from_depth_if_needed(); bool is_post_processing_enabled_in_config_file() const; void avoid_streaming_on_embedded_filters_not_matching_configuration() const; diff --git a/common/sw-update/http-downloader.cpp b/common/sw-update/http-downloader.cpp deleted file mode 100644 index 598e68df43..0000000000 --- a/common/sw-update/http-downloader.cpp +++ /dev/null @@ -1,232 +0,0 @@ -// License: Apache 2.0. See LICENSE file in root directory. -// Copyright(c) 2020 RealSense, Inc. All Rights Reserved. - - - -#ifdef CHECK_FOR_UPDATES -#include -#include -#include -#include -#include -#include -#include -#include -#endif // CHECK_FOR_UPDATES - -#include "http-downloader.h" -#include - - -namespace rs2 -{ - namespace http - { - -#ifndef CHECK_FOR_UPDATES - // Dummy functions - http_downloader::http_downloader() {} - http_downloader::~http_downloader() {} - bool http_downloader::download_to_stream(const std::string& url, std::stringstream &output, user_callback_func_type user_callback_func) { return false; } - bool http_downloader::download_to_file(const std::string& url, const std::string &file_name, user_callback_func_type user_callback_func) { return false; } - bool http_downloader::download_to_bytes_vector(const std::string& url, std::vector &output, user_callback_func_type user_callback_func) { return false; } - -#else - - std::mutex initialize_mutex; - static const curl_off_t HALF_SEC = 500000; // User call back function delay - static const int CONNECT_TIMEOUT = 5L; // Libcurl connection timeout 5 [Sec] - - struct progress_data { - curl_off_t last_run_time; - user_callback_func_type user_callback_func; - CURL *curl; - }; - - - size_t stream_write_callback(void *input_stream, size_t size, size_t nmemb, void *output_stream) - { - if (input_stream && output_stream) - { - std::string data((const char*)input_stream, (size_t)size * nmemb); - *((std::stringstream*)output_stream) << data; - return size * nmemb; - } - return 0; // Error - } - - size_t vector_write_callback(void *input_stream, size_t size, size_t nmemb, void *output_vec) - { - uint8_t* source_bytes(static_cast(input_stream)); - - if (input_stream && output_vec) - { - int total_size((int)(size * nmemb)); - while (total_size > 0) - { - static_cast *>(output_vec)->push_back(*source_bytes); - source_bytes++; - --total_size; - } - - return size * nmemb; - } - return 0; // Error - } - - size_t file_write_callback(void *input_stream, size_t size, size_t nmemb, void *output) - { - - if (input_stream && output) - { - std::ofstream &out_stream(*static_cast (output)); - - size_t num_of_bytem(nmemb*size); - out_stream.write((char *)input_stream, num_of_bytem); - return size * nmemb; - - } - return 0; // Error - } - - // This function will be called if CURLOPT_NOPROGRESS is set to 0 - // Return value: 0 = continue download / 1 = stop download - int progress_callback(void *p, curl_off_t dltotal, curl_off_t dlnow, curl_off_t ultotal, curl_off_t ulnow) - { - progress_data *myp = static_cast(p); - CURL *curl(myp->curl); - curl_off_t curtime(0); - if( curl_easy_getinfo(curl, CURLINFO_TOTAL_TIME_T, &curtime) == CURLE_OK ) - if (dltotal != 0 && (curtime - myp->last_run_time > HALF_SEC)) - { - myp->last_run_time = curtime; - return myp->user_callback_func(static_cast(dlnow), - static_cast(dltotal)) == callback_result::CONTINUE_DOWNLOAD ? 0 : 1; - } - - return 0; - } - - http_downloader::http_downloader() : _curl(nullptr) - { - // Protect curl_easy_init() it is not considers thread safe - std::lock_guard lock(initialize_mutex); - _curl = curl_easy_init(); - } - - http_downloader::~http_downloader() - { - std::lock_guard lock(initialize_mutex); - curl_easy_cleanup(_curl); - } - - bool http_downloader::download_to_stream(const std::string& url, std::stringstream &output, user_callback_func_type user_callback_func) - { - if (!_curl) return false; - - set_common_options(url); - if( curl_easy_setopt(_curl, CURLOPT_WRITEFUNCTION, stream_write_callback) != CURLE_OK || - curl_easy_setopt(_curl, CURLOPT_WRITEDATA, &output) != CURLE_OK || - curl_easy_setopt(_curl, CURLOPT_SSL_VERIFYPEER ,0L) != CURLE_OK || - curl_easy_setopt(_curl, CURLOPT_SSL_VERIFYHOST ,0L) != CURLE_OK ) - throw std::invalid_argument( "Setting CURL option failed" ); - - progress_data progress_record; // Should stay here - "curl_easy_perform" use it - if (user_callback_func) - { - register_progress_call_back(progress_record, user_callback_func); - } - auto res = curl_easy_perform(_curl); - - if (CURLE_OK != res) - { - LOG_ERROR("Download error from URL: " + url + ", error info: " + std::string(curl_easy_strerror(res))); - return false; - } - return true; - } - - bool http_downloader::download_to_bytes_vector(const std::string& url, std::vector &output, user_callback_func_type user_callback_func) - { - if (!_curl) return false; - - set_common_options(url); - if( curl_easy_setopt(_curl, CURLOPT_WRITEFUNCTION, vector_write_callback) != CURLE_OK || - curl_easy_setopt(_curl, CURLOPT_WRITEDATA, &output) != CURLE_OK ) - throw std::invalid_argument( "Setting CURL option failed" ); - - progress_data progress_record; // Should stay here - "curl_easy_perform" use it - if (user_callback_func) - { - register_progress_call_back(progress_record, user_callback_func); - } - auto res = curl_easy_perform(_curl); - - if (CURLE_OK != res) - { - LOG_ERROR("Download error from URL: " + url + ", error info: " + std::string(curl_easy_strerror(res))); - return false; - } - return true; - } - - - bool http_downloader::download_to_file(const std::string& url, const std::string &file_name, user_callback_func_type user_callback_func) - { - if (!_curl) return false; - - /* open the file */ - std::ofstream out_file(file_name, std::ios::out); - - if (out_file.good()) - { - set_common_options(url); - if( curl_easy_setopt(_curl, CURLOPT_WRITEFUNCTION, file_write_callback) != CURLE_OK || - curl_easy_setopt(_curl, CURLOPT_WRITEDATA, &out_file) != CURLE_OK ) - throw std::invalid_argument( "Setting CURL option failed" ); - - progress_data progress_record; // Should stay here - "curl_easy_perform" use it - if (user_callback_func) - { - register_progress_call_back(progress_record, user_callback_func); - } - auto res = curl_easy_perform(_curl); - out_file.close(); - - if (CURLE_OK != res) - { - LOG_ERROR("Download error from URL: " + url + ", error info: " + std::string(curl_easy_strerror(res))); - return false; - } - } - else - { - LOG_ERROR("Download error - Cannot open local file: " + file_name); - return false; - } - - return true; - } - - void http_downloader::set_common_options(const std::string &url) - { - if( curl_easy_setopt( _curl, CURLOPT_CONNECTTIMEOUT, CONNECT_TIMEOUT ) != CURLE_OK || // timeout for the connect phase - curl_easy_setopt( _curl, CURLOPT_URL, url.c_str() ) != CURLE_OK || // provide the URL to use in the request - curl_easy_setopt( _curl, CURLOPT_FOLLOWLOCATION, 1L ) != CURLE_OK || // follow HTTP 3xx redirects - curl_easy_setopt( _curl, CURLOPT_NOSIGNAL, 1 ) != CURLE_OK || // skip all signal handling - curl_easy_setopt( _curl, CURLOPT_FAILONERROR, 1L ) != CURLE_OK || // request failure on HTTP response >= 400 - curl_easy_setopt( _curl, CURLOPT_NOPROGRESS, 1L ) != CURLE_OK ) // switch off the progress meter - throw std::invalid_argument( "Setting CURL option failed" ); - } - - void http_downloader::register_progress_call_back(progress_data &progress_record, user_callback_func_type user_callback_func) - { - progress_record = { 0, user_callback_func, _curl }; - if( curl_easy_setopt(_curl, CURLOPT_XFERINFOFUNCTION, progress_callback) != CURLE_OK || - curl_easy_setopt(_curl, CURLOPT_XFERINFODATA, &progress_record) != CURLE_OK || - curl_easy_setopt(_curl, CURLOPT_NOPROGRESS, 0L) != CURLE_OK ) - throw std::invalid_argument( "Setting CURL option failed" ); - } -#endif - } -} diff --git a/common/sw-update/versions-db-manager.h b/common/sw-update/versions-db-manager.h index 5c99b668fe..7f57283b7f 100644 --- a/common/sw-update/versions-db-manager.h +++ b/common/sw-update/versions-db-manager.h @@ -7,7 +7,7 @@ #include #include #include -#include "http-downloader.h" +#include "../http/http-downloader.h" #include #include diff --git a/common/updates-model.cpp b/common/updates-model.cpp index 63c0f7932a..1ea44fa631 100644 --- a/common/updates-model.cpp +++ b/common/updates-model.cpp @@ -8,7 +8,7 @@ #include "os.h" #include #include -#include "sw-update/http-downloader.h" +#include "http/http-downloader.h" #include using namespace rs2; diff --git a/common/viewer.cpp b/common/viewer.cpp index 74382e4637..567c493020 100644 --- a/common/viewer.cpp +++ b/common/viewer.cpp @@ -933,6 +933,9 @@ namespace rs2 _hidden_options.emplace(RS2_OPTION_NOISE_ESTIMATION); _hidden_options.emplace(RS2_OPTION_REGION_OF_INTEREST); _hidden_options.emplace(RS2_OPTION_READOUT_SHAPING); + // Rendered as a "more" popup Selectable in device-model.cpp instead of a sensor + // control, so it doesn't need to appear in the sensor's Controls tree. + _hidden_options.emplace(RS2_OPTION_SENSORS_CONFIG_MODE); } void viewer_model::update_configuration(config_file* new_cfg) @@ -2957,19 +2960,21 @@ namespace rs2 temp_cfg.set(configurations::viewer::settings_tab, tab); } ImGui::PopStyleColor(2); +#ifdef BUILD_WITH_LIBCURL + // One "Online Services" tab hosting both curl-backed features (updates + usage stats); + // each section renders only if its feature is compiled in. ImGui::SameLine(); - ImGui::PushStyleColor(ImGuiCol_Text, tab != 3 ? light_grey : light_blue); ImGui::PushStyleColor(ImGuiCol_TextSelectedBg, tab != 3 ? light_grey : light_blue); - - if (ImGui::Button("Updates", { 120, 30 })) + if (ImGui::Button("Online Services", { 160, 30 })) { tab = 3; config_file::instance().set(configurations::viewer::settings_tab, tab); temp_cfg.set(configurations::viewer::settings_tab, tab); } - ImGui::PopStyleColor(2); +#endif + ImGui::PopFont(); ImGui::PopStyleColor(2); // button color @@ -3373,8 +3378,6 @@ namespace rs2 if (tab == 3) { #ifdef CHECK_FOR_UPDATES - ImGui::Separator(); - ImGui::Text("%s", "SW/FW Updates From Server:"); if (ImGui::IsItemHovered()) { @@ -3416,6 +3419,52 @@ namespace rs2 temp_cfg.set(configurations::update::sw_updates_url, url_str); } } + + ImGui::Separator(); +#endif + +#ifdef ENABLE_STATS + ImGui::Text("Real User Monitoring (RUM)"); + ImGui::Text("Anonymous usage statistics are collected locally. Cloud upload happens only with your consent."); + + bool cloud_enabled = temp_cfg.get_or_default(configurations::stats::rum_cloud_enabled, false); + if (ImGui::Checkbox("Enable anonymous cloud upload", &cloud_enabled)) + temp_cfg.set(configurations::stats::rum_cloud_enabled, cloud_enabled); + + if (ImGui::Button("Export RUM data...")) + { + if (auto ret = file_dialog_open(save_file, "JSON\0*.json\0", NULL, NULL)) + { + try + { + std::ofstream(ret) << rs2::rum::get_report(); + } + catch (const std::exception& e) { LOG_ERROR("RUM export failed: " << e.what()); } + } + } + ImGui::SameLine(); + // TODO: "Upload now" (and rum_uploader::upload_async) is a testing affordance to send + // the live session on demand; boot upload is the product path. Drop it once RUM is fully merged. + // Gate on the saved consent, not the checkbox: upload() reads the persisted value, + // so the button must stay disabled until the choice is applied (OK/Apply). + bool consent_saved = config_file::instance().get_or_default(configurations::stats::rum_cloud_enabled, false); + RsImGui::RsImButton([&]() { + if (ImGui::Button("Upload now")) + // Off the UI thread; the uploader skips if one is already in flight. + // Capture not_model by value so the callback (on the upload thread) stays valid. + _rum_uploader.upload_async(rs2::rum::get_report(), + [not_model = not_model](bool ok) { + not_model->add_notification({ ok ? "RUM report uploaded" : "RUM upload failed", + ok ? RS2_LOG_SEVERITY_INFO : RS2_LOG_SEVERITY_ERROR, + RS2_NOTIFICATION_CATEGORY_UNKNOWN_ERROR }); + }); + }, !consent_saved); + // AllowWhenDisabled: the button is disabled until consent is applied, but the + // hint explaining why must still show on hover. + if (ImGui::IsItemHovered(ImGuiHoveredFlags_AllowWhenDisabled)) + RsImGui::CustomTooltip(consent_saved + ? "Send the current report now" + : "Enable cloud upload above and click Apply first"); #endif } } diff --git a/common/viewer.h b/common/viewer.h index 7e232b8659..3de6964288 100644 --- a/common/viewer.h +++ b/common/viewer.h @@ -14,6 +14,9 @@ #include "measurement.h" #include "updates-model.h" #include "bag-conversion-helper.h" +#ifdef ENABLE_STATS +#include "rum-uploader/rum-uploader.h" +#endif #include namespace rs2 @@ -157,6 +160,9 @@ namespace rs2 post_processing_filters ppf; context &ctx; +#ifdef ENABLE_STATS + rs2::rum_uploader _rum_uploader; // owns the "Upload now" worker; joins itself in its dtor +#endif std::shared_ptr not_model = std::make_shared(); bool is_3d_view = false; bool paused = false; diff --git a/doc/rum.md b/doc/rum.md new file mode 100644 index 0000000000..c9ac7c5b54 --- /dev/null +++ b/doc/rum.md @@ -0,0 +1,62 @@ +# Real User Monitoring (RUM) + +RUM collects **anonymous, aggregated** usage statistics about how the RealSense SDK is +used in the field, so the team can prioritize fixes and features on real evidence. +Collection is local; data leaves the machine **only** if you explicitly opt in to cloud +upload. + +## What is collected + +A small JSON report (a few KB), aggregated — counts and configurations, never raw events: + +- **SDK build**: version, build type, backend, and the build-time flags it was compiled with. +- **System**: OS and CPU architecture. +- **Devices**: model, firmware version, connection type, MIPI driver version (where applicable). +- **Streams**: the stream configurations opened (type, format, resolution, fps) and how long they ran. +- **Options changed**: device-sensor options set to a non-default value (name + last value). +- **Filters**: which SDK post-processing filters were actually applied to frames. +- **Notifications**: SDK notification categories, counted. + +## What is NOT collected + +- No serial numbers, IP addresses, or any device/user identifier beyond a random `source_id`. +- No personal data. +- No image, depth, or point-cloud content. + +The `source_id` is a random token generated once per installation to deduplicate reports on +the server. It is not tied to the user or the hardware. Each report also carries a per-run +`session_id` and a `generated_at` timestamp so the server can dedup a session that is uploaded +more than once (e.g. a manual upload followed by the next-boot upload). + +## Consent and control + +- **Opt-in**: nothing is uploaded until you agree. The viewer shows a one-time consent prompt + on first run; you can change the choice any time in **Settings → Online Services**. +- **Disable upload at runtime**: turn it off in Settings → Online Services, or via the environment + variable `RS2_RUM_CLOUD_ENABLED`. The override is asymmetric: `=0` always disables (a kill switch), + while `=1` only enables when you have not explicitly opted out — the env var can never turn upload + on against a saved opt-out. +- **Collection is off by default at build time**: build the SDK with `-DENABLE_STATS=ON` to enable + it. The `rs2_rum_*` API is always available and functional; `ENABLE_STATS` gates only + the instrumentation hooks that feed the collector, so when off the report's collected lists stay + empty but the API itself behaves identically. + +## Where the data lives + +The local report is written to `rum.json` under the SDK's app-data folder +(`%APPDATA%\rum\` on Windows, `~/.rum/` on Linux). Consent and settings are stored in the +shared `realsense-config.json`. + +## Uploading + +The viewer performs the upload (the SDK itself never opens a network socket). +If you have consented, the viewer uploads the previously saved report in the background at +startup (the server deduplicates repeats via each report's `session_id`). You can also trigger +an immediate upload from **Settings → Online Services → "Upload now"**. + +The startup upload is throttled to at most once per `rum_upload_interval_hours` (default 24, `0` +disables the throttle). This is read from `realsense-config.json` and is not exposed in the UI. + +The production ingest endpoint is not live yet, so uploads currently target a local dev-server +stub (`tools/rum-uploader/dev-server/rum_dev_server.py`) — see its header for how to run and +point the viewer at it. diff --git a/include/CMakeLists.txt b/include/CMakeLists.txt index 16b2c6dae1..7ec1abad21 100644 --- a/include/CMakeLists.txt +++ b/include/CMakeLists.txt @@ -19,6 +19,7 @@ target_sources( ${PROJECT_NAME} "${CMAKE_CURRENT_LIST_DIR}/librealsense2/h/rs_config.h" "${CMAKE_CURRENT_LIST_DIR}/librealsense2/h/rs_advanced_mode_command.h" "${CMAKE_CURRENT_LIST_DIR}/librealsense2/h/rs_eth_config.h" + "${CMAKE_CURRENT_LIST_DIR}/librealsense2/h/rs_rum.h" "${CMAKE_CURRENT_LIST_DIR}/librealsense2/hpp/rs_types.hpp" "${CMAKE_CURRENT_LIST_DIR}/librealsense2/hpp/rs_context.hpp" @@ -34,6 +35,7 @@ target_sources( ${PROJECT_NAME} "${CMAKE_CURRENT_LIST_DIR}/librealsense2/hpp/rs_internal.hpp" "${CMAKE_CURRENT_LIST_DIR}/librealsense2/hpp/rs_pipeline.hpp" "${CMAKE_CURRENT_LIST_DIR}/librealsense2/hpp/rs_eth_config.hpp" + "${CMAKE_CURRENT_LIST_DIR}/librealsense2/hpp/rs_rum.hpp" "${CMAKE_CURRENT_LIST_DIR}/librealsense2/rsutil.h" "${CMAKE_CURRENT_LIST_DIR}/librealsense2/rs_advanced_mode.h" diff --git a/include/librealsense2/h/rs_option.h b/include/librealsense2/h/rs_option.h index f9f9257833..7c18d81ff8 100644 --- a/include/librealsense2/h/rs_option.h +++ b/include/librealsense2/h/rs_option.h @@ -140,6 +140,7 @@ extern "C" { RS2_OPTION_DOWNSCALE_RATIO, /**< Embedded filter: secondary-frame downscale ratio (pre-stream only) */ RS2_OPTION_READOUT_SHAPING, /**< IR/depth sensor readout shaping [0-100%]; higher slows readout to avoid dropped frames */ RS2_OPTION_DETECTION_DISTANCE, /**< Enable firmware calculation of per-detection distance (meters) on the object-detection stream */ + RS2_OPTION_SENSORS_CONFIG_MODE, /**< D5x5: 0 = dedicated color sensor (3C), 1 = dual RGB (2C). Requires a hardware_reset after setting; the device then re-enumerates under the new PID. */ RS2_OPTION_COUNT /**< Number of enumeration values. Not a valid input: intended to be used in for-loops. */ } rs2_option; diff --git a/include/librealsense2/h/rs_rum.h b/include/librealsense2/h/rs_rum.h new file mode 100644 index 0000000000..cc69b27d96 --- /dev/null +++ b/include/librealsense2/h/rs_rum.h @@ -0,0 +1,55 @@ +/* License: Apache 2.0. See LICENSE file in root directory. + Copyright(c) 2026 RealSense, Inc. All Rights Reserved. */ + +/** \file rs_rum.h +* \brief +* Exposes RUM (Real User Monitoring) functionality for C compilers. +* +* RUM collects anonymous, aggregated SDK usage statistics locally. Data only leaves +* the machine when the user explicitly opts in to cloud upload. These entry points are +* always compiled and functional: rs2_rum_get_report returns the report (SDK/system +* metadata plus whatever has been collected) and the consent get/set always read and +* write the per-user config file. The ENABLE_STATS build option (default OFF) gates only +* the instrumentation hooks that feed the collector; when it is off the report's +* collected lists (devices, streams, filters, options, notifications) stay empty, but +* the API itself behaves identically. +*/ + + +#ifndef LIBREALSENSE_RS2_RUM_H +#define LIBREALSENSE_RS2_RUM_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include "rs_types.h" + +/** +* Retrieve the live RUM report for the current session as a JSON buffer, reflecting everything +* collected so far in this process. The SDK also persists the report to the app-data folder when a +* context is destroyed (for later upload), but this call reads the in-memory report, not the file. +* No upload is performed. +* \param[out] error If non-null, receives any error that occurs during this call, otherwise, errors are ignored +* \return A raw-data buffer holding the UTF-8 JSON report; release with rs2_delete_raw_data +*/ +const rs2_raw_data_buffer* rs2_rum_get_report(rs2_error** error); + +/** +* Set the cloud-upload consent flag. Persists to the per-user configuration file. +* \param[in] enabled Non-zero to opt in to cloud upload, zero to opt out +* \param[out] error If non-null, receives any error that occurs during this call, otherwise, errors are ignored +*/ +void rs2_rum_set_cloud_enabled(int enabled, rs2_error** error); + +/** +* Query the resolved cloud-upload consent (RS2_RUM_CLOUD_ENABLED env var overrides the config file). +* \param[out] error If non-null, receives any error that occurs during this call, otherwise, errors are ignored +* \return Non-zero if cloud upload is enabled, zero otherwise +*/ +int rs2_rum_is_cloud_enabled(rs2_error** error); + +#ifdef __cplusplus +} +#endif +#endif diff --git a/include/librealsense2/hpp/rs_rum.hpp b/include/librealsense2/hpp/rs_rum.hpp new file mode 100644 index 0000000000..9843e18368 --- /dev/null +++ b/include/librealsense2/hpp/rs_rum.hpp @@ -0,0 +1,55 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#ifndef LIBREALSENSE_RS2_RUM_HPP +#define LIBREALSENSE_RS2_RUM_HPP + +#include "rs_types.hpp" +#include "../h/rs_rum.h" + +#include +#include + +namespace rs2 +{ + namespace rum + { + // The live RUM report for the current session as a JSON string (the in-memory report, + // not the on-disk copy). + inline std::string get_report() + { + rs2_error* e = nullptr; + std::shared_ptr buffer( + rs2_rum_get_report(&e), rs2_delete_raw_data); + error::handle(e); + if (!buffer) + return std::string(); + auto size = rs2_get_raw_data_size(buffer.get(), &e); + error::handle(e); + auto data = rs2_get_raw_data(buffer.get(), &e); + error::handle(e); + if (!data) + return std::string(); + return std::string(reinterpret_cast(data), size); + } + + // Set the cloud-upload consent flag; persists to the per-user config file. + inline void set_cloud_enabled(bool enabled) + { + rs2_error* e = nullptr; + rs2_rum_set_cloud_enabled(enabled ? 1 : 0, &e); + error::handle(e); + } + + // Resolved cloud-upload consent (env var overrides the config file). + inline bool is_cloud_enabled() + { + rs2_error* e = nullptr; + auto enabled = rs2_rum_is_cloud_enabled(&e); + error::handle(e); + return enabled != 0; + } + } +} + +#endif // LIBREALSENSE_RS2_RUM_HPP diff --git a/include/librealsense2/rs.h b/include/librealsense2/rs.h index 374631c4eb..24ed4a624e 100644 --- a/include/librealsense2/rs.h +++ b/include/librealsense2/rs.h @@ -24,6 +24,7 @@ extern "C" { #include "h/rs_sensor.h" #include "h/rs_safety_sensor.h" #include "h/rs_eth_config.h" +#include "h/rs_rum.h" #define RS2_API_MAJOR_VERSION 2 #define RS2_API_MINOR_VERSION 58 diff --git a/include/librealsense2/rs.hpp b/include/librealsense2/rs.hpp index 1c5fbe4540..6be427e2b7 100644 --- a/include/librealsense2/rs.hpp +++ b/include/librealsense2/rs.hpp @@ -15,6 +15,7 @@ #include "hpp/rs_safety_sensor.hpp" #include "hpp/rs_pipeline.hpp" #include "hpp/rs_eth_config.hpp" +#include "hpp/rs_rum.hpp" namespace rs2 { diff --git a/package.xml b/package.xml index c75fdb5b91..5a77437934 100644 --- a/package.xml +++ b/package.xml @@ -8,7 +8,7 @@ 2.58.0 - Library for controlling and capturing data from the Intel(R) RealSense(TM) D400 devices. + Library for controlling and capturing data from the RealSense depth streaming devices. LibRealSense ROS Team diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 649e906eca..d2cc306481 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -11,6 +11,8 @@ include(${_rel_path}/usb/CMakeLists.txt) include(${_rel_path}/fw-logs/CMakeLists.txt) include(${_rel_path}/fw-update/CMakeLists.txt) +include(${_rel_path}/rum/CMakeLists.txt) + message(STATUS "using ${BACKEND}") if(UNIX OR (ANDROID_NDK_TOOLCHAIN_INCLUDED AND (${BACKEND} STREQUAL RS2_USE_V4L2_BACKEND))) diff --git a/src/context.cpp b/src/context.cpp index 8757f533c4..f6c7690ee5 100644 --- a/src/context.cpp +++ b/src/context.cpp @@ -9,6 +9,7 @@ #include "dds/rsdds-device-factory.h" #endif #include "rscore-pp-block-factory.h" +#include "rum/rum-hooks.h" #include // rs2_devices_changed_callback #include // RS2_API_FULL_VERSION_STR @@ -107,6 +108,7 @@ namespace librealsense { context::~context() { + rum::hooks::on_context_closed(); // save this session's RUM report to disk } diff --git a/src/cuda/CMakeLists.txt b/src/cuda/CMakeLists.txt index 8d217b0378..553a554b0c 100644 --- a/src/cuda/CMakeLists.txt +++ b/src/cuda/CMakeLists.txt @@ -9,5 +9,7 @@ target_sources(${LRS_TARGET} "${CMAKE_CURRENT_LIST_DIR}/cuda-frame-memory.cu" "${CMAKE_CURRENT_LIST_DIR}/cuda-frame-memory.h" "${CMAKE_CURRENT_LIST_DIR}/cuda-compat.h" + "${CMAKE_CURRENT_LIST_DIR}/cuda-rggb.cu" + "${CMAKE_CURRENT_LIST_DIR}/cuda-rggb.cuh" "${CMAKE_CURRENT_LIST_DIR}/rscuda_utils.cuh" ) diff --git a/src/cuda/cuda-rggb.cu b/src/cuda/cuda-rggb.cu new file mode 100644 index 0000000000..298cfc2246 --- /dev/null +++ b/src/cuda/cuda-rggb.cu @@ -0,0 +1,324 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#ifdef RS2_USE_CUDA + +#include "cuda-rggb.cuh" +#include // rs2_intrinsics/rs2_extrinsics (used by rscuda_utils.cuh) +#include "rscuda_utils.cuh" +#include +#include + +namespace { + +// One 8-bit Bayer sample at (x,y): the RAW10 8-bit value is just the MSB byte of each 5-byte / 4-px +// group (msb<<2 | lsb) >> 2 == msb. Edge-clamped, black-level subtracted (matches rggb-debayer.cpp). +__device__ __forceinline__ int bayer_at( const uint8_t * src, int stride, int x, int y, + int wmax, int hmax, int bl ) +{ + x = x < 0 ? 0 : ( x > wmax ? wmax : x ); + y = y < 0 ? 0 : ( y > hmax ? hmax : y ); + int v = (int)src[ (size_t)y * stride + (size_t)( x >> 2 ) * 5 + ( x & 3 ) ] - bl; + return v < 0 ? 0 : v; +} + +__device__ __forceinline__ float clamp01( float v ) { return v < 0.f ? 0.f : ( v > 1.f ? 1.f : v ); } + +__device__ __forceinline__ unsigned char clamp_u8( float v ) +{ + int i = (int)( v + 0.5f ); + return (unsigned char)( i < 0 ? 0 : ( i > 255 ? 255 : i ) ); +} + +__global__ void kernel_rggb_debayer( const uint8_t * src, int src_stride, int width, int height, + uint8_t * dst, int dst_stride, rscuda::rggb_isp_params p ) +{ + const int x = blockIdx.x * blockDim.x + threadIdx.x; + const int y = blockIdx.y * blockDim.y + threadIdx.y; + if( x >= width || y >= height ) + return; + + const int wmax = width - 1, hmax = height - 1, bl = p.black_level; + const int xodd = x & 1, yodd = y & 1; + float R, G, B; + + if( !yodd && !xodd ) // R site + { + R = (float)bayer_at( src, src_stride, x, y, wmax, hmax, bl ); + G = ( bayer_at( src, src_stride, x - 1, y, wmax, hmax, bl ) + bayer_at( src, src_stride, x + 1, y, wmax, hmax, bl ) + + bayer_at( src, src_stride, x, y - 1, wmax, hmax, bl ) + bayer_at( src, src_stride, x, y + 1, wmax, hmax, bl ) ) * 0.25f; + B = ( bayer_at( src, src_stride, x - 1, y - 1, wmax, hmax, bl ) + bayer_at( src, src_stride, x + 1, y - 1, wmax, hmax, bl ) + + bayer_at( src, src_stride, x - 1, y + 1, wmax, hmax, bl ) + bayer_at( src, src_stride, x + 1, y + 1, wmax, hmax, bl ) ) * 0.25f; + } + else if( !yodd && xodd ) // Gr site (red row): H=R, V=B + { + G = (float)bayer_at( src, src_stride, x, y, wmax, hmax, bl ); + R = ( bayer_at( src, src_stride, x - 1, y, wmax, hmax, bl ) + bayer_at( src, src_stride, x + 1, y, wmax, hmax, bl ) ) * 0.5f; + B = ( bayer_at( src, src_stride, x, y - 1, wmax, hmax, bl ) + bayer_at( src, src_stride, x, y + 1, wmax, hmax, bl ) ) * 0.5f; + } + else if( yodd && !xodd ) // Gb site (blue row): H=B, V=R + { + G = (float)bayer_at( src, src_stride, x, y, wmax, hmax, bl ); + R = ( bayer_at( src, src_stride, x, y - 1, wmax, hmax, bl ) + bayer_at( src, src_stride, x, y + 1, wmax, hmax, bl ) ) * 0.5f; + B = ( bayer_at( src, src_stride, x - 1, y, wmax, hmax, bl ) + bayer_at( src, src_stride, x + 1, y, wmax, hmax, bl ) ) * 0.5f; + } + else // B site + { + B = (float)bayer_at( src, src_stride, x, y, wmax, hmax, bl ); + G = ( bayer_at( src, src_stride, x - 1, y, wmax, hmax, bl ) + bayer_at( src, src_stride, x + 1, y, wmax, hmax, bl ) + + bayer_at( src, src_stride, x, y - 1, wmax, hmax, bl ) + bayer_at( src, src_stride, x, y + 1, wmax, hmax, bl ) ) * 0.25f; + R = ( bayer_at( src, src_stride, x - 1, y - 1, wmax, hmax, bl ) + bayer_at( src, src_stride, x + 1, y - 1, wmax, hmax, bl ) + + bayer_at( src, src_stride, x - 1, y + 1, wmax, hmax, bl ) + bayer_at( src, src_stride, x + 1, y + 1, wmax, hmax, bl ) ) * 0.25f; + } + + if( p.swap_rb ) { float t = R; R = B; B = t; } // RGGB demosaic -> BGGR (real D401 phase) + + const float gr = p.gain_r * p.digital_gain; + const float gg = p.gain_g * p.digital_gain; + const float gb = p.gain_b * p.digital_gain; + const float inv_g = ( p.gamma > 0.f ) ? 1.f / p.gamma : 1.f; + const float * m = p.ccm; + const float inv_range = 1.f / ( 255.f - (float)p.black_level ); + + // white-balance + digital gain, normalized to [0,1] (matches rggb-debayer.cpp) + float r = clamp01( R * gr * inv_range ); + float g = clamp01( G * gg * inv_range ); + float b = clamp01( B * gb * inv_range ); + // color-correction matrix + float r2 = m[0] * r + m[1] * g + m[2] * b; + float g2 = m[3] * r + m[4] * g + m[5] * b; + float b2 = m[6] * r + m[7] * g + m[8] * b; + // saturation about luma (linear, Rec.709), gamma, then contrast about mid-grey + const float yl = 0.2126f * r2 + 0.7152f * g2 + 0.0722f * b2; + r2 = clamp01( yl + p.saturation * ( r2 - yl ) ); + g2 = clamp01( yl + p.saturation * ( g2 - yl ) ); + b2 = clamp01( yl + p.saturation * ( b2 - yl ) ); + // gamma, then S-curve contrast (matches rggb-debayer.cpp's tone LUT) + float tr = powf( r2, inv_g ); tr = clamp01( tr + p.s_curve * tr * ( 1.f - tr ) * ( 2.f * tr - 1.f ) ); + float tg = powf( g2, inv_g ); tg = clamp01( tg + p.s_curve * tg * ( 1.f - tg ) * ( 2.f * tg - 1.f ) ); + float tb = powf( b2, inv_g ); tb = clamp01( tb + p.s_curve * tb * ( 1.f - tb ) * ( 2.f * tb - 1.f ) ); + const float rd = 255.f * tr, gd = 255.f * tg, bd = 255.f * tb; + + uint8_t * o = dst + (size_t)y * dst_stride + (size_t)x * 3; + o[0] = clamp_u8( ( rd - 128.f ) * p.contrast + 128.f ); + o[1] = clamp_u8( ( gd - 128.f ) * p.contrast + 128.f ); + o[2] = clamp_u8( ( bd - 128.f ) * p.contrast + 128.f ); +} + +__global__ void kernel_remap_rgb8( const uint8_t * src, int src_w, int src_h, int src_stride, + const float * sx, const float * sy, int out_w, int out_h, + uint8_t * dst, int dst_stride ) +{ + const int u = blockIdx.x * blockDim.x + threadIdx.x; + const int v = blockIdx.y * blockDim.y + threadIdx.y; + if( u >= out_w || v >= out_h ) + return; + + const int idx = v * out_w + u; + const float fx = sx[idx], fy = sy[idx]; + const int x0 = (int)floorf( fx ), y0 = (int)floorf( fy ); + uint8_t * o = dst + (size_t)v * dst_stride + (size_t)u * 3; + + if( x0 < 0 || y0 < 0 || x0 + 1 >= src_w || y0 + 1 >= src_h ) + { + o[0] = o[1] = o[2] = 0; + return; + } + const float ax = fx - x0, ay = fy - y0; + const uint8_t * p00 = src + (size_t)y0 * src_stride + (size_t)x0 * 3; + const uint8_t * p01 = p00 + 3; + const uint8_t * p10 = p00 + src_stride; + const uint8_t * p11 = p10 + 3; + for( int ch = 0; ch < 3; ++ch ) + { + float top = p00[ch] * ( 1 - ax ) + p01[ch] * ax; + float bot = p10[ch] * ( 1 - ax ) + p11[ch] * ax; + o[ch] = (uint8_t)( top * ( 1 - ay ) + bot * ay + 0.5f ); + } +} + +// Bilinear scale of a crop rectangle [crop_x,crop_x+crop_w) x [crop_y,crop_y+crop_h) within an +// interleaved RGB8 source to out_w x out_h. Output pixel centers map back into the crop (the +// +0.5/-0.5 keeps sampling centered). Mirrors rggb::crop_scale_rgb8 on the CPU. +__global__ void kernel_crop_scale_rgb8( const uint8_t * src, int src_stride, + int crop_x, int crop_y, int crop_w, int crop_h, + int out_w, int out_h, uint8_t * dst, int dst_stride ) +{ + const int u = blockIdx.x * blockDim.x + threadIdx.x; + const int v = blockIdx.y * blockDim.y + threadIdx.y; + if( u >= out_w || v >= out_h ) + return; + + const float sx = (float)crop_w / (float)out_w; + const float sy = (float)crop_h / (float)out_h; + float fx = ( u + 0.5f ) * sx - 0.5f; if( fx < 0.f ) fx = 0.f; + float fy = ( v + 0.5f ) * sy - 0.5f; if( fy < 0.f ) fy = 0.f; + int x0 = (int)fx; if( x0 > crop_w - 1 ) x0 = crop_w - 1; + int y0 = (int)fy; if( y0 > crop_h - 1 ) y0 = crop_h - 1; + const int x1 = ( x0 + 1 < crop_w ) ? x0 + 1 : x0; + const int y1 = ( y0 + 1 < crop_h ) ? y0 + 1 : y0; + const float ax = fx - x0, ay = fy - y0; + const uint8_t * p00 = src + (size_t)( crop_y + y0 ) * src_stride + (size_t)( crop_x + x0 ) * 3; + const uint8_t * p01 = src + (size_t)( crop_y + y0 ) * src_stride + (size_t)( crop_x + x1 ) * 3; + const uint8_t * p10 = src + (size_t)( crop_y + y1 ) * src_stride + (size_t)( crop_x + x0 ) * 3; + const uint8_t * p11 = src + (size_t)( crop_y + y1 ) * src_stride + (size_t)( crop_x + x1 ) * 3; + uint8_t * o = dst + (size_t)v * dst_stride + (size_t)u * 3; + for( int ch = 0; ch < 3; ++ch ) + { + float top = p00[ch] * ( 1 - ax ) + p01[ch] * ax; + float bot = p10[ch] * ( 1 - ax ) + p11[ch] * ax; + o[ch] = (uint8_t)( top * ( 1 - ay ) + bot * ay + 0.5f ); + } +} + +// 32x8 = 256 threads/block; x warp-aligned so consecutive lanes hit consecutive columns. +inline dim3 block_2d() { return dim3( rscuda::THREADS_IN_WARP, 8 ); } +inline dim3 grid_2d( int w, int h, dim3 b ) { return dim3( ( w + b.x - 1 ) / b.x, ( h + b.y - 1 ) / b.y ); } + +} // namespace + +void rscuda::rggb_debayer_raw10_cuda( const uint8_t * h_src, int src_stride, int width, int height, + uint8_t * h_dst, int dst_stride, const rscuda::rggb_isp_params & isp ) +{ + const size_t src_bytes = (size_t)src_stride * height; + const size_t dst_bytes = (size_t)dst_stride * height; + + uint8_t * src_dev = rscuda::try_device_ptr( h_src ); + uint8_t * dst_dev = rscuda::try_device_ptr( h_dst ); + std::shared_ptr s_src, s_dst; + if( !src_dev ) + { + s_src = rscuda::alloc_dev( (int)src_bytes ); + RS_CUDA_CHECK( cudaMemcpy( s_src.get(), h_src, src_bytes, cudaMemcpyHostToDevice ) ); + src_dev = s_src.get(); + } + if( !dst_dev ) + { + s_dst = rscuda::alloc_dev( (int)dst_bytes ); + dst_dev = s_dst.get(); + } + + const dim3 block = block_2d(); + const dim3 grid = grid_2d( width, height, block ); + kernel_rggb_debayer<<< grid, block >>>( src_dev, src_stride, width, height, dst_dev, dst_stride, isp ); + RS_CUDA_CHECK( cudaGetLastError() ); + + if( s_dst ) + RS_CUDA_CHECK( cudaMemcpy( h_dst, dst_dev, dst_bytes, cudaMemcpyDeviceToHost ) ); + else + RS_CUDA_CHECK( cudaStreamSynchronize( 0 ) ); // mapped output: ensure CPU sees the writes +} + +void rscuda::rggb_debayer_scale_raw10_cuda( const uint8_t * h_src, int src_stride, int native_w, int native_h, + const rscuda::rggb_isp_params & isp, + uint8_t * h_dst, int out_w, int out_h ) +{ + const int native_stride = native_w * 3; + const int dst_stride = out_w * 3; + const size_t src_bytes = (size_t)src_stride * native_h; + const size_t native_bytes = (size_t)native_stride * native_h; + const size_t dst_bytes = (size_t)dst_stride * out_h; + + // Source (RAW10) and destination (RGB8) may be zero-copy mapped frame buffers; native RGB is an + // intermediate device scratch either way. + uint8_t * src_dev = rscuda::try_device_ptr( h_src ); + uint8_t * dst_dev = rscuda::try_device_ptr( h_dst ); + std::shared_ptr s_src, s_dst; + if( !src_dev ) + { + s_src = rscuda::alloc_dev( (int)src_bytes ); + RS_CUDA_CHECK( cudaMemcpy( s_src.get(), h_src, src_bytes, cudaMemcpyHostToDevice ) ); + src_dev = s_src.get(); + } + if( !dst_dev ) + { + s_dst = rscuda::alloc_dev( (int)dst_bytes ); + dst_dev = s_dst.get(); + } + auto native = rscuda::alloc_dev( (int)native_bytes ); + + const dim3 block = block_2d(); + // 1) RAW10 -> demosaic + tone into the native RGB scratch. + kernel_rggb_debayer<<< grid_2d( native_w, native_h, block ), block >>>( + src_dev, src_stride, native_w, native_h, native.get(), native_stride, isp ); + RS_CUDA_CHECK( cudaGetLastError() ); + + // 2) Centered crop-to-aspect (same math as rggb::crop_rect_for_output) + bilinear scale -> dst. + int cw = native_w, ch = native_h; + const long long src_ar = (long long)native_w * out_h; + const long long out_ar = (long long)out_w * native_h; + if( out_ar > src_ar ) ch = (int)( ( (long long)native_w * out_h ) / out_w ); + else if( out_ar < src_ar ) cw = (int)( ( (long long)native_h * out_w ) / out_h ); + if( cw > native_w ) cw = native_w; + if( ch > native_h ) ch = native_h; + if( cw < 1 ) cw = 1; + if( ch < 1 ) ch = 1; + const int cx = ( native_w - cw ) / 2, cy = ( native_h - ch ) / 2; + + kernel_crop_scale_rgb8<<< grid_2d( out_w, out_h, block ), block >>>( + native.get(), native_stride, cx, cy, cw, ch, out_w, out_h, dst_dev, dst_stride ); + RS_CUDA_CHECK( cudaGetLastError() ); + + if( s_dst ) + RS_CUDA_CHECK( cudaMemcpy( h_dst, dst_dev, dst_bytes, cudaMemcpyDeviceToHost ) ); + else + RS_CUDA_CHECK( cudaStreamSynchronize( 0 ) ); // mapped output: ensure CPU sees the writes +} + +void rscuda::rggb_remap_rgb8_cuda( const uint8_t * h_src, int src_w, int src_h, int src_stride, + const float * sx_dev, const float * sy_dev, int out_w, int out_h, + uint8_t * h_dst, int dst_stride ) +{ + const size_t src_bytes = (size_t)src_stride * src_h; + const size_t dst_bytes = (size_t)dst_stride * out_h; + + uint8_t * src_dev = rscuda::try_device_ptr( h_src ); + uint8_t * dst_dev = rscuda::try_device_ptr( h_dst ); + std::shared_ptr s_src, s_dst; + if( !src_dev ) + { + s_src = rscuda::alloc_dev( (int)src_bytes ); + RS_CUDA_CHECK( cudaMemcpy( s_src.get(), h_src, src_bytes, cudaMemcpyHostToDevice ) ); + src_dev = s_src.get(); + } + if( !dst_dev ) + { + s_dst = rscuda::alloc_dev( (int)dst_bytes ); + dst_dev = s_dst.get(); + } + + const dim3 block = block_2d(); + const dim3 grid = grid_2d( out_w, out_h, block ); + kernel_remap_rgb8<<< grid, block >>>( src_dev, src_w, src_h, src_stride, sx_dev, sy_dev, out_w, out_h, dst_dev, dst_stride ); + RS_CUDA_CHECK( cudaGetLastError() ); + + if( s_dst ) + RS_CUDA_CHECK( cudaMemcpy( h_dst, dst_dev, dst_bytes, cudaMemcpyDeviceToHost ) ); + else + RS_CUDA_CHECK( cudaStreamSynchronize( 0 ) ); +} + +void * rscuda::rggb_cuda_alloc_upload( const void * host, size_t bytes ) +{ + void * dev = nullptr; + if( cudaMalloc( &dev, bytes ) != cudaSuccess ) + { + cudaGetLastError(); + return nullptr; + } + if( cudaMemcpy( dev, host, bytes, cudaMemcpyHostToDevice ) != cudaSuccess ) + { + cudaFree( dev ); + cudaGetLastError(); + return nullptr; + } + return dev; +} + +void rscuda::rggb_cuda_free( void * dev_ptr ) +{ + if( dev_ptr ) + cudaFree( dev_ptr ); +} + +#endif // RS2_USE_CUDA diff --git a/src/cuda/cuda-rggb.cuh b/src/cuda/cuda-rggb.cuh new file mode 100644 index 0000000000..7a9859e0d2 --- /dev/null +++ b/src/cuda/cuda-rggb.cuh @@ -0,0 +1,66 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once +#ifndef CUDA_RGGB_CUH +#define CUDA_RGGB_CUH + +#ifdef RS2_USE_CUDA + +// CUDA path for the D401 GMSL dual-RGB pipeline (mirrors src/proc/rggb-debayer.cpp and the remap +// in src/proc/stereo-rectify.cpp). Two fused kernels: +// * rggb_debayer_raw10_cuda : MIPI RAW10 -> RGGB demosaic -> white-balance/digital gain -> tone +// * rggb_remap_rgb8_cuda : bilinear rectification remap (maps precomputed on the host) +// +// Both follow the project's zero-copy convention (see cuda-pointcloud.cu): when the frame buffers +// are CUDA pinned+mapped (integrated GPU, zero-copy build) the kernel reads/writes them in place +// with no host<->device copy; otherwise a per-call staging buffer + copy is used. +// +// This header is intentionally free of CUDA types so host translation units (rggb-converter.cpp, +// dual-rgb-rectify-filter.cpp) can include it under RS2_USE_CUDA without pulling in cuda_runtime.h. + +#include +#include + +namespace rscuda +{ + // Mirrors librealsense::rggb::isp_params (kept separate to avoid an SDK include here). + struct rggb_isp_params + { + int black_level; + float gain_r, gain_g, gain_b; + float digital_gain; + float gamma; + float s_curve; // contrast S-curve strength baked after gamma (the "pop") + float saturation; + float contrast; + int swap_rb; // 1 => BGGR (swap R/B after demosaic); 0 => RGGB + float ccm[9]; // row-major 3x3 sensor-RGB -> display-RGB color-correction matrix + }; + + // RAW10 (4 px / 5 bytes, RGGB) -> interleaved RGB8. src_stride/dst_stride are bytes per row. + // width must be a multiple of 4; src holds width*10/8 active bytes per row (plus alignment). + void rggb_debayer_raw10_cuda( const uint8_t * src, int src_stride, int width, int height, + uint8_t * dst, int dst_stride, const rggb_isp_params & isp ); + + // RAW10 -> demosaic to native RGB8 (native_w x native_h), then center-crop to the output aspect + // ratio and bilinear-scale to out_w x out_h, written tightly into dst (out_w*3 bytes/row). Used + // for the user-selectable dual-RGB output resolutions (crop-to-aspect + scale, no stretch). + // dst may be a zero-copy mapped frame buffer; a device scratch holds the native RGB in between. + void rggb_debayer_scale_raw10_cuda( const uint8_t * src, int src_stride, int native_w, int native_h, + const rggb_isp_params & isp, + uint8_t * dst, int out_w, int out_h ); + + // Bilinear remap of an interleaved RGB8 image. sx_dev/sy_dev are DEVICE pointers (out_w*out_h + // floats each) holding, per output pixel, the source pixel to sample (out-of-range -> black). + void rggb_remap_rgb8_cuda( const uint8_t * src, int src_w, int src_h, int src_stride, + const float * sx_dev, const float * sy_dev, int out_w, int out_h, + uint8_t * dst, int dst_stride ); + + // cudaMalloc + H2D upload of `bytes` from host; returns the device pointer (nullptr on failure). + // Used to stage the (constant) remap tables once. Free with rggb_cuda_free. + void * rggb_cuda_alloc_upload( const void * host, size_t bytes ); + void rggb_cuda_free( void * dev_ptr ); +} + +#endif // RS2_USE_CUDA +#endif // CUDA_RGGB_CUH diff --git a/src/ds/d400/d400-color.cpp b/src/ds/d400/d400-color.cpp index 269d7be5a4..329e890c6b 100644 --- a/src/ds/d400/d400-color.cpp +++ b/src/ds/d400/d400-color.cpp @@ -7,6 +7,7 @@ #include #include #include "proc/color-formats-converter.h" +#include "proc/rggb-converter.h" #include "d400-color.h" #include "d400-info.h" #include @@ -26,7 +27,8 @@ namespace librealsense {rs_fourcc('U','Y','V','Y'), RS2_FORMAT_UYVY}, {rs_fourcc('M','J','P','G'), RS2_FORMAT_MJPEG}, {rs_fourcc('R','W','1','6'), RS2_FORMAT_RAW16}, - {rs_fourcc('B','Y','R','2'), RS2_FORMAT_RAW16} + {rs_fourcc('B','Y','R','2'), RS2_FORMAT_RAW16}, + {rs_fourcc('B','A','8','1'), RS2_FORMAT_RAW8} // D401 GMSL dual-RGB: SBGGR8 (driver PR #459), RAW10 in disguise }; std::map d400_color_fourcc_to_rs2_stream = { {rs_fourcc('Y','U','Y','2'), RS2_STREAM_COLOR}, @@ -34,7 +36,8 @@ namespace librealsense {rs_fourcc('U','Y','V','Y'), RS2_STREAM_COLOR}, {rs_fourcc('R','W','1','6'), RS2_STREAM_COLOR}, {rs_fourcc('B','Y','R','2'), RS2_STREAM_COLOR}, - {rs_fourcc('M','J','P','G'), RS2_STREAM_COLOR} + {rs_fourcc('M','J','P','G'), RS2_STREAM_COLOR}, + {rs_fourcc('B','A','8','1'), RS2_STREAM_COLOR} // D401 GMSL dual-RGB: SBGGR8 (driver PR #459) }; d400_color::d400_color( std::shared_ptr< const d400_info > const & dev_info ) @@ -66,8 +69,24 @@ namespace librealsense _color_extrinsic = std::make_shared< rsutils::lazy< rs2_extrinsics > >( [this]() { return from_pose( get_d400_color_stream_extrinsic( *_color_calib_table_raw ) ); } ); - environment::get_instance().get_extrinsics_graph().register_extrinsics(*_color_stream, *_depth_stream, _color_extrinsic); - register_stream_to_extrinsic_group(*_color_stream, 0); + auto & ext_graph = environment::get_instance().get_extrinsics_graph(); + if (_pid == RS401_GMSL_PID) + { + // D401 GMSL dual-RGB: the two color streams ARE the two stereo imagers. Tie each color + // stream to its imager's pose (left/right IR) so the inter-stream extrinsics carry the + // real stereo baseline (Color0->Color1 == IR1->IR2) and rectification has correct geometry. + // (Otherwise both colors share one pose and Color0->Color1 is zero.) + ext_graph.register_same_extrinsics( *_color_stream, *_left_ir_stream ); + register_stream_to_extrinsic_group(*_color_stream, 0); + _color_stream2 = std::make_shared< stream >( RS2_STREAM_COLOR ); + ext_graph.register_same_extrinsics( *_color_stream2, *_right_ir_stream ); + register_stream_to_extrinsic_group(*_color_stream2, 0); + } + else + { + ext_graph.register_extrinsics(*_color_stream, *_depth_stream, _color_extrinsic); + register_stream_to_extrinsic_group(*_color_stream, 0); + } std::vector color_devs_info; // end point 3 is used for color sensor @@ -93,7 +112,25 @@ namespace librealsense auto enable_global_time_option = std::shared_ptr(new global_time_option()); platform::uvc_device_info info; if (_is_mipi_device) - info = color_devs_info[1]; + { + // The driver names the color node "video-rs-color-". Only when those links are absent + // does the group keep its positional layout: depth, color, IR, IMU. Indexing blindly copies + // a uvc_device_info that may be past the end, which faults on the copied strings. + auto find_path = [&color_devs_info](const char * hint) + { + return std::find_if(color_devs_info.begin(), color_devs_info.end(), + [hint](const platform::uvc_device_info& i) + { return i.device_path.find(hint) != std::string::npos; }); + }; + auto color_node = find_path("video-rs-color"); + if (color_node != color_devs_info.end()) + info = *color_node; + else if (find_path("video-rs-") == color_devs_info.end() && color_devs_info.size() > 1) + info = color_devs_info[1]; + else + throw backend_exception("cannot access color sensor - no color node in a MIPI group of " + + std::to_string(color_devs_info.size())); + } else info = color_devs_info.front(); auto uvcd = get_backend()->create_uvc_device( info ); @@ -342,7 +379,7 @@ namespace librealsense // MIPI on x86 (ADL-P) color_ep.register_processing_block(processing_block_factory::create_pbf_vector(RS2_FORMAT_YUYV, map_supported_color_formats(RS2_FORMAT_YUYV), RS2_STREAM_COLOR)); } - } + } } void d400_color::register_metadata_mipi(const synthetic_sensor &color_ep) const diff --git a/src/ds/d400/d400-device.cpp b/src/ds/d400/d400-device.cpp index 137fe994c2..0fb9698c60 100644 --- a/src/ds/d400/d400-device.cpp +++ b/src/ds/d400/d400-device.cpp @@ -25,6 +25,8 @@ #include #include #include +#include +#include #include #include @@ -34,6 +36,7 @@ #include #include using rsutils::type::fourcc; +#include #include #include @@ -62,7 +65,8 @@ namespace librealsense {fourcc('Z','1','6',' '), RS2_FORMAT_Z16}, {fourcc('R','G','B','2'), RS2_FORMAT_BGR8}, {fourcc('M','J','P','G'), RS2_FORMAT_MJPEG}, - {fourcc('B','Y','R','2'), RS2_FORMAT_RAW16} + {fourcc('B','Y','R','2'), RS2_FORMAT_RAW16}, + {fourcc('B','A','8','1'), RS2_FORMAT_RAW8} // D401 GMSL dual-RGB: SBGGR8 8-bit Bayer (RAW8 CSI passthrough, driver PR #459) }; std::map d400_depth_fourcc_to_rs2_stream = { @@ -78,9 +82,38 @@ namespace librealsense {fourcc('Z','1','6',' '), RS2_STREAM_DEPTH}, {fourcc('Z','1','6','H'), RS2_STREAM_DEPTH}, {fourcc('B','Y','R','2'), RS2_STREAM_COLOR}, - {fourcc('M','J','P','G'), RS2_STREAM_COLOR} + {fourcc('M','J','P','G'), RS2_STREAM_COLOR}, + {fourcc('B','A','8','1'), RS2_STREAM_COLOR} // D401 GMSL dual-RGB: SBGGR8, expose each OV9782 imager as color }; + // D401 GMSL dual-RGB stream-id resolver. The two OV9782 imagers each arrive on a separate backend pin, + // both advertising identical SBGGR8 (fourcc BA81). Rank the BA81 color pins by ascending pin_index and + // route them to Color 0 / Color 1 (distinct streams), mirroring the IR1/IR2 split. Uses the upstream + // per-pin _stream_id_resolver mechanism (cf. d500_dual_rgb::resolve_color_stream). + static void resolve_d401_color_stream( const std::vector< platform::stream_profile > & all, + const platform::stream_profile & p, rs2_stream & type, int & index ) + { + const auto ba81 = fourcc( 'B', 'A', '8', '1' ); + if( p.format != ba81 ) + return; // not a D401 color pin - leave type/index as resolved by the fourcc map + + std::set< uint32_t > color_pins; + for( auto & q : all ) + if( q.format == ba81 ) + color_pins.insert( q.pin_index ); + + int rank = 0; + for( auto cp : color_pins ) + { + if( cp == p.pin_index ) + break; + ++rank; + } + + type = RS2_STREAM_COLOR; + index = rank; // Color 0 (left imager), Color 1 (right) + } + std::vector d400_device::send_receive_raw_data(const std::vector& input) { return _hw_monitor->send(input); @@ -167,7 +200,12 @@ namespace librealsense processing_blocks d400_depth_sensor::get_recommended_processing_blocks() const { - return get_ds_depth_recommended_proccesing_blocks(); + auto res = get_ds_depth_recommended_proccesing_blocks(); + // D401 GMSL dual-RGB: rectify the two color streams (default-on, toggleable in the viewer's + // Post-Processing). The filter self-configures from the color profiles' SDK calibration. + if( _owner->_pid == ds::RS401_GMSL_PID ) + res.push_back( std::make_shared< dual_rgb_rectify_filter >() ); + return res; } rs2_intrinsics d400_depth_sensor::get_intrinsics( const stream_profile & profile ) const @@ -222,7 +260,42 @@ namespace librealsense rs2_intrinsics d400_depth_sensor::get_color_intrinsics( const stream_profile & profile ) const { - if( _owner->_pid == ds::RS405_PID || _owner->_pid == ds::RS401_GMSL_PID ) + if( _owner->_pid == ds::RS401_GMSL_PID ) + { + // D401 dual-RGB: every output resolution is produced by demosaicing the native 1288x808 + // image, center-cropping to the output aspect ratio, then bilinear-scaling (see + // rggb_converter / rggb::crop_scale_rgb8). So a non-native resolution's intrinsics are + // the NATIVE intrinsics transformed by that crop + scale -- NOT a plain resize of the + // calibration (which get_d405_color_stream_intrinsic would give and which ignores the + // crop). Compute native, then apply crop-offset + scale-factor consistently with the + // image path so rectify / deprojection / pointcloud stay correct at every resolution. + const int native_w = 1288, native_h = 808; + rs2_intrinsics in = ds::get_d405_color_stream_intrinsic( *_owner->_color_calib_table_raw, + native_w, native_h ); + const int out_w = (int)profile.width, out_h = (int)profile.height; + if( out_w == native_w && out_h == native_h ) + return in; + + int cx, cy, cw, ch; + rggb::crop_rect_for_output( native_w, native_h, out_w, out_h, &cx, &cy, &cw, &ch ); + const float sx = (float)out_w / (float)cw; + const float sy = (float)out_h / (float)ch; + + rs2_intrinsics out = in; // model + distortion coeffs are unchanged + out.width = out_w; + out.height = out_h; + out.fx = in.fx * sx; + out.fy = in.fy * sy; + // Shift the principal point into crop coordinates, then scale. crop_scale_rgb8 samples + // with the pixel-center convention (src = (out+0.5)/scale - 0.5), so carry the matching + // +0.5/-0.5 half-pixel terms; otherwise ppx/ppy are biased by 0.5*(sx-1) (~0.3 px at the + // smallest resolution). fx/fy need no such term (focal length is origin-independent). + out.ppx = ( in.ppx - (float)cx + 0.5f ) * sx - 0.5f; + out.ppy = ( in.ppy - (float)cy + 0.5f ) * sy - 0.5f; + return out; + } + + if( _owner->_pid == ds::RS405_PID ) return ds::get_d405_color_stream_intrinsic( *_owner->_color_calib_table_raw, profile.width, profile.height ); @@ -261,7 +334,11 @@ namespace librealsense } else if (p->get_stream_type() == RS2_STREAM_COLOR) { - assign_stream(_owner->_color_stream, p); + // D401 GMSL dual-RGB: color index 0 = left imager, index 1 = right (distinct streams). + if (p->get_stream_index() == 1 && _owner->_color_stream2) + assign_stream(_owner->_color_stream2, p); + else + assign_stream(_owner->_color_stream, p); } auto&& vid_profile = dynamic_cast(p.get()); @@ -691,6 +768,62 @@ namespace librealsense { {RS2_FORMAT_Y16, RS2_STREAM_INFRARED, 1}, {RS2_FORMAT_Y16, RS2_STREAM_INFRARED, 2} }, []() {return std::make_shared(); } ); + + // D401 GMSL dual-RGB: the two OV9782 imagers stream 8-bit RGGB Bayer via the FW RAW8 + // CSI passthrough. Expose each as a color stream - crop the transport padding + // (1612 -> 1288 px) and demosaic RGGB -> RGB8. The per-imager stream index (0/1) is + // carried through from the source profile, mirroring the IR1/IR2 split. + if( _pid == RS401_GMSL_PID ) + { + // Route the two identical BGGR color pins to Color 0 / Color 1 (ascending pin order). + raw_depth_sensor->set_stream_id_resolver( resolve_d401_color_stream ); + + // Both imagers share one hardware frame counter; without a per-stream counter the + // reported color FPS reads 2x. + raw_depth_sensor->enable_software_color_frame_numbers(); + + // The camera always delivers one native color resolution (1288x808 after the + // 1612 transport crop). For the user we expose the standard resolutions too: each + // is produced by demosaicing to native then center-cropping to that aspect ratio + // and bilinear-scaling (crop-to-aspect + scale, no stretch; see rggb_converter / + // cuda-rggb). We mirror the depth resolution set so the viewer offers one shared + // resolution across depth + Color 0/1 (no per-stream resolution UI needed). + // + // resolution_transform is a plain function pointer (no captures), so each output + // resolution needs its own captureless transform; the converter factory (a + // std::function) captures the target size. Index 0 = left imager, 1 = right; the + // resolver tags the two RGGB sources 0/1 and formats-converter matches by index. + static const int NATIVE_W = 1288; + struct color_res { int w, h; void ( *xf )( uint32_t &, uint32_t & ); }; + // Mirror the depth resolution set exactly (top out at 1280x720, not the native + // 1288x808) so color shares every resolution with depth/IR. That keeps the viewer + // on a single shared Resolution dropdown and lets depth + IR + Color 0/1 always be + // selected together (depth has no 1288x808 mode). The native 1288x808 is still the + // internal capture/demosaic size; 1280x720 is its center-cropped 16:9 output. + static const color_res color_resolutions[] = { + { 1280, 720, []( uint32_t & w, uint32_t & h ) { w = 1280; h = 720; } }, + { 848, 480, []( uint32_t & w, uint32_t & h ) { w = 848; h = 480; } }, + { 640, 480, []( uint32_t & w, uint32_t & h ) { w = 640; h = 480; } }, + { 640, 360, []( uint32_t & w, uint32_t & h ) { w = 640; h = 360; } }, + { 480, 270, []( uint32_t & w, uint32_t & h ) { w = 480; h = 270; } }, + { 424, 240, []( uint32_t & w, uint32_t & h ) { w = 424; h = 240; } }, + }; + for( auto & r : color_resolutions ) + { + const int rw = r.w, rh = r.h; + depth_sensor.register_processing_block( + { { RS2_FORMAT_RAW8, RS2_STREAM_COLOR } }, + { { RS2_FORMAT_RGB8, RS2_STREAM_COLOR, 0, 0, 0, 0, r.xf }, + { RS2_FORMAT_RGB8, RS2_STREAM_COLOR, 1, 0, 0, 0, r.xf } }, + [rw, rh]() { + rggb::isp_params isp; + isp.swap_rb = true; // OV9782 is BGGR (driver declares SBGGR8); the base + // demosaic is RGGB-pattern, so swap R<->B to correct it + return std::make_shared< rggb_converter >( RS2_FORMAT_RGB8, NATIVE_W, rw, rh, isp ); + } + ); + } + } } diff --git a/src/ds/d400/d400-device.h b/src/ds/d400/d400-device.h index 1517596c2a..3830872658 100644 --- a/src/ds/d400/d400-device.h +++ b/src/ds/d400/d400-device.h @@ -147,6 +147,7 @@ namespace librealsense std::shared_ptr _left_ir_stream; std::shared_ptr _right_ir_stream; std::shared_ptr _color_stream; + std::shared_ptr _color_stream2; // D401 GMSL dual-RGB: 2nd color (right imager) uint8_t _depth_device_idx; diff --git a/src/ds/d400/d400-factory.cpp b/src/ds/d400/d400-factory.cpp index 490d364a82..2ff1a6aed2 100644 --- a/src/ds/d400/d400-factory.cpp +++ b/src/ds/d400/d400-factory.cpp @@ -97,6 +97,21 @@ namespace librealsense std::shared_ptr create_matcher(const frame_holder& frame) const override; + // D401 GMSL dual-RGB + depth coexistence (FW 5.17.3.151+): the two COLOR streams come from + // independent EP imagers (1288x808), decoupled from depth/IR (e.g. 1280x720). COLOR must NOT + // contradict another stream on resolution, else depth+color can't be requested together (the + // base rule rejects any width/height mismatch). Depth/IR still cross-check among themselves. + bool contradicts( const stream_profile_interface * a, const std::vector< stream_profile > & others ) const override + { + if( a->get_stream_type() == RS2_STREAM_COLOR ) + return false; + std::vector< stream_profile > non_color; + for( auto & sp : others ) + if( sp.stream != RS2_STREAM_COLOR ) + non_color.push_back( sp ); + return device::contradicts( a, non_color ); + } + std::vector get_profiles_tags() const override { std::vector tags; @@ -1252,6 +1267,10 @@ namespace librealsense std::shared_ptr rs401_gmsl_device::create_matcher(const frame_holder& frame) const { std::vector streams = { _depth_stream.get() , _left_ir_stream.get() , _right_ir_stream.get(), _color_stream.get() }; + // D401 GMSL dual-RGB: the second color stream (right imager) must be known to the syncer, + // otherwise its frames have no matcher -> create_matcher recurses -> stack/heap corruption. + if( _color_stream2 ) + streams.push_back( _color_stream2.get() ); return matcher_factory::create(RS2_MATCHER_DEFAULT, streams); } diff --git a/src/ds/d400/d400-motion.cpp b/src/ds/d400/d400-motion.cpp index cc9048a0aa..6210d06a3c 100644 --- a/src/ds/d400/d400-motion.cpp +++ b/src/ds/d400/d400-motion.cpp @@ -282,6 +282,12 @@ namespace librealsense if (!is_fisheye_avaialable) return; + if (fisheye_infos.empty()) + { + LOG_WARNING("FishEye sensor reported as available with no UVC node - sensor not created"); + return; + } + std::unique_ptr< frame_timestamp_reader > ds_timestamp_reader_backup( new ds_timestamp_reader() ); std::unique_ptr ds_timestamp_reader_metadata(new ds_timestamp_reader_from_metadata(std::move(ds_timestamp_reader_backup))); auto enable_global_time_option = std::shared_ptr(new global_time_option()); diff --git a/src/ds/d500/d500-color.cpp b/src/ds/d500/d500-color.cpp index 0d528c4727..f4e1a0cef8 100644 --- a/src/ds/d500/d500-color.cpp +++ b/src/ds/d500/d500-color.cpp @@ -107,9 +107,9 @@ namespace librealsense this ); auto color_ep = std::make_shared(this, - raw_color_ep, - d500_color_fourcc_to_rs2_format, - d500_color_fourcc_to_rs2_stream); + raw_color_ep, + d500_color_fourcc_to_rs2_format, + d500_color_fourcc_to_rs2_stream); color_ep->register_option(RS2_OPTION_GLOBAL_TIME_ENABLED, enable_global_time_option); @@ -162,8 +162,16 @@ namespace librealsense RS2_FORMAT_M420, map_supported_color_formats( RS2_FORMAT_M420 ), RS2_STREAM_COLOR ) ); - color_ep.register_processing_block( - processing_block_factory::create_id_pbf( RS2_FORMAT_YUYV, RS2_STREAM_COLOR ) ); + // MIPI FW currently delivers YUYV rather than NV12, so convert it to RGB until NV12 is supported. + // On USB, YUY2 remains exposed passthrough-only. + if( _is_mipi_device ) + color_ep.register_processing_block( processing_block_factory::create_pbf_vector< yuy2_converter >( + RS2_FORMAT_YUYV, + map_supported_color_formats( RS2_FORMAT_YUYV ), + RS2_STREAM_COLOR ) ); + else + color_ep.register_processing_block( + processing_block_factory::create_id_pbf( RS2_FORMAT_YUYV, RS2_STREAM_COLOR ) ); break; default: throw invalid_value_exception( "invalid native color format " diff --git a/src/ds/d500/d500-device.cpp b/src/ds/d500/d500-device.cpp index 2b2e92c5b1..63920c5f23 100644 --- a/src/ds/d500/d500-device.cpp +++ b/src/ds/d500/d500-device.cpp @@ -567,6 +567,18 @@ namespace librealsense depth_sensor.register_option(RS2_OPTION_PROJECTOR_TEMPERATURE, proj_temperature); } + if (d5x5_family_pids.count(_pid)) + { + depth_sensor.register_option( RS2_OPTION_SENSORS_CONFIG_MODE, + std::make_shared< uvc_xu_option< uint8_t > >( + raw_depth_sensor, + depth_xu, + d500_xu_id::DUAL_RGB_MODE, + "Dedicated color sensor (0) vs dual RGB (1). Requires a hardware reset to take effect.", + std::map< float, std::string >{ { 0.f, "Dedicated Color Sensor" }, { 1.f, "Dual RGB" } }, + false /* not settable while streaming */ ) ); + } + auto error_control = std::make_shared< uvc_xu_option< uint8_t > >( raw_depth_sensor, depth_xu, DS5_ERROR_REPORTING, diff --git a/src/ds/d500/d500-factory.cpp b/src/ds/d500/d500-factory.cpp index b5c4609e7f..b667224b00 100644 --- a/src/ds/d500/d500-factory.cpp +++ b/src/ds/d500/d500-factory.cpp @@ -103,6 +103,16 @@ namespace librealsense }; + void add_motion_streams( const std::shared_ptr< ds_motion_common > & motion_common, + std::vector< std::shared_ptr< librealsense::stream_interface > > & streams ) + { + if( motion_common ) + { + streams.push_back( motion_common->get_accel_stream() ); + streams.push_back( motion_common->get_gyro_stream() ); + } + } + // D585 or D535, dual color variant. No dedicated color sensor. class rs5x5_device : public d500_active @@ -133,9 +143,8 @@ namespace librealsense { std::vector< std::shared_ptr< stream_interface > > streams = { _depth_stream, _left_ir_stream, _right_ir_stream, - _color_stream_1, _color_stream_2, - _ds_motion_common->get_accel_stream(), - _ds_motion_common->get_gyro_stream() }; + _color_stream_1, _color_stream_2 }; + add_motion_streams( _ds_motion_common, streams ); return create_default_matcher( streams ); } @@ -155,64 +164,6 @@ namespace librealsense }; - // D585 GMSL (MIPI) variant with dedicated color sensor. On MIPI the color and IMU are exposed as - // separate V4L2 nodes; d500_motion selects the UVC-motion path at runtime from _is_mipi_device, - // so this uses the single-color (d500_color) path rather than the USB dual-color path of rs5x5_device. - class rs5x5_gmsl_dedicated_color_device - : public d500_active - , public d500_color - , public d500_motion - , public ds_advanced_mode_base - , public extended_firmware_logger_device - { - public: - rs5x5_gmsl_dedicated_color_device( std::shared_ptr< const d500_info > const & dev_info ) - : device( dev_info ) - , backend_device( dev_info ) - , d500_device( dev_info ) - , d500_active( dev_info ) - , d500_color( dev_info, RS2_FORMAT_YUYV ) - , d500_motion( dev_info ) - , ds_advanced_mode_base() - , extended_firmware_logger_device( dev_info, d500_device::_hw_monitor, get_firmware_logs_command() ) - { - ds_advanced_mode_base::initialize_advanced_mode( this ); - - // Improved Close Range Depth - USB toggle - // Disabled on D585 GMSL: the MIPI V4L2 backend has no CID for the close-range depth-XU selector (0x14). - //register_feature( std::make_shared< close_range_filter_feature >( - // dynamic_cast< d500_depth_sensor & >( get_depth_sensor() ) ) ); - } - - std::shared_ptr create_matcher(const frame_holder& frame) const override - { - - std::vector< std::shared_ptr< stream_interface > > streams = { _depth_stream, _left_ir_stream, _right_ir_stream, _color_stream }; - if( ! _has_motion_module_failed && _ds_motion_common ) - { - streams.push_back( _ds_motion_common->get_accel_stream() ); - streams.push_back( _ds_motion_common->get_gyro_stream() ); - } - return create_default_matcher( streams ); - } - - std::vector get_profiles_tags() const override - { - std::vector tags; - - tags.push_back({ RS2_STREAM_COLOR, -1, 1280, 720, RS2_FORMAT_RGB8, 30, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); - tags.push_back({ RS2_STREAM_DEPTH, -1, 1280, 720, RS2_FORMAT_Z16, 30, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); - tags.push_back({ RS2_STREAM_INFRARED, -1, 1280, 720, RS2_FORMAT_Y8, 30, profile_tag::PROFILE_TAG_SUPERSET }); - // UVC motion requires accel and gyro at equal fps (see uvc_sensor::verify_supported_requests), - // so both defaults must match or the viewer fails to start the motion module. - tags.push_back({ RS2_STREAM_GYRO, -1, 0, 0, RS2_FORMAT_MOTION_XYZ32F, (int)odr::IMU_FPS_200, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); - tags.push_back({ RS2_STREAM_ACCEL, -1, 0, 0, RS2_FORMAT_MOTION_XYZ32F, (int)odr::IMU_FPS_200, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); - - return tags; - }; - }; - - // D585 or D535 with dedicated color sensor. Can be with IR filter on lens or without. class rs5x5_dedicated_color_device : public d500_active @@ -236,18 +187,17 @@ namespace librealsense { ds_advanced_mode_base::initialize_advanced_mode( this ); - // Improved Close Range Depth - USB toggle - register_feature( std::make_shared< close_range_filter_feature >( - dynamic_cast< d500_depth_sensor & >( get_depth_sensor() ) ) ); + // Skipped on MIPI: the V4L2 backend has no CID for the close-range depth-XU selector (0x14). + if( ! _is_mipi_device ) + register_feature( std::make_shared< close_range_filter_feature >( dynamic_cast< d500_depth_sensor & >( get_depth_sensor() ) ) ); } std::shared_ptr create_matcher(const frame_holder& frame) const override { std::vector< std::shared_ptr< stream_interface > > streams = { _depth_stream, _left_ir_stream, _right_ir_stream, _color_stream, - _ds_motion_common->get_accel_stream(), - _ds_motion_common->get_gyro_stream(), _object_detection_stream }; + add_motion_streams( _ds_motion_common, streams ); return create_default_matcher( streams ); } @@ -255,13 +205,17 @@ namespace librealsense { std::vector tags; + // MIPI requires accel and gyro at equal fps, USB uses 100 for accel. + int gyro_fps = static_cast< int >( odr::IMU_FPS_200 ); + int accel_fps = _is_mipi_device ? static_cast< int >( odr::IMU_FPS_200 ) : static_cast< int >( odr::IMU_FPS_100 ); + tags.push_back({ RS2_STREAM_COLOR, -1, 1280, 720, RS2_FORMAT_RGB8, 30, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); tags.push_back({ RS2_STREAM_DEPTH, -1, 1280, 720, RS2_FORMAT_Z16, 30, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); tags.push_back({ RS2_STREAM_INFRARED, -1, 1280, 720, RS2_FORMAT_Y8, 30, profile_tag::PROFILE_TAG_SUPERSET }); - tags.push_back({ RS2_STREAM_GYRO, -1, 0, 0, RS2_FORMAT_MOTION_XYZ32F, (int)odr::IMU_FPS_200, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); - tags.push_back({ RS2_STREAM_ACCEL, -1, 0, 0, RS2_FORMAT_MOTION_XYZ32F, (int)odr::IMU_FPS_100, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); + tags.push_back({ RS2_STREAM_GYRO, -1, 0, 0, RS2_FORMAT_MOTION_XYZ32F, gyro_fps, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); + tags.push_back({ RS2_STREAM_ACCEL, -1, 0, 0, RS2_FORMAT_MOTION_XYZ32F, accel_fps, profile_tag::PROFILE_TAG_SUPERSET | profile_tag::PROFILE_TAG_DEFAULT }); tags.push_back({ RS2_STREAM_OBJECT_DETECTION, -1, -1, -1, RS2_FORMAT_Y8, -1, profile_tag::PROFILE_TAG_SUPERSET }); - + return tags; }; }; @@ -294,9 +248,8 @@ namespace librealsense { std::vector< std::shared_ptr< stream_interface > > streams = { _depth_stream, _left_ir_stream, _right_ir_stream, _color_stream, - _ds_motion_common->get_accel_stream(), - _ds_motion_common->get_gyro_stream(), _object_detection_stream }; + add_motion_streams( _ds_motion_common, streams ); return create_default_matcher( streams ); } @@ -356,9 +309,8 @@ namespace librealsense std::shared_ptr create_matcher(const frame_holder& frame) const override { std::vector< std::shared_ptr< stream_interface > > streams = { _depth_stream, _left_ir_stream, _right_ir_stream, _color_stream, - _safety_stream, _occupancy_stream, _point_cloud_stream, - _ds_motion_common->get_accel_stream(), - _ds_motion_common->get_gyro_stream() }; + _safety_stream, _occupancy_stream, _point_cloud_stream }; + add_motion_streams( _ds_motion_common, streams ); return create_default_matcher( streams ); } @@ -443,9 +395,8 @@ namespace librealsense { std::vector< std::shared_ptr< stream_interface > > streams = { _depth_stream, _left_ir_stream, _right_ir_stream, _color_stream, - _ds_motion_common->get_accel_stream(), - _ds_motion_common->get_gyro_stream(), _object_detection_stream }; + add_motion_streams( _ds_motion_common, streams ); return create_default_matcher( streams ); } @@ -475,7 +426,6 @@ namespace librealsense auto dev_info = std::dynamic_pointer_cast< const d500_info >( shared_from_this() ); auto pid = _group.uvc_devices.front().pid; - bool is_mipi = _group.uvc_devices.front().is_mipi; try { @@ -496,9 +446,6 @@ namespace librealsense case ds::D585_3C_PID: case ds::D585F_PID: case ds::D585_3C_PROTO_PID: - // On MIPI/GMSL the color and IMU are exposed as dedicated V4L2 nodes. - if( is_mipi ) - return std::make_shared< rs5x5_gmsl_dedicated_color_device >( dev_info ); return std::make_shared< rs5x5_dedicated_color_device >( dev_info ); default: throw std::runtime_error( rsutils::string::from() << "unsupported D500 PID 0x" << hexdump( pid ) ); diff --git a/src/ds/d500/d500-private.h b/src/ds/d500/d500-private.h index ea7e59b244..6059bcddab 100644 --- a/src/ds/d500/d500-private.h +++ b/src/ds/d500/d500-private.h @@ -36,6 +36,7 @@ namespace librealsense { DETECTION_DISTANCE = 0x01, // Enable FW depth-derived distance for detections ALIGN_DEPTH = 0x10, // Enable depth-to-RGB alignment for OD distance; must be sent before depth streaming starts + DUAL_RGB_MODE = 0x12, // FW spec: csEU_CONTROL_ADVANCED_DEVICE_MODE. 1-byte GET/SET: 0 = dedicated color sensor (3C), 1 = dual RGB (2C). SET triggers PID change on next enumeration. PVT_TEMPERATURE = 0x15, PROJECTOR_TEMPERATURE = 0x16, OHM_TEMPERATURE = 0x17 @@ -74,9 +75,12 @@ namespace librealsense D585_3C_PROTO_PID }; - // D5x5 (non-safety, non-legacy) interactive Triggered Calibration flow. - // D555 stays on the D400 OCC path; D585S and D585_LEGACY_PID stay on the current D500 triggered-calibration flow. - static const std::set d5x5_interactive_triggered_calibration_pids = { + // D5x5 (non-safety, non-legacy) SKU family: D535 and D585 in their 2C/3C/F/proto variants. + // Used to gate features that are only exposed by the modern D5x5 FW branch — currently: + // - interactive Triggered Calibration flow (D555 stays on the D400 OCC path; + // D585S and D585_LEGACY_PID stay on the current D500 triggered-calibration flow) + // - the DUAL_RGB_MODE XU (0x12) selector that toggles Dual-RGB (2C) vs Dedicated-Color (3C) + static const std::set d5x5_family_pids = { D535_2C_PID, D535_3C_PID, D535F_PID, @@ -89,7 +93,7 @@ namespace librealsense inline bool uses_interactive_triggered_calibration( uint16_t pid ) { - return d5x5_interactive_triggered_calibration_pids.find( pid ) != d5x5_interactive_triggered_calibration_pids.end(); + return d5x5_family_pids.find( pid ) != d5x5_family_pids.end(); } static const std::map< std::uint16_t, std::string > rs500_sku_names = { diff --git a/src/gl/pointcloud-gl.cpp b/src/gl/pointcloud-gl.cpp index 5f6e135830..dd0760e240 100644 --- a/src/gl/pointcloud-gl.cpp +++ b/src/gl/pointcloud-gl.cpp @@ -423,9 +423,18 @@ const librealsense::float3* pointcloud_gl::depth_to_points( _depth_data = depth_frame; _depth_scale = depth_frame.get_units(); _depth_intr = depth_intrinsics; + + // With no texture mapped the base class skips get_texture_map, leaving the output empty. + // Render here instead against the depth stream itself, so the shader passes UVs through. + if (!_extrinsics || !_other_intrinsics) + { + get_texture_map(output, nullptr, depth_frame.get_width(), depth_frame.get_height(), + depth_intrinsics, identity_matrix(), nullptr); + } }, [&]{ _enabled = false; }); + return nullptr; } diff --git a/src/pose.h b/src/pose.h index 3f06be518c..a379e35668 100644 --- a/src/pose.h +++ b/src/pose.h @@ -26,8 +26,9 @@ inline pose operator*( const pose & a, const pose & b ) { return{ a.orientation inline pose inverse( const pose & a ) { + // inverse of {R, t} is { R^T, R^T * (-t) } auto inv = transpose( a.orientation ); - return { inv, inv * a.position * -1 }; + return { inv, inv * (a.position * -1) }; } inline pose to_pose( const rs2_extrinsics & a ) diff --git a/src/proc/CMakeLists.txt b/src/proc/CMakeLists.txt index be25d3f199..2f07332d77 100644 --- a/src/proc/CMakeLists.txt +++ b/src/proc/CMakeLists.txt @@ -39,6 +39,10 @@ target_sources(${LRS_TARGET} "${CMAKE_CURRENT_LIST_DIR}/units-transform.cpp" "${CMAKE_CURRENT_LIST_DIR}/rotation-transform.cpp" "${CMAKE_CURRENT_LIST_DIR}/color-formats-converter.cpp" + "${CMAKE_CURRENT_LIST_DIR}/rggb-debayer.cpp" + "${CMAKE_CURRENT_LIST_DIR}/rggb-converter.cpp" + "${CMAKE_CURRENT_LIST_DIR}/stereo-rectify.cpp" + "${CMAKE_CURRENT_LIST_DIR}/dual-rgb-rectify-filter.cpp" "${CMAKE_CURRENT_LIST_DIR}/depth-formats-converter.cpp" "${CMAKE_CURRENT_LIST_DIR}/motion-transform.cpp" "${CMAKE_CURRENT_LIST_DIR}/auto-exposure-processor.cpp" @@ -72,6 +76,10 @@ target_sources(${LRS_TARGET} "${CMAKE_CURRENT_LIST_DIR}/units-transform.h" "${CMAKE_CURRENT_LIST_DIR}/rotation-transform.h" "${CMAKE_CURRENT_LIST_DIR}/color-formats-converter.h" + "${CMAKE_CURRENT_LIST_DIR}/rggb-debayer.h" + "${CMAKE_CURRENT_LIST_DIR}/rggb-converter.h" + "${CMAKE_CURRENT_LIST_DIR}/stereo-rectify.h" + "${CMAKE_CURRENT_LIST_DIR}/dual-rgb-rectify-filter.h" "${CMAKE_CURRENT_LIST_DIR}/depth-formats-converter.h" "${CMAKE_CURRENT_LIST_DIR}/motion-transform.h" "${CMAKE_CURRENT_LIST_DIR}/auto-exposure-processor.h" diff --git a/src/proc/dual-rgb-rectify-filter.cpp b/src/proc/dual-rgb-rectify-filter.cpp new file mode 100644 index 0000000000..eb84416356 --- /dev/null +++ b/src/proc/dual-rgb-rectify-filter.cpp @@ -0,0 +1,182 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "dual-rgb-rectify-filter.h" +#include +#include +#include + +#ifdef RS2_USE_CUDA +#include "cuda/cuda-rggb.cuh" +#include "rsutils/accelerators/gpu.h" // rsutils::rs2_is_cuda_available +#endif + +namespace librealsense { + +dual_rgb_rectify_filter::dual_rgb_rectify_filter() + : stream_filter_processing_block( "RGB Rectification" ) +{ + // Only act on color streams; everything else passes through. + _stream_filter.stream = RS2_STREAM_COLOR; + _stream_filter.format = RS2_FORMAT_RGB8; +} + +dual_rgb_rectify_filter::~dual_rgb_rectify_filter() +{ + // Lock so free_device_maps() can't race an in-flight process_frame() on the other pin's thread. + std::lock_guard< std::mutex > lock( _mutex ); + free_device_maps(); +} + +void dual_rgb_rectify_filter::free_device_maps() +{ +#ifdef RS2_USE_CUDA + for( int i = 0; i < 2; ++i ) + { + rscuda::rggb_cuda_free( _dmap_sx[i] ); + rscuda::rggb_cuda_free( _dmap_sy[i] ); + _dmap_sx[i] = nullptr; + _dmap_sy[i] = nullptr; + } +#endif +} + +void dual_rgb_rectify_filter::ensure_maps() +{ + if( ! _p0 || ! _p1 ) + return; + + // Both color eyes stream at the same selected output resolution; build the rectification at + // that geometry. The color profile intrinsics are already resolution-correct -- for the D401 + // GMSL get_color_intrinsics reports the native 1288x808 calibration transformed by the same + // center-crop + bilinear-scale the image path applies (rggb_converter / crop_scale_rgb8), so we + // use them as-is. (The old path hardcoded the native 1288x808 size, which no advertised + // resolution matches -> the map geometry never lined up with the frame and rectify never ran.) + const int out_w = _p0.width(), out_h = _p0.height(); + if( out_w <= 0 || out_h <= 0 ) + return; + + rs2_intrinsics inL = _p0.get_intrinsics(); + rs2_intrinsics inR = _p1.get_intrinsics(); + + rs2_extrinsics lr = _p0.get_extrinsics_to( _p1 ); // left -> right (carries the baseline) + + _rc = rect::compute( inL, inR, lr, out_w, out_h ); + _maps_w = out_w; // remember the geometry these tables were built for, so a resolution change + _maps_h = out_h; // invalidates them (see process_frame) instead of cutting the frame. + _ready = true; + + // The rectified image is a pinhole projection with a new common focal and the principal point at + // the image center (see rect::build_table), and the source distortion has been remapped out. Clone + // each eye's profile with those intrinsics so consumers that deproject (rs2_deproject_pixel_to_point, + // pointcloud, align) get the geometry of the image we actually hand them. + auto rectified_intrinsics = [&]( const rs2::video_stream_profile & src ) { + rs2_intrinsics in = src.get_intrinsics(); + in.fx = in.fy = _rc.new_f; + in.ppx = out_w * 0.5f; + in.ppy = out_h * 0.5f; + in.model = RS2_DISTORTION_NONE; + for( auto & c : in.coeffs ) + c = 0.f; + return in; + }; + _tgt_p[0] = _p0.clone( _p0.stream_type(), _p0.stream_index(), _p0.format(), out_w, out_h, rectified_intrinsics( _p0 ) ); + _tgt_p[1] = _p1.clone( _p1.stream_type(), _p1.stream_index(), _p1.format(), out_w, out_h, rectified_intrinsics( _p1 ) ); + +#ifdef RS2_USE_CUDA + // Upload the (constant) remap tables to the device once so the GPU remap reads them directly. + if( rsutils::rs2_is_cuda_available() ) + { + const size_t lbytes = _rc.left.sx.size() * sizeof( float ); + const size_t rbytes = _rc.right.sx.size() * sizeof( float ); + _dmap_sx[0] = rscuda::rggb_cuda_alloc_upload( _rc.left.sx.data(), lbytes ); + _dmap_sy[0] = rscuda::rggb_cuda_alloc_upload( _rc.left.sy.data(), lbytes ); + _dmap_sx[1] = rscuda::rggb_cuda_alloc_upload( _rc.right.sx.data(), rbytes ); + _dmap_sy[1] = rscuda::rggb_cuda_alloc_upload( _rc.right.sy.data(), rbytes ); + } +#endif +} + +rs2::frame dual_rgb_rectify_filter::process_frame( const rs2::frame_source & source, const rs2::frame & f ) +{ + auto vf = f.as< rs2::video_frame >(); + if( ! vf || vf.get_profile().format() != RS2_FORMAT_RGB8 ) + return f; + + // Both color pins' threads call this on one shared instance; serialize all shared-state access + // (profiles, _ready, maps, _rc, _tmp, device pointers) and the internal free_device_maps() call. + std::lock_guard< std::mutex > lock( _mutex ); + + const int idx = vf.get_profile().stream_index(); + + // If the stream resolution changed since the maps were built, they are for the old geometry and + // would remap into only part of the frame (e.g. 848-wide maps into a 1280 frame -> right ~34% + // black). Drop the cached maps + captured profiles so they rebuild at the new size. + if( _ready && ( vf.get_width() != _maps_w || vf.get_height() != _maps_h ) ) + { + free_device_maps(); + _p0 = rs2::video_stream_profile{}; + _p1 = rs2::video_stream_profile{}; + _tgt_p[0] = rs2::stream_profile{}; + _tgt_p[1] = rs2::stream_profile{}; + _ready = false; + } + + if( auto vsp = vf.get_profile().as< rs2::video_stream_profile >() ) + { + if( idx == 0 && ! _p0 ) _p0 = vsp; + else if( idx == 1 && ! _p1 ) _p1 = vsp; + } + + if( ! _ready ) + { + ensure_maps(); + if( ! _ready ) + return f; // pass through until both eyes' calibration is available + } + + const int eye = ( idx == 1 ) ? 1 : 0; + const rect::remap_table & t = ( eye == 1 ) ? _rc.right : _rc.left; + const int w = vf.get_width(), h = vf.get_height(); + const int src_stride = vf.get_stride_in_bytes(); // honor the frame's real (possibly padded) stride + if( t.w > w || t.h > h ) + return f; // unexpected geometry; don't touch + if( ! _tgt_p[eye] ) + return f; // no rectified profile to advertise; better to pass the frame through unchanged + + rs2::frame tgt = source.allocate_video_frame( _tgt_p[eye], f ); + auto tvf = tgt.as< rs2::video_frame >(); + if( ! tvf ) + return f; + uint8_t * dst = static_cast< uint8_t * >( const_cast< void * >( tvf.get_data() ) ); + const int dstride = tvf.get_stride_in_bytes(); + +#ifdef RS2_USE_CUDA + // GPU remap straight into the output frame (in place under zero-copy, no host round-trip). + if( rsutils::rs2_is_cuda_available() && _dmap_sx[eye] && _dmap_sy[eye] ) + { + rscuda::rggb_remap_rgb8_cuda( static_cast< const uint8_t * >( vf.get_data() ), w, h, src_stride, + static_cast< const float * >( _dmap_sx[eye] ), + static_cast< const float * >( _dmap_sy[eye] ), + t.w, t.h, dst, dstride ); + return tgt; + } +#endif + + // CPU: rectify the real-width content into scratch, then place it (left-aligned) into the + // output frame (any padding columns stay zero), so the stream's advertised geometry is unchanged. + _tmp.resize( (size_t)t.w * t.h * 3 ); + rect::remap_rgb8( static_cast< const uint8_t * >( vf.get_data() ), w, h, src_stride, t, _tmp.data() ); + const int rows = std::min( h, t.h ); // never read past the (t.h-row) remap scratch + for( int y = 0; y < rows; ++y ) + { + std::memcpy( dst + (size_t)y * dstride, _tmp.data() + (size_t)y * t.w * 3, (size_t)t.w * 3 ); + if( w > t.w ) + std::memset( dst + (size_t)y * dstride + t.w * 3, 0, (size_t)( w - t.w ) * 3 ); + } + for( int y = rows; y < h; ++y ) // frame taller than the table: zero the remaining rows + std::memset( dst + (size_t)y * dstride, 0, (size_t)w * 3 ); + return tgt; +} + +} // namespace librealsense diff --git a/src/proc/dual-rgb-rectify-filter.h b/src/proc/dual-rgb-rectify-filter.h new file mode 100644 index 0000000000..348321da74 --- /dev/null +++ b/src/proc/dual-rgb-rectify-filter.h @@ -0,0 +1,54 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once + +#include "synthetic-stream.h" // stream_filter_processing_block +#include "stereo-rectify.h" // rect::rectification +#include +#include +#include + +namespace librealsense { + +// D401 GMSL dual-RGB rectification, as a recommended post-processing filter (shows up in the +// viewer's Post-Processing with an on/off toggle, on by default). It self-configures from the two +// color frames' own profiles (intrinsics + the left->right extrinsics, both provided by the SDK): +// once it has seen both color streams it computes the stereo-rectify maps once, then remaps each +// color frame to a rectified image. Pure C++ (no OpenCV) via the stereo-rectify module. +class LRS_EXTENSION_API dual_rgb_rectify_filter : public stream_filter_processing_block +{ +public: + dual_rgb_rectify_filter(); + ~dual_rgb_rectify_filter() override; + +protected: + rs2::frame process_frame( const rs2::frame_source & source, const rs2::frame & f ) override; + +private: + void ensure_maps(); + void free_device_maps(); // release the uploaded CUDA remap tables (no-op w/o CUDA) + + rs2::video_stream_profile _p0, _p1; // captured per-eye color profiles (for calibration) + // Output profiles carrying the *rectified* pinhole intrinsics (indexed by eye, 0=left/1=right). + // The rectified image no longer matches the source calibration, so deprojection / pointcloud / + // align must see fx=fy=new_f, principal point at the image center and no distortion. + rs2::stream_profile _tgt_p[2]; + bool _ready = false; + int _maps_w = 0; // output geometry the current maps were built for; a frame + int _maps_h = 0; // at a different size invalidates and rebuilds them + rect::rectification _rc; + std::vector< uint8_t > _tmp; // scratch for the rectified image + + // CUDA: the remap tables uploaded to the device once (indexed by eye, 0=left/1=right). Plain + // void* so the header stays CUDA-free; populated/used/freed only under RS2_USE_CUDA. + void * _dmap_sx[2] = { nullptr, nullptr }; + void * _dmap_sy[2] = { nullptr, nullptr }; + + // The filter accumulates both eyes' profiles (_p0/_p1) in one instance and is invoked from both + // color pins' backend threads, so process_frame() can run concurrently. Serialize all access to + // the shared state above (profiles, _ready, maps, _rc, _tmp, device pointers) and guard the + // destructor's free_device_maps() so it can't race an in-flight process_frame(). + std::mutex _mutex; +}; + +} // namespace librealsense diff --git a/src/proc/formats-converter.cpp b/src/proc/formats-converter.cpp index e5d1933a90..77f6763cc7 100644 --- a/src/proc/formats-converter.cpp +++ b/src/proc/formats-converter.cpp @@ -114,8 +114,9 @@ stream_profiles formats_converter::get_all_possible_profiles( const stream_profi for( const auto & target : pbf->get_target_info() ) { // When a converter declares multiple indexed targets for one source stream (e.g. interleaved - // infrared split into IR1/IR2, or the two color pins routed to Color 1/2), match each raw - // profile to the target whose index equals the raw stream index. + // infrared split into IR1/IR2, or dual-RGB color pins routed to distinct Color streams - + // D500 Color 1/2, D401 GMSL Color 0/1), match each raw profile to the target whose index + // equals the raw stream index. (Single-stream color: raw index 0 == target index 0 → still matches.) if( ( source.stream == RS2_STREAM_INFRARED || source.stream == RS2_STREAM_COLOR ) && raw_profile->get_stream_index() != target.index ) continue; @@ -328,7 +329,14 @@ void formats_converter::update_target_profiles_data( const stream_profiles & fro // Hack for L515 confidence. // Requesting source resolution from the camera, getting frame size of target (*2 y axis resolution) - video_raw_profile->set_dims( video_from_profile->get_width(), video_from_profile->get_height() ); + // Do NOT shrink the raw/source profile below its native resolution: a resolution-reducing + // converter (e.g. the D401 GMSL RGGB crop 1612 -> 1288) must keep the backend capturing at + // the full source resolution, otherwise the V4L2 buffer is sized for the (smaller) target and + // the packed raw data is truncated. For every non-reducing conversion (all others today) the + // target dims equal the native dims, so this guard is a no-op and behavior is unchanged. + if( ! ( video_raw_profile->get_width() > video_from_profile->get_width() + || video_raw_profile->get_height() > video_from_profile->get_height() ) ) + video_raw_profile->set_dims( video_from_profile->get_width(), video_from_profile->get_height() ); } } } diff --git a/src/proc/rggb-converter.cpp b/src/proc/rggb-converter.cpp new file mode 100644 index 0000000000..bc9f3ea972 --- /dev/null +++ b/src/proc/rggb-converter.cpp @@ -0,0 +1,223 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "rggb-converter.h" +#include // video_stream_profile_interface +#include // get_image_bpp +#include // struct rs2_stream_profile (->profile) +#include // rs2::video_stream_profile +#include +#include +#include + +#ifdef RS2_USE_CUDA +#include "cuda/cuda-rggb.cuh" +#include "rsutils/accelerators/gpu.h" // rsutils::rs2_is_cuda_available +#endif + +namespace librealsense +{ + void rggb_converter::init_profiles_info( const rs2::frame * f ) + { + auto p = f->get_profile(); + if( p.get() != _source_stream_profile.get() ) + { + _source_stream_profile = p; + + // Source dimensions. RAW8 passthrough is 1 byte/pixel and the V4L2 profile width is + // the padded transport width (e.g. 1612). The real row stride is larger still — the + // kernel pads each row to 64 bytes (1612 -> 1664) — so process_function() derives the + // true stride from the frame's raw size rather than trusting width. + if( auto vsp = p.as< rs2::video_stream_profile >() ) + { + _src_width = vsp.width(); + _src_height = vsp.height(); + } + else + { + _src_width = _native_width; + } + + _target_stream_profile = p.clone( p.stream_type(), p.stream_index(), _target_format ); + _target_bpp = get_image_bpp( _target_format ) / 8; + + // Set the target profile dims to the requested OUTPUT resolution (crop-to-aspect + + // scale from the native sensor image). This matches the advertised resolution + // (registration's resolution_transform) and the frame allocated in prepare_frame(). + auto target_spi = (stream_profile_interface *)_target_stream_profile.get()->profile; + if( auto target_vspi = dynamic_cast< video_stream_profile_interface * >( target_spi ) ) + target_vspi->set_dims( static_cast< uint32_t >( _out_width ), + static_cast< uint32_t >( _out_height ) ); + } + } + + rs2::frame rggb_converter::prepare_frame( const rs2::frame_source & source, const rs2::frame & f ) + { + init_profiles_info( &f ); + // Allocate the output at the requested resolution (out_width x out_height), tight stride. + return source.allocate_video_frame( _target_stream_profile, f, _target_bpp, + _out_width, _out_height, _out_width * _target_bpp, _extension_type ); + } + + rs2::frame rggb_converter::process_frame( const rs2::frame_source & source, const rs2::frame & f ) + { + // One shared instance serves both color pins (see _proc_mutex) - serialize the whole + // convert so the per-instance scratch / AWB / _src_data_size can't be written concurrently. + std::lock_guard< std::mutex > lock( _proc_mutex ); + // Capture the source frame's *actual* byte count - the authoritative way to recover the + // real row stride regardless of whether the backend keeps the padded V4L2 buffer (1664) + // or repacks to width*bpp (1612). + // The source frame's actual byte count gives the true row stride (data_size / height), + // independent of whether the backend keeps 1664-padded rows or hands us a 1612 frame. + _src_data_size = f.get_data_size(); + return functional_processing_block::process_frame( source, f ); + } + + void rggb_converter::process_function( uint8_t * const dest[], const uint8_t * source, + int /*width*/, int /*height*/, int /*actual_size*/, int /*input_size*/ ) + { + // The 'RGGB 8-bit' node actually carries MIPI RAW10 (4 px / 5 bytes). Recover the row + // stride from the frame's real byte count (fallback: 64-byte-aligned source width), unpack + // RAW10 -> 8-bit Bayer at the native sensor width, demosaic to native RGB, then center-crop + // to the output aspect ratio and bilinear-scale to the requested output resolution. + int src_stride = ( _src_width + 63 ) & ~63; + if( _src_data_size > 0 && _src_height > 0 && ( _src_data_size % _src_height ) == 0 ) + src_stride = _src_data_size / _src_height; + + const int native_w = _native_width; // real sensor width, e.g. 1288 (multiple of 4) + const int native_h = _src_height; // native rows, e.g. 808 + const int real_width = native_w; // AWB samples the native image + const int height = native_h; // AWB/debayer iterate native rows + + // RAW10 packs 4 px / 5 bytes, so a full native row occupies (native_w / 4) * 5 bytes and both + // the AWB sampling below and unpack_raw10 read that far into every row. A truncated frame (a + // partial V4L2 buffer) would make them read past the end of the source buffer. + const int min_src_stride = ( native_w / 4 ) * 5; + if( src_stride < min_src_stride ) + { + LOG_WARNING( "RGGB converter: truncated frame, row stride " << src_stride << " < " << min_src_stride + << " bytes - dropping" ); + return; + } + + // White-patch auto white balance: balance the brightest *unclipped* surfaces (the white/ + // light objects) to neutral, so white reads as white. Gray-world (scene average) leaves a + // cast on white objects that are cooler than the average; white-patch targets them directly. + // Two sparse passes over the packed source: (1) find the brightest unclipped green, (2) + // average R/G/B over the top brightness band. EMA-smoothed; feeds CPU + CUDA debayer. + { + const int bl = _isp.black_level; + auto bval = [&]( int x, int y ) -> int { + int v = (int)source[ (size_t)y * src_stride + (size_t)( x >> 2 ) * 5 + ( x & 3 ) ] - bl; + return v < 0 ? 0 : v; + }; + const int step = 16; // sample one RGGB cell every 16 px + const int hi = 220; // clip threshold: a channel at/above this is blown -> skip + int gmax = 1; // brightest unclipped green among the samples + for( int y = 0; y + 1 < height; y += step ) + for( int x = 0; x + 1 < real_width; x += step ) + { + const int g = ( bval( x + 1, y ) + bval( x, y + 1 ) ) >> 1; + if( g < hi && g > gmax ) gmax = g; + } + const int gthr = ( gmax * 3 ) / 5; // top ~40% brightness band = light/white surfaces + double sR = 0, sG = 0, sB = 0; + long n = 0; + for( int y = 0; y + 1 < height; y += step ) + for( int x = 0; x + 1 < real_width; x += step ) // x,y even -> land on R sites + { + int rr = bval( x, y ); // R site (BGGR: this is B) + const int gg2 = ( bval( x + 1, y ) + bval( x, y + 1 ) ) >> 1; // (Gr + Gb) / 2 + int bb = bval( x + 1, y + 1 ); // B site (BGGR: this is R) + if( _isp.swap_rb ) { int t = rr; rr = bb; bb = t; } // BGGR: real R/B are swapped + if( gg2 < gthr ) continue; // not a bright surface + if( rr >= hi || gg2 >= hi || bb >= hi ) continue; // any channel clipped + sR += rr; sG += gg2; sB += bb; ++n; + } + if( n > 20 && sR > 1.0 && sB > 1.0 ) // enough bright-surface samples to trust it + { + const double mR = sR / n, mG = sG / n, mB = sB / n; + auto clampg = []( float g ) { return g < 0.5f ? 0.5f : ( g > 4.f ? 4.f : g ); }; + // No warm bias: validated against the captured raw, the unbiased white-patch gains + // (gR~2.15, gB~1.72) land the bright surfaces neutral. A bias only re-introduces a tint. + const float tR = clampg( float( mG / mR ) ), tB = clampg( float( mG / mB ) ); + const float a = 0.1f; // EMA: converges in ~30 frames + _awb_gain_r += a * ( tR - _awb_gain_r ); + _awb_gain_b += a * ( tB - _awb_gain_b ); + } + } + rggb::isp_params isp = _isp; // per-frame ISP with the auto-white-balance gains + isp.gain_r = _awb_gain_r; + isp.gain_g = 1.f; + isp.gain_b = _awb_gain_b; + + (void)real_width; (void)height; // aliases for the AWB loop above; native_w/native_h below + +#ifdef RS2_USE_CUDA + // GPU path: fused RAW10 unpack + demosaic + tone to native RGB, then crop-to-aspect + + // bilinear scale, writing the output frame in place under zero-copy (no host round-trip). + if( rsutils::rs2_is_cuda_available() ) + { + rscuda::rggb_isp_params ip{}; + ip.black_level = isp.black_level; + ip.gain_r = isp.gain_r; ip.gain_g = isp.gain_g; ip.gain_b = isp.gain_b; + ip.digital_gain = isp.digital_gain; ip.gamma = isp.gamma; ip.s_curve = isp.s_curve; + ip.saturation = isp.saturation; ip.contrast = isp.contrast; + ip.swap_rb = isp.swap_rb ? 1 : 0; + for( int i = 0; i < 9; ++i ) ip.ccm[i] = isp.ccm[i]; + rscuda::rggb_debayer_scale_raw10_cuda( source, src_stride, native_w, native_h, ip, + dest[0], _out_width, _out_height ); + return; + } +#endif + + // CPU: demosaic to native RGB scratch, then crop-to-aspect + scale to the output frame. + _bayer.resize( static_cast< size_t >( native_w ) * native_h ); + rggb::unpack_raw10( source, src_stride, native_w, native_h, _bayer.data() ); + _rgb_native.resize( static_cast< size_t >( native_w ) * native_h * 3 ); + // The tone LUT depends only on gamma / s_curve, so build it once here rather than letting each + // of the bands below rebuild it (1024 std::pow per band). + if( _tone_gamma != isp.gamma || _tone_s_curve != isp.s_curve ) + { + rggb::build_tone_lut( isp, _tone ); + _tone_gamma = isp.gamma; + _tone_s_curve = isp.s_curve; + } + // Demosaic is the CPU hot path; split it across row bands (the Jetson has spare cores). + { + const int nthreads = 4; + const int band = ( native_h + nthreads - 1 ) / nthreads; + std::vector< std::thread > pool; + for( int t = 1; t < nthreads; ++t ) + { + const int b0 = t * band, b1 = ( native_h < b0 + band ) ? native_h : b0 + band; + if( b0 >= b1 ) break; + pool.emplace_back( [&, b0, b1]() { + rggb::debayer_rggb8( _bayer.data(), native_w, native_w, native_h, + _rgb_native.data(), isp, native_w, b0, b1, _tone ); + } ); + } + rggb::debayer_rggb8( _bayer.data(), native_w, native_w, native_h, _rgb_native.data(), isp, native_w, + 0, ( native_h < band ) ? native_h : band, _tone ); + for( auto & th : pool ) th.join(); + } + // Crop-to-aspect + bilinear scale native RGB -> output frame (threaded over output rows). + { + const int nthreads = 4; + const int band = ( _out_height + nthreads - 1 ) / nthreads; + std::vector< std::thread > pool; + for( int t = 1; t < nthreads; ++t ) + { + const int b0 = t * band, b1 = ( _out_height < b0 + band ) ? _out_height : b0 + band; + if( b0 >= b1 ) break; + pool.emplace_back( [&, b0, b1]() { + rggb::crop_scale_rgb8( _rgb_native.data(), native_w, native_h, native_w, + dest[0], _out_width, _out_height, b0, b1 ); + } ); + } + rggb::crop_scale_rgb8( _rgb_native.data(), native_w, native_h, native_w, + dest[0], _out_width, _out_height, 0, ( _out_height < band ) ? _out_height : band ); + for( auto & th : pool ) th.join(); + } + } +} diff --git a/src/proc/rggb-converter.h b/src/proc/rggb-converter.h new file mode 100644 index 0000000000..997a495706 --- /dev/null +++ b/src/proc/rggb-converter.h @@ -0,0 +1,72 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once + +#include "color-formats-converter.h" // color_converter +#include "rggb-debayer.h" // rggb::isp_params, unpack_raw10, debayer_rggb8 + +#include +#include + +namespace librealsense +{ + // Processing block for the D401 GMSL "dual RGB" mode: an 8-bit RGGB Bayer frame delivered + // by the FW RAW8 CSI passthrough (V4L2 fourcc 'RGGB', mapped to RS2_FORMAT_RAW8) is cropped + // from the padded transport width (e.g. 1612) to the real sensor width (e.g. 1288), demosaiced + // and white-balanced to RGB8. Unlike the YUV color converters this changes resolution, so it + // overrides init_profiles_info() to set the cropped output dimensions (cf. rotation_transform). + class LRS_EXTENSION_API rggb_converter : public color_converter + { + public: + // native_width : real sensor width after cropping the transport padding (e.g. 1288). + // out_width/out_height : the requested OUTPUT resolution. The native image (native_width x + // source_height) is demosaiced, then center-cropped to the output aspect ratio and + // bilinear-scaled to out_width x out_height (crop-to-aspect + scale, no stretch). For the + // native output (out == native), the scale collapses to a straight copy. One converter + // instance is registered per output resolution. + rggb_converter( rs2_format target_format, + int native_width, + int out_width, + int out_height, + rggb::isp_params isp = {}, + rs2_stream target_stream = RS2_STREAM_COLOR ) + : color_converter( "RGGB Converter", target_format, target_stream ) + , _native_width( native_width ) + , _out_width( out_width ) + , _out_height( out_height ) + , _isp( isp ) + { + } + + protected: + void init_profiles_info( const rs2::frame * f ) override; + // Allocate the output at the requested output resolution (not the source's padded width). + rs2::frame prepare_frame( const rs2::frame_source & source, const rs2::frame & f ) override; + rs2::frame process_frame( const rs2::frame_source & source, const rs2::frame & f ) override; + void process_function( uint8_t * const dest[], const uint8_t * source, + int width, int height, int actual_size, int input_size ) override; + + int _native_width; // real sensor width in px (e.g. 1288), pre-scale + int _out_width; // requested output width in px + int _out_height; // requested output height in px + int _src_width = 0; // source profile width in px (e.g. 1612, padded) + int _src_height = 0; // source profile height in px (e.g. 808) + int _src_data_size = 0;// source frame's actual byte count (authoritative for stride) + rggb::isp_params _isp; + float _awb_gain_r = 1.7f; // gray-world auto-white-balance gains (EMA per frame) + float _awb_gain_b = 1.4f; + std::vector< uint8_t > _bayer; // scratch: RAW10 unpacked to 8-bit Bayer (native_width*height) + std::vector< uint8_t > _rgb_native; // scratch: demosaiced native RGB8 before crop+scale + + // Tone LUT, built once instead of per demosaic band (4x per frame). Only gamma / s_curve feed + // it, so it is rebuilt only if those change (_tone_gamma < 0 => not built yet). + uint8_t _tone[1024]; + float _tone_gamma = -1.f; + float _tone_s_curve = -1.f; + // formats_converter shares one converter instance across BOTH color pins (Color 0 and + // Color 1), whose frames are delivered on separate backend threads. Serialize process_frame + // so the per-instance scratch (_bayer/_rgb_native), the AWB gains and _src_data_size aren't + // written concurrently. The shared AWB is intentional - it keeps the stereo pair color-matched. + std::mutex _proc_mutex; + }; +} diff --git a/src/proc/rggb-debayer.cpp b/src/proc/rggb-debayer.cpp new file mode 100644 index 0000000000..8dde64c6ef --- /dev/null +++ b/src/proc/rggb-debayer.cpp @@ -0,0 +1,238 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "rggb-debayer.h" + +#include // size_t +#include // std::pow +#include // std::memcpy + +namespace librealsense { +namespace rggb { + +namespace { + +inline int clampi( int v, int lo, int hi ) +{ + return v < lo ? lo : ( v > hi ? hi : v ); +} + +inline uint8_t to_u8( float v ) +{ + int i = static_cast< int >( v + 0.5f ); + return static_cast< uint8_t >( i < 0 ? 0 : ( i > 255 ? 255 : i ) ); +} + +inline float clamp01( float v ) { return v < 0.f ? 0.f : ( v > 1.f ? 1.f : v ); } + +} // namespace + +void unpack_raw10( const uint8_t * src, int src_stride, int real_width, int height, uint8_t * bayer8 ) +{ + const int groups = real_width / 4; // 5 source bytes per 4 pixels + for( int y = 0; y < height; ++y ) + { + const uint8_t * s = src + static_cast< size_t >( y ) * src_stride; + uint8_t * d = bayer8 + static_cast< size_t >( y ) * real_width; + for( int g = 0; g < groups; ++g ) + { + const uint8_t * q = s + g * 5; // [m0 m1 m2 m3 lsb]; 8-bit value == MSB byte + d[ g * 4 + 0 ] = q[ 0 ]; + d[ g * 4 + 1 ] = q[ 1 ]; + d[ g * 4 + 2 ] = q[ 2 ]; + d[ g * 4 + 3 ] = q[ 3 ]; + } + } +} + +void build_tone_lut( const isp_params & p, uint8_t tone[1024] ) +{ + // The sensor data is linear; encode with 1/gamma (sRGB-like) so midtones aren't crushed on a + // display. 1024-entry LUT indexed by the normalized [0,1] post-CCM value. + const float inv_g = ( p.gamma > 0.f ) ? 1.f / p.gamma : 1.f; + const float sc = p.s_curve; + for( int i = 0; i < 1024; ++i ) + { + float x = std::pow( i / 1023.f, inv_g ); + x = x + sc * x * ( 1.f - x ) * ( 2.f * x - 1.f ); // S-curve contrast (the "pop") + tone[i] = to_u8( 255.f * clamp01( x ) ); + } +} + +void debayer_rggb8( const uint8_t * bayer, int bayer_stride, int width, int height, + uint8_t * dst, const isp_params & p, int dst_stride_px, + int y_begin, int y_end, const uint8_t * tone_in ) +{ + const int wmax = width - 1; + const int hmax = height - 1; + const int yb = y_begin; + const int ye = ( y_end < 0 ) ? height : y_end; + const int bl = p.black_level; + const int row_px = ( dst_stride_px > width ) ? dst_stride_px : width; + + uint8_t tone_local[1024]; + const uint8_t * tone = tone_in; + if( ! tone ) + { + build_tone_lut( p, tone_local ); + tone = tone_local; + } + + const float gr = p.gain_r * p.digital_gain; + const float gg = p.gain_g * p.digital_gain; + const float gb = p.gain_b * p.digital_gain; + const float sat = p.saturation, con = p.contrast; + const float * m = p.ccm; + // Normalize the black-subtracted value to [0,1] using the post-black range (255 - black), so a + // full-scale sensor reading maps to 1.0 (matches the reference raw decode). + const float inv_range = 1.f / ( 255.f - (float)p.black_level ); + + // Black-level-subtracted, edge-clamped Bayer sample at (x,y). + auto S = [&]( int x, int y ) -> int { + x = clampi( x, 0, wmax ); + y = clampi( y, 0, hmax ); + int v = static_cast< int >( bayer[ y * bayer_stride + x ] ) - bl; + return v < 0 ? 0 : v; + }; + + for( int y = yb; y < ye; ++y ) + { + uint8_t * row = dst + static_cast< size_t >( y ) * row_px * 3; + const int yodd = y & 1; + for( int x = 0; x < width; ++x ) + { + const int xodd = x & 1; + float R, G, B; + + if( !yodd && !xodd ) // R site + { + R = (float)S( x, y ); + G = ( S( x - 1, y ) + S( x + 1, y ) + S( x, y - 1 ) + S( x, y + 1 ) ) * 0.25f; + B = ( S( x - 1, y - 1 ) + S( x + 1, y - 1 ) + S( x - 1, y + 1 ) + S( x + 1, y + 1 ) ) * 0.25f; + } + else if( !yodd && xodd ) // Gr site (red row): H=R, V=B + { + G = (float)S( x, y ); + R = ( S( x - 1, y ) + S( x + 1, y ) ) * 0.5f; + B = ( S( x, y - 1 ) + S( x, y + 1 ) ) * 0.5f; + } + else if( yodd && !xodd ) // Gb site (blue row): H=B, V=R + { + G = (float)S( x, y ); + R = ( S( x, y - 1 ) + S( x, y + 1 ) ) * 0.5f; + B = ( S( x - 1, y ) + S( x + 1, y ) ) * 0.5f; + } + else // B site + { + B = (float)S( x, y ); + G = ( S( x - 1, y ) + S( x + 1, y ) + S( x, y - 1 ) + S( x, y + 1 ) ) * 0.25f; + R = ( S( x - 1, y - 1 ) + S( x + 1, y - 1 ) + S( x - 1, y + 1 ) + S( x + 1, y + 1 ) ) * 0.25f; + } + + if( p.swap_rb ) { float t = R; R = B; B = t; } // RGGB demosaic -> BGGR (real D401 phase) + + // White-balance + digital gain, normalized to [0,1]. + float r = clamp01( R * gr * inv_range ); + float g = clamp01( G * gg * inv_range ); + float b = clamp01( B * gb * inv_range ); + // Color-correction matrix (sensor RGB -> display primaries). + float r2 = m[0] * r + m[1] * g + m[2] * b; + float g2 = m[3] * r + m[4] * g + m[5] * b; + float b2 = m[6] * r + m[7] * g + m[8] * b; + // Saturation about luma (linear, Rec.709), then gamma (LUT) + contrast about mid-grey. + const float yl = 0.2126f * r2 + 0.7152f * g2 + 0.0722f * b2; + r2 = clamp01( yl + sat * ( r2 - yl ) ); + g2 = clamp01( yl + sat * ( g2 - yl ) ); + b2 = clamp01( yl + sat * ( b2 - yl ) ); + float rd = tone[ static_cast< int >( r2 * 1023.f ) ]; + float gd = tone[ static_cast< int >( g2 * 1023.f ) ]; + float bd = tone[ static_cast< int >( b2 * 1023.f ) ]; + row[ x * 3 + 0 ] = to_u8( ( rd - 128.f ) * con + 128.f ); + row[ x * 3 + 1 ] = to_u8( ( gd - 128.f ) * con + 128.f ); + row[ x * 3 + 2 ] = to_u8( ( bd - 128.f ) * con + 128.f ); + } + // Zero any padding columns so a narrower image sits cleanly in a wider output frame. + for( int x = width; x < row_px; ++x ) + { + row[ x * 3 + 0 ] = 0; + row[ x * 3 + 1 ] = 0; + row[ x * 3 + 2 ] = 0; + } + } +} + +void crop_rect_for_output( int src_w, int src_h, int out_w, int out_h, + int * crop_x, int * crop_y, int * crop_w, int * crop_h ) +{ + // Centered crop matching the output aspect ratio (so the subsequent scale doesn't stretch). + // Compare src_w*out_h vs out_w*src_h to avoid float rounding: target "wider" than source -> + // crop the height; "narrower" -> crop the width. + int cw = src_w, ch = src_h; + const long long src_ar = (long long)src_w * out_h; // src_w/src_h vs + const long long out_ar = (long long)out_w * src_h; // out_w/out_h + if( out_ar > src_ar ) // output is wider -> limit by width, crop height + ch = (int)( ( (long long)src_w * out_h ) / out_w ); + else if( out_ar < src_ar ) // output is narrower/taller -> crop width + cw = (int)( ( (long long)src_h * out_w ) / out_h ); + if( cw > src_w ) cw = src_w; + if( ch > src_h ) ch = src_h; + if( cw < 1 ) cw = 1; + if( ch < 1 ) ch = 1; + *crop_w = cw; *crop_h = ch; + *crop_x = ( src_w - cw ) / 2; + *crop_y = ( src_h - ch ) / 2; +} + +void crop_scale_rgb8( const uint8_t * src, int src_w, int src_h, int src_stride_px, + uint8_t * dst, int out_w, int out_h, int y_begin, int y_end ) +{ + if( y_end < 0 ) y_end = out_h; + + int cx, cy, cw, ch; + crop_rect_for_output( src_w, src_h, out_w, out_h, &cx, &cy, &cw, &ch ); + + // Fast path: crop already equals output (e.g. native res requested) -> straight row copy. + if( cw == out_w && ch == out_h ) + { + for( int y = y_begin; y < y_end; ++y ) + { + const uint8_t * s = src + ( (size_t)( cy + y ) * src_stride_px + cx ) * 3; + std::memcpy( dst + (size_t)y * out_w * 3, s, (size_t)out_w * 3 ); + } + return; + } + + // Bilinear scale of the crop rect [cx,cx+cw) x [cy,cy+ch) -> out_w x out_h. Map output pixel + // centers back into the crop (the +0.5/-0.5 keeps the sampling centered, no half-pixel shift). + const float sx = (float)cw / (float)out_w; + const float sy = (float)ch / (float)out_h; + for( int oy = y_begin; oy < y_end; ++oy ) + { + float fy = ( oy + 0.5f ) * sy - 0.5f; + int y0 = (int)( fy < 0.f ? 0.f : fy ); + if( y0 > ch - 1 ) y0 = ch - 1; + int y1 = ( y0 + 1 < ch ) ? y0 + 1 : y0; + float wy = fy - (float)y0; if( wy < 0.f ) wy = 0.f; + const uint8_t * r0 = src + ( (size_t)( cy + y0 ) * src_stride_px + cx ) * 3; + const uint8_t * r1 = src + ( (size_t)( cy + y1 ) * src_stride_px + cx ) * 3; + uint8_t * orow = dst + (size_t)oy * out_w * 3; + for( int ox = 0; ox < out_w; ++ox ) + { + float fx = ( ox + 0.5f ) * sx - 0.5f; + int x0 = (int)( fx < 0.f ? 0.f : fx ); + if( x0 > cw - 1 ) x0 = cw - 1; + int x1 = ( x0 + 1 < cw ) ? x0 + 1 : x0; + float wx = fx - (float)x0; if( wx < 0.f ) wx = 0.f; + for( int c = 0; c < 3; ++c ) + { + float top = r0[ x0 * 3 + c ] * ( 1.f - wx ) + r0[ x1 * 3 + c ] * wx; + float bot = r1[ x0 * 3 + c ] * ( 1.f - wx ) + r1[ x1 * 3 + c ] * wx; + float v = top * ( 1.f - wy ) + bot * wy; + orow[ ox * 3 + c ] = (uint8_t)( v < 0.f ? 0.f : ( v > 255.f ? 255.f : v + 0.5f ) ); + } + } + } +} + +} // namespace rggb +} // namespace librealsense diff --git a/src/proc/rggb-debayer.h b/src/proc/rggb-debayer.h new file mode 100644 index 0000000000..9240aa967d --- /dev/null +++ b/src/proc/rggb-debayer.h @@ -0,0 +1,96 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once + +#include + +// RAW10 RGGB -> RGB8 for the D401 GMSL "dual RGB" mode. +// +// Each OV9782 imager is delivered over GMSL via the FW CSI passthrough. Although the V4L2 node +// advertises 'RGGB' 8-bit, the payload is actually MIPI **RAW10**: 4 pixels packed into 5 bytes +// (4 MSB bytes + 1 byte holding the four 2-bit LSBs). The transport width is 1612 (1610 active +// RAW10 bytes + alignment) while the real image is 1288 px; the row is padded to a 64-byte +// V4L2 stride (1664) on the kernel side, though librealsense hands us a 1612-stride frame. +// +// The host therefore: (1) unpacks RAW10 -> 8-bit Bayer (the 8-bit value is just the MSB byte, +// since (msb<<2 | lsb) >> 2 == msb), (2) demosaics RGGB -> RGB, (3) applies white-balance gains +// (OV9782 is green-dominant). This header has no SDK dependency so it can be unit-tested alone. + +namespace librealsense { +namespace rggb { + +// ISP knobs. Defaults gray-balance the green-dominant OV9782 (measured R/G/B ~ 41/69/48). +struct isp_params +{ + int black_level = 16; // subtracted per channel before gains, clamped to 0 + float gain_r = 1.9f; // white-balance gains (boost R/B relative to green) + float gain_g = 1.0f; + float gain_b = 1.6f; + float digital_gain = 1.0f; // brightness multiplier (auto-exposure sets the actual exposure) + float gamma = 1.8f; // display gamma (teammate-validated tone; brighter than 2.2) + float s_curve = 0.6f; // contrast S-curve baked into the tone LUT - the "pop"/contrast, + // f(x) = x + sc*x*(1-x)*(2x-1); replaces the plain contrast multiply + float saturation = 1.40f; // teammate value; correct now the Bayer phase (BGGR) is right + float contrast = 1.0f; // separate contrast off - the S-curve handles contrast/de-haze + bool swap_rb = false; // true => treat the Bayer as BGGR (the D401 GMSL's ACTUAL phase). + // The delivered data is BGGR, NOT the wiki's RGGB; decoding it as + // RGGB swaps red<->blue (red objects render blue). GMSL sets this true. + // Color-correction matrix (teammate-validated: boosts R/B purity, suppresses green crosstalk). + // Correct ONLY with the right Bayer phase (swap_rb); on wrong-phase data it produces a colour cast. + float ccm[9] = { 1.5f, -0.4f, -0.1f, + -0.1f, 1.2f, -0.1f, + -0.1f, -0.4f, 1.5f }; +}; + +// Unpack MIPI RAW10 (4 px / 5 bytes) to 8-bit Bayer. +// src : RAW10-packed bytes, top-left origin +// src_stride : bytes per source row (e.g. 1612 from the SDK frame, or 1664 raw V4L2) +// real_width : real pixel columns to produce, multiple of 4 (e.g. 1288) +// height : rows +// bayer8 : caller buffer of real_width*height bytes (8-bit RGGB Bayer) +void unpack_raw10( const uint8_t * src, int src_stride, int real_width, int height, uint8_t * bayer8 ); + +// Bilinear RGGB demosaic of 8-bit Bayer -> interleaved RGB8, with black-level + WB gains. +// bayer : 8-bit RGGB Bayer (row0: R G R G..., row1: G B G B...) +// bayer_stride : bytes per Bayer row +// width,height : pixels to demosaic (even dims assumed) +// dst : RGB8 output (R first) +// p : ISP params +// dst_stride_px: output row width in px (0 => contiguous = width). If > width, the extra +// columns [width, dst_stride_px) are zeroed - lets a 1288-px image sit inside a +// 1612-px output frame without a profile-dimension change. +// y_begin,y_end: process only rows [y_begin, y_end) (y_end < 0 => height). Lets the caller split +// the image across threads (the demosaic is the CPU hot path); reads clamp to the +// full image at borders, each call writes only its own rows. +// tone : optional precomputed tone LUT (see build_tone_lut). Pass one when demosaicing a +// frame across several threads/bands, or every call rebuilds it (1024 std::pow). +void debayer_rggb8( const uint8_t * bayer, int bayer_stride, int width, int height, + uint8_t * dst, const isp_params & p = {}, int dst_stride_px = 0, + int y_begin = 0, int y_end = -1, const uint8_t * tone = nullptr ); + +// Build the 1024-entry tone curve LUT (1/gamma encode + S-curve contrast) that debayer_rggb8 applies. +// Depends only on isp_params::gamma and ::s_curve, both constant per converter instance. +void build_tone_lut( const isp_params & p, uint8_t tone[1024] ); + +// Center-crop an interleaved RGB8 image to the target aspect ratio, then bilinear-scale that crop +// to out_w x out_h. The D401 color always arrives at one native resolution (e.g. 1288x808); this +// produces the user-selected output resolutions without stretching (crop-to-aspect preserves +// geometry, at the cost of a little FOV on the longer axis). A no-op fast copy when the crop +// already equals the output size. +// src : interleaved RGB8, top-left origin +// src_w, src_h : native image size in px +// src_stride_px : source row width in px (>= src_w) +// dst : RGB8 output, tight (out_w*3 bytes/row) +// out_w, out_h : target output size in px +// y_begin,y_end : output rows to fill [y_begin,y_end) (y_end<0 => out_h); lets callers thread it. +void crop_scale_rgb8( const uint8_t * src, int src_w, int src_h, int src_stride_px, + uint8_t * dst, int out_w, int out_h, int y_begin = 0, int y_end = -1 ); + +// Given a native size and a target output size, compute the centered crop rectangle (matching the +// output aspect ratio) that crop_scale_rgb8 uses. Exposed so the device layer can derive the +// per-resolution color intrinsics (crop offset + scale factor) consistently with the image path. +void crop_rect_for_output( int src_w, int src_h, int out_w, int out_h, + int * crop_x, int * crop_y, int * crop_w, int * crop_h ); + +} // namespace rggb +} // namespace librealsense diff --git a/src/proc/stereo-rectify.cpp b/src/proc/stereo-rectify.cpp new file mode 100644 index 0000000000..e511cd13f7 --- /dev/null +++ b/src/proc/stereo-rectify.cpp @@ -0,0 +1,156 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "stereo-rectify.h" +#include // rs2_project_point_to_pixel (handles RS distortion models) + +#include // std::min / std::max (used in inv_rodrigues) +#include +#include + +namespace librealsense { +namespace rect { + +namespace { + +// --- minimal 3x3 (row-major) helpers --- +void mat_mul( const float a[9], const float b[9], float o[9] ) +{ + for( int r = 0; r < 3; ++r ) + for( int c = 0; c < 3; ++c ) + o[r * 3 + c] = a[r * 3 + 0] * b[0 * 3 + c] + a[r * 3 + 1] * b[1 * 3 + c] + a[r * 3 + 2] * b[2 * 3 + c]; +} + +void rodrigues( const float v[3], float R[9] ) // rotation vector -> 3x3 +{ + float th = std::sqrt( v[0] * v[0] + v[1] * v[1] + v[2] * v[2] ); + if( th < 1e-9f ) + { + R[0] = R[4] = R[8] = 1.f; + R[1] = R[2] = R[3] = R[5] = R[6] = R[7] = 0.f; + return; + } + float x = v[0] / th, y = v[1] / th, z = v[2] / th, c = std::cos( th ), s = std::sin( th ), C = 1 - c; + R[0] = c + x * x * C; R[1] = x * y * C - z * s; R[2] = x * z * C + y * s; + R[3] = y * x * C + z * s; R[4] = c + y * y * C; R[5] = y * z * C - x * s; + R[6] = z * x * C - y * s; R[7] = z * y * C + x * s; R[8] = c + z * z * C; +} + +void inv_rodrigues( const float R[9], float v[3] ) // 3x3 -> rotation vector +{ + float tr = R[0] + R[4] + R[8]; + float th = std::acos( std::max( -1.f, std::min( 1.f, ( tr - 1 ) * 0.5f ) ) ); + if( th < 1e-9f ) { v[0] = v[1] = v[2] = 0.f; return; } + float s = 2.f * std::sin( th ); + v[0] = ( R[7] - R[5] ) / s * th; + v[1] = ( R[2] - R[6] ) / s * th; + v[2] = ( R[3] - R[1] ) / s * th; +} + +} // namespace + +remap_table build_table( const rs2_intrinsics & src, const float R[9], float new_f, int out_w, int out_h ) +{ + remap_table t; + t.w = out_w; t.h = out_h; + t.sx.resize( (size_t)out_w * out_h ); + t.sy.resize( (size_t)out_w * out_h ); + const float cx = out_w * 0.5f, cy = out_h * 0.5f; + for( int v = 0; v < out_h; ++v ) + { + for( int u = 0; u < out_w; ++u ) + { + // rectified pinhole ray + float x = ( u - cx ) / new_f, y = ( v - cy ) / new_f, z = 1.f; + // orig = R^T * rect (R: orig -> rectified, row-major) + float pt[3] = { R[0] * x + R[3] * y + R[6] * z, + R[1] * x + R[4] * y + R[7] * z, + R[2] * x + R[5] * y + R[8] * z }; + float px[2]; + rs2_project_point_to_pixel( px, &src, pt ); // applies src distortion (incl. inverse-BC) + t.sx[(size_t)v * out_w + u] = px[0]; + t.sy[(size_t)v * out_w + u] = px[1]; + } + } + return t; +} + +rectification compute( const rs2_intrinsics & inL, const rs2_intrinsics & inR, + const rs2_extrinsics & lr, int out_w, int out_h ) +{ + rectification rc; + + // Bouguet: rotate each camera halfway to a common (coplanar) orientation, then rotate so the + // baseline lies along x (rows become epipolar lines). lr.rotation is column-major (left->right). + float R_lr[9]; + for( int r = 0; r < 3; ++r ) + for( int c = 0; c < 3; ++c ) + R_lr[r * 3 + c] = lr.rotation[c * 3 + r]; + + float om[3]; inv_rodrigues( R_lr, om ); + float omh[3] = { om[0] * 0.5f, om[1] * 0.5f, om[2] * 0.5f }; + float r_r[9]; rodrigues( omh, r_r ); // right rotated +half + float omn[3] = { -omh[0], -omh[1], -omh[2] }; + float r_l[9]; rodrigues( omn, r_l ); // left rotated -half + + // baseline in the half-rotated left frame: t = r_l * (lr.translation) + float T[3] = { lr.translation[0], lr.translation[1], lr.translation[2] }; + float t[3] = { r_l[0] * T[0] + r_l[1] * T[1] + r_l[2] * T[2], + r_l[3] * T[0] + r_l[4] * T[1] + r_l[5] * T[2], + r_l[6] * T[0] + r_l[7] * T[1] + r_l[8] * T[2] }; + float tn = std::sqrt( t[0] * t[0] + t[1] * t[1] + t[2] * t[2] ); + if( tn < 1e-9f ) tn = 1.f; + // rectified basis: e1 along baseline, e2 = z x e1, e3 = e1 x e2 + float e1[3] = { t[0] / tn, t[1] / tn, t[2] / tn }; + // Orient the rectified x-axis to +x. The D400 baseline is along -x (T.x < 0), which would make + // Rrect a 180-degree rotation (flipped image). Keeping x ~ +x yields Rrect ~ identity for these + // near-parallel imagers (a mirror of the disparity sign, irrelevant for display/rectification). + if( e1[0] < 0.f ) { e1[0] = -e1[0]; e1[1] = -e1[1]; e1[2] = -e1[2]; } + float e2[3] = { -e1[1], e1[0], 0.f }; + float e2n = std::sqrt( e2[0] * e2[0] + e2[1] * e2[1] ); if( e2n < 1e-9f ) e2n = 1.f; + e2[0] /= e2n; e2[1] /= e2n; e2[2] = 0.f; + float e3[3] = { e1[1] * e2[2] - e1[2] * e2[1], e1[2] * e2[0] - e1[0] * e2[2], e1[0] * e2[1] - e1[1] * e2[0] }; + float Rrect[9] = { e1[0], e1[1], e1[2], e2[0], e2[1], e2[2], e3[0], e3[1], e3[2] }; + + mat_mul( Rrect, r_l, rc.R_left ); + mat_mul( Rrect, r_r, rc.R_right ); + + rc.new_f = inL.fx; // inL is expected already at the output resolution + rc.left = build_table( inL, rc.R_left, rc.new_f, out_w, out_h ); + rc.right = build_table( inR, rc.R_right, rc.new_f, out_w, out_h ); + return rc; +} + +void remap_rgb8( const uint8_t * src, int src_w, int src_h, int src_stride, + const remap_table & t, uint8_t * dst ) +{ + for( int v = 0; v < t.h; ++v ) + { + uint8_t * drow = dst + (size_t)v * t.w * 3; + for( int u = 0; u < t.w; ++u ) + { + float fx = t.sx[(size_t)v * t.w + u], fy = t.sy[(size_t)v * t.w + u]; + int x0 = (int)std::floor( fx ), y0 = (int)std::floor( fy ); + uint8_t * o = drow + u * 3; + if( x0 < 0 || y0 < 0 || x0 + 1 >= src_w || y0 + 1 >= src_h ) + { + o[0] = o[1] = o[2] = 0; + continue; + } + float ax = fx - x0, ay = fy - y0; + const uint8_t * p00 = src + (size_t)y0 * src_stride + x0 * 3; + const uint8_t * p01 = p00 + 3; + const uint8_t * p10 = p00 + src_stride; + const uint8_t * p11 = p10 + 3; + for( int ch = 0; ch < 3; ++ch ) + { + float top = p00[ch] * ( 1 - ax ) + p01[ch] * ax; + float bot = p10[ch] * ( 1 - ax ) + p11[ch] * ax; + o[ch] = (uint8_t)( top * ( 1 - ay ) + bot * ay + 0.5f ); + } + } + } +} + +} // namespace rect +} // namespace librealsense diff --git a/src/proc/stereo-rectify.h b/src/proc/stereo-rectify.h new file mode 100644 index 0000000000..844215a414 --- /dev/null +++ b/src/proc/stereo-rectify.h @@ -0,0 +1,55 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once + +#include // rs2_intrinsics +#include // rs2_extrinsics +#include +#include + +// Pure-C++ stereo rectification for the D401 GMSL dual-RGB pair. No OpenCV dependency (so it can +// live in an SDK processing block): the remap tables are built with RealSense's own projection +// model (rs2_project_point_to_pixel, which handles RS2_DISTORTION_INVERSE_BROWN_CONRADY used by +// the color imagers), and the per-frame work is a bilinear remap. +// +// Pipeline per eye: for each output (rectified, pinhole) pixel -> back-project to a ray -> apply +// the rectifying rotation -> project through the source intrinsics (with distortion) -> source +// pixel. The table is built once; remap_rgb8() runs per frame. + +namespace librealsense { +namespace rect { + +// Precomputed source-pixel lookup for one eye. +struct remap_table +{ + int w = 0, h = 0; // output (rectified) size + std::vector< float > sx, sy; // per output pixel, the source pixel to sample (bilinear) +}; + +// Rectifying rotations + common focal length for a stereo pair, computed from the two intrinsics +// and the left->right extrinsics (Bouguet). R_left/R_right are 3x3 row-major (orig -> rectified). +struct rectification +{ + float R_left[9]; + float R_right[9]; + float new_f; // common focal (px) for the rectified images, scaled to out_w + remap_table left; + remap_table right; +}; + +// Compute the full rectification for an output size out_w x out_h. `inL`/`inR` are the (already +// resolution-correct) source intrinsics for left/right; `lr` is the left->right extrinsics. +rectification compute( const rs2_intrinsics & inL, + const rs2_intrinsics & inR, + const rs2_extrinsics & lr, + int out_w, int out_h ); + +// Build one eye's table directly (used internally / for undistort-only with R = identity). +remap_table build_table( const rs2_intrinsics & src, const float R[9], float new_f, int out_w, int out_h ); + +// Bilinear remap of an interleaved RGB8 source into dst (out=t.w x t.h, 3 bytes/px, tight). +void remap_rgb8( const uint8_t * src, int src_w, int src_h, int src_stride, + const remap_table & t, uint8_t * dst ); + +} // namespace rect +} // namespace librealsense diff --git a/src/proc/synthetic-stream.cpp b/src/proc/synthetic-stream.cpp index 3c8da5a19d..2d0f1613fa 100644 --- a/src/proc/synthetic-stream.cpp +++ b/src/proc/synthetic-stream.cpp @@ -17,6 +17,8 @@ #include +#include "rum/rum-hooks.h" + namespace librealsense { @@ -83,6 +85,8 @@ namespace librealsense { if (should_process(f)) { + if( ! _rum_applied.exchange( true ) ) // first processed frame -> filter actually used + rum::hooks::on_filter( get_info( RS2_CAMERA_INFO_NAME ) ); auto res = process_frame(source, f); if (!res) continue; if (auto composite = res.as()) diff --git a/src/proc/synthetic-stream.h b/src/proc/synthetic-stream.h index 9be07131d1..6fbda23ad8 100644 --- a/src/proc/synthetic-stream.h +++ b/src/proc/synthetic-stream.h @@ -13,6 +13,8 @@ #include #include +#include + namespace librealsense { @@ -66,6 +68,7 @@ namespace librealsense std::mutex _mutex; rs2_frame_processor_callback_sptr _callback; synthetic_source _source_wrapper; + std::atomic< bool > _rum_applied{ false }; // RUM: report first frame through this block once }; class LRS_EXTENSION_API generic_processing_block : public processing_block diff --git a/src/realsense.def b/src/realsense.def index e43f5f8522..5c2e204b64 100644 --- a/src/realsense.def +++ b/src/realsense.def @@ -126,6 +126,10 @@ EXPORTS rs2_delete_raw_data rs2_get_raw_data + rs2_rum_get_report + rs2_rum_set_cloud_enabled + rs2_rum_is_cloud_enabled + rs2_get_device_info rs2_supports_device_info rs2_get_sensor_info diff --git a/src/rs.cpp b/src/rs.cpp index b1d013a6bc..b8840d3822 100644 --- a/src/rs.cpp +++ b/src/rs.cpp @@ -59,6 +59,9 @@ #include "debug-stream-sensor.h" #include "max-usable-range-sensor.h" #include "fw-update/fw-update-device-interface.h" +#include "rum/rum-config.h" +#include "rum/rum-collector.h" +#include "rum/rum-hooks.h" #include "core/frame-callback.h" #include "color-sensor.h" #include "perception-sensor.h" @@ -292,6 +295,7 @@ NOEXCEPT_RETURN(nullptr, what, name, args, type) void notifications_processor::raise_notification(const notification n) { + librealsense::rum::hooks::on_notification(n.category); _dispatcher.invoke([this, n](dispatcher::cancellable_timer ct) { std::lock_guard lock(_callback_mutex); @@ -412,7 +416,10 @@ rs2_device* rs2_create_device(const rs2_device_list* info_list, int index, rs2_e VALIDATE_NOT_NULL(info_list); VALIDATE_RANGE(index, 0, (int)info_list->list.size() - 1); - return new rs2_device{ info_list->list[index]->create_device() }; + auto dev = info_list->list[index]->create_device(); + if( dev ) + librealsense::rum::hooks::on_device( *dev ); + return new rs2_device{ dev }; } HANDLE_EXCEPTIONS_AND_RETURN(nullptr, info_list, index) @@ -801,6 +808,25 @@ void rs2_delete_raw_data(const rs2_raw_data_buffer* buffer) BEGIN_API_CALL } NOEXCEPT_RETURN(, buffer) +const rs2_raw_data_buffer* rs2_rum_get_report(rs2_error** error) BEGIN_API_CALL +{ + auto report = librealsense::rum::rum_collector::instance().get_report(); + return new rs2_raw_data_buffer{ std::vector( report.begin(), report.end() ) }; +} +NOARGS_HANDLE_EXCEPTIONS_AND_RETURN(nullptr) + +void rs2_rum_set_cloud_enabled(int enabled, rs2_error** error) BEGIN_API_CALL +{ + librealsense::rum::rum_config::instance().set_cloud_enabled( enabled != 0 ); +} +HANDLE_EXCEPTIONS_AND_RETURN(, enabled) + +int rs2_rum_is_cloud_enabled(rs2_error** error) BEGIN_API_CALL +{ + return librealsense::rum::rum_config::instance().is_cloud_enabled() ? 1 : 0; +} +NOARGS_HANDLE_EXCEPTIONS_AND_RETURN(0) + void rs2_open(rs2_sensor* sensor, const rs2_stream_profile* profile, rs2_error** error) BEGIN_API_CALL { VALIDATE_NOT_NULL(sensor); @@ -907,6 +933,7 @@ void rs2_set_option(const rs2_options* options, rs2_option option, float value, VALIDATE_OPTION_ENABLED(options, option); auto& option_ref = options->options->get_option(option); auto range = option_ref.get_range(); + float applied = value; // value actually set (integer options are truncated); reported to RUM switch (option_ref.get_value_type()) { case RS2_OPTION_TYPE_FLOAT: @@ -921,7 +948,8 @@ void rs2_set_option(const rs2_options* options, rs2_option option, float value, if ((int)value != value) LOG_WARNING("Float value " << value << " given to integer option " << rs2_get_option_name(options, option, error) << ", truncating to " << std::trunc(value)); - option_ref.set(std::trunc(value)); + applied = std::trunc(value); + option_ref.set(applied); break; case RS2_OPTION_TYPE_BOOLEAN: @@ -946,6 +974,7 @@ void rs2_set_option(const rs2_options* options, rs2_option option, float value, } throw not_implemented_exception("use rs2_set_option_value to set string values"); } + librealsense::rum::hooks::on_set_option( *options->options, option, applied, range.def ); } HANDLE_EXCEPTIONS_AND_RETURN(, options, option, value) diff --git a/src/rum/CMakeLists.txt b/src/rum/CMakeLists.txt new file mode 100644 index 0000000000..a6f31fb6f5 --- /dev/null +++ b/src/rum/CMakeLists.txt @@ -0,0 +1,14 @@ +# License: Apache 2.0. See LICENSE file in root directory. +# Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +# Always compiled; ENABLE_STATS gates behavior (hook bodies and the rs2_rum_* C-API), not the +# build. When off, the hooks no-op and the collector/config are simply never invoked. +target_sources(${LRS_TARGET} + PRIVATE + "${CMAKE_CURRENT_LIST_DIR}/rum-config.h" + "${CMAKE_CURRENT_LIST_DIR}/rum-config.cpp" + "${CMAKE_CURRENT_LIST_DIR}/rum-collector.h" + "${CMAKE_CURRENT_LIST_DIR}/rum-collector.cpp" + "${CMAKE_CURRENT_LIST_DIR}/rum-hooks.h" + "${CMAKE_CURRENT_LIST_DIR}/rum-hooks.cpp" +) diff --git a/src/rum/rum-collector.cpp b/src/rum/rum-collector.cpp new file mode 100644 index 0000000000..5fead8ec42 --- /dev/null +++ b/src/rum/rum-collector.cpp @@ -0,0 +1,300 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "rum-collector.h" +#include "rum-config.h" + +#include // RS2_API_VERSION_STR + +#include +#include +#include // get_os_name, cpu_arch +#include +#include + +#include +#include +#include + + +#ifdef _WIN32 +#include +#else +#include +#endif + +using json = rsutils::json; + + +namespace librealsense { +namespace rum { + +static constexpr int rum_schema_version = 1; + + +namespace { + + +// Report file: /rum/rum.json. Forward slashes work on Windows and POSIX. +std::string report_path() +{ + return rsutils::os::get_special_folder( rsutils::os::special_folder::app_data ) + "rum/rum.json"; +} + + +// Create the report's parent directory; benign if it already exists. +void ensure_report_directory() +{ + auto dir = rsutils::os::get_special_folder( rsutils::os::special_folder::app_data ) + "rum"; +#ifdef _WIN32 + CreateDirectoryA( dir.c_str(), nullptr ); +#else + mkdir( dir.c_str(), 0700 ); +#endif +} + + +// Random anonymous id, formatted to look like a UUID. +std::string generate_source_id() +{ + std::random_device rd; + // Draw the random values first; snprintf may read its arguments in any order. + unsigned a = rd(), b = rd() & 0xFFFF, c = rd() & 0xFFFF, d = rd() & 0xFFFF, e = rd() & 0xFFFF, f = rd(); + char buf[37]; + std::snprintf( buf, sizeof( buf ), "%08x-%04x-%04x-%04x-%04x%08x", a, b, c, d, e, f ); + return std::string( buf ); +} + + +// Reuse the id saved in rum.json, or make a new one on first run. Kept out of +// realsense-config.json so the viewer's config writes can't overwrite it. +std::string load_or_create_source_id() +{ + try + { + auto id = rsutils::json_config::load_from_file( report_path() ) + .nested( "source_id", &json::is_string ).string_ref_or_empty(); + if( ! id.empty() ) + return id; + } + catch( ... ) + { + } + return generate_source_id(); +} + +char const * build_type() +{ +#ifdef NDEBUG + return "Release"; +#else + return "Debug"; +#endif +} + +// Build-time configuration, read from the SDK's existing compile macros (no RUM-specific defines). +#ifdef BUILD_WITH_DDS +constexpr bool cmake_build_with_dds = true; +#else +constexpr bool cmake_build_with_dds = false; +#endif + +#ifdef RS2_USE_CUDA +constexpr bool cmake_build_with_cuda = true; +#else +constexpr bool cmake_build_with_cuda = false; +#endif + +#ifdef ENABLE_STATS +constexpr bool cmake_enable_stats = true; +#else +constexpr bool cmake_enable_stats = false; +#endif + +char const * backend() +{ +#if defined( RS2_USE_WMF_BACKEND ) + return "wmf"; +#elif defined( RS2_USE_V4L2_BACKEND ) + return "v4l2"; +#elif defined( RS2_USE_LIBUVC_BACKEND ) + return "libuvc"; +#elif defined( RS2_USE_WINUSB_UVC_BACKEND ) + return "winusb_uvc"; +#elif defined( RS2_USE_ANDROID_BACKEND ) + return "android"; +#else + return "unknown"; +#endif +} + + +} // namespace + + +rum_collector::rum_collector() + : _source_id( load_or_create_source_id() ) + , _session_id( generate_source_id() ) +{ +} + + +rum_collector & rum_collector::instance() +{ + static rum_collector inst; + return inst; +} + + +void rum_collector::record_device( std::string const & type, + std::string const & fw_version, + std::string const & connection, + std::string const & mipi_driver_version ) +{ + std::lock_guard< std::mutex > lk( _mutex ); + ++_device_counts[device_key{ type, fw_version, connection, mipi_driver_version }]; +} + + +void rum_collector::record_stream( std::string const & stream_type, + std::string const & format, + std::string const & resolution, + int fps ) +{ + std::lock_guard< std::mutex > lk( _mutex ); + ++_stream_counts[stream_key{ stream_type, format, resolution, fps }].count; +} + + +void rum_collector::record_stream_duration( std::string const & stream_type, + std::string const & format, + std::string const & resolution, + int fps, + double seconds ) +{ + std::lock_guard< std::mutex > lk( _mutex ); + _stream_counts[stream_key{ stream_type, format, resolution, fps }].duration_seconds += seconds; +} + + +void rum_collector::record_option_change( std::string const & option, float value ) +{ + std::lock_guard< std::mutex > lk( _mutex ); + auto & entry = _option_changes[option]; + ++entry.first; + entry.second = value; +} + + +void rum_collector::add_recommended_filter( std::string const & name ) +{ + std::lock_guard< std::mutex > lk( _mutex ); + _recommended_filters.insert( name ); +} + + +void rum_collector::record_filter( std::string const & name ) +{ + std::lock_guard< std::mutex > lk( _mutex ); + // Count only recommended (user-facing) filters; ignore viewer/internal blocks. + if( _recommended_filters.count( name ) ) + ++_filter_counts[name]; +} + + +void rum_collector::record_notification( std::string const & category ) +{ + std::lock_guard< std::mutex > lk( _mutex ); + ++_notification_counts[category]; +} + + +std::string rum_collector::get_report() const +{ + std::lock_guard< std::mutex > lk( _mutex ); + json report = json::object(); + report["schema_version"] = rum_schema_version; + report["source_id"] = _source_id; + report["session_id"] = _session_id; + report["generated_at"] = std::chrono::duration_cast< std::chrono::seconds >( + std::chrono::system_clock::now().time_since_epoch() ).count(); + report["sdk"] = json::object(); + report["sdk"]["version"] = RS2_API_VERSION_STR; + report["sdk"]["build_type"] = build_type(); + report["sdk"]["backend"] = backend(); + report["sdk"]["cmake_flags"] = { + { "ENABLE_STATS", cmake_enable_stats }, + { "BUILD_WITH_DDS", cmake_build_with_dds }, + { "BUILD_WITH_CUDA", cmake_build_with_cuda }, + }; + report["system"] = json::object(); + report["system"]["os"] = rsutils::os::get_os_name(); + report["system"]["arch"] = rsutils::os::cpu_arch(); + + report["devices"] = json::array(); + for( auto const & entry : _device_counts ) + { + auto const & key = entry.first; + json device = json::object(); + device["type"] = key.type; + device["fw_version"] = key.fw_version; + device["connection"] = key.connection; + device["mipi_driver_version"] = key.mipi_driver_version; + device["count"] = entry.second; + report["devices"].push_back( device ); + } + + report["streams"] = json::array(); + for( auto const & entry : _stream_counts ) + { + auto const & key = entry.first; + json stream = json::object(); + stream["type"] = key.type; + stream["format"] = key.format; + stream["resolution"] = key.resolution; + stream["fps"] = key.fps; + stream["count"] = entry.second.count; + stream["duration_seconds"] = entry.second.duration_seconds; + report["streams"].push_back( stream ); + } + + report["options_changed"] = json::array(); + for( auto const & entry : _option_changes ) + { + json option = json::object(); + option["option"] = entry.first; + option["set_count"] = entry.second.first; + option["last_value"] = entry.second.second; + report["options_changed"].push_back( option ); + } + + report["filters"] = json::array(); + for( auto const & entry : _filter_counts ) + { + json filter = json::object(); + filter["name"] = entry.first; + filter["count"] = entry.second; + report["filters"].push_back( filter ); + } + + report["notifications"] = json::array(); + for( auto const & entry : _notification_counts ) + { + json notification = json::object(); + notification["category"] = entry.first; + notification["count"] = entry.second; + report["notifications"].push_back( notification ); + } + return report.dump( 2 ); +} + + +void rum_collector::flush() +{ + ensure_report_directory(); + rsutils::os::atomic_write_file( report_path(), get_report() ); +} + + +} // namespace rum +} // namespace librealsense diff --git a/src/rum/rum-collector.h b/src/rum/rum-collector.h new file mode 100644 index 0000000000..8df5ab3617 --- /dev/null +++ b/src/rum/rum-collector.h @@ -0,0 +1,109 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once + +#include +#include +#include +#include +#include +#include + + +namespace librealsense { +namespace rum { + + +// Process-wide collector of RUM data: SDK metadata plus per-session tallies +// (devices, streams, options, filters, notifications) filled by the hooks. +// Builds the JSON report on demand. Thread-safe. +class rum_collector +{ +public: + static rum_collector & instance(); + + // Record a created device, counted by (type, fw version, connection, mipi driver). + // Safe to call repeatedly. + void record_device( std::string const & type, + std::string const & fw_version, + std::string const & connection, + std::string const & mipi_driver_version ); + + // Record an opened stream config, counted by (type, format, resolution, fps). + // Safe to call repeatedly. + void record_stream( std::string const & stream_type, + std::string const & format, + std::string const & resolution, + int fps ); + + // Add streamed seconds (start->stop) to a stream config's running total. + void record_stream_duration( std::string const & stream_type, + std::string const & format, + std::string const & resolution, + int fps, + double seconds ); + + // Record an option set to a non-default value; tallies set-count and last value per option. + void record_option_change( std::string const & option, float value ); + + // Add a filter name that counts as user-facing post-processing (a sensor's recommended block). + // record_filter only tallies names added here, so viewer/internal blocks (colorizer, pointcloud, + // align, format converters, ...) never pollute the report. + void add_recommended_filter( std::string const & name ); + + // Record that a filter processed a frame (first time per block); tallied only if recommended. + void record_filter( std::string const & name ); + + // Record a raised notification, tallied per category. + void record_notification( std::string const & category ); + + // Write the current report to the local file. No network. + void flush(); + + // The live in-memory report as JSON. This is what rs2_rum_get_report returns. + std::string get_report() const; + +private: + rum_collector(); + + struct device_key + { + std::string type, fw_version, connection, mipi_driver_version; + bool operator<( device_key const & o ) const + { + return std::tie( type, fw_version, connection, mipi_driver_version ) + < std::tie( o.type, o.fw_version, o.connection, o.mipi_driver_version ); + } + }; + struct stream_key + { + std::string type, format, resolution; + int fps; + bool operator<( stream_key const & o ) const + { + return std::tie( type, format, resolution, fps ) < std::tie( o.type, o.format, o.resolution, o.fps ); + } + }; + + mutable std::mutex _mutex; + std::string const _source_id; // loaded from rum.json or created at construction; stable across runs + std::string const _session_id; // new per run; lets the server dedup a session uploaded twice + // Deduplicated device tallies -> count. + std::map< device_key, int > _device_counts; + // Deduplicated stream tallies -> (open count, total streamed seconds). + struct stream_stat { int count = 0; double duration_seconds = 0.0; }; + std::map< stream_key, stream_stat > _stream_counts; + // Per-option change tallies, keyed by option name -> (set_count, last_value). + std::map< std::string, std::pair< int, float > > _option_changes; + // Filter usage tallies (first frame through each block), keyed by filter name -> count. + std::map< std::string, int > _filter_counts; + // Names that count as user-facing post-processing (sensors' recommended blocks); record_filter + // ignores anything not in here. + std::set< std::string > _recommended_filters; + // Notification tallies, keyed by category -> count. + std::map< std::string, int > _notification_counts; +}; + + +} // namespace rum +} // namespace librealsense diff --git a/src/rum/rum-config.cpp b/src/rum/rum-config.cpp new file mode 100644 index 0000000000..ad3c9f53a9 --- /dev/null +++ b/src/rum/rum-config.cpp @@ -0,0 +1,110 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "rum-config.h" + +#include // RS2_CONFIG_FILENAME + +#include +#include +#include +#include +#include + +#include + +using json = rsutils::json; + + +namespace librealsense { +namespace rum { + + +namespace { + + +std::string default_config_path() +{ + return rsutils::os::get_special_folder( rsutils::os::special_folder::app_data ) + RS2_CONFIG_FILENAME; +} + + +bool parse_bool( std::string const & s ) +{ + return s == "1"; // anything else (incl. "0", empty, malformed) -> opt-out +} + + +} // namespace + + +rum_config & rum_config::instance() +{ + static rum_config inst( default_config_path() ); + return inst; +} + + +rum_config::rum_config( std::string filename ) + : _filename( std::move( filename ) ) +{ +} + + +json rum_config::load_config() const +{ + try + { + auto j = rsutils::json_config::load_from_file( _filename ); + if( j.is_object() ) + return j; + } + catch( ... ) + { + } + return json::object(); +} + + +bool rum_config::save_config( json const & j ) +{ + return rsutils::os::atomic_write_file( _filename, j.dump( 2 ) ); +} + + +bool rum_config::is_cloud_enabled() const +{ + std::string config_str; + { + std::lock_guard< std::mutex > lk( _mutex ); + config_str = load_config().nested( "rum_cloud_enabled", &json::is_string ).string_ref_or_empty(); + } + bool const config_consent = ! config_str.empty() && parse_bool( config_str ); + + // Env override: =0 always disables (kill switch); =1 enables only if the user hasn't opted out. + // The env var can never turn upload on against a saved opt-out. + auto env = std::getenv( "RS2_RUM_CLOUD_ENABLED" ); + if( env && *env ) + { + if( ! parse_bool( env ) ) + return false; + return config_str.empty() || config_consent; + } + return config_consent; +} + + +void rum_config::set_cloud_enabled( bool enabled ) +{ + std::lock_guard< std::mutex > lk( _mutex ); + auto j = load_config(); + // Store as "1"/"0" (string) to match the viewer's config_file; a native bool would + // make the first-run popup re-prompt. + j["rum_cloud_enabled"] = enabled ? "1" : "0"; + if( ! save_config( j ) ) + LOG_WARNING( "RUM: failed to persist rum_cloud_enabled to " << _filename ); +} + + +} // namespace rum +} // namespace librealsense diff --git a/src/rum/rum-config.h b/src/rum/rum-config.h new file mode 100644 index 0000000000..8ea2dc29bf --- /dev/null +++ b/src/rum/rum-config.h @@ -0,0 +1,43 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once + +#include + +#include +#include + + +namespace librealsense { +namespace rum { + + +// RUM cloud-upload consent, saved in the shared realsense-config.json (the file the viewer's +// config_file also uses). Stored as a string so SDK and viewer writes agree. (The source_id +// lives with the report file, not here.) +class rum_config +{ +public: + // Process-wide instance backed by realsense-config.json under the app-data folder. + static rum_config & instance(); + + // Explicit file path — used by tests to avoid touching the real user config. + explicit rum_config( std::string filename ); + + // Resolved consent: env var wins, then the config key, else false (no decision = no upload). + bool is_cloud_enabled() const; + + void set_cloud_enabled( bool enabled ); + +private: + // Load/save the backing config file (owns _filename). Callers hold _mutex. + rsutils::json load_config() const; + bool save_config( rsutils::json const & j ); + + std::string _filename; + mutable std::mutex _mutex; +}; + + +} // namespace rum +} // namespace librealsense diff --git a/src/rum/rum-hooks.cpp b/src/rum/rum-hooks.cpp new file mode 100644 index 0000000000..d9ea1e2a9b --- /dev/null +++ b/src/rum/rum-hooks.cpp @@ -0,0 +1,131 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "rum-hooks.h" +#include "rum-collector.h" + +#include "core/device-interface.h" // device_interface, supports_info/get_info +#include "core/video.h" // stream_profile_interface, video_stream_profile_interface +#include "core/sensor-interface.h" // sensor_interface, get_recommended_processing_blocks +#include "core/processing-block-interface.h" // processing_block_interface (recommended-filter names) +#include "core/options-interface.h" // options_interface +#include "core/enum-helpers.h" // get_string( rs2_stream / rs2_format / rs2_option / rs2_notification_category ) + +#include + + +#ifdef ENABLE_STATS + + +namespace librealsense { +namespace rum { +namespace hooks { + + +void on_device( device_interface & dev ) +{ + auto info = [&]( rs2_camera_info i ) -> std::string { + return dev.supports_info( i ) ? dev.get_info( i ) : std::string(); + }; + rum_collector::instance().record_device( info( RS2_CAMERA_INFO_NAME ), + info( RS2_CAMERA_INFO_FIRMWARE_VERSION ), + info( RS2_CAMERA_INFO_CONNECTION_TYPE ), + info( RS2_CAMERA_INFO_MIPI_DRIVER_VERSION ) ); + + // Mark this device's recommended post-processing filters as the ones worth recording, so + // record_filter ignores viewer/internal blocks (colorizer/pointcloud/align, format converters). + for( size_t i = 0; i < dev.get_sensors_count(); ++i ) + for( auto const & block : dev.get_sensor( i ).get_recommended_processing_blocks() ) + if( block && block->supports_info( RS2_CAMERA_INFO_NAME ) ) + rum_collector::instance().add_recommended_filter( block->get_info( RS2_CAMERA_INFO_NAME ) ); +} + + +namespace { + +// Extract the (type, format, resolution, fps) stream-tally key from a profile. +void stream_key_of( std::shared_ptr< stream_profile_interface > const & p, + std::string & type, std::string & format, std::string & resolution, int & fps ) +{ + type = get_string( p->get_stream_type() ); + format = get_string( p->get_format() ); + fps = static_cast< int >( p->get_framerate() ); + resolution.clear(); + if( auto vp = std::dynamic_pointer_cast< video_stream_profile_interface >( p ) ) + resolution = std::to_string( vp->get_width() ) + "x" + std::to_string( vp->get_height() ); +} + +} // namespace + + +void on_open( std::vector< std::shared_ptr< stream_profile_interface > > const & profiles ) +{ + for( auto const & p : profiles ) + { + if( ! p ) + continue; + std::string type, format, resolution; + int fps; + stream_key_of( p, type, format, resolution, fps ); + rum_collector::instance().record_stream( type, format, resolution, fps ); + } +} + + +void on_stream_duration( std::vector< std::shared_ptr< stream_profile_interface > > const & profiles, double seconds ) +{ + for( auto const & p : profiles ) + { + if( ! p ) + continue; + std::string type, format, resolution; + int fps; + stream_key_of( p, type, format, resolution, fps ); + rum_collector::instance().record_stream_duration( type, format, resolution, fps, seconds ); + } +} + + +void on_set_option( options_interface & target, rs2_option option, float value, float default_value ) +{ + if( value == default_value ) + return; + // Only record options set on a device sensor; processing-block options are set + // internally, not user tuning. + if( dynamic_cast< sensor_interface * >( &target ) == nullptr ) + return; + rum_collector::instance().record_option_change( get_string( option ), value ); +} + + +void on_filter( std::string const & name ) +{ + // Record any processing block that processes a frame; narrowing to recommended filters + // is left to the consumer. + rum_collector::instance().record_filter( name ); +} + + +void on_notification( rs2_notification_category category ) +{ + rum_collector::instance().record_notification( get_string( category ) ); +} + + +void on_context_closed() noexcept +{ + try + { + rum_collector::instance().flush(); + } + catch( ... ) + { + } +} + + +} // namespace hooks +} // namespace rum +} // namespace librealsense + +#endif // ENABLE_STATS diff --git a/src/rum/rum-hooks.h b/src/rum/rum-hooks.h new file mode 100644 index 0000000000..b6a6884a75 --- /dev/null +++ b/src/rum/rum-hooks.h @@ -0,0 +1,93 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. +#pragma once + +#include // rs2_option +#include // rs2_notification_category +#include + +#include +#include +#include +#include + + +namespace librealsense { + + +class device_interface; +class stream_profile_interface; +class options_interface; + + +// Instrumentation facade. Call sites invoke these one-liners; all data extraction lives here +// and in the collector, so changing a reported field never touches the call site. When +// ENABLE_STATS is off these become inline no-ops, so call sites need no guard of their own. +namespace rum { +namespace hooks { + + +#ifdef ENABLE_STATS + +// A device was created — record its type, firmware version, connection and MIPI driver version. +void on_device( device_interface & dev ); + +// A sensor was opened with these stream profiles — record each configuration. +void on_open( std::vector< std::shared_ptr< stream_profile_interface > > const & profiles ); + +// A sensor stopped after streaming `seconds` — add that to each active profile's running total. +void on_stream_duration( std::vector< std::shared_ptr< stream_profile_interface > > const & profiles, double seconds ); + +// An option was set — recorded only when the value is non-default and the target is a +// device sensor (processing-block options are ignored). +void on_set_option( options_interface & target, rs2_option option, float value, float default_value ); + +// A processing block processed a frame (once per block) — real usage, not construction. +// Restricting to recommended filters is left to the consumer. +void on_filter( std::string const & name ); + +// A notification was raised — record it by category. +void on_notification( rs2_notification_category category ); + +// An SDK session (context) is closing — save the report to the local file. Called from the +// context destructor, so it never throws. +void on_context_closed() noexcept; + +#else // ENABLE_STATS — inline no-ops so call sites compile away when stats are disabled + +inline void on_device( device_interface & ) {} +inline void on_open( std::vector< std::shared_ptr< stream_profile_interface > > const & ) {} +inline void on_stream_duration( std::vector< std::shared_ptr< stream_profile_interface > > const &, double ) {} +inline void on_set_option( options_interface &, rs2_option, float, float ) {} +inline void on_filter( std::string const & ) {} +inline void on_notification( rs2_notification_category ) {} +inline void on_context_closed() noexcept {} + +#endif // ENABLE_STATS + + +} // namespace hooks + + +// Times a sensor's streaming intervals and reports each to RUM. A sensor holds one of these +// instead of a raw stopwatch: restart() when streaming begins, record() when it ends (a no-op +// unless actually streaming, so re-start/close/teardown never double-count or drop an interval). +class stream_timer +{ +public: + void restart() { _sw.reset(); } + + void record( bool streaming, + std::vector< std::shared_ptr< stream_profile_interface > > const & active ) + { + if( streaming ) + hooks::on_stream_duration( active, std::chrono::duration< double >( _sw.get_elapsed() ).count() ); + } + +private: + rsutils::time::stopwatch _sw; +}; + + +} // namespace rum +} // namespace librealsense diff --git a/src/sensor.cpp b/src/sensor.cpp index 1149b90656..543217bb82 100644 --- a/src/sensor.cpp +++ b/src/sensor.cpp @@ -25,6 +25,8 @@ #include #include +#include "rum/rum-hooks.h" + #include #include #include @@ -633,11 +635,14 @@ void log_callback_end( uint32_t fps, } set_active_streams(requests); + + rum::hooks::on_open( requests ); } void synthetic_sensor::close() { std::lock_guard lock(_synthetic_configure_lock); + record_rum_stream_duration(); // closing while still streaming would otherwise drop the interval _raw_sensor->close(); std::vector< std::shared_ptr< processing_block > > active_pbs = _formats_converter.get_active_converters(); @@ -658,14 +663,29 @@ void log_callback_end( uint32_t fps, set_frames_callback(callback); _formats_converter.set_frames_callback( callback ); // TODO duplicate?! Something fishy here! + record_rum_stream_duration(); // flush any prior interval if start() is called without a stop() between + // Call the processing block on the frame _raw_sensor->start( make_frame_callback( [&, this]( frame_holder f ) { _formats_converter.convert_frame( f ); } ) ); + + _rum_timer.restart(); + } + + + void sensor_base::record_rum_stream_duration() + { + // Call before the sensor actually stops/closes/starts, so is_streaming() still reflects + // the interval being closed. + _rum_timer.record( is_streaming(), get_active_streams() ); } void synthetic_sensor::stop() { std::lock_guard lock(_synthetic_configure_lock); + + record_rum_stream_duration(); + _raw_sensor->stop(); } diff --git a/src/sensor.h b/src/sensor.h index 3dee5f931a..b64dc3615f 100644 --- a/src/sensor.h +++ b/src/sensor.h @@ -7,6 +7,7 @@ #include "source.h" #include "core/extension.h" #include "proc/formats-converter.h" +#include "rum/rum-hooks.h" // rum::stream_timer #include #include #include @@ -14,6 +15,7 @@ #include #include #include +#include #include #include @@ -132,6 +134,9 @@ namespace librealsense void sort_profiles( stream_profiles & ); + void record_rum_stream_duration(); // hand any in-flight streamed interval to RUM; no-op when not streaming + rum::stream_timer _rum_timer; // owns the streamed-duration stopwatch + reporting + std::shared_ptr< frame > generate_frame_from_data( const platform::frame_object & fo, rs2_time_t system_time, frame_timestamp_reader * timestamp_reader, diff --git a/src/software-sensor.cpp b/src/software-sensor.cpp index a950150d30..303007dde6 100644 --- a/src/software-sensor.cpp +++ b/src/software-sensor.cpp @@ -8,6 +8,7 @@ #include "core/video-frame.h" #include "core/notification.h" #include "depth-sensor.h" +#include "rum/rum-hooks.h" #include #include @@ -79,6 +80,7 @@ software_sensor::software_sensor( std::string const & name, software_device * ow software_sensor::~software_sensor() { + try { record_rum_stream_duration(); } catch( ... ) {} // flush in-flight interval on teardown/disconnect } @@ -199,6 +201,7 @@ void software_sensor::open( const stream_profiles & requests ) throw wrong_api_call_sequence_exception( "open(...) failed. Software device is already opened!" ); _is_opened = true; set_active_streams( requests ); + rum::hooks::on_open( requests ); } @@ -224,6 +227,7 @@ void software_sensor::start( frame_callback_ptr callback ) _source.set_sensor( this->shared_from_this() ); _source.set_callback( callback ); _is_streaming = true; + _rum_timer.restart(); raise_on_before_streaming_changes( true ); } @@ -233,6 +237,7 @@ void software_sensor::stop() if( ! _is_streaming ) throw wrong_api_call_sequence_exception( "stop_streaming() failed. Software device is not streaming!" ); + record_rum_stream_duration(); // while is_streaming() still true _is_streaming = false; raise_on_before_streaming_changes( false ); _source.flush(); diff --git a/src/to-string.cpp b/src/to-string.cpp index 942d2a55f8..7d3afb1999 100644 --- a/src/to-string.cpp +++ b/src/to-string.cpp @@ -581,6 +581,7 @@ std::string const & get_string_( rs2_option value ) CASE( DOWNSCALE_RATIO ) CASE( READOUT_SHAPING ) CASE( DETECTION_DISTANCE ) + CASE( SENSORS_CONFIG_MODE ) #undef CASE return arr; }(); diff --git a/src/uvc-sensor.cpp b/src/uvc-sensor.cpp index 59c65c7fd4..cd6eb2ee38 100644 --- a/src/uvc-sensor.cpp +++ b/src/uvc-sensor.cpp @@ -146,6 +146,17 @@ void uvc_sensor::open( const stream_profiles & requests ) _zc_inflight.clear(); // drained/read in close(); repopulated per stream below + // On devices that opt in (enable_software_color_frame_numbers), the color pins share a single + // hardware frame counter, so each stream's frame number jumps by the number of color streams per + // interval - which makes the reported (hardware) FPS read 2x. Give each color stream its own + // software frame counter, mirroring the accel/gyro override below. + int color_stream_count = 0; + if( _sw_color_frame_numbers ) + for( auto && rp : requests ) + if( rp->get_stream_type() == RS2_STREAM_COLOR ) + ++color_stream_count; + const bool per_stream_color_fn = ( color_stream_count > 1 ); + for( auto && req_profile : requests ) { auto && req_profile_base = std::dynamic_pointer_cast< stream_profile_base >( req_profile ); @@ -160,7 +171,8 @@ void uvc_sensor::open( const stream_profiles & requests ) _zc_inflight.push_back( zc_inflight ); // close() drains this before buffers are freed _device->probe_and_commit( req_profile_base->get_backend_profile(), - [this, req_profile_base, req_profile, last_frame_number, last_timestamp, zc_inflight]( + [this, req_profile_base, req_profile, last_frame_number, last_timestamp, zc_inflight, + per_stream_color_fn, color_fn = 0ull]( platform::stream_profile p, platform::frame_object f, std::function< void() > continuation ) mutable @@ -203,6 +215,15 @@ void uvc_sensor::open( const stream_profiles & requests ) frame_counter = fr->additional_data.frame_number; } + // Dual color streams share one hardware frame counter; give each its own + // per-stream software counter so the reported (hardware) FPS isn't doubled + // (see per_stream_color_fn). Mirrors the accel/gyro override above. + if( per_stream_color_fn && req_profile_base->get_stream_type() == RS2_STREAM_COLOR ) + { + fr->additional_data.frame_number = ++color_fn; + frame_counter = fr->additional_data.frame_number; + } + LOG_DEBUG( "FrameAccepted," << librealsense::get_string( req_profile_base->get_stream_type() ) << ",Counter," << std::dec << fr->additional_data.frame_number << ",Index," diff --git a/src/uvc-sensor.h b/src/uvc-sensor.h index 9b585460d2..72bf19314f 100644 --- a/src/uvc-sensor.h +++ b/src/uvc-sensor.h @@ -45,6 +45,12 @@ class uvc_sensor : public raw_sensor_base int & index ) >; void set_stream_id_resolver( stream_id_resolver resolver ) { _stream_id_resolver = std::move( resolver ); } + // Opt in to a per-stream software frame number for color, for devices whose color pins share one + // hardware frame counter (D401 GMSL dual-RGB). Off by default: it makes get_frame_number() count + // 1,2,3... so counter-gap frame-drop detection no longer works and the value no longer matches + // RS2_FRAME_METADATA_FRAME_COUNTER, which still reports the raw hardware counter. + void enable_software_color_frame_numbers() { _sw_color_frame_numbers = true; } + std::vector< platform::stream_profile > get_configuration() const { return _internal_config; } std::shared_ptr< platform::uvc_device > get_uvc_device() { return _device; } platform::usb_spec get_usb_specification() const { return _device->get_usb_specification(); } @@ -81,6 +87,7 @@ class uvc_sensor : public raw_sensor_base void reset_streaming(); std::atomic _gyro_counter; std::atomic _accel_counter; + bool _sw_color_frame_numbers = false; // see enable_software_color_frame_numbers() struct power diff --git a/third-party/rsutils/include/rsutils/os/os.h b/third-party/rsutils/include/rsutils/os/os.h index 42632cbc4c..c516b2d4aa 100644 --- a/third-party/rsutils/include/rsutils/os/os.h +++ b/third-party/rsutils/include/rsutils/os/os.h @@ -11,5 +11,8 @@ namespace rsutils std::string get_os_name(); std::string get_platform_name(); + // CPU architecture the binary was built for: "x86_64", "arm64", "x86", "arm", or "unknown". + std::string cpu_arch(); + } } diff --git a/third-party/rsutils/src/os.cpp b/third-party/rsutils/src/os.cpp index 9872676896..62a1870836 100644 --- a/third-party/rsutils/src/os.cpp +++ b/third-party/rsutils/src/os.cpp @@ -24,6 +24,21 @@ namespace rsutils #endif } + std::string cpu_arch() + { + #if defined( _M_X64 ) || defined( __x86_64__ ) + return "x86_64"; + #elif defined( _M_ARM64 ) || defined( __aarch64__ ) + return "arm64"; + #elif defined( _M_IX86 ) || defined( __i386__ ) + return "x86"; + #elif defined( __arm__ ) + return "arm"; + #else + return "unknown"; + #endif + } + std::string get_platform_name() { #ifdef _WIN64 diff --git a/tools/realsense-viewer/realsense-viewer.cpp b/tools/realsense-viewer/realsense-viewer.cpp index 2a729c2113..217431872d 100644 --- a/tools/realsense-viewer/realsense-viewer.cpp +++ b/tools/realsense-viewer/realsense-viewer.cpp @@ -25,10 +25,13 @@ #include +#include "rum-uploader/rum-uploader.h" + using namespace rs2; using namespace rs400; + void update_viewer_configuration(viewer_model& viewer_model) { // Hide options from the Viewer application @@ -388,12 +391,18 @@ int run_viewer( int argc, const char ** argv, if( on_setup ) on_setup( *device_models, viewer_model ); + // Its destructor joins the boot-upload worker on any exit from run_viewer (normal return or an + // exception out of the render loop), so the thread is never left running after shutdown. + rs2::rum_uploader rum_boot; + // Closing the window while (window) { refresh_devices(m, ctx, devices_connection_changes, connected_devs, device_names, *device_models, viewer_model, error_message); + rum_boot.upload_data(window); + auto output_height = viewer_model.get_output_height(); rect viewer_rect = { viewer_model.panel_width, @@ -675,5 +684,7 @@ int run_viewer( int argc, const char ** argv, sub->stop(viewer_model.not_model); } + rs2::rum_uploader::join_pending_stops(device_models); + return EXIT_SUCCESS; } diff --git a/tools/realsense-viewer/tests/controls/test-options-filter.cpp b/tools/realsense-viewer/tests/controls/test-options-filter.cpp new file mode 100644 index 0000000000..e2eb8f4113 --- /dev/null +++ b/tools/realsense-viewer/tests/controls/test-options-filter.cpp @@ -0,0 +1,60 @@ +// License: Apache 2.0. See LICENSE file in root directory. +// Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +#include "viewer-test-helpers.h" + + +// Type into the Controls search box and verify the option list is filtered live: +// case-insensitive substring match on the control name, empty input restores the full list +VIEWER_TEST( "controls", "options_filter" ) +{ + auto & model = test.find_first_device_or_exit(); + bool tested = false; + + for( auto && sub : model.subdevices ) + { + test.expand_sensor_panel( model, sub ); + test.expand_controls( model, sub ); + + // the options the UI currently renders inside this sensor's Controls section + auto options = test.controls_options( model, sub ); + if( options.size() < 2 ) + { + test.collapse_sensor_panel( model, sub ); + continue; + } + + // filter = first control's name minus its last char; pick another control whose + // name does not contain it + std::string filter = test.control_name( sub, options[0] ); + filter.pop_back(); + rs2_option other = RS2_OPTION_COUNT; + for( auto o : options ) + if( test.control_name( sub, o ).find( filter ) == std::string::npos ) + { + other = o; + break; + } + if( other == RS2_OPTION_COUNT ) // filter hides nothing — can't verify on this sensor + { + test.collapse_sensor_panel( model, sub ); + continue; + } + + // case-insensitive substring match: non-matching control disappears, matching stays + test.set_controls_filter( model, sub, filter ); + IM_CHECK( test.wait_until( 10, 0.3f, [&] { return ! test.control_visible( model, sub, other ); } ) ); + IM_CHECK( test.control_visible( model, sub, options[0] ) ); + + // clearing the box restores the full list + test.set_controls_filter( model, sub, "" ); + IM_CHECK( test.wait_until( 10, 0.3f, [&] { return test.control_visible( model, sub, other ); } ) ); + + test.collapse_controls( model, sub ); + test.collapse_sensor_panel( model, sub ); + tested = true; + break; // one sensor is enough — the filter code path is per-sensor identical + } + + IM_CHECK( tested ); +} diff --git a/tools/realsense-viewer/tests/viewer-test-helpers.cpp b/tools/realsense-viewer/tests/viewer-test-helpers.cpp index 4abe6965a8..9aee6137a9 100644 --- a/tools/realsense-viewer/tests/viewer-test-helpers.cpp +++ b/tools/realsense-viewer/tests/viewer-test-helpers.cpp @@ -4,6 +4,9 @@ #include "viewer-test-helpers.h" #include "imgui_te_context.h" +#include +#include + // --------------------------------------------------------------------------- // viewer_test method implementations @@ -213,6 +216,54 @@ std::string viewer_test::get_control_value( rs2::device_model & model, return get_value_by_seed( opt, seed ); } +void viewer_test::set_controls_filter( rs2::device_model & model, + std::shared_ptr< rs2::subdevice_model > sub, + const std::string & text ) +{ + imgui->SetRef( "Control Panel" ); + imgui->ItemInput( ImHashStr( "##options_filter", 0, controls_id_seed( model, sub ) ) ); + imgui->KeyCharsReplaceEnter( text.c_str() ); + imgui->SleepNoSkip( 0.3f, 0.1f ); +} + +std::vector< rs2_option > viewer_test::controls_options( rs2::device_model & model, + std::shared_ptr< rs2::subdevice_model > sub ) +{ + imgui->SetRef( "Control Panel" ); + ImGuiID seed = controls_id_seed( model, sub ); + ImGuiTestItemList items; + imgui->GatherItems( &items, seed, -1 ); + + std::vector< rs2_option > result; + for( auto & kvp : sub->options_metadata ) + { + auto & opt = kvp.second; + const std::string & widget = opt.is_checkbox() ? opt.label : opt.id; + ImGuiID id = ImHashStr( widget.c_str(), 0, seed ); + for( auto const & item : items ) + if( item.ID == id ) + { + result.push_back( kvp.first ); + break; + } + } + return result; +} + +std::string viewer_test::control_name( std::shared_ptr< rs2::subdevice_model > sub, rs2_option option ) +{ + // label format is "##" + auto & label = find_option( sub, option ).label; + return rsutils::string::to_lower( label.substr( 0, label.find( "##" ) ) ); +} + +bool viewer_test::control_visible( rs2::device_model & model, + std::shared_ptr< rs2::subdevice_model > sub, rs2_option option ) +{ + auto v = controls_options( model, sub ); + return std::find( v.begin(), v.end(), option ) != v.end(); +} + void viewer_test::select_combo_item( ImGuiID combo_id, const std::string & item ) { imgui->ItemClick( combo_id ); diff --git a/tools/realsense-viewer/tests/viewer-test-helpers.h b/tools/realsense-viewer/tests/viewer-test-helpers.h index 1fe7a5f571..bb04c82910 100644 --- a/tools/realsense-viewer/tests/viewer-test-helpers.h +++ b/tools/realsense-viewer/tests/viewer-test-helpers.h @@ -125,6 +125,20 @@ class viewer_test std::shared_ptr< rs2::subdevice_model > sub, rs2_option option ); + // Replace the text in the Controls section's search/filter box ("" clears it) + void set_controls_filter( rs2::device_model & model, + std::shared_ptr< rs2::subdevice_model > sub, + const std::string & text ); + // Options whose control widgets are currently rendered inside the Controls section + // (single gather pass; requires the sensor panel and Controls section to be expanded) + std::vector< rs2_option > controls_options( rs2::device_model & model, + std::shared_ptr< rs2::subdevice_model > sub ); + // Lowercased display name of an option's control + std::string control_name( std::shared_ptr< rs2::subdevice_model > sub, rs2_option option ); + // Whether an option's control is currently rendered inside the Controls section + bool control_visible( rs2::device_model & model, + std::shared_ptr< rs2::subdevice_model > sub, rs2_option option ); + // Open a combo dropdown by ID and select the named item void select_combo_item( ImGuiID combo_id, const std::string & item ); // Select a resolution from the sensor's resolution combo box diff --git a/tools/rum-uploader/dev-server/rum_dev_server.py b/tools/rum-uploader/dev-server/rum_dev_server.py new file mode 100644 index 0000000000..22b8136ca3 --- /dev/null +++ b/tools/rum-uploader/dev-server/rum_dev_server.py @@ -0,0 +1,112 @@ +# License: Apache 2.0. See LICENSE file in root directory. +# Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +""" +Smallest-possible RUM ingest stub for local development. + +Accepts HTTPS/HTTP POST /v1/rum, writes each received body to a timestamped JSON +file under ./received/ so the payload can be inspected, and replies 200 OK. This is +NOT the production server (no validation, auth, or storage) -- it exists only to +confirm the viewer's uploader sends well-formed reports end-to-end. + +Usage: + python rum_dev_server.py [--port 8080] [--dir received] + +The viewer's uploader targets http://127.0.0.1:8080/v1/rum by default (the production endpoint +is not live yet), so just run this on port 8080 and consented uploads land in ./received/. +""" + +import argparse +import json +import os +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + + +class _Handler(BaseHTTPRequestHandler): + out_dir = "received" + _counter = 0 + + def do_POST(self): + if self.path != "/v1/rum": + self.send_response(404) + self.end_headers() + return + + length = int(self.headers.get("Content-Length", 0)) + body = self.rfile.read(length) if length else b"" + + os.makedirs(self.out_dir, exist_ok=True) + _Handler._counter += 1 + stamp = time.strftime("%Y%m%d-%H%M%S") + path = os.path.join(self.out_dir, "rum-{}-{:03d}.json".format(stamp, _Handler._counter)) + + # Pretty-print if it parses as JSON; otherwise store raw bytes. + try: + parsed = json.loads(body.decode("utf-8")) + with open(path, "w", encoding="utf-8") as f: + json.dump(parsed, f, indent=2) + self._print_summary(parsed, len(body), path) + except Exception: + with open(path, "wb") as f: + f.write(body) + print("\n[{}] received {} bytes (non-JSON) -> {}".format( + time.strftime("%H:%M:%S"), len(body), path)) + + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b'{"status":"ok"}') + + @staticmethod + def _print_summary(r, nbytes, path): + sdk = r.get("sdk", {}) + sys_ = r.get("system", {}) + flags = sdk.get("cmake_flags", {}) + flags_str = ", ".join("{}={}".format(k, v) for k, v in flags.items()) or "-" + + def join(arr, fmt): + return "; ".join(fmt(x) for x in r.get(arr, [])) or "-" + + print("\n" + "=" * 70) + print("[{}] RUM report ({} bytes)".format(time.strftime("%H:%M:%S"), nbytes)) + print(" source_id : {}".format(r.get("source_id", "?"))) + print(" sdk : {} ({}, backend={})".format( + sdk.get("version"), sdk.get("build_type"), sdk.get("backend"))) + print(" cmake : {}".format(flags_str)) + print(" system : {} / {}".format(sys_.get("os"), sys_.get("arch"))) + print(" devices : {}".format(join("devices", + lambda d: "{} fw={} {} mipi={} (x{})".format(d.get("type"), d.get("fw_version"), d.get("connection"), d.get("mipi_driver_version") or "-", d.get("count"))))) + print(" streams : {}".format(join("streams", + lambda s: "{} {} {}@{} (x{}, {:.1f}s)".format(s.get("type"), s.get("format"), s.get("resolution"), s.get("fps"), s.get("count"), s.get("duration_seconds", 0))))) + print(" options : {}".format(join("options_changed", + lambda o: "{}={} (x{})".format(o.get("option"), o.get("last_value"), o.get("set_count"))))) + print(" filters : {}".format(join("filters", + lambda f: "{} (x{})".format(f.get("name"), f.get("count"))))) + print(" notifs : {}".format(join("notifications", + lambda n: "{} (x{})".format(n.get("category"), n.get("count"))))) + print(" saved -> {}".format(path)) + print("=" * 70) + + def log_message(self, *args): + pass # quiet default access logging + + +def main(): + parser = argparse.ArgumentParser(description="RUM local dev ingest stub") + parser.add_argument("--port", type=int, default=8080) + parser.add_argument("--dir", default="received") + args = parser.parse_args() + + _Handler.out_dir = args.dir + server = ThreadingHTTPServer(("127.0.0.1", args.port), _Handler) + print("RUM dev server listening on http://127.0.0.1:{}/v1/rum (writing to '{}/')".format(args.port, args.dir)) + try: + server.serve_forever() + except KeyboardInterrupt: + pass + server.server_close() + + +if __name__ == "__main__": + main() diff --git a/tools/tools_common.cmake b/tools/tools_common.cmake index ea56a73ef4..cfc6f3e841 100644 --- a/tools/tools_common.cmake +++ b/tools/tools_common.cmake @@ -17,11 +17,16 @@ macro(tools_target_config _target) else() target_link_libraries(${_target} dl) endif() - if(CHECK_FOR_UPDATES) - message( STATUS "Check for updates capability added to ${_target}" ) + if(CHECK_FOR_UPDATES OR ENABLE_STATS) add_dependencies(${_target} libcurl) set(RS_VIEWER_LIBS ${RS_VIEWER_LIBS} curl) endif() + if(CHECK_FOR_UPDATES) + message( STATUS "Check for updates capability added to ${_target}" ) + endif() + if(ENABLE_STATS) + message( STATUS "Stats monitoring capability added to ${_target}" ) + endif() target_link_libraries(${_target} ${DEPENDENCIES} ${RS_VIEWER_LIBS} tclap) set_target_properties(${_target} PROPERTIES CXX_STANDARD 11 FOLDER Tools) endmacro() diff --git a/unit-tests/conftest.py b/unit-tests/conftest.py index 10fae1f586..d839862c00 100644 --- a/unit-tests/conftest.py +++ b/unit-tests/conftest.py @@ -193,6 +193,27 @@ def pytest_addoption(parser): dest="repeat_count", help="Run all tests in each file N times (module-scoped alias for pytest-repeat's --count). Use --count for per-test repetition." ) + group.addoption( + "--custom-fw-d400", + action="store", + default=None, + help="Path to a custom D400 firmware image; pytest-fw-update flashes it if it differs " + "from the installed FW." + ) + group.addoption( + "--custom-fw-d555", + action="store", + default=None, + help="Path to a custom D555 firmware image; pytest-fw-update flashes it if it differs " + "from the installed FW." + ) + group.addoption( + "--custom-fw-d585", + action="store", + default=None, + help="Path to a custom D585 (non-safety) firmware image; pytest-fw-update flashes it if " + "it differs from the installed FW. Applies to D585 only -- never flashed onto D585S." + ) # --debug and -r/--regex conflict with pytest built-ins and are consumed before # pytest parses args. Document them here so they show up in --help: group.addoption( @@ -210,7 +231,7 @@ def pytest_addoption(parser): action="append", default=[], help="Restrict pytest discovery to tests under this directory or file. " - "May be repeated (e.g. `--test-dir live/image-quality --test-dir test-fw-update.py`). " + "May be repeated (e.g. `--test-dir live/image-quality --test-dir pytest-fw-update.py`). " "Matches run-unit-tests.py --test-dir for shared UNIT_TESTS_ARGS." ) diff --git a/unit-tests/infra-tests/e2e/pytest-custom-fw.py b/unit-tests/infra-tests/e2e/pytest-custom-fw.py new file mode 100644 index 0000000000..dcd4512049 --- /dev/null +++ b/unit-tests/infra-tests/e2e/pytest-custom-fw.py @@ -0,0 +1,16 @@ +# License: Apache 2.0. See LICENSE file in root directory. +# Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +"""E2E: --custom-fw-* options are registered and their values reach the tests.""" + + +def test_values(request): + assert request.config.getoption('--custom-fw-d400') == 'd400.bin' + assert request.config.getoption('--custom-fw-d555') == 'd555.bin' + assert request.config.getoption('--custom-fw-d585') == 'd585.bin' + + +def test_defaults(request): + assert request.config.getoption('--custom-fw-d400') is None + assert request.config.getoption('--custom-fw-d555') is None + assert request.config.getoption('--custom-fw-d585') is None diff --git a/unit-tests/infra-tests/test_collection.py b/unit-tests/infra-tests/test_collection.py index 41e0a3dc33..c4d9729fa4 100644 --- a/unit-tests/infra-tests/test_collection.py +++ b/unit-tests/infra-tests/test_collection.py @@ -154,6 +154,51 @@ def test_default_priority_is_500(self): assert names[0] == "test_below" assert names.index("test_no_prio") < names.index("test_above") + def test_priority_orders_modules(self): + """A module's priority must beat alphabetical module order in both directions: + mod_zzz_fw_update (priority 1) runs first, mod_0bbb (priority 900) runs last, + even though the alphabet would put them the other way around.""" + items = [ + make_mock_item("test_default", module_name="mod_aaa"), + make_mock_item("test_urgent", module_name="mod_zzz_fw_update", + markers=[pytest.mark.priority(1)]), + make_mock_item("test_late", module_name="mod_0bbb", + markers=[pytest.mark.priority(900)]), + ] + filter_and_sort_items(make_mock_config(), items) + + names = [i.name for i in items] + assert names == ["test_urgent", "test_default", "test_late"] + + def test_module_priority_uses_own_default(self): + """A module whose tests are all above the 500 default must sort after a default + module, not be clamped to 500 and fall back to alphabetical order.""" + items = [ + make_mock_item("test_late", module_name="mod_aaa", + markers=[pytest.mark.priority(900)]), + make_mock_item("test_default", module_name="mod_zzz"), + ] + filter_and_sort_items(make_mock_config(), items) + + names = [i.name for i in items] + assert names == ["test_default", "test_late"] + + def test_module_priority_keeps_module_grouping(self): + """A single high-priority test pulls its whole module forward, but items never + interleave across modules — grouping by module is preserved.""" + items = [ + make_mock_item("test_a1", module_name="mod_aaa"), + make_mock_item("test_z1", module_name="mod_zzz"), + make_mock_item("test_z2_first", module_name="mod_zzz", + markers=[pytest.mark.priority(1)]), + make_mock_item("test_a2", module_name="mod_aaa"), + ] + filter_and_sort_items(make_mock_config(), items) + + names = [i.name for i in items] + # mod_zzz (min priority 1) runs first as a contiguous block, priority order inside + assert names == ["test_z2_first", "test_z1", "test_a1", "test_a2"] + class TestDeviceGrouping: """Tests should be grouped by (module, device_serial) so hub recycling is minimized.""" diff --git a/unit-tests/infra-tests/test_e2e_cli_options.py b/unit-tests/infra-tests/test_e2e_cli_options.py index c9d73ba529..8af5d184ee 100644 --- a/unit-tests/infra-tests/test_e2e_cli_options.py +++ b/unit-tests/infra-tests/test_e2e_cli_options.py @@ -165,6 +165,20 @@ def test_not_live(self): rc, out, *_ = run_e2e("pytest-live.py", "--not-live") assert_outcomes(out, passed=1, skipped=1) + def test_custom_fw_options(self): + """--custom-fw-d400/-d555/-d585 are accepted and their values reach the tests + via request.config.getoption (consumed by pytest-fw-update).""" + rc, out, *_ = run_e2e("pytest-custom-fw.py", "-k", "test_values", + "--custom-fw-d400", "d400.bin", + "--custom-fw-d555", "d555.bin", + "--custom-fw-d585", "d585.bin") + assert_outcomes(out, passed=1) + + def test_custom_fw_defaults(self): + """Without --custom-fw-* flags the options default to None (pytest-fw-update skips).""" + rc, out, *_ = run_e2e("pytest-custom-fw.py", "-k", "test_defaults") + assert_outcomes(out, passed=1) + def test_tag_filters_by_marker(self): """--tag should run only tests with pytest.mark. (alias for -m).""" rc, out, *_ = run_e2e("pytest-priority.py", "--tag", "priority") diff --git a/unit-tests/live/fw/pytest-fw-errors.py b/unit-tests/live/fw/pytest-fw-errors.py index 3a5abfc5b0..1ebce7d338 100644 --- a/unit-tests/live/fw/pytest-fw-errors.py +++ b/unit-tests/live/fw/pytest-fw-errors.py @@ -16,7 +16,7 @@ pytestmark = [ pytest.mark.device_each("D400*"), pytest.mark.device_each("D500*"), - pytest.mark.device_exclude("D585 Proto*"), + pytest.mark.device_exclude("D585 Proto"), pytest.mark.context("nightly"), ] diff --git a/unit-tests/py/rspy/pytest/collection.py b/unit-tests/py/rspy/pytest/collection.py index 89acbe6648..94dec0014e 100644 --- a/unit-tests/py/rspy/pytest/collection.py +++ b/unit-tests/py/rspy/pytest/collection.py @@ -133,11 +133,22 @@ def get_priority(item): # Within a (module, device) bucket, also sort by pytest-repeat step so pass 0 # completes before pass 1 — preserves --repeat N module-scoped ordering so # module-scoped fixtures see one pass at a time. + # Modules are ordered by their most urgent (lowest) item priority so a high-priority + # module (e.g. pytest-fw-update, priority 1, which must run before anything that reads + # the FW version) runs before all other modules — matching run-unit-tests.py, which + # sorted whole test files by priority. Within a bucket the stable sort preserves the + # per-item priority order from above. + module_priority = {} + for item in items: + module = item.module.__name__ + p = get_priority(item) + module_priority[module] = min(module_priority.get(module, p), p) + def get_device_group_key(item): module = item.module.__name__ params = item.callspec.params if hasattr(item, 'callspec') else {} device_serial = params.get('_test_device_serial', '') step = params.get('__pytest_repeat_step_number', 0) - return (module, device_serial, step) + return (module_priority[module], module, device_serial, step) items.sort(key=get_device_group_key) diff --git a/unit-tests/pytest-fw-update.py b/unit-tests/pytest-fw-update.py new file mode 100644 index 0000000000..7045f61eb0 --- /dev/null +++ b/unit-tests/pytest-fw-update.py @@ -0,0 +1,380 @@ +# License: Apache 2.0. See LICENSE file in root directory. +# Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +import sys +import os +import subprocess +import re +import pytest +from pytest_check import check +import pyrealsense2 as rs +import pyrsutils as rsutils +from rspy import devices, repo, fw_compat, config_file, libci +from rspy.timer import Timer +import time +import logging + +log = logging.getLogger(__name__) + +# We want this test to run right after camera detection phase, so that all tests will run +# with updated FW versions, so we give it high priority +pytestmark = [ + pytest.mark.device_each("D400*"), + pytest.mark.device_each("D555"), + pytest.mark.device_each("D585"), + pytest.mark.device_exclude("D585S"), + pytest.mark.priority(1), + pytest.mark.timeout(500), + pytest.mark.skipif(bool(os.environ.get('GITHUB_ACTIONS')), reason="not runnable on GHA"), +] + + +def wait_for_reboot( same_version ): + """ + Wait for the camera to finish rebooting after a FW update. + The test exit flow may cut USB power (via hub port disable), so we must ensure + the device has had enough time to complete its reboot before we exit. + When updating to a different version, FW may need time to flash a new ISP FW. + """ + sleep_time = 60 if not same_version else 3 + log.debug( f"Waiting {sleep_time} seconds for device to finish rebooting after FW update..." ) + time.sleep( sleep_time ) + + +def send_hardware_monitor_command(device, command): + # byte_index = -1 + raw_result = rs.debug_protocol(device).send_and_receive_raw_data(command) + + return raw_result[4:] + + +def extract_version_from_filename(file_path): + """ + Extracts the version string from a filename like: + FlashGeneratedImage_Image5_16_7_0.bin -> 5.16.7 + FlashGeneratedImage_RELEASE_DS5_5_16_3_1.bin -> 5.16.3.1 + rvp-flash-dfu-release-7.56.37749.4831.img -> 7.56.37749.4831 + 20260727_7.58.40846.12889.img (NIGHTLY build) -> 7.58.40846.12889 + + Args: + file_path (str): Full path to the file. + + Returns: + str: Extracted version in format x.y.z or x.y.z.w, or None if not found or if path is invalid. + """ + if not file_path or not os.path.exists(file_path): + log.info(f"File not found: {file_path}") + return None + + filename = os.path.basename(file_path) + + # Match *last* 4 numeric groups before .img/.bin + # following matching patterns for cases: + # FlashGeneratedImage_Image5_16_7_0.bin -> 5.16.7 + # FlashGeneratedImage_RELEASE_DS5_5_16_3_1.bin -> 5.16.3.1 + match = re.search(r'(\d+)_(\d+)_(\d+)_(\d+)\.(bin|img)$', filename) + if not match: + # Match a dot-separated x.y.z.w version immediately before the extension, + # regardless of what precedes it (hyphen, underscore, a date prefix, etc.), e.g.: + # rvp-flash-dfu-release-7.56.37749.4831.img -> 7.56.37749.4831 + # 20260727_7.58.40846.12889.img (NIGHTLY build) -> 7.58.40846.12889 + match = re.search(r'(? path is required + # for every product line; otherwise we cannot exercise the update flow. + same_version = False + custom_fw_path = None + custom_fw_version = None + if product_line == "D400" and custom_fw_d400: + custom_fw_path = custom_fw_d400 + elif "D555" in product_name and custom_fw_d555: + custom_fw_path = custom_fw_d555 + elif "D585" in product_name and custom_fw_d585: + custom_fw_path = custom_fw_d585 + + if not custom_fw_path: + pytest.skip("No custom FW path provided (use --custom-fw-d400 / --custom-fw-d555 / --custom-fw-d585); skipping FW update test") + + # check if recovery on the configured domain (e.g. a D400 USB recovery device). If so recover. + # (recovered may already be True from the domain-0 DDS recovery handled above.) + if device.is_in_recovery_mode(): + log.debug( "recovering device ..." ) + # rs-fw-update -r needs a known-good image, which isn't always the caller's + # --custom-fw- path (e.g. D400 -r expects a *signed* FW, while the custom + # image is typically unsigned). Fetch the per-product-line gold FW to recover with. + gold_fw = fw_compat.download_gold_fw( product_line, product_name ) + if not gold_fw: + pytest.fail( f"Could not download gold recovery FW for {product_name}; cannot recover DFU device" ) + cmd = [fw_updater_exe, '-r', '-f', gold_fw, '-s', serial] + del device, ctx + log.debug( f'running: {cmd}' ) + subprocess.run( cmd ) + recovered = True + fw_compat.reload_d4xx_driver_on_jetson( context ) + # The device's identity changed: in DFU it exposed firmware_update_id only, + # now in normal mode it exposes its real serial_number (optic_serial). The + # firmware_update_id (asic_serial) is still exposed and matches what the + # harness was tracking. Poll for the device to re-enumerate in normal mode + # (a fresh rs.context() needs time after rs-fw-update exits) -- up to 60s. + log.debug( "waiting for recovered device to re-enumerate in normal mode..." ) + recovered_device = None + timer = Timer( 60 ) + timer.start() + while not timer.has_expired(): + for d in rs.context().devices: + if d.supports( rs.camera_info.firmware_update_id ) \ + and d.get_info( rs.camera_info.firmware_update_id ) == serial \ + and not d.is_in_recovery_mode(): + recovered_device = d + break + if recovered_device is not None: + break + time.sleep( 2 ) + if recovered_device is None: + pytest.fail( f"Recovered device with firmware_update_id '{serial}' did not " + f"re-enumerate within {timer.get_timeout()}s after gold FW flash" ) + # Re-pin the serial to the device's normal-mode SN so downstream + # rs-fw-update -s finds the device (rs-fw-update.cpp:480 uses SN when supported). + if recovered_device.supports( rs.camera_info.serial_number ): + new_sn = recovered_device.get_info( rs.camera_info.serial_number ) + if new_sn != serial: + log.debug( f're-pinning serial: {serial} (FWID) -> {new_sn} (SN)' ) + serial = new_sn + device, ctx = find_device_or_fail( serial ) + current_fw_version = rsutils.version(device.get_info(rs.camera_info.firmware_version)) + log.debug(f"FW version after recovery: {current_fw_version}") + + + custom_fw_version = extract_version_from_filename(custom_fw_path) + log.debug(f'Using custom FW version: {custom_fw_version}') + + if current_fw_version == custom_fw_version: + same_version = True + if recovered or 'nightly' not in context: + log.debug('versions are same; skipping FW update') + return + + downgrade_counter = get_downgrade_counter( device ) + log.debug( f'downgrade counter: {downgrade_counter}' ) + if downgrade_counter == 0xFFFF: + log.debug( 'downgrade counter is uninitialized (0xFFFF), skipping reset' ) + downgrade_counter = 0 + elif downgrade_counter >= 19: + log.debug( f'resetting downgrade counter (was {downgrade_counter})' ) + reset_downgrade_counter( device ) + log.debug( 'sleeping for 3 sec...' ) + time.sleep( 3 ) + downgrade_counter = get_downgrade_counter( device ) + log.debug( f'downgrade counter after reset is: {downgrade_counter}' ) + check.equal( downgrade_counter, 0 ) + downgrade_counter = 0 + + image_file = custom_fw_path + + cmd = [fw_updater_exe, '-f', image_file] + if serial: + cmd += ['-s', serial] + # Add '-u' only if the path doesn't include 'signed' + if ('signed' not in custom_fw_path.lower() + and "d555" not in product_name.lower() # currently -u is not supported for D555 + and "d585" not in product_name.lower()): # nor for D585/D585S + cmd.insert(1, '-u') + + # for DDS devices we need to close device and context to detect it back after FW update + del device, ctx + log.debug( f'running: {cmd}' ) + sys.stdout.flush() + result = subprocess.run( cmd ) # may throw + + # Wait for the camera to finish rebooting before doing anything else, REGARDLESS of + # rs-fw-update's exit code. A non-zero exit doesn't necessarily mean no flash started: + # rs-fw-update may have begun a section flash before erroring out, leaving the device + # mid-reboot. The test exit flow may cut USB power (hub port disable), so we must not + # exit while the device is still rebooting. + wait_for_reboot( same_version ) + + if result.returncode != 0: + pytest.fail( f'rs-fw-update should return exit code 0 (got {result.returncode})' ) + + # make sure update worked and check FW version and update counter + device, ctx = find_device_or_fail( serial ) + current_fw_version = rsutils.version( device.get_info( rs.camera_info.firmware_version )) + + # camera_locked returns "YES" (locked) or "NO" (unlocked) + if device.supports( rs.camera_info.camera_locked ) and device.get_info( rs.camera_info.camera_locked ) == 'YES': + log.warning( 'Device is flash-locked' ) + + check.equal(current_fw_version, custom_fw_version) + new_downgrade_counter = get_downgrade_counter( device ) + log.debug( f'downgrade counter after update: {new_downgrade_counter}' ) + # + ############################################################################### diff --git a/unit-tests/rum/pytest-rum-config.py b/unit-tests/rum/pytest-rum-config.py new file mode 100644 index 0000000000..a9de644624 --- /dev/null +++ b/unit-tests/rum/pytest-rum-config.py @@ -0,0 +1,73 @@ +# License: Apache 2.0. See LICENSE file in root directory. +# Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +import os +import json +import pytest +import pyrealsense2 as rs +from rspy import config_file + + +def test_rum_submodule_is_exposed(): + assert hasattr( rs, "rum" ) + + +def test_cloud_consent_round_trips(): + if os.environ.get( "RS2_RUM_CLOUD_ENABLED" ): + pytest.skip( "RS2_RUM_CLOUD_ENABLED is set; env overrides the config value" ) + # Leave the config as we found it: restore the value if the key was there, drop the key if only + # the file was there, or remove the file entirely if the test created it (e.g. a fresh CI runner). + key = "rum_cloud_enabled" + cfg_path = config_file.get_config_path() + had_file = os.path.exists( cfg_path ) + had_key = had_file and key in config_file.get_config_file() + saved = rs.rum.is_cloud_enabled() + try: + rs.rum.set_cloud_enabled( True ) + assert rs.rum.is_cloud_enabled() + rs.rum.set_cloud_enabled( False ) + assert not rs.rum.is_cloud_enabled() + finally: + if had_key: + rs.rum.set_cloud_enabled( saved ) + elif had_file: + cfg = config_file.get_config_file() + cfg.pop( key, None ) + with open( cfg_path, "w", encoding="utf-8" ) as f: + json.dump( cfg, f ) + elif os.path.exists( cfg_path ): + os.remove( cfg_path ) + + +def test_report_is_valid_json_with_expected_fields(): + report = json.loads( rs.rum.get_report() ) + assert report.get( "schema_version" ) == 1 + source_id = report.get( "source_id", "" ) + assert isinstance( source_id, str ) + assert len( source_id ) == 36 + session_id = report.get( "session_id", "" ) + assert isinstance( session_id, str ) + assert len( session_id ) == 36 + assert isinstance( report.get( "generated_at" ), int ) + assert report.get( "sdk", {} ).get( "version" ) + assert isinstance( report.get( "sdk", {} ).get( "cmake_flags" ), dict ) + assert report.get( "sdk", {} ).get( "backend" ) + assert report.get( "system", {} ).get( "os" ) + assert report.get( "system", {} ).get( "arch" ) + # Aggregation arrays are always present (possibly empty) so the schema is stable. + for key in ( "devices", "streams", "options_changed", "filters", "notifications" ): + assert isinstance( report.get( key ), list ) + + +def test_source_id_is_stable_across_calls(): + first = json.loads( rs.rum.get_report() ).get( "source_id" ) + again = json.loads( rs.rum.get_report() ).get( "source_id" ) + assert first == again + + +def test_processing_block_option_excluded_from_options_changed(): + # A processing-block option must never land in options_changed (only device options do). + th = rs.threshold_filter() + th.set_option( rs.option.min_distance, 0.5 ) + names = [ o.get( "option" ) for o in json.loads( rs.rum.get_report() ).get( "options_changed", [] ) ] + assert "Min Distance" not in names diff --git a/unit-tests/rum/pytest-rum-device.py b/unit-tests/rum/pytest-rum-device.py new file mode 100644 index 0000000000..1c88595782 --- /dev/null +++ b/unit-tests/rum/pytest-rum-device.py @@ -0,0 +1,95 @@ +# License: Apache 2.0. See LICENSE file in root directory. +# Copyright(c) 2026 RealSense, Inc. All Rights Reserved. + +import json +import pytest +import pyrealsense2 as rs + + +def _stats_enabled(): + flags = json.loads( rs.rum.get_report() ).get( "sdk", {} ).get( "cmake_flags", {} ) + return bool( flags.get( "ENABLE_STATS", False ) ) + + +# The collector is fed only by the instrumentation hooks, which compile to no-ops when +# ENABLE_STATS is off (the build default). Without them the report stays empty, so these +# live-device checks only make sense on a stats-enabled build. +pytestmark = [ + pytest.mark.device_each( "D400*" ), + pytest.mark.skipif( not _stats_enabled(), reason="SDK built with ENABLE_STATS=OFF" ), +] + + +def depth_z16_profile( sensor ): + profile = next( ( p for p in sensor.get_stream_profiles() + if p.stream_type() == rs.stream.depth and p.format() == rs.format.z16 ), None ) + assert profile is not None, "device exposes no Z16 depth profile" + return profile + + +def test_created_device_appears_in_report( test_device ): + dev, _ = test_device + name = dev.get_info( rs.camera_info.name ) + devices = json.loads( rs.rum.get_report() ).get( "devices", [] ) + entry = next( ( d for d in devices if d.get( "type" ) == name ), None ) + assert entry is not None + assert entry.get( "fw_version" ) + assert entry.get( "connection" ) + assert entry.get( "count", 0 ) >= 1 + + +def test_opened_stream_appears_in_report( test_device ): + dev, _ = test_device + sensor = dev.first_depth_sensor() + sensor.open( depth_z16_profile( sensor ) ) # triggers the stream hook + try: + streams = json.loads( rs.rum.get_report() ).get( "streams", [] ) + depth = next( ( s for s in streams if s.get( "type" ) == "Depth" ), None ) + assert depth is not None + assert depth.get( "format" ) == "Z16" + assert "x" in depth.get( "resolution", "" ) + assert depth.get( "fps", 0 ) > 0 + finally: + sensor.close() + + +def test_applied_filter_and_stream_duration( test_device ): + dev, _ = test_device + sensor = dev.first_depth_sensor() + queue = rs.frame_queue( 8 ) + spatial = rs.spatial_filter() + sensor.open( depth_z16_profile( sensor ) ) + sensor.start( queue ) + try: + for _ in range( 10 ): + spatial.process( queue.wait_for_frame() ) # run frames through the filter -> applied + finally: + sensor.stop() + sensor.close() + report = json.loads( rs.rum.get_report() ) + flt = next( ( f for f in report.get( "filters", [] ) if f.get( "name" ) == "Spatial Filter" ), None ) + assert flt is not None + assert flt.get( "count", 0 ) >= 1 + # The depth stream above (start -> stop) accumulates duration on its stream config. + depth = next( ( s for s in report.get( "streams", [] ) if s.get( "type" ) == "Depth" ), None ) + assert depth is not None + assert depth.get( "duration_seconds", 0 ) > 0 + + +def test_non_default_option_in_options_changed( test_device ): + dev, _ = test_device + sensor = dev.first_depth_sensor() + opt = rs.option.laser_power + if not sensor.supports( opt ): + pytest.skip( "device has no Laser Power option" ) + rng = sensor.get_option_range( opt ) + newval = rng.min if rng.default != rng.min else rng.max + sensor.set_option( opt, newval ) + try: + changed = json.loads( rs.rum.get_report() ).get( "options_changed", [] ) + entry = next( ( o for o in changed if o.get( "option" ) == "Laser Power" ), None ) + assert entry is not None + assert entry.get( "set_count", 0 ) >= 1 + assert entry.get( "last_value" ) == newval + finally: + sensor.set_option( opt, rng.default ) # restore device state diff --git a/unit-tests/test-fw-update.py b/unit-tests/test-fw-update.py deleted file mode 100644 index f2e8ecd02b..0000000000 --- a/unit-tests/test-fw-update.py +++ /dev/null @@ -1,367 +0,0 @@ -# License: Apache 2.0. See LICENSE file in root directory. -# Copyright(c) 2021 RealSense, Inc. All Rights Reserved. - -#We want this test to run right after camera detection phase, so that all tests will run with updated FW versions, so we give it high priority -#test:priority 1 -#test:timeout 500 -#test:donotrun:gha -#test:device each(D400*) -#test:device each(D555) -# each(D585) below name-matches D585S too (substring match); the script itself refuses -# to flash the D585 custom FW onto a D585S device (see the product_name check below). -#test:device each(D585) - -import sys -import os -import subprocess -import re -import platform -import pyrealsense2 as rs -import pyrsutils as rsutils -from rspy import log, test, file, repo, fw_compat, config_file -from rspy.timer import Timer -import time -import argparse - -# Parse command-line arguments -parser = argparse.ArgumentParser(description="Test firmware update") -parser.add_argument('--custom-fw-d400', type=str, help='Path to custom D400 firmware file') -parser.add_argument('--custom-fw-d555', type=str, help='Path to custom D555 firmware file') -parser.add_argument('--custom-fw-d585', type=str, help='Path to custom D585 (non-safety) firmware file; NOT applied to D585S') -parser.add_argument('--serial', type=str, default=None, help='Serial number of the device to update (for multi-device rigs)') -args = parser.parse_args() - - -def wait_for_reboot( same_version ): - """ - Wait for the camera to finish rebooting after a FW update. - The test exit flow may cut USB power (via hub port disable), so we must ensure - the device has had enough time to complete its reboot before we exit. - When updating to a different version, FW may need time to flash a new ISP FW. - """ - sleep_time = 60 if not same_version else 3 - log.d( "Waiting", sleep_time, "seconds for device to finish rebooting after FW update..." ) - time.sleep( sleep_time ) - - -def send_hardware_monitor_command(device, command): - # byte_index = -1 - raw_result = rs.debug_protocol(device).send_and_receive_raw_data(command) - - return raw_result[4:] - -import os -import re - -def extract_version_from_filename(file_path): - """ - Extracts the version string from a filename like: - FlashGeneratedImage_Image5_16_7_0.bin -> 5.16.7 - FlashGeneratedImage_RELEASE_DS5_5_16_3_1.bin -> 5.16.3.1 - rvp-flash-dfu-release-7.56.37749.4831.img -> 7.56.37749.4831 - 20260727_7.58.40846.12889.img (NIGHTLY build) -> 7.58.40846.12889 - - Args: - file_path (str): Full path to the file. - - Returns: - str: Extracted version in format x.y.z or x.y.z.w, or None if not found or if path is invalid. - """ - if not file_path or not os.path.exists(file_path): - log.i(f"File not found: {file_path}") - return None - - filename = os.path.basename(file_path) - - # Match *last* 4 numeric groups before .img/.bin - # following matching patterns for cases: - # FlashGeneratedImage_Image5_16_7_0.bin -> 5.16.7 - # FlashGeneratedImage_RELEASE_DS5_5_16_3_1.bin -> 5.16.3.1 - match = re.search(r'(\d+)_(\d+)_(\d+)_(\d+)\.(bin|img)$', filename) - if not match: - # Match a dot-separated x.y.z.w version immediately before the extension, - # regardless of what precedes it (hyphen, underscore, a date prefix, etc.), e.g.: - # rvp-flash-dfu-release-7.56.37749.4831.img -> 7.56.37749.4831 - # 20260727_7.58.40846.12889.img (NIGHTLY build) -> 7.58.40846.12889 - match = re.search(r'(? path is required -# for every product line; otherwise we cannot exercise the update flow. -same_version = False -custom_fw_path = None -custom_fw_version = None -if product_line == "D400" and args.custom_fw_d400: - custom_fw_path = args.custom_fw_d400 -elif "D555" in product_name and args.custom_fw_d555: - custom_fw_path = args.custom_fw_d555 -# "D585" also matches "D585S" (safety SKU) as a substring, so explicitly exclude it here: -# this custom FW is for the non-safety D585 (name may read "D585 Prototype" etc.) only. -elif "D585" in product_name and "D585S" not in product_name and args.custom_fw_d585: - custom_fw_path = args.custom_fw_d585 - -if not custom_fw_path: - log.w("No custom FW path provided (use --custom-fw-d400 / --custom-fw-d555 / --custom-fw-d585); skipping FW update test") - exit(0) - - -test.start( "Update FW" ) -# check if recovery on the configured domain (e.g. a D400 USB recovery device). If so recover. -# (recovered may already be True from the domain-0 DDS recovery handled above.) -if device.is_in_recovery_mode(): - log.d( "recovering device ..." ) - try: - # rs-fw-update -r needs a known-good image, which isn't always the caller's - # --custom-fw- path (e.g. D400 -r expects a *signed* FW, while the custom - # image is typically unsigned). Fetch the per-product-line gold FW to recover with. - gold_fw = fw_compat.download_gold_fw( product_line, product_name ) - if not gold_fw: - log.f( f"Could not download gold recovery FW for {product_name}; cannot recover DFU device" ) - sys.exit( 1 ) # defensive: log.f normally exits; never build a -f command with gold_fw=None - cmd = [fw_updater_exe, '-r', '-f', gold_fw, '-s', args.serial] - del device, ctx - log.d( 'running:', cmd ) - subprocess.run( cmd ) - recovered = True - fw_compat.reload_d4xx_driver_on_jetson( test.context ) - except Exception as e: - test.unexpected_exception() - log.f( "Unexpected error while trying to recover device:", e ) - else: - # The device's identity changed: in DFU it exposed firmware_update_id only, - # now in normal mode it exposes its real serial_number (optic_serial). The - # firmware_update_id (asic_serial) is still exposed and matches what the - # harness was tracking. Poll for the device to re-enumerate in normal mode - # (a fresh rs.context() needs time after rs-fw-update exits) -- up to 60s. - log.d( "waiting for recovered device to re-enumerate in normal mode..." ) - recovered_device = None - timer = Timer( 60 ) - timer.start() - while not timer.has_expired(): - for d in rs.context().devices: - if d.supports( rs.camera_info.firmware_update_id ) \ - and d.get_info( rs.camera_info.firmware_update_id ) == args.serial \ - and not d.is_in_recovery_mode(): - recovered_device = d - break - if recovered_device is not None: - break - time.sleep( 2 ) - if recovered_device is None: - log.f( f"Recovered device with firmware_update_id '{args.serial}' did not " - f"re-enumerate within {timer.get_timeout()}s after gold FW flash" ) - # Re-pin args.serial to the device's normal-mode SN so downstream - # rs-fw-update -s finds the device (rs-fw-update.cpp:480 uses SN when supported). - if recovered_device.supports( rs.camera_info.serial_number ): - new_sn = recovered_device.get_info( rs.camera_info.serial_number ) - if new_sn != args.serial: - log.d( f're-pinning args.serial: {args.serial} (FWID) -> {new_sn} (SN)' ) - args.serial = new_sn - device, ctx = test.find_first_device_or_exit( args.serial ) - current_fw_version = rsutils.version(device.get_info(rs.camera_info.firmware_version)) - log.d("FW version after recovery:", current_fw_version) - - -custom_fw_version = extract_version_from_filename(custom_fw_path) -log.d('Using custom FW version: ', custom_fw_version) - -if current_fw_version == custom_fw_version: - same_version = True - if recovered or 'nightly' not in test.context: - log.d('versions are same; skipping FW update') - test.finish() - test.print_results_and_exit() - -downgrade_counter = get_downgrade_counter( device ) -log.d( 'downgrade counter:', downgrade_counter ) -if downgrade_counter == 0xFFFF: - log.d( 'downgrade counter is uninitialized (0xFFFF), skipping reset' ) - downgrade_counter = 0 -elif downgrade_counter >= 19: - log.d( 'resetting downgrade counter (was', str(downgrade_counter) + ')' ) - reset_downgrade_counter( device ) - log.d( 'sleeping for 3 sec...' ) - time.sleep( 3 ) - downgrade_counter = get_downgrade_counter( device ) - log.d( 'downgrade counter after reset is:', str(downgrade_counter)) - test.check_equal( downgrade_counter, 0 ) - downgrade_counter = 0 - -image_file = custom_fw_path - -cmd = [fw_updater_exe, '-f', image_file] -if args.serial: - cmd += ['-s', args.serial] -# Add '-u' only if the path doesn't include 'signed' -if ('signed' not in custom_fw_path.lower() - and "d555" not in product_name.lower() # currently -u is not supported for D555 - and "d585" not in product_name.lower()): # nor for D585/D585S - cmd.insert(1, '-u') - -# for DDS devices we need to close device and context to detect it back after FW update -del device, ctx -log.d( 'running:', cmd ) -sys.stdout.flush() -result = subprocess.run( cmd ) # may throw - -# Wait for the camera to finish rebooting before doing anything else, REGARDLESS of -# rs-fw-update's exit code. A non-zero exit doesn't necessarily mean no flash started: -# rs-fw-update may have begun a section flash before erroring out, leaving the device -# mid-reboot. The test exit flow may cut USB power (hub port disable), so we must not -# exit while the device is still rebooting. -wait_for_reboot( same_version ) - -if result.returncode != 0: - log.e( 'rs-fw-update returned exit code', result.returncode ) - test.check( False, description='rs-fw-update should return exit code 0' ) - test.finish() - test.print_results_and_exit() - -# make sure update worked and check FW version and update counter -device, ctx = test.find_first_device_or_exit( args.serial ) -current_fw_version = rsutils.version( device.get_info( rs.camera_info.firmware_version )) - -# camera_locked returns "YES" (locked) or "NO" (unlocked) -if device.supports( rs.camera_info.camera_locked ) and device.get_info( rs.camera_info.camera_locked ) == 'YES': - log.w( 'Device is flash-locked' ) - -test.check_equal(current_fw_version, custom_fw_version) -new_downgrade_counter = get_downgrade_counter( device ) -log.d( 'downgrade counter after update:', new_downgrade_counter ) - -test.finish() -# -############################################################################### - -test.print_results_and_exit() diff --git a/unit-tests/types/test-pose.cpp b/unit-tests/types/test-pose.cpp index ffbadc2bf8..d14e39fd42 100644 --- a/unit-tests/types/test-pose.cpp +++ b/unit-tests/types/test-pose.cpp @@ -234,7 +234,9 @@ TEST_CASE( "inverse of inverse (rot mat)", "[types]" ) INFO( "\ninv(p)=\n" << std::setprecision( 15 ) << inv( p ) ); INFO( "\ninv(inv(p))=\n" << inv( inv( p ) ) ); INFO( "\ninverse(inverse(p))=\n" << inverse( inverse( p ) ) ); - CHECK( inverse( inverse( p ) ) == p ); + // inverse(inverse(p)) computes R * (R^T * t); R * R^T is not exactly the identity in float, + // so the round-trip is only approximate -- compare like the (extr) case below + CHECK( eq( inverse( inverse( p ) ), p ) ); } TEST_CASE( "inverse of inverse (extr)", "[types]" ) diff --git a/wrappers/python/CMakeLists.txt b/wrappers/python/CMakeLists.txt index f28eda2b3a..c2f0b88076 100644 --- a/wrappers/python/CMakeLists.txt +++ b/wrappers/python/CMakeLists.txt @@ -16,6 +16,7 @@ set(PYRS_CPP pyrs_pipeline.cpp pyrs_processing.cpp pyrs_record_playback.cpp + pyrs_rum.cpp pyrs_sensor.cpp pyrs_types.cpp pyrsutil.cpp diff --git a/wrappers/python/pyrealsense2.cpp b/wrappers/python/pyrealsense2.cpp index 4ba4f131ba..25ade3c1c3 100644 --- a/wrappers/python/pyrealsense2.cpp +++ b/wrappers/python/pyrealsense2.cpp @@ -37,6 +37,7 @@ PYBIND11_MODULE(NAME, m) { init_serializable_device(m); init_util(m); init_eth_config(m); + init_rum(m); /** rs_export.hpp **/ py::class_(m, "save_to_ply") diff --git a/wrappers/python/pyrealsense2.h b/wrappers/python/pyrealsense2.h index 43eb484d8d..69a2a2c546 100644 --- a/wrappers/python/pyrealsense2.h +++ b/wrappers/python/pyrealsense2.h @@ -244,3 +244,4 @@ void init_advanced_mode(py::module &m); void init_serializable_device(py::module& m); void init_util(py::module &m); void init_eth_config(py::module &m); +void init_rum(py::module &m); diff --git a/wrappers/python/pyrs_rum.cpp b/wrappers/python/pyrs_rum.cpp new file mode 100644 index 0000000000..33deb62d33 --- /dev/null +++ b/wrappers/python/pyrs_rum.cpp @@ -0,0 +1,16 @@ +/* License: Apache 2.0. See LICENSE file in root directory. +Copyright(c) 2026 RealSense, Inc. All Rights Reserved. */ + +#include "pyrealsense2.h" +#include + +void init_rum(py::module &m) +{ + auto rum = m.def_submodule( "rum", "Real User Monitoring (RUM) usage statistics" ); + rum.def( "get_report", &rs2::rum::get_report, + "The live RUM report for the current session as a JSON string." ); + rum.def( "set_cloud_enabled", &rs2::rum::set_cloud_enabled, "enabled"_a, + "Set the cloud-upload consent flag (persists to the per-user config file)." ); + rum.def( "is_cloud_enabled", &rs2::rum::is_cloud_enabled, + "Resolved cloud-upload consent (RS2_RUM_CLOUD_ENABLED env var overrides the config file)." ); +}