feat: add --camera-api to skip firmware detection

CHANGELOG.md

@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## [Unreleased]
 
+### Added
+
+- Add `--camera-api` and Docker `CAMERA_API` to select automatic, legacy, or modern recording APIs.
+
 ### Changed
 
 - Internal: adopt the Python blueprint for development tools, agent setup, and CI checks.

Dockerfile

@@ -19,6 +19,7 @@ COPY entrypoint.sh /entrypoint.sh
 COPY crontab /var/spool/cron/crontabs/dashcam
 
 ENV ADDRESS="" \
+    CAMERA_API=auto \
     PUID="" \
     PGID="" \
     KEEP="" \

README.md

@@ -120,6 +120,7 @@ Other options:
   - `none`: No grouping, the default.
 - `--priority`: Downloads recordings with different priorities: `date` downloads oldest to newest; `rdate` downloads newest to oldest; `type` downloads manual, event (all types), normal and (non-event) parking recordings in that order. Defaults to `date`.
 - `--max-used-disk`: Downloads stop once the specified used disk percentage threshold is reached. Defaults to `90` (i.e. 90%.)
+- `--camera-api`: Selects `auto` (default), `legacy`, or `modern`. `auto` probes `/accessible` on every sync; `legacy` skips the probe and uses `/blackvue_vod.cgi`; `modern` skips the probe and uses `/vodList`. Select an explicit mode only when the camera API is known. The selected mode also preserves the matching video and metadata behavior. Invalid values are rejected.
 - `--timeout`: Sets a timeout for establishing a connection to the dashcam, in seconds. Defaults to `10.0` seconds.
 - `--retry-failed-after`: Sets the minimum elapsed time before retrying a failed download. Accepted units are `s` for seconds, `h` for hours, `d` for days and `w` for weeks. If no unit is indicated, days are assumed. Defaults to `1d`.
 - `--skip-metadata`: Skips downloading metadata file types. Takes a string of characters: `t` for thumbnail (`.thm`), `3` for accelerometer (`.3gf`)[^1], `g` for GPS (`.gps`). For example, `--skip-metadata t3g` skips all metadata files, downloading only the `.mp4` video recordings.
@@ -288,6 +289,7 @@ Other parameters:
 - `KEEP`: Sets the retention period of downloaded recordings. Recordings prior to the retention period will be removed from the destination. Accepted units are `d` for days and `w` for weeks. If no unit is indicated, days are assumed. (Default: empty, meaning recordings are kept forever.)
 - `PRIORITY`: Sets the priority to download recordings. Pick `date` to download from oldest to newest; pick `rdate` to download from newset to oldest; pick `type` to download manual, event (all types), normal and (non-event) parking recordings in that order. Defaults to `date`.
 - `MAX_USED_DISK`: If set to a percentage value, stops downloading if the amount of used disk space exceeds the indicated percentage value. (Default: `90`, i.e. 90%.)
+- `CAMERA_API`: Selects the recording API: `CAMERA_API=auto` (default) detects firmware using `/accessible`; `CAMERA_API=legacy` uses `/blackvue_vod.cgi`; `CAMERA_API=modern` uses `/vodList`. Both explicit modes skip `/accessible`. Use `auto` for unknown or changing firmware, `legacy` for cameras known to serve the legacy index, and `modern` for cameras known to serve the JSON index. Values are lowercase. An empty or unset value defaults to `auto`; invalid nonempty values are rejected at container startup.
 - `TIMEOUT`: If set to a float value, sets the timeout in seconds for connecting to the dashcam. (Default: `10.0` seconds.)
 - `RETRY_FAILED_AFTER`: If set, sets the minimum elapsed time before retrying a failed download. Accepted units are `s` for seconds, `h` for hours, `d` for days and `w` for weeks. If no unit is indicated, days are assumed. (Default: `1d`.)
 - `VERBOSE`: If set to a number greater than zero, increases logging verbosity. (Default: `0`.)
@@ -305,6 +307,20 @@ Other parameters:
 - `DRY_RUN`: If set to any value, makes it so that the script communicates what it would do without actually doing anything. (Default: empty.)
 - `RUN_ONCE`: If set to any value, the docker image runs the sync operation once and exits without setting up the cron job. (Default: empty. Not supported in Docker Compose.)
 
+### Explicit camera API selection
+
+For a camera verified to serve the legacy index:
+
+```sh
+blackvuesync dashcam.example.net --camera-api legacy --destination /path/to/recordings
+```
+
+In Docker or Portainer, set `CAMERA_API=legacy` (Compose: `CAMERA_API: legacy`). Leave it unset or use `CAMERA_API=auto` to retain automatic detection. Set `CAMERA_API=modern` only for a camera serving `/vodList`; recheck this choice after a firmware update. Forced modes do not fall back to another API on failure. If a forced recording index returns HTTP 404, the error identifies the selected mode and suggests `auto`.
+
+The selected mode is logged at DEBUG level at startup (`--verbose --verbose` or Docker `VERBOSE=2`). Default cron runs do not log the mode; normal verbosity and quiet settings still apply. Only the mode is included in this new log message.
+
+Skipping the probe removes one request per synchronization. It does not fix an unreachable host or another network failure; inventory and download requests still require camera connectivity.
+
 ## License
 
 This project is licensed under the MIT License - see the [COPYING](COPYING) file for details

blackvuesync.py

@@ -46,6 +46,9 @@
 import urllib.request
 from collections.abc import Callable
 from dataclasses import dataclass
+from typing import Literal
+
+CAMERA_API_MODES = ("auto", "legacy", "modern")
 
 # logging
 TEXT_LOG_FORMAT = "%(asctime)s: %(levelname)s %(message)s"
@@ -839,10 +842,16 @@ def get_dashcam_filenames_legacy(base_url: str) -> list[str]:
     return get_filenames(file_lines)
 
 
-def get_dashcam_filenames(base_url: str) -> tuple[bool, list[str]]:
+def get_dashcam_filenames(
+    base_url: str, camera_api: Literal["auto", "legacy", "modern"] = "auto"
+) -> tuple[bool, list[str]]:
     """gets whether the camera is legacy and its recording filenames"""
     try:
-        legacy = is_legacy_camera(base_url)
+        legacy = (
+            is_legacy_camera(base_url)
+            if camera_api == "auto"
+            else camera_api == "legacy"
+        )
         if legacy:
             return legacy, get_dashcam_filenames_legacy(base_url)
 
@@ -863,6 +872,15 @@ def get_dashcam_filenames(base_url: str) -> tuple[bool, list[str]]:
                 ) from e
 
         return legacy, filenames
+    except urllib.error.HTTPError as e:
+        if camera_api != "auto" and e.code == 404:
+            raise RuntimeError(
+                f"Recording API '{camera_api}' returned HTTP 404 at {base_url}; "
+                "try --camera-api auto (CAMERA_API=auto) to detect the camera API."
+            ) from e
+        raise RuntimeError(
+            f"Cannot obtain list of recordings from dashcam at address : {base_url}; error : {e}"
+        ) from e
     except urllib.error.URLError as e:
         if isinstance(e.reason, OSError) and (
             isinstance(e.reason, (TimeoutError, socket.timeout))
@@ -1741,12 +1759,13 @@ def sync(  # pylint: disable=too-many-arguments,too-many-positional-arguments
     include: tuple[str, ...] | None,
     exclude: tuple[str, ...] | None,
     metrics: SyncMetrics | None = None,
+    camera_api: Literal["auto", "legacy", "modern"] = "auto",
 ) -> None:
     """synchronizes the recordings at the dashcam address with the destination directory"""
     prepare_destination(destination, grouping)
 
     base_url = f"http://{address}"
-    legacy, dashcam_filenames = get_dashcam_filenames(base_url)
+    legacy, dashcam_filenames = get_dashcam_filenames(base_url, camera_api)
     dashcam_recordings = to_dashcam_recordings(dashcam_filenames, grouping)
     if metrics:
         metrics.dashcam_recordings_seen = len(dashcam_recordings)
@@ -1858,6 +1877,12 @@ def parse_args() -> argparse.Namespace:
     arg_parser.add_argument(
         "address", metavar="ADDRESS", help="dashcam IP address or name"
     )
+    arg_parser.add_argument(
+        "--camera-api",
+        choices=CAMERA_API_MODES,
+        default="auto",
+        help="selects the recording API; auto detects firmware (default), legacy uses blackvue_vod.cgi, modern uses vodList",
+    )
     arg_parser.add_argument(
         "-d",
         "--destination",
@@ -2020,6 +2045,12 @@ def main() -> int:
     configure_logging(args.log_format)
     set_logging_levels(-1 if args.quiet else args.verbose, args.cron)
 
+    logger.debug(
+        "Camera API mode : %s",
+        args.camera_api,
+        extra={"event": "camera_api_configured", "camera_api": args.camera_api},
+    )
+
     dry_run = args.dry_run
     affinity_key = args.affinity_key
     skip_metadata = args.skip_metadata
@@ -2098,6 +2129,7 @@ def main() -> int:
                 args.include,
                 args.exclude,
                 metrics,
+                camera_api=args.camera_api,
             )
             sync_success = True
         finally:

blackvuesync.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 set -euo pipefail
 
-set -- "${ADDRESS}" --destination /recordings
+set -- "${ADDRESS}" --destination /recordings --camera-api "${CAMERA_API:-auto}"
 
 # keep option if KEEP set
 [ -n "${KEEP:-}" ] && set -- "$@" --keep "$KEEP"

docker-compose.yml

@@ -11,6 +11,9 @@ services:
       # Dashcam address
       # ADDRESS: dashcam.example.net
 
+      # selects auto detection, legacy blackvue_vod.cgi, or modern vodList.
+      CAMERA_API: auto
+
       # Set these to the desired destination directory's user id and group id.
       PUID: 1000
       PGID: 1000

entrypoint.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
 
+case "${CAMERA_API:-auto}" in
+    auto|legacy|modern) ;;
+    *) echo "Invalid CAMERA_API: expected auto, legacy or modern" >&2; exit 2 ;;
+esac
+
 /setuid.sh && su -m dashcam /blackvuesync.sh
 status=$?
 

features/environment.py

@@ -201,3 +201,4 @@ def before_scenario(context: Context, scenario: Scenario) -> None:
     context.grouping = "none"
     # numbers the run logs of the scenario
     context.run_count = 0
+    context.docker_environment = {}

features/lib/dashcam.py

@@ -16,7 +16,9 @@ def __init__(self, url: str, affinity_key: str) -> None:
         self._url = url
 
     # JSON responses are untyped, hence Any
-    def _request(self, method: str, path: str, json: object = None) -> dict[str, Any]:
+    def _request(
+        self, method: str, path: str, json: dict[str, Any] | None = None
+    ) -> dict[str, Any]:
         """sends a control request and returns the decoded JSON response."""
         response = requests.request(
             method,
@@ -41,15 +43,22 @@ def fail_downloads(self, filenames: Collection[str]) -> None:
         """makes the dashcam answer downloads of the filenames with a 500 error."""
         self._request("PUT", "/mock/download-errors", {"filenames": list(filenames)})
 
-    def fail_listing(self) -> None:
-        """makes the dashcam answer requests for its recording list with a 500 error."""
-        self._request("PUT", "/mock/listing-error", {"failing": True})
+    def fail_listing(self, status_code: int = 500) -> None:
+        """makes the dashcam answer recording list requests with the given status."""
+        self._request(
+            "PUT", "/mock/listing-error", {"failing": True, "status_code": status_code}
+        )
 
     def requested_files(self) -> list[str]:
         """returns the recording filenames requested since the log was cleared."""
         filenames: list[str] = self._request("GET", "/mock/requests")["filenames"]
         return filenames
 
+    def requested_paths(self) -> list[str]:
+        """returns all camera paths requested since the log was cleared."""
+        paths: list[str] = self._request("GET", "/mock/requests")["paths"]
+        return paths
+
     def clear_requests(self) -> None:
         """clears the session's request log."""
         self._request("DELETE", "/mock/requests")

features/lib/runner.py

@@ -26,6 +26,9 @@
 # long options with values and the environment variables blackvuesync.sh maps
 # them from in the docker image
 DOCKER_OPTION_ENV_VARS = {
+    "camera-api": "CAMERA_API",
+    "cron": "CRON",
+    "verbose": "VERBOSE",
     "exclude": "EXCLUDE",
     "grouping": "GROUPING",
     "include": "INCLUDE",
@@ -85,7 +88,12 @@ def run_direct(
         affinity_key,
     ]
     for name, value in options.items():
-        cmd += [f"--{name}", value]
+        if name == "cron":
+            cmd.append("--cron")
+        elif name == "verbose":
+            cmd += ["--verbose"] * int(value)
+        else:
+            cmd += [f"--{name}", value]
 
     logger.info("running (direct): %s", cmd)
     result = subprocess.run(
@@ -101,6 +109,7 @@ def run_docker(
     destination: Path,
     affinity_key: str,
     options: Mapping[str, str],
+    environment: Mapping[str, str | None] | None = None,
 ) -> RunResult:
     """runs the blackvuesync image once on the network.
 
@@ -133,6 +142,12 @@ def run_docker(
     for name, value in options.items():
         container.with_env(DOCKER_OPTION_ENV_VARS[name], value)
 
+    for env_name, env_value in (environment or {}).items():
+        if env_value is not None:
+            container.with_env(env_name, env_value)
+    if environment and environment.get("CAMERA_API", "") is None:
+        container.with_kwargs(entrypoint=["env", "-u", "CAMERA_API", "/entrypoint.sh"])
+
     logger.info("running (docker): %s with %s", image_tag, dict(options))
     with container:
         status = container.get_wrapped_container().wait(timeout=RUN_TIMEOUT)

features/mock_dashcam/server.py

@@ -35,11 +35,13 @@ class _Session:
 
     recordings: list[str] = field(default_factory=list)
     download_errors: set[str] = field(default_factory=set)
-    # answers the recording list requests with a 500 error
+    # selects whether recording list requests return an error
     listing_error: bool = False
+    listing_status_code: int = 500
     legacy_api: bool = False
     # recording filenames blackvuesync requested, in order
     requested_files: list[str] = field(default_factory=list)
+    requested_paths: list[str] = field(default_factory=list)
 
 
 def _metadata_filename(video_filename: str, metadata_type: str) -> str:
@@ -121,14 +123,17 @@ def _serve_recording_file(self, session: _Session, filename: str) -> flask.Respo
         return flask.send_file(MOCK_FILES_DIR / f"mock.{extension}")
 
     def _abort_on_listing_error(self, session: _Session) -> None:
-        """aborts 500 when the session is configured to fail its recording list."""
+        """aborts with the configured status when the recording list fails."""
         with self._lock:
             failing = session.listing_error
+            status_code = session.listing_status_code
         if failing:
             logger.info(
-                "answers %s with 500: configured listing error", flask.request.path
+                "answers %s with %s: configured listing error",
+                flask.request.path,
+                status_code,
             )
-            flask.abort(500)
+            flask.abort(status_code)
 
     def _record_request(self, session: _Session, filename: str) -> None:
         """appends a requested recording filename to the session's request log."""
@@ -138,6 +143,14 @@ def _record_request(self, session: _Session, filename: str) -> None:
     def _setup_routes(self) -> None:
         """registers the dashcam and /mock/ control routes."""
 
+        @self.app.before_request
+        def record_camera_request() -> None:
+            """records camera paths separately from test control requests."""
+            if not flask.request.path.startswith("/mock/"):
+                session = self._session()
+                with self._lock:
+                    session.requested_paths.append(flask.request.path)
+
         @self.app.route("/accessible", methods=["GET"])
         def accessible() -> tuple[dict[str, str], int]:
             """reports the V1.009+ API is reachable; absent on legacy firmware."""
@@ -263,22 +276,28 @@ def set_listing_error() -> tuple[dict[str, object], int]:
             """makes the session fail or serve its recording list."""
             session = self._session()
             with self._lock:
-                session.listing_error = bool(flask.request.get_json()["failing"])
+                payload = flask.request.get_json()
+                session.listing_error = bool(payload["failing"])
+                session.listing_status_code = int(payload.get("status_code", 500))
             return {"failing": session.listing_error}, 200
 
         @self.app.route("/mock/requests", methods=["GET"])
         def get_requests() -> tuple[dict[str, object], int]:
-            """returns the recording filenames the session received requests for."""
+            """returns the session's recording filenames and camera request paths."""
             session = self._session()
             with self._lock:
-                return {"filenames": list(session.requested_files)}, 200
+                return {
+                    "filenames": list(session.requested_files),
+                    "paths": list(session.requested_paths),
+                }, 200
 
         @self.app.route("/mock/requests", methods=["DELETE"])
         def clear_requests() -> tuple[dict[str, object], int]:
             """clears the session's request log."""
             session = self._session()
             with self._lock:
                 session.requested_files.clear()
+                session.requested_paths.clear()
             return {}, 200
 
         @self.app.route("/mock/session", methods=["DELETE"])

features/steps/blackvuesync_steps.py

@@ -5,7 +5,7 @@
 import datetime
 import re
 
-from behave import when
+from behave import given, when
 from behave.runner import Context
 
 from features.lib.runner import RunResult, run_direct, run_docker
@@ -47,6 +47,7 @@ def _run_blackvuesync(context: Context, options: dict[str, str]) -> None:
             context.dest_dir,
             context.dashcam.affinity_key,
             options,
+            context.docker_environment,
         )
     else:
         context.result = run_direct(
@@ -60,6 +61,20 @@ def _run_blackvuesync(context: Context, options: dict[str, str]) -> None:
     _write_run_log(context, context.result)
 
 
+@given('the docker environment variable "{name}" is empty')
+def docker_environment_empty(context: Context, name: str) -> None:
+    """sets an empty environment variable for the next Docker run."""
+    context.docker_environment[name] = ""
+
+
+@given('the docker environment variable "{name}" is unset')
+def docker_environment_unset(context: Context, name: str) -> None:
+    """unsets the camera API variable before the entrypoint runs."""
+    if name != "CAMERA_API":
+        raise ValueError("only CAMERA_API unsetting is supported")
+    context.docker_environment[name] = None
+
+
 @when("blackvuesync runs")
 def run_blackvuesync(context: Context) -> None:
     """runs blackvuesync with the default options."""