From 59acf536eb7352fae85d2c0bd513677118c0853d Mon Sep 17 00:00:00 2001 From: Gerardo Puga Date: Tue, 15 Sep 2026 16:43:22 -0300 Subject: [PATCH 1/2] Sunset beluga_benchmarks, thanks for the ride Signed-off-by: Gerardo Puga --- README.md | 1 - beluga_amcl/README.md | 12 +- beluga_benchmark/CHANGELOG.rst | 41 --- beluga_benchmark/CMakeLists.txt | 27 -- beluga_benchmark/README.md | 9 - beluga_benchmark/beluga_benchmark/__init__.py | 23 -- .../beluga_benchmark/compare_results.py | 290 ------------------ .../beluga_benchmark/exceptions.py | 17 - .../beluga_benchmark/timem_results.py | 100 ------ beluga_benchmark/docs/BENCHMARKING.md | 105 ------- beluga_benchmark/docs/PROFILING.md | 35 --- .../docs/reports/2023-03-16/REPORT.md | 58 ---- .../docs/reports/2023-03-16/params.yaml | 93 ------ .../docs/reports/2023-03-16/params_par.yaml | 93 ------ .../docs/reports/2023-03-16/results-1.png | Bin 107207 -> 0 bytes .../docs/reports/2023-03-16/results-2.png | Bin 96952 -> 0 bytes .../docs/reports/2023-05-20/REPORT.md | 123 -------- .../2023-05-20/beam_par_beluga_vs_amcl.png | Bin 180356 -> 0 bytes .../docs/reports/2023-05-20/beam_params.yaml | 93 ------ .../reports/2023-05-20/beam_params_par.yaml | 93 ------ .../2023-05-20/beam_seq_beluga_vs_amcl.png | Bin 183136 -> 0 bytes .../likelihood_par_beluga_vs_amcl.png | Bin 177190 -> 0 bytes .../reports/2023-05-20/likelihood_params.yaml | 93 ------ .../2023-05-20/likelihood_params_par.yaml | 93 ------ .../likelihood_seq_beluga_vs_amcl.png | Bin 178835 -> 0 bytes .../docs/reports/2023-06-03/REPORT.md | 144 --------- .../beam_beluga_seq_vs_amcl_log.png | Bin 154231 -> 0 bytes .../beam_beluga_vs_beluga_vs_amcl.png | Bin 248847 -> 0 bytes .../docs/reports/2023-06-03/beam_params.yaml | 93 ------ .../reports/2023-06-03/beam_params_par.yaml | 93 ------ .../likelihood_beluga_seq_vs_amcl_log.png | Bin 196793 -> 0 bytes .../likelihood_beluga_vs_beluga_vs_amcl.png | Bin 245614 -> 0 bytes .../reports/2023-06-03/likelihood_params.yaml | 93 ------ .../2023-06-03/likelihood_params_par.yaml | 93 ------ .../docs/reports/2023-09-02/REPORT.md | 122 -------- .../beam_beluga_vs_beluga_vs_amcl.png | Bin 187416 -> 0 bytes .../beam_beluga_vs_beluga_vs_amcl_linear.png | Bin 140633 -> 0 bytes .../likelihood_beluga_vs_beluga_vs_amcl.png | Bin 189106 -> 0 bytes ...lihood_beluga_vs_beluga_vs_amcl_linear.png | Bin 148939 -> 0 bytes .../baseline_configurations/beam_params.yaml | 93 ------ .../beam_params_par.yaml | 93 ------ .../likelihood_params.yaml | 93 ------ .../likelihood_params_par.yaml | 93 ------ beluga_benchmark/package.xml | 28 -- .../scripts/benchmarking/compare_results | 26 -- .../scripts/benchmarking/parameterized_run | 134 -------- .../scripts/benchmarking/timem_results | 26 -- beluga_benchmark/scripts/profiling/flamegraph | 21 -- .../profiling/profile_amcl_with_bagfile | 30 -- tools/build-and-test.sh | 2 +- 50 files changed, 3 insertions(+), 2673 deletions(-) delete mode 100644 beluga_benchmark/CHANGELOG.rst delete mode 100644 beluga_benchmark/CMakeLists.txt delete mode 100644 beluga_benchmark/README.md delete mode 100644 beluga_benchmark/beluga_benchmark/__init__.py delete mode 100644 beluga_benchmark/beluga_benchmark/compare_results.py delete mode 100644 beluga_benchmark/beluga_benchmark/exceptions.py delete mode 100644 beluga_benchmark/beluga_benchmark/timem_results.py delete mode 100644 beluga_benchmark/docs/BENCHMARKING.md delete mode 100644 beluga_benchmark/docs/PROFILING.md delete mode 100644 beluga_benchmark/docs/reports/2023-03-16/REPORT.md delete mode 100644 beluga_benchmark/docs/reports/2023-03-16/params.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-03-16/params_par.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-03-16/results-1.png delete mode 100644 beluga_benchmark/docs/reports/2023-03-16/results-2.png delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/REPORT.md delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/beam_par_beluga_vs_amcl.png delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/beam_params.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/beam_params_par.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/beam_seq_beluga_vs_amcl.png delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/likelihood_par_beluga_vs_amcl.png delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/likelihood_params.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/likelihood_params_par.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-05-20/likelihood_seq_beluga_vs_amcl.png delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/REPORT.md delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/beam_beluga_seq_vs_amcl_log.png delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/beam_beluga_vs_beluga_vs_amcl.png delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/beam_params.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/beam_params_par.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/likelihood_beluga_seq_vs_amcl_log.png delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/likelihood_beluga_vs_beluga_vs_amcl.png delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/likelihood_params.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-06-03/likelihood_params_par.yaml delete mode 100644 beluga_benchmark/docs/reports/2023-09-02/REPORT.md delete mode 100644 beluga_benchmark/docs/reports/2023-09-02/beam_beluga_vs_beluga_vs_amcl.png delete mode 100644 beluga_benchmark/docs/reports/2023-09-02/beam_beluga_vs_beluga_vs_amcl_linear.png delete mode 100644 beluga_benchmark/docs/reports/2023-09-02/likelihood_beluga_vs_beluga_vs_amcl.png delete mode 100644 beluga_benchmark/docs/reports/2023-09-02/likelihood_beluga_vs_beluga_vs_amcl_linear.png delete mode 100644 beluga_benchmark/docs/reports/baseline_configurations/beam_params.yaml delete mode 100644 beluga_benchmark/docs/reports/baseline_configurations/beam_params_par.yaml delete mode 100644 beluga_benchmark/docs/reports/baseline_configurations/likelihood_params.yaml delete mode 100644 beluga_benchmark/docs/reports/baseline_configurations/likelihood_params_par.yaml delete mode 100644 beluga_benchmark/package.xml delete mode 100755 beluga_benchmark/scripts/benchmarking/compare_results delete mode 100755 beluga_benchmark/scripts/benchmarking/parameterized_run delete mode 100755 beluga_benchmark/scripts/benchmarking/timem_results delete mode 100755 beluga_benchmark/scripts/profiling/flamegraph delete mode 100755 beluga_benchmark/scripts/profiling/profile_amcl_with_bagfile diff --git a/README.md b/README.md index befbd98685..d92e14ff5d 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,6 @@ This repository contains the following packages: | [`beluga_ros`](beluga) | A ROS library, providing utilities to interface ROS with Beluga. | | [`beluga_amcl`](beluga_amcl) | A ROS wrapper, providing an executable node and component (or nodelet).
It provides interface parity with `nav2_amcl` (and `amcl`). | | [`beluga_example`](beluga_example) | Example launch files, showing how to run Beluga-based nodes. | -| [`beluga_benchmark`](beluga_benchmark) | Scripts to benchmark, profile and also compare Beluga with other MCL implementations. | | [`beluga_system_tests`](beluga_system_tests) | System integration tests for Beluga. | | [`beluga_vdb`](beluga_vdb) | A library extension for `beluga` facilitating the use of OpenVDB for 3D localization. | diff --git a/beluga_amcl/README.md b/beluga_amcl/README.md index b667b13744..83c056472f 100644 --- a/beluga_amcl/README.md +++ b/beluga_amcl/README.md @@ -14,7 +14,7 @@ The compatibility between `beluga_amcl` and its longstanding counterparts in the - [Published Transforms](#published-transforms) - [Advertised Services](#advertised-services) - [Performance](#performance) - - [Compatibility](#compatibility-notes) + - [Compatibility notes](#compatibility-notes) - [Next Steps](#next-steps) ## ROS Interface @@ -67,14 +67,7 @@ Defaults are `map`, `odom` and `base`. ## Performance -Performance reports are periodically generated to track improvements and regressions. These reports are produced using the scripts in the [beluga_benchmark](../beluga_benchmark) package, which allow comparing the performance of `beluga_amcl` against `nav2_amcl` using a synthetic dataset. - -
- Beluga Vs Nav2 -
Typical trajectory plot: the ground-truth and both AMCL trajectories are so close that they effectively overlap.
-
- -Further details on the benchmarking methodology and datasets can be found in the blog post [Big shoes to fill: Validating the performance of Beluga AMCL](https://ekumenlabs.com/blog/posts/big-shoes-to-fill-beluga-performance-report/) +Performance reports are periodically generated to track improvements and regressions. Further details on the benchmarking methodology and datasets can be found in the blog post [Big shoes to fill: Validating the performance of Beluga AMCL](https://ekumenlabs.com/blog/posts/big-shoes-to-fill-beluga-performance-report/) ## Compatibility notes @@ -85,7 +78,6 @@ Notes on parameter and feature availability between Beluga AMCL and Nav2 AMCL ca ## Next Steps - See [example launch files](../beluga_example) showing how to run Beluga-based nodes. -- See [available benchmarks](../beluga_benchmark) for scripts and comparison with other AMCL implementations. [nav2_amcl]: https://github.com/ros-planning/navigation2/tree/main/nav2_amcl [nav2_configuration_guide]: https://navigation.ros.org/configuration/packages/configuring-amcl.html diff --git a/beluga_benchmark/CHANGELOG.rst b/beluga_benchmark/CHANGELOG.rst deleted file mode 100644 index 1c477b6d08..0000000000 --- a/beluga_benchmark/CHANGELOG.rst +++ /dev/null @@ -1,41 +0,0 @@ -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Changelog for package beluga_benchmark -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -2.1.1 (2026-04-20) ------------------- - -2.1.0 (2025-11-25) ------------------- -* Drop Noetic support (`#520 `_) -* Contributors: Michel Hidalgo - -2.0.2 (2024-06-18) ------------------- - -2.0.1 (2024-05-24) ------------------- - -2.0.0 (2024-05-21) ------------------- -* Drop ament_lint_auto dependencies (`#370 `_) -* Update missed ROS version detection logic (`#368 `_) -* Add option to compare_results.py to save data to csv instead of plotting (`#261 `_) -* Add updated performance report (`#255 `_) -* Add cmake-format to pre-commit hooks (`#243 `_) -* Wrap up ROS Noetic support (`#225 `_) -* Fix timem output in parameterized runs (`#221 `_) -* Refactor launch files (`#211 `_) -* Add new report after recent performance updates (`#208 `_) -* Fix small issues with plotter and update MessageFilter tolerance before generating new report -* Optimize likelihood field model runtime performance (`#199 `_) -* Remove shared_mutexes to address concurrency performance (`#195 `_) -* Add performance report for latest version with minor fixes to scripts (`#193 `_) -* Improve documentation and guidelines (`#186 `_) -* Remove Git LFS from the repository (`#162 `_) -* Fix benchmarking instructions (`#159 `_) -* Performance comparison with nav2_amcl (`#142 `_) -* Update list of maintainers (`#130 `_) -* Add script to run benchmarks (`#117 `_) - -* Contributors: Gerardo Puga, Ivan Santiago Paunovic, Michel Hidalgo, Nahuel Espinosa diff --git a/beluga_benchmark/CMakeLists.txt b/beluga_benchmark/CMakeLists.txt deleted file mode 100644 index 6cc1e8f48e..0000000000 --- a/beluga_benchmark/CMakeLists.txt +++ /dev/null @@ -1,27 +0,0 @@ -# Copyright 2023 Ekumen, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -cmake_minimum_required(VERSION 3.16) -project(beluga_benchmark) - -find_package(ament_cmake REQUIRED) -find_package(ament_cmake_python REQUIRED) -find_package(beluga_amcl REQUIRED) - -ament_python_install_package(${PROJECT_NAME}) - -install(DIRECTORY scripts/profiling scripts/benchmarking USE_SOURCE_PERMISSIONS - DESTINATION lib/${PROJECT_NAME}) - -ament_package() diff --git a/beluga_benchmark/README.md b/beluga_benchmark/README.md deleted file mode 100644 index b942fbccd6..0000000000 --- a/beluga_benchmark/README.md +++ /dev/null @@ -1,9 +0,0 @@ -# Beluga Benchmark - -This package contains scripts to benchmark, profile and also compare Beluga AMCL with other implementations. - -## Getting Started - -- View [reports](docs/reports/) comparing Beluga's performance to other AMCL implementations. -- [How to benchmark Beluga?](docs/BENCHMARKING.md) -- [How to profile Beluga?](docs/PROFILING.md) diff --git a/beluga_benchmark/beluga_benchmark/__init__.py b/beluga_benchmark/beluga_benchmark/__init__.py deleted file mode 100644 index 4be18feaa1..0000000000 --- a/beluga_benchmark/beluga_benchmark/__init__.py +++ /dev/null @@ -1,23 +0,0 @@ -# Copyright 2023 Ekumen, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from .exceptions import ScriptError -from . import compare_results -from . import timem_results - -__all__ = [ - 'ScriptError', - 'compare_results', - 'timem_results', -] diff --git a/beluga_benchmark/beluga_benchmark/compare_results.py b/beluga_benchmark/beluga_benchmark/compare_results.py deleted file mode 100644 index d522276bdf..0000000000 --- a/beluga_benchmark/beluga_benchmark/compare_results.py +++ /dev/null @@ -1,290 +0,0 @@ -# Copyright 2023 Ekumen, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import argparse -from pathlib import Path -import re - -from evo import main_ape -from evo.tools import file_interface -import matplotlib.pyplot as plt -from rosbag2_py._info import Info as Bag2Info -import pandas as pd -from itertools import cycle - -from . import timem_results -from .exceptions import ScriptError - -DIR_NAME_PATTERN = 'benchmark_([0-9]+)_particles_output' -SUPPORTED_TOPIC_NAMES = ('/pose', '/amcl_pose') -EVO_RESULTS_FILE_NAME = 'evo_results.zip' -TERMINAL_OUTPUT_LOG_FILE_NAME = 'output.log' - - -def get_bag_est_topic(bag_path: Path): - metadata = Bag2Info().read_metadata(str(bag_path), '') - for topic_name in SUPPORTED_TOPIC_NAMES: - if any( - topic_info.topic_metadata.name == topic_name - for topic_info in metadata.topics_with_message_count - ): - return topic_name - raise ScriptError( - f'Estimate pose topic was not found in bag file, expected names: {SUPPORTED_TOPIC_NAMES}' - ) - - -def run_evo_ape(dir_path: Path): - evo_results_path = dir_path / EVO_RESULTS_FILE_NAME - if evo_results_path.exists(): - return - bag_path = dir_path / 'rosbag' - est_topic = get_bag_est_topic(bag_path) - arg_parser = main_ape.parser() - args = arg_parser.parse_args( - [ - 'bag2', - str(bag_path), - '/odometry/ground_truth', - est_topic, - '--save_results', - str(evo_results_path.resolve()), - ] - ) - main_ape.run(args) - - -def read_evo_stats(zip_file_path): - # keys: "rmse", "mean", "median", "std", "min", "max", "sse" - return file_interface.load_res_file(zip_file_path).stats - - -def parse_latency_data(output_file_path): - import re - - # Example line: - pattern = re.compile( - r'\[.*\] \[INFO\] \[([0-9]*\.[0-9]*)\] \[amcl\]: Particle filter update iteration stats: ' - + r'[0-9]* particles [0-9]* points - ([0-9]*\.[0-9]*)ms' - ) - latencies_seq = [] - with open(output_file_path, 'r') as f: - for line in f: - if not pattern.match(line): - continue - # first match is the whole line, second match the timestamp, and third the latency - latency = pattern.match(line).group(2) - latencies_seq.append(float(latency) * 1e-3) # convert to seconds - return ( - { - 'latency_min': min(latencies_seq), - 'latency_max': max(latencies_seq), - 'latency_mean': sum(latencies_seq) / len(latencies_seq), - 'latency_median': sorted(latencies_seq)[len(latencies_seq) // 2], - } - if latencies_seq - else { - 'latency_min': 0.0, - 'latency_max': 0.0, - 'latency_mean': 0.0, - 'latency_median': 0.0, - } - ) - - -def create_parameterized_series(results_path: Path): - particles = [] - peak_rss = [] - cpu_usage = [] - ape_rmse = [] - ape_mean = [] - ape_std = [] - ape_median = [] - ape_max = [] - latency_min = [] - latency_max = [] - latency_mean = [] - latency_median = [] - - for dir in results_path.iterdir(): - if not dir.is_dir(): - continue - m = re.match(DIR_NAME_PATTERN, dir.name) - if not m: - continue - particles.append(int(m[1])) - timem_results_output = timem_results.read_timem_output(dir) - rss, cpu = ( - timem_results.get_timem_metrics_values(timem_results_output) - if timem_results_output - else (None, None) - ) - peak_rss.append(rss) - cpu_usage.append(cpu) - run_evo_ape(dir) - evo_results_path = dir / EVO_RESULTS_FILE_NAME - stats = read_evo_stats(evo_results_path) - ape_rmse.append(stats["rmse"]) - ape_max.append(stats["max"]) - ape_mean.append(stats["mean"]) - ape_std.append(stats["std"]) - ape_median.append(stats["median"]) - terminal_output_path = dir / TERMINAL_OUTPUT_LOG_FILE_NAME - latency_stats = parse_latency_data(terminal_output_path) - latency_min.append(latency_stats['latency_min']) - latency_max.append(latency_stats['latency_max']) - latency_mean.append(latency_stats['latency_mean']) - latency_median.append(latency_stats['latency_median']) - - return ( - pd.DataFrame( - { - 'particles_n': particles, - 'peak_rss': peak_rss, - 'cpu_usage': cpu_usage, - 'ape_rmse': ape_rmse, - 'ape_mean': ape_mean, - 'ape_std': ape_std, - 'ape_median': ape_median, - 'ape_max': ape_max, - 'latency_min': latency_min, - 'latency_max': latency_max, - 'latency_mean': latency_mean, - 'latency_median': latency_median, - } - ) - .set_index('particles_n') - .sort_index() - ) - - -def main(): - arg_parser = argparse.ArgumentParser( - description='Script to compare parameterized run results of beluga with another' - ' implementation (e.g. nav2_amcl, another beluga version, etc).' - ) - arg_parser.add_argument( - '--series', - '-s', - type=Path, - action='append', - help='Folder with parameterized benchmark results', - required=True, - ) - arg_parser.add_argument( - '--label', - '-l', - type=str, - action='append', - help='Label for a series', - required=True, - ) - - arg_parser.add_argument( - '--use-ylog', - action='store_true', - help='Use log scale on y axis', - ) - - default_plots = ['cpu_usage', 'peak_rss', 'ape_rmse', 'latency_median'] - arg_parser.add_argument( - '--plot-names', - type=str, - nargs='*', # 0 or more values expected => creates a list - default=default_plots, - help='List of plots to generate. Default: ' + ', '.join(default_plots), - ) - - arg_parser.add_argument( - '--save-csv', - type=str, - action='store', - help='Instead of plotting, save all the results to a csv file', - ) - - args = arg_parser.parse_args() - - assert len(args.series) == len(args.label), 'Number of series and labels must match' - - color_gen = cycle( - [ - 'red', - 'green', - 'blue', - 'brown', - 'purple', - 'gray', - 'orange', - ] - ) - - if args.save_csv: - # create list of dataframes with no filtered columns - series = [ - create_parameterized_series(series).add_prefix(label + '_') - for label, series in zip(args.label, args.series) - ] - # leave only the index - full_table = series[0].filter(items=[]) - # concatenate data in a single table - for s in series: - full_table = pd.concat([full_table, s], axis=1) - print('Saving to CSV file: ', args.save_csv) - full_table.to_csv(args.save_csv) - return 0 - - series = [ - create_parameterized_series(series) - .filter(items=args.plot_names) - .add_prefix(label + '_') - for label, series in zip(args.label, args.series) - ] - - print('Plotting data...') - - marker_gen = cycle('o^sDvP*') - - ax = series[0].plot( - subplots=True, - color=next(color_gen), - marker=next(marker_gen), - linestyle='dashed', - ) - - for other_series in series[1:]: - other_series.plot( - ax=ax, - subplots=True, - color=next(color_gen), - marker=next(marker_gen), - linestyle='dashed', - ) - - for ax in plt.gcf().axes: - ax.set_xscale('log') - if args.use_ylog: - ax.set_yscale('log') - ax.grid(True, which="both", ls="-") - ax.legend(fontsize='small', loc='upper left', bbox_to_anchor=(1.01, 1)) - current_bounds = ax.get_position().bounds - new_bounds = (0.05, *current_bounds[1:]) - ax.set_position(new_bounds) - current_ylimits = ax.get_ylim() - new_limits = ( - min(0.0, current_ylimits[0]), # include zero below - max(0.0, current_ylimits[1]), # include zero above - ) - ax.set_ylim(new_limits) - - plt.show() diff --git a/beluga_benchmark/beluga_benchmark/exceptions.py b/beluga_benchmark/beluga_benchmark/exceptions.py deleted file mode 100644 index 9c653d65fa..0000000000 --- a/beluga_benchmark/beluga_benchmark/exceptions.py +++ /dev/null @@ -1,17 +0,0 @@ -# Copyright 2023 Ekumen, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -class ScriptError(Exception): - pass diff --git a/beluga_benchmark/beluga_benchmark/timem_results.py b/beluga_benchmark/beluga_benchmark/timem_results.py deleted file mode 100644 index a2f752ea68..0000000000 --- a/beluga_benchmark/beluga_benchmark/timem_results.py +++ /dev/null @@ -1,100 +0,0 @@ -# Copyright 2023 Ekumen, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import argparse -import json -from pathlib import Path - -import matplotlib.pyplot as plt -import numpy as np -import pandas as pd - -from .exceptions import ScriptError - -DEFAULT_FILE_NAME = "timem-output.json" - - -def read_timem_output(dir_path: Path): - file = dir_path / DEFAULT_FILE_NAME - try: - with open(file, mode='r') as f: - output = json.load(f) - except (FileExistsError, FileNotFoundError) as ex: - raise ScriptError(f"Failed to open file '{file.absolute()}': {ex}") - except json.decoder.JSONDecodeError: - return None - return output['timemory']['timem'][0] - - -def get_value_with_unit(data): - return f"{data['value']:.2f}{data['unit_repr']}" - - -def get_timem_metrics(output): - return output['peak_rss'], output['cpu_util'] - - -def get_timem_metrics_values(output): - return tuple(data['value'] for data in get_timem_metrics(output)) - - -def get_timem_metrics_with_units(output): - return tuple(get_value_with_unit(data) for data in get_timem_metrics(output)) - - -def create_timeseries(output): - data = output['wall_clock'] - elapsed_time = data['value'] - time = np.arange(0.0, elapsed_time, 0.2) - wall_clock_to_time_offset = output['history'][0]['wall_clock']['value'] - wall_clock_to_time_scale = elapsed_time / ( - output['history'][-1]['wall_clock']['value'] - wall_clock_to_time_offset - ) - time = [] - rss = [] - virtual_memory = [] - for sample in output['history']: - time.append( - (sample['wall_clock']['value'] - wall_clock_to_time_offset) - * wall_clock_to_time_scale - ) - rss.append(sample['page_rss']['value']) - virtual_memory.append(sample['virtual_memory']['value']) - - series = pd.DataFrame( - {'time': time, 'rss': rss, 'virtual_memory': virtual_memory} - ).set_index('time') - return series - - -def main(): - arg_parser = argparse.ArgumentParser( - description='Script to postprocess timem results.' - ) - arg_parser.add_argument( - 'dir', help='Directory with timem-output.json file', type=Path - ) - args = arg_parser.parse_args() - dir_path = Path(args.dir) - name = dir_path.name - output = read_timem_output(dir_path) - peak_rss_str, cpu_usage_str = get_timem_metrics_with_units(output) - series = create_timeseries(output) - elapsed_time_str = get_value_with_unit(output['wall_clock']) - print(f"timem metrics for run '{name}':") - print(f"\telapsed_time: {elapsed_time_str}") - print(f"\tcpu_usage: {cpu_usage_str}") - print(f"\tpeak_rss: {peak_rss_str}") - series.plot(subplots=True) - plt.show() diff --git a/beluga_benchmark/docs/BENCHMARKING.md b/beluga_benchmark/docs/BENCHMARKING.md deleted file mode 100644 index 75a8f05bfe..0000000000 --- a/beluga_benchmark/docs/BENCHMARKING.md +++ /dev/null @@ -1,105 +0,0 @@ -# How to benchmark Beluga - -## Run rosbag benchmark parameterizing the number of particles - -This script will run the benchmark with the specified number of particles and record: - -- The `timem` output: CPU usage, RSS memory, virtual memory. -- A rosbag with the reference and estimated trajectories. -- Perf events (optionally). - -To run, use: - -```bash -ros2 run beluga_benchmark parameterized_run -``` - -The results of the different runs will be stored in folders named `benchmark_${N_PARTICLES}_particles_output`, where `N_PARTICLES` are the numbers specified in the above command. - -To run the same experiment using another AMCL node, e.g. nav2, use: - -```bash -ros2 run beluga_benchmark parameterized_run --package nav2_amcl --executable amcl -``` - -For other options, e.g. using a different rosbag, see: - -```bash -ros2 run beluga_benchmark parameterized_run --help -``` - -## Visualizing timem results of one run - -Use the following command: - -``` -ros2 run beluga_benchmark timem_results -``` - -Where the specified directory needs to have a `timem-output.json` file, e.g. the output directory generated by `parameterized_run`. -The script will print cpu usage, peak rss and elapsed time. -It will also plot virtual and RSS memory over time. - -## Visualizing estimated trajectory APE of one run - -Use the following command: - -``` -evo_ape bag2 /odometry/ground_truth /pose -p -``` - -For `nav2_amcl`, replace `/pose` with `/amcl_pose`. -The bagfiles generated by `parameterized_run` can be found in the generated output directories for each run. -This will print APE metrics (mean, median, max, std, rmse, etc), and also plot APE over time. - -## Comparing parameterized runs - -The following command allows to compare the results of different benchmarking runs in a single plot for each metric being measured. - -This can be used to compare different `beluga_amcl` and/or `nav2_amcl` runs, or to compare the same node with different base configuration settings. - -The command is -``` -ros2 run beluga_benchmark compare_results -s -l -s -l ... -``` -where `PATH1` and `PATH2` are the paths to the output directories of the benchmarking runs to compare, and `LABEL1` and `LABEL2` are the labels to use in the plot for each of them. Any number of runs can be added in the same plot by providing additional `-s -l