Reorganize into top-level packages, drop dynamic sys.path inserts

.github/workflows/ci.yml

@@ -31,14 +31,15 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           python -m pip install -r simulation/requirements.txt
+          python -m pip install -e . --no-deps
 
       - name: Run unit tests
         run: |
-          python -m pytest simulation/tests/unit -m "not simtest"
+          python -m pytest tests/unit -m "not simtest"
 
       - name: Run simtests
         run: |
-          python -m pytest simulation/tests/simtests -m simtest
+          python -m pytest tests/simtests -m simtest
 
   stack-matrix:
     name: Discover stack test files
@@ -54,7 +55,7 @@ jobs:
         id: discover
         shell: bash
         run: |
-          files_json=$(find simulation/tests/sitl -name 'test_*.py' | sort | jq -R -s -c 'split("\n") | map(select(length > 0))')
+          files_json=$(find tests/sitl -name 'test_*.py' | sort | jq -R -s -c 'split("\n") | map(select(length > 0))')
           echo "files=${files_json}" >> "$GITHUB_OUTPUT"
 
   stack-build-image:

.gitignore

@@ -19,7 +19,7 @@ simulation/logs/
 *.log
 
 # One-off / diagnostic scripts (kept locally for reproducibility, not checked in by default)
-simulation/tests/oneoff/
+tests/oneoff/
 
 # Generated flight report plots
 *.png
@@ -30,6 +30,9 @@ tmp/
 # Logs
 *.log
 
+# Python packaging build artifacts
+*.egg-info/
+
 # Node
 node_modules/
 

AGENTS.md

@@ -12,6 +12,33 @@ Current focus:
 - Run `bash test.sh stack -n 1 -k test_lua_flight_steady_sitl` to validate the steady flight SITL stack.
 - After steady stack passes, validate pumping and landing stack tests.
 
+## Repository Layout
+
+The repo root is a single Python distribution (`pyproject.toml`, name `rawes`) containing
+8 first-party top-level packages, installed in editable mode
+(`pip install -e . --no-deps`, done by `setup.cmd`/`setup.sh`). Import them as plain dotted
+packages (`from simulation.controller import ...`, `from analysis.flight_log import ...`) —
+there are no `sys.path.insert()` hacks anywhere in the codebase.
+
+| Package | Contents |
+|---|---|
+| `simulation/` | Physics/aero/EKF-adjacent simulation runtime, Lua/Python flight-stack modules (`mediator.py`, `controller.py`, `param_defaults.py`, `swashplate.py`, `frames.py`, `gcs.py`, ...), `requirements.txt`, `Dockerfile`, `logs/` |
+| `arduloop/` | Self-contained Python port of ArduPilot's traditional-heli attitude/rate-control stack (used by the in-process mock ArduPilot) |
+| `calibrate/` | Bench calibration REPL/tooling for hardware bring-up |
+| `envelope/` | Flight-envelope map computation (`compute_map.py`) and related analysis |
+| `analysis/` | Post-run diagnosis/report scripts (all read `simulation/logs/{test_name}/...`) |
+| `viz3d/` | 3D telemetry playback and torque visualizers |
+| `scripts/` | Deployed Lua flight scripts (`rawes.lua`, `rawes_test_surface.lua`) and standalone runtime scripts (`sitl_bench.py`, `query_hardware.py`) |
+| `tests/` | All test suites: `tests/unit`, `tests/simtests`, `tests/sitl` (Docker/SITL), `tests/hil`, `tests/oneoff`, `tests/common` |
+
+Non-package top-level directories: `design/` (owner docs), `documents/`, `hardware/`,
+`presentations/`, `felix/`, `am32config/` (ESC config tool, separate `package.json`), `tmp/`
+(scratch/working files only).
+
+`simulation/logs/` is the single log root for every test tier (unit fixtures, simtests, and
+SITL stack runs all write there) — it did not move when the other packages were promoted to
+top-level.
+
 ## Read Order (for agents)
 
 1. `design/flight_stack.md` (system behavior and control ownership)
@@ -61,7 +88,7 @@ ast-grep run -p 'def $NAME($$$ARGS):
 # Find Lua function definitions in a script
 ast-grep run -p 'function $NAME($$$ARGS)
   $$$BODY
-end' -l lua simulation/scripts/rawes.lua
+end' -l lua scripts/rawes.lua
 ```
 
 When to still use grep/`grep_search`: matching exact substrings/regex in prose,
@@ -85,19 +112,19 @@ Use the primary doc for each topic. Other docs should link, not restate.
 |---|---|---|
 | Flight architecture, mode ownership, AP/Lua boundaries | `design/flight_stack.md` | `design/tension_collective_control_loop.md`, `design/GUIDED_CONTROL_LOOPS.md` |
 | Simulation internals (physics, sensors, controller plumbing, module map) | `design/simulation.md` | `simulation/README.md`, code docstrings |
-| SITL stack workflow, lockstep, diagnosis procedure | `design/sitl_testing.md` | `simulation/analysis/diagnose_sitl.py` usage text |
-| SITL IC-start timeline and event anchors | `design/sitl_flight_timeline.md` | `design/sitl_testing.md`, `simulation/tests/sitl/flight/conftest.py` |
+| SITL stack workflow, lockstep, diagnosis procedure | `design/sitl_testing.md` | `analysis/diagnose_sitl.py` usage text |
+| SITL IC-start timeline and event anchors | `design/sitl_flight_timeline.md` | `design/sitl_testing.md`, `tests/sitl/flight/conftest.py` |
 | Aero interfaces and conventions | `design/aero_conventions.md` | `design/aero.md` |
 | EKF gating and GPS yaw bring-up | `design/EKF_GATING.md` | `design/ekf_const_pos_mode.md` |
 | ArduPilot heli control-loop behavior | `design/GUIDED_CONTROL_LOOPS.md` | `design/flight_stack.md` |
 | Swashplate geometry and sign mapping | `simulation/swashplate.py` | `design/flight_stack.md` |
 | Hardware assembly and components | `design/hardware.md` | `design/components.md`, `design/dshot.md`, `design/flap_sensor_bench.md` |
-| Testing taxonomy and Lua/Python test conventions | `design/testing.md` | `simulation/pytest.ini` |
+| Testing taxonomy and Lua/Python test conventions | `design/testing.md` | `pyproject.toml` (`[tool.pytest.ini_options]`) |
 | Milestones and decisions history | `design/history.md` | this file (summary only) |
 
 Parameter-reference ownership note:
-- Canonical place for ArduPilot parameter defaults and inline explanations is `simulation/tests/sitl/copter-heli.parm`.
-- Canonical place for RAWES_* parameter defaults and inline explanations is `simulation/tests/sitl/rawes_common_defaults.parm`.
+- Canonical place for ArduPilot parameter defaults and inline explanations is `tests/sitl/copter-heli.parm`.
+- Canonical place for RAWES_* parameter defaults and inline explanations is `tests/sitl/rawes_common_defaults.parm`.
 - If a parameter explanation changes, update the owning `.parm` file first; other docs should link to it instead of duplicating bitmasks/tables.
 
 ## Core Invariants (summary)
@@ -188,7 +215,7 @@ For signs, frame details, EKF gating, and mixer conventions, read the primary do
 Use this as the quick contract-level reference for the yaw-motor ESC path.
 Canonical long-form owner doc is `design/dshot.md`.
 
-Active hardware-default parameters (from `simulation/tests/sitl/rawes_common_defaults.parm`):
+Active hardware-default parameters (from `tests/sitl/rawes_common_defaults.parm`):
 - `SERVO9_FUNCTION=36` (Motor4 on output 9), `SERVO9_MIN=1000`, `SERVO9_MAX=2000`, `SERVO9_TRIM=1000`
 - `SERVO_BLH_MASK=256` (output 9)
 - `SERVO_BLH_BDMASK=256` (bidirectional DShot on output 9)
@@ -202,7 +229,7 @@ Active hardware-default parameters (from `simulation/tests/sitl/rawes_common_def
 
 SITL behavior:
 - These BLHeli/DShot params are intentionally excluded from SITL boot verification
-    (`simulation/tests/sitl/stack_utils.py` -> `SITL_UNSUPPORTED_PARAMS`) because
+    (`tests/sitl/stack_utils.py` -> `SITL_UNSUPPORTED_PARAMS`) because
     ArduCopter-heli SITL does not compile the BLHeli backend and drives output 9 as PWM.
 
 ## Workflow Rules
@@ -222,9 +249,9 @@ SITL behavior:
   `TelRow` fields in `telemetry_csv.py`, NVF maps in `torque_test_utils.py`, and row-write
   dicts in `mediator.py` in the same commit. Mismatch causes `AttributeError` in
   `TelRow.to_dict()` at runtime.
-- Keep `controller.py` aligned with `simulation/scripts/rawes.lua` behavior.
-- Keep `simulation/scripts/rawes_test_surface.lua` exports in sync with needed Lua test symbols.
-- `_PumpingPythonMode` in `simulation/tests/common/mock_ardupilot.py` is a mechanical translation
+- Keep `controller.py` aligned with `scripts/rawes.lua` behavior.
+- Keep `scripts/rawes_test_surface.lua` exports in sync with needed Lua test symbols.
+- `_PumpingPythonMode` in `tests/common/mock_ardupilot.py` is a mechanical translation
   of `rawes.lua do_steady_loop_inner()`. Variable names mirror Lua. When changing Lua altitude PID
   logic, update the Python in the same commit. Key state that must stay in sync:
   `_tension_for_bz` is a RAMPED value (τ=RAWES_TRP≈2 s) toward `_tension_cmd_n` — not a step.
@@ -258,8 +285,8 @@ There are three tiers, each with a different scope and runtime:
 
 | Tier | Command | Marker | Notes |
 |---|---|---|---|
-| Unit | `.venv/Scripts/python.exe -m pytest simulation/tests/unit` | (none) | Fast; no physics sim |
-| Simtest | `.venv/Scripts/python.exe -m pytest simulation/tests/simtests` | `simtest` | Python physics loop; seconds–minutes |
+| Unit | `.venv/Scripts/python.exe -m pytest tests/unit` | (none) | Fast; no physics sim |
+| Simtest | `.venv/Scripts/python.exe -m pytest tests/simtests` | `simtest` | Python physics loop; seconds–minutes |
 | Stack | `bash test.sh stack [-n N]` | `sitl` | ArduPilot SITL in Docker |
 
 SITL IC-start timeline rule (agent-critical):
@@ -269,9 +296,9 @@ SITL IC-start timeline rule (agent-critical):
 - Canonical definition and per-phase markers live in `design/sitl_flight_timeline.md`.
 
 Stack-test execution rule (agent-critical):
-- For any test under `simulation/tests/sitl/**`, ALWAYS use `bash test.sh stack -n 4 ...`.
+- For any test under `tests/sitl/**`, ALWAYS use `bash test.sh stack -n 4 ...`.
 - DO NOT run SITL tests with host-side pytest commands like
-    `.venv/Scripts/python.exe -m pytest simulation/tests/sitl/...`.
+    `.venv/Scripts/python.exe -m pytest tests/sitl/...`.
     Those bypass the Docker stack harness and can fail with host-path issues
     (for example `/ardupilot/scripts` not existing on Windows host).
 - For a single SITL test, use:
@@ -313,29 +340,29 @@ Long-running test commands (agent-critical, efficiency):
 
 Flight telemetry (pumping, steady, passive SITL runs):
 ```
-.venv/Scripts/python.exe simulation/viz3d/visualize_3d.py simulation/logs/<test_name>/telemetry.csv
+.venv/Scripts/python.exe viz3d/visualize_3d.py simulation/logs/<test_name>/telemetry.csv
 ```
 Example — most recent pumping SITL run:
 ```
-.venv/Scripts/python.exe simulation/viz3d/visualize_3d.py simulation/logs/test_pumping_cycle_lua_sitl/telemetry.csv
+.venv/Scripts/python.exe viz3d/visualize_3d.py simulation/logs/test_pumping_cycle_lua_sitl/telemetry.csv
 ```
 
 Counter-torque motor telemetry (torque SITL runs):
 ```
-.venv/Scripts/python.exe simulation/viz3d/visualize_torque.py simulation/logs/<test_name>/telemetry.csv
+.venv/Scripts/python.exe viz3d/visualize_torque.py simulation/logs/<test_name>/telemetry.csv
 ```
 Example — yaw regulation run:
 ```
-.venv/Scripts/python.exe simulation/viz3d/visualize_torque.py simulation/logs/test_yaw_regulation_sitl/telemetry.csv
+.venv/Scripts/python.exe viz3d/visualize_torque.py simulation/logs/test_yaw_regulation_sitl/telemetry.csv
 ```
 
 Controls (both visualizers): Space = play/pause, Left/Right = step frame, +/- = speed.
 
 ## File Placement Rules
 
 - Temporary and working files must go in `tmp/` at repo root.
-- One-off diagnostic scripts belong in `simulation/tests/oneoff/`.
-- Reusable analysis tooling belongs in `simulation/analysis/`.
+- One-off diagnostic scripts belong in `tests/oneoff/`.
+- Reusable analysis tooling belongs in `analysis/`.
 
 ## Agent Editing Policy for Docs
 

README.md

@@ -154,10 +154,10 @@ Then run tests in three sequential stages. Always run them in order.
 
 ```bash
 # Stage 1 -- Unit tests (Windows, no Docker, ~460 tests, ~65 s)
-.venv/Scripts/python.exe -m pytest simulation/tests/unit -m "not simtest" -q
+.venv/Scripts/python.exe -m pytest tests/unit -m "not simtest" -q
 
 # Stage 2 -- Simtests (Windows, no Docker, ~29 tests, ~5 min)
-.venv/Scripts/python.exe -m pytest simulation/tests/simtests -m simtest -q
+.venv/Scripts/python.exe -m pytest tests/simtests -m simtest -q
 
 # Stage 3 -- Stack tests (Docker, ArduPilot SITL)
 bash test.sh stack -v

analysis/__init__.py

@@ -0,0 +1 @@
+"""analysis — diagnostic tooling for SITL/simtest telemetry and MAVLink logs."""

simulation/analysis/analyse_descent.py → analysis/analyse_descent.py

@@ -23,9 +23,9 @@
 
 Usage
 -----
-  python simulation/analysis/analyse_descent.py
-  python simulation/analysis/analyse_descent.py --wind 0 --col 0.02 --tension 176
-  python simulation/analysis/analyse_descent.py --wind 5 --col 0.02 --tension 200
+  python analysis/analyse_descent.py
+  python analysis/analyse_descent.py --wind 0 --col 0.02 --tension 176
+  python analysis/analyse_descent.py --wind 5 --col 0.02 --tension 200
 """
 from __future__ import annotations
 
@@ -36,11 +36,10 @@
 
 import numpy as np
 
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 
 from dynbem import rotor_definition as rd
 from dynbem   import create_aero
-from frames import build_orb_frame
+from simulation.frames import build_orb_frame
 
 
 _BZ_VERT = np.array([0.0, 0.0, -1.0])
@@ -182,7 +181,7 @@ def simulate_descent_bem_ode(
     Same point-mass ODE as simulate_descent() but uses q_spin_from_aero +
     step_spin_ode explicitly, with I_ode_kgm2 (not I_spin) — matching PhysicsCore.
     """
-    from physics_core import q_spin_from_aero, step_spin_ode
+    from simulation.physics_core import q_spin_from_aero, step_spin_ode
 
     rotor  = rd.default()
     dk     = rotor.dynamics_kwargs()
@@ -258,8 +257,8 @@ def simulate_descent_physics(
     angles, making comparison with the other two functions meaningless.
     """
     from types import SimpleNamespace
-    from physics_core import PhysicsCore, q_spin_from_aero
-    from param_defaults import load_collective_phys_range as _lr
+    from simulation.physics_core import PhysicsCore, q_spin_from_aero
+    from simulation.param_defaults import load_collective_phys_range as _lr
 
     rotor   = rd.default()
     bz_vert = np.array([0.0, 0.0, -1.0])

simulation/analysis/analyse_gps_fusion.py → analysis/analyse_gps_fusion.py

@@ -13,17 +13,18 @@
 
 Usage
 -----
-    python3 simulation/analysis/analyse_gps_fusion.py test_stationary_gps_fusion
-    python3 simulation/analysis/analyse_gps_fusion.py test_guided_flight --plot
-    python3 simulation/analysis/analyse_gps_fusion.py --log /path/to/gcs.log
+    python3 analysis/analyse_gps_fusion.py test_stationary_gps_fusion
+    python3 analysis/analyse_gps_fusion.py test_guided_flight --plot
+    python3 analysis/analyse_gps_fusion.py --log /path/to/gcs.log
 """
 
 import argparse
 import re
 import sys
 from pathlib import Path
 
-_SIM_DIR = Path(__file__).resolve().parents[1]
+import simulation as _simulation_pkg
+_SIM_DIR = Path(_simulation_pkg.__file__).resolve().parent  # simulation/
 
 # ---------------------------------------------------------------------------
 # EKF flag definitions (EKF_STATUS_REPORT.flags)

simulation/analysis/analyse_hardware_log.py → analysis/analyse_hardware_log.py

@@ -12,8 +12,8 @@
 
 Usage
 -----
-    python simulation/analysis/analyse_hardware_log.py <log.BIN>
-    python simulation/analysis/analyse_hardware_log.py  # auto-finds latest in simulation/logs/hardware/
+    python analysis/analyse_hardware_log.py <log.BIN>
+    python analysis/analyse_hardware_log.py  # auto-finds latest in simulation/logs/hardware/
 """
 from __future__ import annotations
 
@@ -22,8 +22,6 @@
 import sys
 
 _SIM_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
-if _SIM_DIR not in sys.path:
-    sys.path.insert(0, _SIM_DIR)
 
 
 def _find_latest_log() -> str:

simulation/analysis/analyse_ic_attitude.py → analysis/analyse_ic_attitude.py

@@ -10,10 +10,10 @@
 
 import numpy as np
 
-_SIM_DIR = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(_SIM_DIR))
+import simulation as _simulation_pkg
+_SIM_DIR = Path(_simulation_pkg.__file__).resolve().parent  # simulation/
 
-from telemetry_csv import read_csv  # noqa: E402
+from simulation.telemetry_csv import read_csv  # noqa: E402
 
 
 def _angle_deg(a: np.ndarray, b: np.ndarray) -> float:

simulation/analysis/analyse_landing.py → analysis/analyse_landing.py

@@ -17,8 +17,8 @@
 
 Usage
 -----
-    .venv/Scripts/python.exe simulation/analysis/analyse_landing.py
-    .venv/Scripts/python.exe simulation/analysis/analyse_landing.py --test test_landing_lua_sitl --bucket 2
+    .venv/Scripts/python.exe analysis/analyse_landing.py
+    .venv/Scripts/python.exe analysis/analyse_landing.py --test test_landing_lua_sitl --bucket 2
 """
 
 import sys, math, csv, argparse
@@ -27,7 +27,8 @@
 
 import numpy as np
 
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+import simulation
+
 
 BREAK_LOAD_N  = 620.0
 FLOOR_ALT_M   = 1.5
@@ -336,7 +337,7 @@ def frac(pred):
     if args.csv:
         csv_path = Path(args.csv)
     else:
-        csv_path = Path(__file__).resolve().parents[1] / "logs" / args.test / "telemetry.csv"
+        csv_path = Path(simulation.__file__).resolve().parent / "logs" / args.test / "telemetry.csv"
 
     if not csv_path.exists():
         print(f"No telemetry at {csv_path}")

simulation/analysis/analyse_pumping.py → analysis/analyse_pumping.py

@@ -17,10 +17,10 @@
 
 Usage
 -----
-  .venv/Scripts/python.exe simulation/analysis/analyse_pumping.py
-  .venv/Scripts/python.exe simulation/analysis/analyse_pumping.py --wind 8
-  .venv/Scripts/python.exe simulation/analysis/analyse_pumping.py --el 30 --tension 400 --wind 10
-  .venv/Scripts/python.exe simulation/analysis/analyse_pumping.py --col -0.21  # single run, verbose
+  .venv/Scripts/python.exe analysis/analyse_pumping.py
+  .venv/Scripts/python.exe analysis/analyse_pumping.py --wind 8
+  .venv/Scripts/python.exe analysis/analyse_pumping.py --el 30 --tension 400 --wind 10
+  .venv/Scripts/python.exe analysis/analyse_pumping.py --col -0.21  # single run, verbose
 """
 from __future__ import annotations
 
@@ -31,12 +31,11 @@
 
 import numpy as np
 
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 
 from dynbem import rotor_definition as rd
-from frames      import build_orb_frame
-from physics_core import PhysicsCore
-from ic import load_ic
+from simulation.frames import build_orb_frame
+from simulation.physics_core import PhysicsCore
+from simulation.ic import load_ic
 
 
 _G = 9.81
@@ -99,7 +98,7 @@ def simulate_one(
     _ic = load_ic()
     if omega_init is None:
         omega_init = _ic.omega_spin
-    from param_defaults import thrust_to_coll_rad as _t2c  # noqa: F811
+    from simulation.param_defaults import thrust_to_coll_rad as _t2c  # noqa: F811
     col_warm = _t2c(_ic.eq_thrust)
 
     el     = math.radians(elevation_deg)