This repository provides the implementation of Coupled Control-JEPA (C-JEPA) and Wireless-JEPA (W-JEPA) for communication-aware remote robotic control.
The framework combines robot observations and dynamics from ROS/Gazebo with wireless channel information generated using Sionna RT. C-JEPA is first pretrained in a Gym racing-car environment and then fine-tuned in Gazebo. W-JEPA is trained using CSI data collected from the synchronized Sionna RT environment.
The main dependencies are:
- Ubuntu 20.04
- ROS Noetic (ROS1)
- Gazebo Classic
- Python 3
- PyTorch
- OpenCV
- Gym / Gymnasium
- Sionna / Sionna RT
The framework uses two synchronized simulation environments: Gazebo for robot simulation and Sionna RT for wireless channel simulation. Both simulators represent the same physical environment with identical geometry and spatial configuration, enabling consistent robot and wireless simulations.
The environment is first constructed in Gazebo, which provides robot dynamics, camera observations, robot states, and control interfaces through ROS.
The same environment is reconstructed in Sionna RT while preserving the corresponding geometry and coordinate system. Sionna RT is then used for physics-based ray tracing, wireless channel modeling, and CSI generation.
The robot position, orientation, and motion are synchronized between Gazebo and Sionna RT, allowing the wireless channel to be evaluated according to the robot's movement in the Gazebo environment.The training procedure consists of three main stages.
C-JEPA is first trained using the Gym racing-car environment.
cd c_jepa/control_jepa/test/
python3 train.pyThe trained model weights are saved and used to initialize the Gazebo training stage.
Load the pretrained C-JEPA weights and fine-tune the model using observations and robot dynamics from the Gazebo environment.
First, launch the Gazebo environment:
roslaunch gz_sionna jetbot_tellus.launch Then load the pretrained C-JEPA weights and fine-tune the model using observations and robot dynamics from Gazebo:
python3 c_jepa/control_jepa/test/train_gazebo.pyCSI data are generated from Sionna RT and synchronized with the corresponding robot pose, velocity, and latent control state from C-JEPA. The resulting dataset is then used to train W-JEPA.
python3 c_jepa/wireless_jepa/src/train.pyLaunch the Gazebo environment:
roslaunch gz_sionna jetbot_tellus.launch Once the Gazebo environment is running, start the Control-JEPA (C-JEPA) module in a separate terminal:
python3 c_jepa/control_jepa/test/Gazebo_model_test.pyWith Gazebo, C-JEPA, and W-JEPA running simultaneously, the coupled remote-control demo is ready to run.
python3 c_jepa/wireless_jepa/src/wireless_jepa.py- H.P. Madushanka (madushanka.hewapathiranage@oulu.fi)
- Sumudu Samarakoon (sumudu.samarakoon@oulu.fi)
- Mehdi Bennis (mehdi.bennis@oulu.fi)
