VYX - Voice and Y-Vision Execution Engine for LeRobot.

[shashankbhosagi]

VYX (/vɪks/)

Voice & Y-Vision X-Execution Engine for LeRobot.

VYX is a lightweight, asynchronous framework designed to connect local multimodal AI models (Whisper and Ollama Vision-Language Models) directly to robotic hardware. Using Hugging Face's LeRobot as its underlying hardware abstraction layer, VYX translates natural language commands and camera frames into actionable servo motor instructions.

[ Audio / Voice ] -> Whisper STT ->
                                   -> Local VLM Agent -> LeRobot Engine -> SO-101 Servos
[ Camera Frame  ] -> Frame Grab  ->

Objective

Building voice-controlled robotics setups typically requires managing custom serial protocols, camera streams, and model integration across multiple disjointed scripts. VYX provides a structured, modular Python library to streamline this workflow:

• Hardware Abstraction: Leverages LeRobot for motor configuration, port detection, and kinematics management on supported arms like the SO-101.
• Local Execution: Runs entirely on local infrastructure, utilizing CPU-optimized transcription and local network GPU nodes for model inference.
• Unified Control Loop: Captures audio input and camera frames simultaneously, processes the payload into structured coordinate or joint commands, and drives the hardware over serial.

Roadmap (v0.1.0)

• Core Agent and ArmController abstractions wrapping LeRobot
• Multimodal prompt parser for text and image to joint angle payloads
• Voice trigger and camera snapshot event loop
• Execution script for the SO-101 robot arm

[Aliqyan-21]

Sounds great @shashankbhosagi, waiting for it's repository.

[shashankbhosagi]

repo created - https://github.com/bifrion/vyx
project for vyx created https://github.com/orgs/bifrion/projects/1