Your LLM just got a body: a runtime for physical AI agents

What it does

OM1 is a modular, Python-oriented runtime that deploys multimodal AI agents across digital environments and physical robots. It ingests sensor data—cameras, LIDAR, web feeds—and routes it through configurable LLM or VLM endpoints, then translates the model’s decisions into high-level hardware commands like navigation, motion, or speech. The project includes pre-built connectors for ROS2, Zenoh, and CycloneDDS, along with a bundled Prometheus and Grafana stack for monitoring pipeline latency.

The interesting bit

Rather than handling low-level kinematics, OM1 assumes your hardware already exposes a high-level HAL that accepts commands such as backflip or gently pick up the red apple. If your robot does not offer that, the README bluntly notes you will need to build one yourself using reinforcement learning and simulation before OM1 can talk to it.

Key highlights

• Pre-configured endpoints for a broad range of LLMs and VLMs, including local Ollama instances and hosted providers like OpenAI, Anthropic, DeepSeek, and Gemini.
• Plugin-based hardware abstraction supporting ROS2, CycloneDDS, and Zenoh, with Zenoh recommended for new development.
• Includes a pre-configured Prometheus and Grafana dashboard for real-time monitoring of LLM and ASR latencies.
• A starter spot agent uses a webcam for object detection and runs entirely as terminal output, making it easy to test the pipeline without hardware.
• Full autonomy support—navigation, SLAM, auto-charging, face detection—for Unitree Go2 and G1 robots via the open-source BrainPack companion.

Caveats

• The default spot quick-start logs actions but does not execute real robot movements; physical integration requires additional configuration.
• Some features rely on an OpenMind API key, and the free plan is limited to 50 OMCU per month.
• If your robot lacks a high-level SDK, you are responsible for building a suitable HAL first; OM1 will not do low-level motion planning for you.

Verdict

A practical starting point if you already have a robot with a decent high-level SDK and want to layer LLM-driven behavior on top. Look elsewhere if you need fundamental motion-planning or a completely key-free, self-hosted stack.