miHoYo trains a VLM to actually play its own games
Gamer is a vision-language-action foundation model that plays long-horizon 3D games through raw keyboard and mouse, from the Honkai AI R&D team.

What it does
Gamer looks at the screen and emits native keyboard/mouse actions — the same interface a human uses — across both 2D interfaces and 3D open worlds. It’s trained in three stages: continued pre-training on tens of thousands of hours of device-level interaction trajectories, sparse-reasoning SFT, and online RL. The result reportedly plays challenging 3D open-world games at efficiency comparable to skilled players.
The interesting bit
The “sparse thinking” design: the model acts directly on routine steps and only generates reasoning or memory updates when the situation actually changes. Paired with memory summaries that carry task progress across context windows, this is how it survives long-horizon quests without an infinitely long prompt. The RL stage (Temporal-Aware GRPO) trains on memory-linked trajectory segments with temporal credit assignment — a pragmatic answer to finite context and sparse rewards.
Key highlights
- Unified screen-in / keyboard-mouse-out interface across all games, no per-game action APIs
- TA-GRPO adds 7.8–14.5 percentage points over SFT on Honkai: Star Rail side quests (80.0% and 83.6% milestone scores)
- Streaming-vLLM serving engine: KV-cache reuse, FSM speculative decoding, early action dispatch — 1306 ms down to 65.12 ms per request (20.1×) on H200 with a Qwen3.5 MoE backbone
- Plans actions in 200 ms windows of six 33 ms steps — real-time pacing, not turn-based deliberation
- Models are on Hugging Face under the THUSI-Lab collection
Caveats
- Streaming-vLLM, arguably the most engineering-interesting part, is marked “coming soon” — the repo currently ships the paper, prompts, and demos, not the inference engine
- The latency comparison spans baseline TP1 vs. optimized TP4, so the 20.1× figure bundles several changes at once
- Evaluation is centered on Honkai: Star Rail; generalization claims to other 3D titles rest on the paper, not reproducible code here
Verdict
Worth a look if you care about VLA models, game agents, or low-latency multimodal serving — the memory-plus-sparse-reasoning recipe is a genuinely interesting pattern. If you wanted runnable code today, the TODO list says otherwise.
Frequently asked
- What is RainBowLuoCS/Gamer?
- Gamer is a vision-language-action foundation model that plays long-horizon 3D games through raw keyboard and mouse, from the Honkai AI R&D team.
- Is Gamer open source?
- Yes — RainBowLuoCS/Gamer is open source, released under the Apache-2.0 license.
- How popular is Gamer?
- RainBowLuoCS/Gamer has 1.2k stars on GitHub.
- Where can I find Gamer?
- RainBowLuoCS/Gamer is on GitHub at https://github.com/RainBowLuoCS/Gamer.