ResNet + GPT plays Honor of Kings—and honestly, it feeds

What it does The project pipes screenshots from an Android phone running Honor of Kings through ResNet101, then feeds the visual features into a Transformer decoder to predict touch and keyboard actions. It is a behavior-cloning rig: a human plays via keyboard while the script records frames and actions, and the model learns to imitate. The resulting bot pilots the hero Hou Yi through minitouch and scrcpy, though the author admits it frequently “feeds” and struggles to beat bronze-level AI.

The interesting bit The author is refreshingly candid that this was a quick experiment open-sourced only because Bilibili viewers asked, and the README spends more time apologizing for messy code than hyping the architecture. It also contains a surprisingly detailed guide to reconciling Android pointer coordinates with minitouch’s physically-fixed origin, complete with axis rotation math for different phone orientations.

Key highlights

• Vision backbone is PyTorch’s pre-trained resnet101-5d3b4d8f.pth; the sequence head is borrowed from SamLynnEvans’s Transformer decoder.
• Data pipeline captures human gameplay via scrcpy and pyminitouch, storing ResNet-extracted features alongside actions as numpy arrays.
• Hardcoded for a 1080×2160 phone resolution with a specific in-game UI layout; other devices require manual coordinate remapping.
• The author estimates the model has enough capacity to digest tens of thousands of games, but the released checkpoint saw only about a hundred.
• Deprecated: the author has moved on to a reinforcement-learning successor, leaving this repo frozen.

Caveats

• The trained model frequently “feeds” (dies and gives enemy kills) and may lose to beginner AI opponents; the code itself is described as messy and hastily open-sourced.
• minitouch does not support Android 10, and game updates can break compatibility at any time.
• Setup is brittle: Windows-only, NVIDIA GPU with 6 GB+ VRAM recommended, and the phone must match the exact screen layout and coordinate system assumptions.

Verdict Worth a look if you want a frank, warts-and-all reference for piping mobile-game screenshots into a PyTorch vision-sequence model. Skip it if you want a plug-and-play MOBA bot or clean production code.