An Android overlay that pipes screenshots through OCR and back

What it does

overlay-translator is an Android app that captures your screen, runs the image through OCR, translates the text, and draws the results back over the original pixels. It targets games, visual novels, and manga, offering both full-screen and word-level translation without requiring root access.

The interesting bit

The app treats screen translation as a compositing pipeline: it can detect manga speech bubbles, erase original text, repair the background, and render vertical translations back into the bubble shape. It also supports offline LLMs via llama-cpp and local HTTP OCR bridges, letting you offload heavy inference to a PC on the same LAN.

Key highlights

• No root required; uses MediaProjection and Shizuku to grab frames.
• OCR options span on-device ML Kit, PaddleOCR, and manga-specific models, plus cloud fallbacks and local HTTP relays like Umi-OCR.
• Translation backends include cloud LLMs (DeepSeek, ChatGPT, Claude), self-hosted endpoints, DeepL (with deeplx proxy support), and on-device engines like Sakura and Hy-MT2.
• Adaptive rendering modes place translations inline or in a draggable floating window, with automatic bubble detection and text-direction handling for Japanese comics.
• A “word select” mode uses LLMs to generate full dictionary entries—definitions, phonetics, synonyms, and usage notes—instead of just a single-line translation.

Caveats

• Offline features (local LLMs, some OCR models) require downloading large model files first; the app handles this in the background but needs notification permissions on Android 13+.
• Bubble detection and adaptive rendering fall back to plain rectangular blocks if confidence is low or processing fails.
• Some vendor system TTS implementations lack position callbacks, so resuming playback restarts from the beginning of the current segment.

Verdict

Worth a look if you play untranslated Japanese mobile games or read manga on your phone and want subtitles without switching apps. Skip it if you already live in a fully localized ecosystem or avoid floating-window utilities.