Giving your reMarkable an LLM pen pal

What it does

Ghostwriter is a Rust binary that runs natively on the reMarkable 2 and Paper Pro, quietly watching the screen. Tap a corner and it snaps a screenshot, sends it to a vision-capable LLM—Claude, GPT-4o, Gemini, or Groq—and renders the reply right back onto the e-ink page as either typed text or a vector drawing. It is essentially glue between the tablet’s low-level input devices and remote multimodal APIs, packaged as a single cross-compiled executable.

The interesting bit

The project treats handwriting and free-form sketching as the UI itself, not as data to be exported. The author is also experimenting with image segmentation to feed the model spatial coordinates, so the LLM knows where on the page to place its answer instead of scribbling blindly.

Key highlights

• Runs on the tablet itself, not a tethered phone or desktop helper.
• Supports multiple vision backends through a unified engine abstraction.
• Replies via a virtual keyboard layer or by drawing SVG paths as e-ink dots.
• Includes an offline evaluation mode that replays captured screenshots on a laptop for faster prompt iteration.
• Lets you override the system prompt with a local JSON file to change personality or tasking.

Caveats

• SVG rendering is currently dot-based and rough; the author notes the tablet can “flip out” when filling large areas.
• Spatial awareness is improving but historically shaky; the optional segmenter is still explicitly experimental.
• A reMarkable OS upgrade (3.16.2.3) previously broke screenshot capture, so future updates may require downgrading or waiting for fixes.

Verdict

Worth a look if you treat your reMarkable as a thinking tool and want an AI collaborator that lives inside the device rather than beside it. Skip it if you need polished, production-grade handwriting recognition or reliable local-only operation.