LLM NPCs That Hijack Your Screen, Not Your Source Code

What it does The system listens to your microphone and uses facial recognition to identify which NPC is on screen. It feeds your transcribed speech—along with world lore and character memory drawn from vector stores—to an LLM, then converts the reply into speech. A sadtalker animation of the NPC’s face is generated and layered directly over the game’s facial pixels, so the character appears to talk back without any modification to the game itself. A webcam watches your expressions so the NPC can adjust its tone.

The interesting bit Rather than injecting code into game engines, it treats the rendered frame as a canvas and simply overwrites the face region. That architectural dodge means it works with any game, but only if you first build out a small content pipeline of character bios, voice samples, and lore databases for that specific title.

Key highlights

• Recognizes both named side characters and generic background NPCs, generating personalities on the fly for the latter.
• Vector stores provide long-term memory for world knowledge and conversation history.
• Supports “audio-only” interactions triggered by saying an NPC’s name when their face is off-screen, such as during combat or mounted travel.
• Uses pre_conversation.json files filled with iconic lines to steer the LLM toward a specific dialogue style.
• Requires cloud LLM access (Cohere by default, GPT-4 optionally) and substantial per-game manual setup.

Caveats

• Setup is labor-intensive. The README outlines a lengthy manual process requiring game-specific folders, lore text files, vector database creation, and multiple images per character before the first conversation starts.
• It depends on third-party cloud APIs for language generation; local or offline LLM support is not documented.
• The README does not address how the facial pixel replacement behaves under occlusion, dynamic lighting, or non-frontal camera angles.

Verdict Worth exploring for patient tinkerers who want to role-play with their favorite single-player worlds, but expect a weekend of data entry first. If you want a one-click mod, this is not it.