Teaching LLMs to Edit Video Without Watching It

What it does

video-use is a skill framework that hands your raw footage to a coding agent—Claude Code, Codex, or similar—and asks it to produce a finished final.mp4. The agent inventories your takes, proposes an edit strategy, waits for your approval, then handles the cuts, color grading, audio fades, subtitles, and even parallel animation overlays via Remotion or Manim. Everything is driven through chat, not a timeline interface.

The interesting bit

The central trick is that the LLM never “watches” the video. It reads a compact, ~12KB text transcript from ElevenLabs Scribe with word-level timestamps and speaker labels, and only requests on-demand PNG composites—filmstrips with waveforms—when it needs to sanity-check a cut. This avoids dumping 45 million tokens of frame noise into the context window and keeps the reasoning grounded in text.

Key highlights

• Audio-first editing: Cuts are driven by speech boundaries, silence gaps, and filler-word detection (umm, uh), with 30ms fades to prevent pops.
• Self-evaluation loop: After rendering, the agent runs timeline_view on the output at every cut boundary to catch visual jumps or hidden subtitles, retrying up to three times before showing you anything.
• Session memory: State persists in a local project.md, so you can pause and resume editing days later without losing context.
• Agent-native, not standalone: Designed to be registered as a skill inside Claude Code or another shell-capable agent; it does not provide its own GUI or direct CLI workflow.
• Zero-presets philosophy: Claims to work for talking heads, tutorials, travel, or interviews without content-specific templates.

Caveats

• Requires a shell-capable coding agent (Claude Code, Codex, etc.) and an ElevenLabs API key for transcription; it is not a standalone application you can run in isolation.
• The “12 hard rules” of editing craft and full production logic live in SKILL.md, not the README, so evaluating the actual edit quality requires digging deeper into the repo.
• Animation overlays depend on external frameworks (HyperFrames, Remotion, Manim, or PIL), which you must manage separately.

Verdict

Content creators who already live inside Claude Code and want to automate rough-cut assembly will find this a clever force multiplier. If you prefer a traditional NLE or need to edit offline without cloud transcription, this is not your tool.