One image, whole video: storyboards sized by cut density

What it does

dsh-video-storyboard gives a multimodal model a way to actually “watch” a video without token bankruptcy. Instead of extracting 600 frames from a 10-minute clip, it packs the whole thing into one or a few contact-sheet images (capped at 384 tokens each) alongside a text manifest: per-cell start/end timestamps, dialogue from local ASR, and cut counts. Six tools cover the workflow — video_probe for cheap metadata, video_storyboard for the main grid, video_zoom to blow up a segment, video_locate for text-to-timestamp, video_subtitles for burned-in hardsubs, and video_help as an on-demand manual.

The interesting bit

Since 0.6.0, one cell = one shot, and cell size is computed from cut density at that point in the video — a 20-second long take gets a full-width cell, a 0.3-second flash gets 1/8 edge length. The layout is row-packed with DP and the canvas is cropped to content. So the image itself communicates rhythm: you can see where the fast cutting starts without reading a number. There’s also a genuinely thoughtful economics layer — video_help exists because tool descriptions are re-sent every turn, so long docs cost money every request; the manual is fetched only when needed, at zero image tokens.

Key highlights

• Cell tiers (1/1 through 1/8) are derived via convolution window → per-frame cut probability → smoothing → 7 thresholds with 5% hysteresis, with window widths normalized by frame rate so 24/60/120fps footage tiers identically.
• The auto strategy (default since 0.6.1) picks sheet count by duration (~240s per image) and the cheapest tier that stays readable — after a real incident where a model picked full and burned 6912 tokens on a 90-second PV.
• dryRun reports cut counts and per-strategy image/token costs before committing; scans are cached, so the second call is nearly free.
• Works with any yt-dlp-supported site or local files; vertical video, DASH, and multi-part videos handled. ASR is optional, local, and offline — without it you get pictures but no dialogue.
• video_subtitles crops burned-in subtitles into a vertical strip with per-line timestamps, for the PVs and music videos where no subtitle track and no ASR can help.

Caveats

• Requires your own ffmpeg/ffprobe; yt-dlp and ASR are optional but their absence visibly degrades the experience (no links, no dialogue). The README itself notes most user reports are missing-dependency issues, not bugs.
• The desktop (Electron) profile rejects dsh plugin installs — source install only there.
• 0.6.0 is a breaking rewrite: cells, sheets, adaptive and friends are gone, replaced by maxFrames/strategy/maxSheets/tierMap.

Verdict

Anyone running DeepSeek Harness against video — especially long or densely-cut material — gets a rare combination of token discipline and real visual detail. If you’re not in the DSH ecosystem, the tier-by-cut-density layout idea is still worth stealing, but the plugin itself won’t follow you.