FFmpeg for PyTorch without the FFmpeg headache

What it does

TorchCodec decodes video and audio into PyTorch tensors on CPU or CUDA, and encodes them back out on CPU. It wraps your system-installed FFmpeg (versions 4 through 8) in a Pythonic API that supports slice indexing, frame-level lookups, and time-based seeking. The output is ready for transforms or model training without intermediate format shuffling.

The interesting bit

Instead of hiding FFmpeg behind a subprocess call, it abstracts the library’s complexity into tensor-native operations—so decoder[0] returns a uint8 tensor and get_frames_played_at returns timestamps alongside the pixels. It is essentially a carefully written glue layer between a mature C library and PyTorch’s expectations.

Key highlights

• Returns decoded frames directly as PyTorch tensors with shape [C, H, W] or batched [N, C, H, W]
• Supports both index-based (get_frames_at) and time-based (get_frames_played_at) retrieval with PTS metadata
• CUDA decoding available on Linux via NVDEC, falling back to CPU wheels on macOS and Windows
• Encoding API supports multiplexing video and audio streams into a single output file
• Tight version coupling with PyTorch releases (e.g., torchcodec 0.13 requires torch >=2.11)

Caveats

• Encoding is CPU-only; the README does not mention GPU-accelerated encoding
• Windows users need FFmpeg “shared” library builds, and CUDA wheel installation requires manual index-url selection
• NVDEC hardware support depends on your specific Nvidia GPU model

Verdict

Worth a look if you build video or audio ML pipelines in PyTorch and are tired of brittle FFmpeg wrappers. Skip it if you only need inference on pre-extracted frames or already have a stable decoding stack you trust.