Text, image, or video in; vector animation out

What it does

OmniLottie is an end-to-end generator that consumes text, images, or video and emits Lottie JSON files—vector animations with timing, layers, and transforms. It is built on a 4B-parameter vision-language model and runs fully offline given the weights. The project also ships MMLottie-2M, a dataset of two million annotated animations, and MMLottieBench, a 900-sample benchmark split between real and synthetic examples.

The interesting bit

Rather than rendering pixels, the model predicts parameterized Lottie tokens—essentially writing the JSON source code for the animation. That keeps the output in an editable vector format instead of a fixed raster sequence.

Key highlights

• Multimodal input: accepts text, single images, or video clips as prompts
• Two model formats: original PyTorch weights and HuggingFace safetensors with from_pretrained() support
• GPU footprint of 15.2 GB and inference latency that scales linearly with token count (roughly 8–133 seconds for 256–4096 tokens)
• Community ComfyUI node available for node-based workflows
• Training code is still on the open-source roadmap; only inference, data, and benchmarks are released

Caveats

• The README notes that inference times in the repository are measured per “OmniLottie Lottie tokens,” while the paper reports per JSON code tokens, so direct comparison with baselines requires reading the fine print
• Training code has not been released yet

Verdict

Motion designers and frontend developers who need programmatic or prompt-driven vector animations should look here; if you only need static icons or raster video, this is overkill.