A unified VLM node pack for ComfyUI that respects your GPU

What it does This repository provides custom ComfyUI nodes for vision-language models, large language models, and audio generation. The centerpiece is a single Modern VLM node that abstracts a wide catalog of models—ranging from sub-billion-parameter edge checkpoints to 30B+ instruction-tuned VLMs—behind one stable interface. It also includes specialized nodes for Florence-2, PaLI-Gemma, Qwen2-VL, and several GGUF loaders, plus API wrappers for OpenAI, Anthropic, Gemini, and Groq.

The interesting bit The project treats GPU memory management as a first-class concern rather than an afterthought. Nodes ask ComfyUI to free the exact memory they need before loading a model, offload layers intelligently across CUDA/ROCm/Metal/XPU/CPU, and cache weights between prompts instead of assuming you want everything on GPU zero. Live token streaming through ComfyUI’s native WebSocket channel means you can watch a model think without blocking the graph.

Key highlights

• One Modern VLM node covers Qwen, Gemma, SmolVLM2, Granite Vision, InternVL, and others, with 16 curated sub-4B presets for low-VRAM machines.
• Native support for ComfyUI’s selected PyTorch backend (CUDA, ROCm, Metal, Intel XPU, or CPU) without forcing you to reinstall torch.
• Specialized nodes preserve model-specific capabilities like Florence-2 segmentation masks, PaLI-Gemma VQ-VAE decoding, and llama.cpp GGUF projector loading.
• API nodes for OpenAI Responses/Chat Completions, Anthropic, Gemini, and Groq with key injection via environment variables or node input.
• Zero network access on import; missing optional backends fail only the nodes that require them.

Caveats

• The README notes that Moondream2 is currently not marked passed on the tested Torch/Transformers stack, so production workflows should fall back to a small Modern VLM preset.
• Qwen 3.5/3.6 thinking mode is off by default for speed; enabling it can slow inference if optimized linear-attention kernels are unavailable.
• Avoid stacking multiple independently quantized VLMs in one workflow unless your GPU can hold them all simultaneously.

Verdict Worth a look if you run ComfyUI workflows that need captioning, OCR, visual Q&A, or prompt generation and you are tired of manually juggling model loaders. Skip it if you only need a single, static VLM and do not care about ComfyUI’s node graph.