kohjingyu/fromage
A multimodal language model that grounds frozen text LLMs to images for retrieval and generation.
Not currently ranked — collecting fresh signals.
star history
FROMAGe is a research implementation featuring a language model with visual grounding capabilities. The model uses linear projection layers and special [RET] tokens to bridge frozen pretrained LLMs with image embeddings, enabling image retrieval and multimodal dialogue. Precomputed visual embeddings on Conceptual Captions images support efficient retrieval-augmented generation. Model checkpoints are small (around 11MB) and included in the repository.
Frequently asked
- What is kohjingyu/fromage?
- A multimodal language model that grounds frozen text LLMs to images for retrieval and generation.
- Is fromage open source?
- Yes — kohjingyu/fromage is open source, released under the Apache-2.0 license.
- What language is fromage written in?
- kohjingyu/fromage is primarily written in Jupyter Notebook.
- How popular is fromage?
- kohjingyu/fromage has 485 stars on GitHub.
- Where can I find fromage?
- kohjingyu/fromage is on GitHub at https://github.com/kohjingyu/fromage.