McGill-NLP/llm2vec
A recipe to convert decoder-only LLMs into bidirectional text encoders through attention masking and contrastive fine-tuning.
Not currently ranked — collecting fresh signals.
star history
LLM2Vec provides code to transform standard decoder-only language models into bidirectional text encoders. The approach involves three steps: enabling bidirectional attention, training with masked next token prediction, and applying unsupervised contrastive learning. The resulting encoders can be fine-tuned further for various natural language understanding tasks, achieving competitive performance on embedding benchmarks.
Frequently asked
- What is McGill-NLP/llm2vec?
- A recipe to convert decoder-only LLMs into bidirectional text encoders through attention masking and contrastive fine-tuning.
- Is llm2vec open source?
- Yes — McGill-NLP/llm2vec is open source, released under the MIT license.
- What language is llm2vec written in?
- McGill-NLP/llm2vec is primarily written in Python.
- How popular is llm2vec?
- McGill-NLP/llm2vec has 1.7k stars on GitHub.
- Where can I find llm2vec?
- McGill-NLP/llm2vec is on GitHub at https://github.com/McGill-NLP/llm2vec.