lucidrains/MEGABYTE-pytorch
A PyTorch implementation of MEGABYTE, a multiscale transformer for predicting million-byte sequences across multiple granularities.
Not currently ranked — collecting fresh signals.
star history
Implementation of the MEGABYTE architecture that uses hierarchical transformers to model sequences at multiple scales. The global transformer handles coarse representations while local transformers refine finer details. Supports variable depth hierarchies, flash attention, and training/generation on arbitrary byte sequences.
Frequently asked
- What is lucidrains/MEGABYTE-pytorch?
- A PyTorch implementation of MEGABYTE, a multiscale transformer for predicting million-byte sequences across multiple granularities.
- Is MEGABYTE-pytorch open source?
- Yes — lucidrains/MEGABYTE-pytorch is open source, released under the MIT license.
- What language is MEGABYTE-pytorch written in?
- lucidrains/MEGABYTE-pytorch is primarily written in Python.
- How popular is MEGABYTE-pytorch?
- lucidrains/MEGABYTE-pytorch has 655 stars on GitHub.
- Where can I find MEGABYTE-pytorch?
- lucidrains/MEGABYTE-pytorch is on GitHub at https://github.com/lucidrains/MEGABYTE-pytorch.