bytedance/effective_transformer
Optimized inference engine for BERT that dynamically removes and restores padding values to reduce memory and computation waste on variable-length sequences.
Not currently ranked — collecting fresh signals.
star history
Effective Transformer is a CUDA-accelerated inference optimization library built on NVIDIA FasterTransformer. It addresses the inefficiency of padding variable-length sequences into uniform batch matrices by computing prefix sums of attention masks to access only valid tokens. During computation stages, padding values are dynamically removed and restored, significantly reducing execution time and memory consumption especially for large batch sizes with highly variable sequence lengths.
Frequently asked
- What is bytedance/effective_transformer?
- Optimized inference engine for BERT that dynamically removes and restores padding values to reduce memory and computation waste on variable-length sequences.
- Is effective_transformer open source?
- Yes — bytedance/effective_transformer is open source, released under the Apache-2.0 license.
- What language is effective_transformer written in?
- bytedance/effective_transformer is primarily written in C++.
- How popular is effective_transformer?
- bytedance/effective_transformer has 479 stars on GitHub.
- Where can I find effective_transformer?
- bytedance/effective_transformer is on GitHub at https://github.com/bytedance/effective_transformer.