hpcaitech/SwiftInfer
TensorRT-based implementation of StreamingLLM for production-grade LLM inference serving.
Not currently ranked — collecting fresh signals.
star history
SwiftInfer provides an optimized implementation of StreamingLLM using NVIDIA TensorRT and TensorRT-LLM, enabling efficient LLM inference with support for infinite input lengths through the Attention Sink mechanism. Built upon TensorRT-LLM v0.6.0, it aims to make streaming LLM inference production-grade by leveraging hardware-accelerated optimization for faster serving of large language models.
Frequently asked
- What is hpcaitech/SwiftInfer?
- TensorRT-based implementation of StreamingLLM for production-grade LLM inference serving.
- Is SwiftInfer open source?
- Yes — hpcaitech/SwiftInfer is open source, released under the Apache-2.0 license.
- What language is SwiftInfer written in?
- hpcaitech/SwiftInfer is primarily written in Python.
- How popular is SwiftInfer?
- hpcaitech/SwiftInfer has 479 stars on GitHub.
- Where can I find SwiftInfer?
- hpcaitech/SwiftInfer is on GitHub at https://github.com/hpcaitech/SwiftInfer.