Wrap LoRA, INT4, and DeepSpeed into one fine-tuning API

What it does

xTuring is a Python library that wraps the end-to-end lifecycle of open-source LLMs—data loading, fine-tuning, evaluation, and inference—into a single BaseModel API. It targets privacy-conscious users who want to run everything locally or inside a VPC, and it supports a wide model zoo including LLaMA, Mistral, Qwen3, GPT-OSS, and even the 0.6B-parameter Qwen3 for CPU experimentation. Under the hood it handles the tedious parts: LoRA adapters, INT8/INT4 quantization via GenericLoraKbitModel, DeepSpeed integration, and Intel-optimized CPU kernels.

The interesting bit

The project treats hardware elasticity as a first-class concern. You can start with a toy model on a laptop using the qwen3_0_6b_lora checkpoint, then promote the same code to a multi-GPU node for a 120B parameter GPT-OSS model by simply changing a string key. The README also includes a striking performance table: on 4×A100s, training LLaMA-7B with LoRA and DeepSpeed drops from 21 hours per epoch to 20 minutes compared to full offloading. That is not a minor optimization; it is the difference between an overnight experiment and a coffee break.

Key highlights

• Supports a broad registry of models—LLaMA/LLaMA 2, Mistral, MiniMax M2, GPT-J, GPT-2, Mamba, and others—addressable through simple string keys like llama2_lora_int8.
• Built-in low-precision training paths: INT8 and INT4 quantization, plus LoRA, including a combined LoRA + INT4 mode via GenericLoraKbitModel.
• Evaluation support is currently limited to perplexity, but it works on any causal LM and any dataset you load.
• Ships with both a CLI chat interface and a localhost UI playground for rapid manual testing after fine-tuning.
• Provides pre-fine-tuned Alpaca checkpoints for DistilGPT-2 and LLaMA LoRA that you can load immediately with BaseModel.load.

Caveats

• Evaluation metrics are thin: only perplexity is supported so far, so you will need external tooling for task-specific benchmarks.
• The INT4 path uses a separate GenericLoraKbitModel class rather than the standard BaseModel.create registry, which fractures the otherwise uniform API slightly.
• Several roadmap items—INT3/INT2/INT1 quantization and Stable Diffusion support—remain unchecked, so the feature set is still evolving.

Verdict

Good for teams that want to experiment with private fine-tuning without hand-rolling HuggingFace, PEFT, and DeepSpeed integrations. Skip it if you need advanced evaluation suites, non-causal architectures, or quantization levels beyond INT4.