This repository provides scripts and integration for training large language models with QLoRA (Quantized Low-Rank Adaptation) combined with FSDP (Fully Sharded Data Parallel). It enables efficient fine-tuning of models like Llama-2 70B on limited GPU memory by combining 4-bit quantization with gradient checkpointing and CPU offloading. The project integrates with Axolotl training framework and uses libraries like bitsandbytes and Hugging Face transformers.