SecGPT: a security LLM that beats Qwen at CISSP, loses at math

What it does

SecGPT is a family of fine-tuned models (1.5B to 14B parameters) built on Qwen2.5-Instruct and DeepSeek-R1, designed for security operations. It ingests logs, traffic captures, malware disassembly, and vulnerability reports to answer questions, trace attack paths, generate payloads, and suggest fixes. A lightweight SecGPT-Mini variant runs on CPU for simpler tasks.

The interesting bit

The project was trained on a self-built corpus exceeding 5 TB across 14 structured security categories, using 8 A100 GPUs for over a week. The maintainers explicitly target both sides of the firewall—red-team payload generation and blue-team log forensics—in a single model.

Key highlights

• Outperforms base Qwen2.5 on security-specific benchmarks (CISSP and CS-EVAL) across all parameter sizes
• Claims end-to-end capabilities: penetration-testing logic, reverse-engineering assistance, code auditing, and IOC extraction from traffic
• Supports high-throughput deployment via vLLM with tensor parallelism and bfloat16 quantization
• Open-sources both the model weights (HuggingFace/ModelScope) and a security paper dataset

Caveats

• The README claims SecGPT surpasses Qwen2.5 on “all evaluation metrics,” but the published tables show it underperforms the base model on general reasoning and math (CEVAL, GSM8K, BBH) at multiple sizes—an inconsistency that undermines the analysis
• Training details and corpus curation methodology are described in broad strokes (e.g., “70+ fields / 14 classes”) without reproducible specifics
• No visible license information in the provided sources

Verdict

Worth a look for security researchers and SOC engineers who need an on-premise, domain-specific assistant and are willing to validate its outputs. General-purpose LLM users will find better reasoning elsewhere.