How to run near-Opus LLMs at home for $40k (or $2k)

What it does

This repo is a field guide to running state-of-the-art LLMs on bare metal without renting GPUs from the cloud. The author documents two complete builds: a budget ~$2k rig with dual RTX 3090s (48GB VRAM) good for Qwen3.6-27B and local speech-to-text, and a ~$40k “almost-Opus” machine built from last-gen EPYC hardware, four RTX PRO 6000s, and a boutique PCIe switch. Alongside the hardware bills of materials, it ships ready-to-run Docker configs for models like GLM-5.2-594B and a sandboxed agent harness called “clankhouse.”

The interesting bit

The project treats multi-GPU inference as a firmware and carpentry problem. To save roughly $10,000 in host costs, the author scrounges DDR4 parts from eBay and splurges on VRAM, then wires the GPUs through an indie c-payne PCIe4 switch so allreduce traffic stays off the CPU root complex. The README includes the exact ROMED8-2T BIOS settings, kernel boot parameters, and a shell script to disable PCIe ACS via setpci—the kind of tribal knowledge usually trapped in forum threads.

Key highlights

• Complete BOMs from ~$2k to ~$40k, with explicit eBay sourcing for the base system and a custom-fabricated wood enclosure for the switch and GPUs.
• Indie PCIe4 switch integration (c-payne PM40100) delivering measured Gen4 line-rate P2P: 27.5/50.4 GB/s with sub-µs latency.
• Ready-to-run Docker Compose configs for GLM-5.2-594B (~80 tok/s at 460k context) and local speech-to-text using cohere-transcribe.
• A sandboxed agent setup (“clankhouse”) running opencode against the local inference API, equipped with web search, Telegram, and a private Gitea instance.
• Deep hardware tuning: redriver gain levels, ASPM disable, bifurcation fixes, and an iommu=off workaround for NCCL hangs.

Caveats

• The ~$20k tier is marked TODO; the author admits no direct experience there.
• Much of the guidance is tightly coupled to specific firmware versions, cable part numbers, and motherboard quirks that may drift over time.
• The repo is explicitly a personal log; the author notes he was “lucky/dumb enough” to buy GPUs before prices moved, so your mileage may vary.

Verdict

Hardware hackers with rack space and a tolerance for BIOS archaeology will find a goldmine. If you are looking for a pip-installable framework or a cloud abstraction layer, look elsewhere.