An open-source MoE code model claiming GPT-4 Turbo parity

What it does

DeepSeek-Coder-V2 is a family of open-source code language models built on a Mixture-of-Experts architecture. It offers two sizes—a 16B-parameter Lite variant and a 236B flagship—both handling 128K context and 338 programming languages. The project publishes base and instruct checkpoints along with benchmark results positioning the largest model as competitive with GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro on coding and mathematical reasoning tasks.

The interesting bit

The models are not trained from scratch but continued from an intermediate checkpoint of DeepSeek-V2 on an additional 6 trillion tokens, effectively specializing a generalist into a code-and-math heavyweight. The MoE design keeps active parameters modest—2.4B for Lite, 21B for the 236B model—while still claiming top-tier benchmark scores.

Key highlights

• Benchmark tables claim the 236B instruct model matches or beats GPT-4 Turbo and Claude 3 Opus on HumanEval, MBPP+, LiveCodeBench, MATH, and AIME 2024
• Expands language support from 86 to 338 programming languages and context length from 16K to 128K tokens
• Released in base and instruct variants on HuggingFace under a dual license: MIT for code, custom Model Agreement for weights
• Lite version (16B total / 2.4B active) offers a smaller entry point with the same 128K context window

Caveats

• The visible repository content centers on model release and evaluation rather than training infrastructure or novel architecture code.
• The 236B model demands substantial inference hardware despite only 21B active parameters.
• Model weights are governed by a custom “Model Agreement” rather than a standard open-source license.

Verdict

Worth a look if you need an open-weights coding model with long context and broad language support, but only if you have the GPUs—or budget for the API—to handle its scale. Pass if you’re hoping to hack on the training infrastructure itself.