SWE-Lancer moved out; this repo just forwards the mail
It tests whether frontier LLMs can earn real money on freelance software engineering gigs.

What it does
This repository originally housed the dataset and code for OpenAI’s SWE-Lancer paper, which asks whether frontier LLMs can earn $1 million from real-world freelance software engineering tasks. The README now states that the entire codebase has been merged into openai/preparedness, leaving this location as a redirect. If you are looking for the benchmark itself, it now lives elsewhere.
The interesting bit The premise reframes model evaluation as a gig-economy stress test: can a model invoice like a senior contractor rather than merely pass a unit test? The actual tasks, payouts, and results have moved out along with the code, so none of that detail is visible here anymore.
Key highlights
- Accompanied the paper “SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?”
- 1,440 stars suggest the community is curious about LLMs as billable labor.
- All code and data now live in the
openai/preparednessrepository. - This repo functions strictly as a forwarding address.
Caveats
- The README is a two-sentence redirect; no methodology, dataset details, or results remain at this URL.
- The repository currently contains no code, dataset, or runnable instructions.
Verdict
The active benchmark now lives in openai/preparedness. This repository is useful only as a historical citation stub; developers looking to experiment will find nothing to run.
Frequently asked
- What is openai/SWELancer-Benchmark?
- It tests whether frontier LLMs can earn real money on freelance software engineering gigs.
- Is SWELancer-Benchmark open source?
- Yes — openai/SWELancer-Benchmark is an open-source project tracked on heatdrop.
- How popular is SWELancer-Benchmark?
- openai/SWELancer-Benchmark has 1.4k stars on GitHub.
- Where can I find SWELancer-Benchmark?
- openai/SWELancer-Benchmark is on GitHub at https://github.com/openai/SWELancer-Benchmark.