← all repositories
openai/SWELancer-Benchmark

SWE-Lancer moved out; this repo just forwards the mail

It tests whether frontier LLMs can earn real money on freelance software engineering gigs.

SWELancer-Benchmark
Not currently ranked — collecting fresh signals.
star history

What it does This repository originally housed the dataset and code for OpenAI’s SWE-Lancer paper, which asks whether frontier LLMs can earn $1 million from real-world freelance software engineering tasks. The README now states that the entire codebase has been merged into openai/preparedness, leaving this location as a redirect. If you are looking for the benchmark itself, it now lives elsewhere.

The interesting bit The premise reframes model evaluation as a gig-economy stress test: can a model invoice like a senior contractor rather than merely pass a unit test? The actual tasks, payouts, and results have moved out along with the code, so none of that detail is visible here anymore.

Key highlights

  • Accompanied the paper “SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?”
  • 1,440 stars suggest the community is curious about LLMs as billable labor.
  • All code and data now live in the openai/preparedness repository.
  • This repo functions strictly as a forwarding address.

Caveats

  • The README is a two-sentence redirect; no methodology, dataset details, or results remain at this URL.
  • The repository currently contains no code, dataset, or runnable instructions.

Verdict The active benchmark now lives in openai/preparedness. This repository is useful only as a historical citation stub; developers looking to experiment will find nothing to run.

Frequently asked

What is openai/SWELancer-Benchmark?
It tests whether frontier LLMs can earn real money on freelance software engineering gigs.
Is SWELancer-Benchmark open source?
Yes — openai/SWELancer-Benchmark is an open-source project tracked on heatdrop.
How popular is SWELancer-Benchmark?
openai/SWELancer-Benchmark has 1.4k stars on GitHub.
Where can I find SWELancer-Benchmark?
openai/SWELancer-Benchmark is on GitHub at https://github.com/openai/SWELancer-Benchmark.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.