← all repositories
huggingface/Repo2RLEnv

Pull requests in, verifiable RL environments out

Hand-building a single verifiable RL task takes hours, so this mines thousands from material repositories already have — merged PRs, commit history, security advisories.

Repo2RLEnv
Not currently ranked.
star history

What it does Repo2RLEnv turns an existing repository into training and evaluation tasks for coding agents. Twenty-three generators work from different raw material — merged pull requests, commit history, security advisories, a library’s own functions, terminal recordings — and each emits a standard Harbor task: a directory with an instruction, a starting container, a private verifier, and a reference solution that stays hidden until grading. Tasks run with any Harbor agent, and publishing to or pulling from the Hugging Face Hub is built in; the project already ships 21 datasets totaling 1,930 tasks. The simplest pipeline needs nothing exotic — no Docker, no LLM key, just Python 3.12+ and the GitHub CLI.

The interesting bit Tasksmith is an agent that builds environments for other agents. Point it at a merged PR and it investigates the change, bootstraps the repo on a remote worker, writes the instruction plus a private verifier, then reviews and repairs its own work until the controls pass — the reference solution must score 1.0 and a no-op must score 0. That oracle/no-op pair is the unglamorous heart of the whole project: it mechanically catches tasks that are trivially passable or quietly impossible, which is exactly where hand-built RL environments tend to fail.

Key highlights

  • Five task families: repository repair (scored by the repo’s own tests — failing ones must pass, passing ones must stay green), implementation and reconstruction, patch similarity, terminal state tasks, and reasoning/optimization with no repository at all
  • pr_diff scores similarity to the merged diff with an optional LLM judge — the fallback for repos with no usable test suite
  • FrontierSmith recasts closed-ended problems as optimization tasks with continuous rewards in [0, 1], where no optimum is known and a better solution simply earns more
  • Fifteen research recipes (SWE-smith, R2E-Gym, SWE-gen, SETA, SCALER) reimplemented on one shared execution and review layer
  • Tasksmith’s reference cohort, HF_ML_Tasksmith, holds 50 verified tasks mined from Accelerate, Diffusers, PEFT, Transformers and TRL

Caveats

  • The README says it plainly: “a generated task is a starting point, not a guarantee.” Evaluation labels and the review loop record how far each task was checked — read them before training on a dataset
  • Only the six native pipelines run locally with Docker; Tasksmith and the research recipes need a Modal or Daytona account and execute their target code on remote workers
  • Agent-driven generation isn’t free: Tasksmith records the evidence and cost of every stage, and the docs publish per-dataset yield and cost figures

Verdict If you’re training or benchmarking coding agents and need verifiable tasks at RL scale, this is the factory — and the published datasets are a free head start. If you only trust hand-verified, SWE-bench-style tasks, treat the output as raw ore and check the labels.

Frequently asked

What is huggingface/Repo2RLEnv?
Hand-building a single verifiable RL task takes hours, so this mines thousands from material repositories already have — merged PRs, commit history, security advisories.
Is Repo2RLEnv open source?
Yes — huggingface/Repo2RLEnv is open source, released under the Apache-2.0 license.
What language is Repo2RLEnv written in?
huggingface/Repo2RLEnv is primarily written in Python.
How popular is Repo2RLEnv?
huggingface/Repo2RLEnv has 802 stars on GitHub.
Where can I find Repo2RLEnv?
huggingface/Repo2RLEnv is on GitHub at https://github.com/huggingface/Repo2RLEnv.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.