Rebuilding real robot tasks in sim, with an agent holding the wrench

What it does RoboSimStudio is an open toolkit for Real2Sim: reconstructing real robot manipulation tasks as interactive physics simulations. You can build environments manually, or delegate to a coding agent like Codex or Claude Code by pointing it at a task instruction, a robot URDF, and a video or demonstration. It accepts human video, ego-centric video, robot video, and teleoperation data as input.

The interesting bit The agent-driven workflow is the hook — the README even recommends different agents for different jobs (Codex Astra for scene construction, Claude Code for robot motion) and suggests keeping a human in the loop to nudge the agent on physics and geometry. It’s built on a serious physics stack: Newton, Warp, MuJoCo Warp, with robot models from i2rt and robosuite.

Key highlights

• Runs on CPU-only machines, with a recent release rendering 500 demos in 4.2 minutes on a single GPU.
• Fifteen material types supported, including rope, knots, paper, bags, tape, cutting, granular, liquid, dough, and water — each with a worked example.
• Twenty example manipulation tasks ship in the repo’s demo collection.
• Apache 2.0 licensed, with a tech report on arXiv and a CoRL 2026 workshop oral.

Caveats

• The agent workflow still needs a human occasionally reminding it to reconstruct physics and geometry correctly — this is supervised automation, not fire-and-forget.
• The README doesn’t quantify how faithful the reconstructions are to the real tasks; you’ll need the tech report for that.

Verdict Worth a look if you do sim-to-real or real-to-sim research and want a fast path from demonstration footage to a physics environment. If you just want a general-purpose robot simulator, Isaac Lab or Genesis are the more established homes.