Is factorio-learning-environment open source?

Yes — JackHopkins/factorio-learning-environment is an open-source project tracked on heatdrop.

What language is factorio-learning-environment written in?

JackHopkins/factorio-learning-environment is primarily written in Python.

How popular is factorio-learning-environment?

JackHopkins/factorio-learning-environment has 1k stars on GitHub.

Where can I find factorio-learning-environment?

JackHopkins/factorio-learning-environment is on GitHub at https://github.com/JackHopkins/factorio-learning-environment.

← all repositories

JackHopkins/factorio-learning-environment

Factorio as an LLM torture test

An open-ended benchmark where LLMs write Python to survive Factorio’s industrial complexity.

★1k stars Python LLMOps · Eval Agents

View on GitHub ↗ Homepage ↗

Not currently ranked — collecting fresh signals.

star history

What it does

FLE is an evaluation framework that drops LLM agents into Factorio, the notoriously intricate factory-building sim. Agents interact by writing Python programs that the environment executes in a REPL loop; they observe the world through the stdout and stderr of their own code, then generate the next program to react. The authors pitch it as a non-saturating benchmark—one that should stay challenging as models improve.

The interesting bit

Most game benchmarks eventually top out. Factorio’s combinatorial resource-management complexity makes that hard. The REPL interface is also unusually raw: the agent observes the world only through the output streams of its previous program, then must synthesize new Python to respond.

Key highlights

Open-ended evaluation designed not to saturate as frontier models improve
Agents interact via code synthesis through a REPL: they write Python, the environment executes it, and they read the resulting stdout/stderr
Built on the actual Factorio game (v2.0.73+), not a simplified clone
Ships with a CLI, leaderboard, and an arXiv paper for reproducible research
Optional extras for MCP protocol and PostgreSQL logging

Caveats

Requires owning Factorio for rendering, plus Docker and Python 3.10+
README is light on concrete task definitions or API surface details

Verdict

Worth a look if you are building LLM benchmarks or researching open-ended agent evaluation. If you just want a Factorio bot that works out of the box, this is a research framework, not a finished player.

Frequently asked

What is JackHopkins/factorio-learning-environment?: An open-ended benchmark where LLMs write Python to survive Factorio’s industrial complexity.
Is factorio-learning-environment open source?: Yes — JackHopkins/factorio-learning-environment is an open-source project tracked on heatdrop.
What language is factorio-learning-environment written in?: JackHopkins/factorio-learning-environment is primarily written in Python.
How popular is factorio-learning-environment?: JackHopkins/factorio-learning-environment has 1k stars on GitHub.
Where can I find factorio-learning-environment?: JackHopkins/factorio-learning-environment is on GitHub at https://github.com/JackHopkins/factorio-learning-environment.