← all repositories
PolyX-Research/Robust-U1

Fix the pixels first, then answer the question

Robust-U1 is a unified MLLM that restores corrupted images itself — then reasons over both the mess and the cleanup — rather than guessing through the noise.

★355 stars Python Language ModelsComputer Vision
Robust-U1
Not currently ranked.
star history

What it does Robust-U1 is a 7B unified multimodal model, fine-tuned from ByteDance’s BAGEL-7B-MoT, that handles degraded images by first generating a restored version of the input, then answering questions using both the corrupted original and its own recovery. It ships with a CLI demo (demo.py), a local Gradio app (app.py), an online demo on Hugging Face, and three public checkpoints: the Stage-I SFT model, the Stage-II RL model, and the final release.

The interesting bit Because the base model is unified — it both understands and generates images — restoration isn’t a preprocessing network bolted on front; the MLLM does its own cleanup before opining. The middle training stage is reinforcement learning (via Flow-GRPO) with rewards scoring both pixel-level fidelity and semantic consistency — the sample config weights a restoration reward at 1.0 against a TinyCLIP semantic reward at 0.2 — so recovery is optimized for what helps downstream reasoning, not just for pretty output.

Key highlights

  • Three-stage recipe: SFT to recover clean images from corrupted ones, RL to align recovery quality, then reasoning training over corrupted + recovered pairs.
  • The RL reward adapter is packaged in rewards/ and plugs straight into Flow-GRPO.
  • Includes a degradation pipeline (add_degradation/) that renders three corruption intensities per image — a reusable artifact for anyone benchmarking VLM robustness, paper or no paper.
  • Evaluation is wired through VLMEvalKit and R-Bench, the latter targeting real-world corruptions.
  • MIT licensed; accepted at ICML 2026.

Caveats

  • The repo is largely orchestration: training actually runs inside external frameworks (MathCanvas for SFT and reasoning, Flow-GRPO for RL), and evaluation inside VLMEvalKit and R-Bench. Reproducing the pipeline means cloning several repos and hand-editing dataset paths and configs.
  • No benchmark numbers appear in the README — for how much self-recovery actually buys you, you’re pointed to the paper.
  • Small but telling: the README’s own instructions run Stage-III training via stage2.sh.

Verdict Worth a look if you care about VLM robustness or unified understanding-generation models — the degradation pipeline alone earns a bookmark. Skip it if you want a self-contained training repo or a drop-in API.

Frequently asked

What is PolyX-Research/Robust-U1?
Robust-U1 is a unified MLLM that restores corrupted images itself — then reasons over both the mess and the cleanup — rather than guessing through the noise.
Is Robust-U1 open source?
Yes — PolyX-Research/Robust-U1 is an open-source project tracked on heatdrop.
What language is Robust-U1 written in?
PolyX-Research/Robust-U1 is primarily written in Python.
How popular is Robust-U1?
PolyX-Research/Robust-U1 has 355 stars on GitHub.
Where can I find Robust-U1?
PolyX-Research/Robust-U1 is on GitHub at https://github.com/PolyX-Research/Robust-U1.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.