Debug ML models by asking them uncomfortable questions

What it does

LIT is a browser-based workbench for dissecting ML predictions. You load a model and dataset, then inspect individual examples with salience maps, slice aggregate metrics across subgroups, and spin up counterfactuals to test whether the model means what it says. It runs as a standalone server or embeds directly inside Colab, Jupyter, and Vertex AI notebooks.

The interesting bit

Most interpretability tools print static reports; LIT treats debugging as a live interrogation. You can manually edit inputs or invoke generator plug-ins to create counterfactuals, compare two models side-by-side in the same pane, and explore embedding spaces without writing a new script. Both the frontend and backend are open to custom components, so the tool grows with your questions rather than limiting them.

Key highlights

• Handles text, image, and tabular data in the same interface.
• Framework-agnostic via a model wrapper API; TensorFlow and PyTorch are explicitly supported.
• Built-in counterfactual generation and side-by-side model comparison.
• Extensible architecture for custom metrics, generators, and UI panels.
• Runs standalone or inside notebooks without rewriting analysis code.

Caveats

• The repository does not ship a pre-built web application; compiling the TypeScript frontend is mandatory for source builds.
• Originally an NLP tool (the package is still lit-nlp), and the academic lineage shows in the citation papers.
• Explicitly a research project maintained by a small team, not an official Google product.

Verdict

Useful if you are an ML researcher or engineer who needs to manually probe failures and test counterfactual hypotheses. Not the right fit if you need automated training orchestration or unsupervised production monitoring.