Spark AutoML that checks your types at compile time

What it does

TransmogrifAI is a Scala AutoML library that sits on Apache Spark and automates the grunt work of structured-data machine learning. It handles feature engineering, validation, and model selection through a pipeline DSL, letting you assemble reusable workflows without wiring DataFrame columns by hand. Salesforce originally built it to speed up internal model development and later open-sourced the toolkit.

The interesting bit

Most Spark ML tools treat your data as a bag of loose columns; TransmogrifAI treats features as strongly typed values and promises to catch wiring errors at compile time. You can call transmogrify() to let the library engineer features automatically, or override every stage manually when the magic feels too strong. It is an unusual attempt to bolt strict Scala typing onto a traditionally dynamic ML stack.

Key highlights

• Automates feature engineering, feature selection, and model selection across algorithms including Logistic Regression and Random Forest.
• Exposes a DSL where features are typed values rather than string-named DataFrame columns.
• Claims accuracy close to hand-tuned models with “almost 100x reduction in time,” per the project’s own estimates.
• Allows full manual specification of extractors and algorithms when automation is not wanted.
• Powers Salesforce Einstein predictions, according to the project’s talk references.

Caveats

• Stable release 0.7.0 targets Spark 2.4 and Scala 2.11; the version matrix lists no newer Spark or Scala releases.
• The “100x reduction in time” and accuracy comparisons are author-reported claims, not independently verified benchmarks.

Verdict

Worth evaluating if you are a Scala team running structured-data ML on Spark and want compile-time type safety in your pipelines. Skip it if you are all-in on Python or if the Spark 2.4 and Scala 2.11 baseline does not match your infrastructure.