← all repositories
tensorflow/ecosystem

TensorFlow's glue-code cookbook for Kubernetes, Spark, and Mesos

A grab-bag of minimal templates for wiring TensorFlow into cluster infrastructure you already have.

ecosystem
Velocity · 7d
+0.4
★ / day
Trend
steady
star history

What it does This repo is a collection of starter templates for running TensorFlow on other people’s systems: Kubernetes, Marathon/Mesos, Docker, Hadoop, and Spark. The README calls them “minimal” and “intended for use as templates” — copy, paste, and customize for your own deployment.

The interesting bit The Hadoop and Spark pieces are the most concrete: a TFRecord InputFormat/OutputFormat for MapReduce, a Spark-TensorFlow connector, and a Python distributor package for running distributed training on Spark clusters. The rest is mostly Jinja-templated YAML and Docker configs.

Key highlights

  • Spark-TensorFlow connector and spark-tensorflow-distributor for bridging PySpark and distributed TF training
  • TFRecord I/O formats for Hadoop MapReduce
  • Kubernetes and Marathon job templates (Jinja2-based expansion)
  • Docker configs for cluster managers
  • Kubeflow links out to a separate project for K8s-native ML workflows

Caveats

  • Most content is thin scaffolding, not production-ready automation
  • The distributed TF examples still use the older between-graph replication with parameter servers, not the newer tf.distribute strategies
  • Framework-specific dependencies are scattered across subdirectories with their own READMEs

Verdict Worth a bookmark if you’re stuck marrying legacy Spark/Hadoop infrastructure with TensorFlow. Skip it if you’re already on modern TF with TFX or Kubeflow — this won’t change your life.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.