uber/petastorm
A library enabling distributed training of deep learning models from Apache Parquet datasets with native TensorFlow, PyTorch, and PySpark support.
Not currently ranked — collecting fresh signals.
star history
Petastorm is a data access library developed at Uber ATG that enables single machine or distributed training and evaluation of deep learning models from datasets stored in Apache Parquet format. It provides native integration with popular ML frameworks including TensorFlow, PyTorch, and PySpark, allowing direct data loading into training pipelines. The library handles dataset generation, schema management, and efficient data sharding for distributed training workflows.
Frequently asked
- What is uber/petastorm?
- A library enabling distributed training of deep learning models from Apache Parquet datasets with native TensorFlow, PyTorch, and PySpark support.
- Is petastorm open source?
- Yes — uber/petastorm is open source, released under the Apache-2.0 license.
- What language is petastorm written in?
- uber/petastorm is primarily written in Python.
- How popular is petastorm?
- uber/petastorm has 1.9k stars on GitHub.
- Where can I find petastorm?
- uber/petastorm is on GitHub at https://github.com/uber/petastorm.