cybertronai/gradient-checkpointing
A TensorFlow library that reduces neural network training memory usage from O(n) to O(sqrt(n)) by checkpointing computation graph nodes and recomputing intermediate values during backpropagation.
Not currently ranked — collecting fresh signals.
star history
This repository provides a memory optimization technique for training deep neural networks in TensorFlow. It works by strategically placing checkpoints in the computation graph and recomputing intermediate activations during backpropagation instead of storing them all in memory. The implementation uses the TensorFlow graph editor to automatically rewrite the backward pass computation graph. For feed-forward networks, this allows fitting models more than 10x larger onto a GPU with approximately 20% additional computation time.
Frequently asked
- What is cybertronai/gradient-checkpointing?
- A TensorFlow library that reduces neural network training memory usage from O(n) to O(sqrt(n)) by checkpointing computation graph nodes and recomputing intermediate values during backpropagation.
- Is gradient-checkpointing open source?
- Yes — cybertronai/gradient-checkpointing is open source, released under the MIT license.
- What language is gradient-checkpointing written in?
- cybertronai/gradient-checkpointing is primarily written in Python.
- How popular is gradient-checkpointing?
- cybertronai/gradient-checkpointing has 2.8k stars on GitHub.
- Where can I find gradient-checkpointing?
- cybertronai/gradient-checkpointing is on GitHub at https://github.com/cybertronai/gradient-checkpointing.