An RNN that trains as fast as a CNN

What it does SRU is a PyTorch recurrent unit pitched as a faster replacement for nn.LSTM. The README claims it runs 10 to 16 times faster than cuDNN LSTM on a GTX 1070 while keeping accuracy on “many tasks,” and can match or beat word-level conv2d throughput. The repo also hosts SRU++, a follow-up that bolts attention onto the same fast-recurrence backbone for language-model training. You use it like nn.LSTM, though the docs suggest stacking more layers than usual.

The interesting bit The trick is “highly parallelizable recurrence”—the paper and README frame it as stripping away the timestep-by-timestep sequential dependency that normally throttles RNN throughput. The authors don’t pretend it’s free lunch; they note you’ll probably need a deeper stack than an LSTM, which is a refreshingly candid caveat.

Key highlights

• Claims 10–16× speedup over cuDNN LSTM on a GTX 1070, with no accuracy loss on “many tasks.”
• Drop-in API similar to nn.LSTM, supporting bidirectional mode, layer norm, and highway gating.
• SRU++ variant layers attention on top of fast recurrence for language modeling with reduced compute.
• Ships with custom CUDA kernels and requires ninja plus PyTorch >=1.6.

Caveats

• The README warns that SRU “likely requires more stacking layers than LSTM” to reach equivalent capacity.
• SRU++ and its experiments currently live on a dev branch, not yet merged to master.
• The “no loss of accuracy” claim is stated broadly across “many tasks,” but the README offers no per-task breakdown.

Verdict Try it if you still need recurrent inductive biases but want to stop paying the LSTM latency tax. Skip it if your architecture is already all-attention.