🔷 Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.

Matrix Minds: RNNs Level Up

The Transformer architecture, the bedrock of AI since 2017, is starting to show its limitations. While undeniably powerful, its reliance on self-attention mechanisms leads to a critical bottleneck: quadratic scaling of computational resources as the input sequence length increases. This means that processing longer pieces of text, code, or any sequential data requires much more compute power, energy, and time.

Enter Recurrent Neural Networks (RNNs), an older architecture that had fallen out of favor due to challenges in training and handling long sequences. RNNs are in competition with state space models like Mamba, Flash Attention, and Infini-Attention for the post-Transformer crown. Two notable advancements in RNN architectures are RWKV’s Eagle/Finch and Google’s Hawk/Griffin. Both approaches tackle the efficiency and performance issues that previously hindered RNNs:

• Tackling the Efficiency Bottleneck: RNNs used to have a sequential bottleneck, ie you needed the results of computation from one token to begin computation of the next. Both new architectures employ clever techniques to parallelize computations and optimize memory usage.

  • Eagle/Finch utilizes custom CUDA implementations and explores parallelization methods like associative scan.

  • Hawk/Griffin leverages model parallelism and custom kernels for efficient training on TPUs. These optimizations significantly improve training and inference speeds, making RNNs competitive with Transformers.

• Conquering Long Sequences: The Achilles' heel of traditional RNNs is the vanishing gradient problem. Succinctly, RNNs lose visibility of the first word as they get deep into a sequence of text. The new architectures address this through certain innovations.

  • Eagle/Finch uses a limited time-step window for their recurrence mechanism, enabling handling of unbounded sequence lengths.

  • Hawk/Griffin employ fixed-size hidden states and local attention, avoiding the quadratic memory growth of Transformers.

• Additionally, both architectures demonstrate a remarkable ability to extrapolate to longer sequences than they were trained on.

These advancements have propelled RNNs back into the spotlight, demonstrating their potential for various applications:

• Long Document Processing: Analyzing lengthy texts, legal documents, or codebases becomes feasible with RNNs' efficient handling of long sequences.

• Real-Time Applications: RNNs' lower latency and higher throughput during inference make them suitable for real-time tasks like chatbots and live translation.

• Resource-Constrained Environments: RNNs' lower computational requirements make them attractive for deployment on edge devices or in situations with limited resources.

I’ve been working with Recursal AI, the corporate entity founded to champion the RWKV architecture, and helping to shepherd the team through the early stages of product market fit. These posts are my effort to get up to speed on what they’ve been up to.

Become a subscriber for daily breakdowns of what’s happening in the AI world: