Recurrent Neural Network
Recurrent Neural Network (RNN) · Also known as: RNN, Elman network, Jordan network, simple recurrent network
A Recurrent Neural Network (RNN) is a class of neural network designed to process sequential data by maintaining a hidden state that carries information across time steps. Introduced in its modern form by Rumelhart et al. (1986) and further shaped by Elman (1990), RNNs became the dominant architecture for sequence modelling in NLP, speech, and time-series analysis before the rise of attention-based models.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+15 more
When to use it
Use an RNN when your data is sequential and order carries meaning — text, speech, sensor streams, financial time series, or biological sequences of moderate length (up to a few hundred steps). It is appropriate when you need a compact recurrent architecture or when downstream integration with LSTM/GRU variants is planned. Do not use a vanilla RNN for sequences longer than a few hundred steps: vanishing gradients make it unable to learn long-range dependencies, where LSTM or GRU are far more effective. For natural language understanding tasks in 2024+, pretrained transformers (BERT, RoBERTa) almost always outperform RNNs. Use RNNs mainly as a component in larger architectures or as a baseline for sequence tasks.
Strengths & limitations
- Naturally processes variable-length sequences without requiring padding to a fixed window size.
- Parameter sharing across time steps keeps model size manageable relative to input length.
- Forms the conceptual and practical foundation for LSTM, GRU, and many modern sequence models.
- Suitable as a lightweight baseline for sequence classification and regression before more expensive architectures.
- Can produce outputs at every time step, enabling many-to-many tasks such as sequence tagging and translation.
- Severe vanishing/exploding gradient problem for sequences longer than roughly 50–200 steps; LSTM or GRU should be preferred.
- Sequential computation cannot be parallelised across time steps, making training slow on long sequences compared to convolutional or transformer architectures.
- Substantially outperformed on most NLP benchmarks by pretrained transformer models such as BERT and RoBERTa.
- Limited memory capacity: the fixed-dimensional hidden state struggles to compress long-range context.
Frequently asked
What is the difference between an RNN, an LSTM, and a GRU?
An RNN is the simplest recurrent architecture: it has a single hidden state updated at each step. LSTM adds input, forget, and output gates that protect long-range information from vanishing gradients. GRU is a streamlined version of LSTM with fewer parameters. Both LSTM and GRU almost always outperform a vanilla RNN on tasks requiring memory beyond ~50–100 steps.
Why does training an RNN sometimes produce NaN losses?
NaN losses almost always result from exploding gradients. Apply gradient clipping (clip norm to 1 or 5) and lower the learning rate. Ensure inputs are normalised. If the problem persists, switch to LSTM or GRU, which are more stable due to their gating mechanisms.
Can I still use an RNN for NLP tasks in 2024?
For production NLP, pretrained transformers such as BERT or RoBERTa dominate. Vanilla RNNs are best reserved as lightweight baselines, components inside larger architectures, or for resource-constrained environments where large model sizes are prohibitive.
How do I choose the hidden state dimension?
Common choices range from 64 to 512 units depending on task complexity and dataset size. Start small (e.g., 128), monitor validation loss, and increase capacity only if underfitting is evident. Larger hidden states increase the risk of overfitting on small datasets.
What loss function should I use for sequence classification?
For binary classification use binary cross-entropy; for multi-class use categorical cross-entropy with softmax on the final hidden state. For sequence labelling (one output per step), apply the loss at every time step and average or sum across steps.
Sources
- Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211. DOI: 10.1207/s15516709cog1402_1 ↗
- Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. DOI: 10.1038/323533a0 ↗
How to cite this page
ScholarGate. (2026, June 3). Recurrent Neural Network (RNN). ScholarGate. https://scholargate.app/en/deep-learning/recurrent-neural-network
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- BERT-based ClassificationDeep learning↔ compare
- Gated Recurrent UnitDeep learning↔ compare
- Long Short-Term MemoryDeep learning↔ compare