Dropout
Dropout Regularization for Deep Neural Networks · Also known as: dropout regularization, stochastic dropout, neuron dropout, inverted dropout
Dropout is a stochastic regularization technique for training deep neural networks, introduced by Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov in 2014. During each training step, each neuron is independently switched off with probability (1 − p), preventing the network from co-adapting its units too tightly and thereby reducing overfitting.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Dropout is appropriate for any moderately large feedforward, convolutional, or recurrent neural network where overfitting is observed or anticipated. It is most effective when the network is sufficiently wide and the dataset is not extremely large relative to model capacity. Standard practice is to apply dropout after fully connected (dense) layers with p = 0.5, and optionally after convolutional layers with a higher retention rate (p = 0.8–0.9). Dropout is less effective — sometimes counterproductive — when combined with batch normalisation in the same layer, because the two regularisers interact in ways that can hurt performance; they should be used together only with care. For very small datasets, simpler regularisation such as L2 weight decay or early stopping may suffice.
Strengths & limitations
- Highly effective regulariser: dramatically reduces overfitting in large fully connected networks with minimal implementation cost.
- Implicit ensemble: approximates averaging over an exponential number of model architectures at the cost of a single model.
- Architecture-agnostic: applicable to feedforward, convolutional, recurrent, and transformer networks.
- No additional parameters: the only hyperparameter is the retention probability p, which is robust to reasonable choices around 0.5.
- Encourages feature robustness by forcing neurons to learn redundant representations that do not depend on specific co-adapted neighbours.
- Increases training time because a larger number of epochs is typically required to reach the same training loss.
- Not universally beneficial: shallow networks, very small datasets, and networks already well-regularised by batch normalisation benefit little or may be hurt.
- The interaction with batch normalisation requires care; naively combining them can degrade performance.
- Choosing p requires cross-validation, and an inappropriate rate (too low or too high) can either under-regularise or impair convergence.
- Recurrent neural networks require specialised dropout variants (e.g., variational dropout) because standard per-step masking disrupts temporal dependencies.
Frequently asked
What retention probability p should I use?
The seminal paper recommends p = 0.5 for hidden fully connected layers and p = 0.8 for the input layer as a default starting point. These values should be tuned via cross-validation; wider layers may tolerate lower p while thinner layers may need p closer to 0.8 to preserve sufficient capacity.
Should I disable dropout when evaluating the model?
Yes. Dropout must be switched off during validation and test inference. Most frameworks (PyTorch, Keras, TensorFlow) handle this automatically when you call model.eval() or set training=False, but it is essential to confirm this is set correctly, as leaving dropout active at inference time introduces noise and degrades predictions.
Does dropout work with batch normalisation?
Combining both in the same layer is problematic. Batch normalisation already acts as a regulariser and its statistics depend on the batch distribution; dropout changes the variance of the inputs seen by the batch norm layer, disrupting its normalisation. If both are used, place batch normalisation before dropout and monitor performance carefully.
How is inverted dropout different from the original formulation?
In the original formulation the weights are scaled by p at test time. Inverted dropout divides the retained activations by p during training, so the expected magnitude is preserved in each forward pass and no rescaling is needed at test time. Inverted dropout is the standard implementation in modern frameworks because it keeps the inference path clean.
Sources
- Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15, 1929–1958. link ↗
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning (Ch. 7: Regularization for Deep Learning). MIT Press. ISBN: 978-0-262-03561-3
How to cite this page
ScholarGate. (2026, June 3). Dropout Regularization for Deep Neural Networks. ScholarGate. https://scholargate.app/en/deep-learning/dropout
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Batch NormalizationDeep learning↔ compare