Multi-layer Perceptron (MLP)
Multi-layer Perceptron (Feedforward Neural Network with Backpropagation) · Also known as: MLP, feedforward neural network, fully connected neural network, artificial neural network, dense neural network
The Multi-layer Perceptron (MLP) is a feedforward neural network architecture trained by backpropagation, formalised by Rumelhart, Hinton, and Williams in their landmark 1986 Nature paper. Composed of an input layer, one or more hidden layers of neurons with nonlinear activation functions, and an output layer, the MLP can approximate any continuous function to arbitrary accuracy and serves as the conceptual bridge between classical machine learning and modern deep learning.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
An MLP is appropriate when the relationship between inputs and outputs is complex and nonlinear, the dataset is large enough to support learning many parameters (typically at least several hundred labelled examples, and often far more for deeper networks), and predictive performance outweighs the need for coefficient-level interpretability. It suits classification (binary and multi-class) and regression on vectorised inputs — tabular data, extracted features, embeddings. It is not the right choice for raw images (prefer CNNs), sequences (prefer RNNs or Transformers), or small datasets where simpler models generalise better. Key assumptions are that training examples are independent and identically distributed, and that a labelled training set is available. Hyperparameter selection (depth, width, activation, learning rate, regularisation) is essential.
Strengths & limitations
- Universal approximation: an MLP with at least one hidden layer and a nonlinear activation can approximate any continuous function to arbitrary precision given sufficient neurons (Cybenko 1989; Hornik et al. 1989).
- Flexible output: the same architecture supports binary classification (sigmoid output, binary cross-entropy), multi-class classification (softmax output, categorical cross-entropy), and regression (linear output, mean squared error) with only a change in the output layer.
- Composable and extensible: serves as the backbone of nearly every deep learning architecture, making skills learned with MLPs transferable to CNNs, RNNs, and Transformers.
- Regularisation options are well understood: dropout, weight decay (L2), early stopping, and batch normalisation all reduce overfitting in established ways.
- Requires substantial labelled data: with small datasets the model overfits easily and simpler models such as logistic regression or random forests typically generalise better.
- Computationally intensive: training large networks demands significant time and (for very deep models) GPU resources.
- Black-box predictions: individual weights are not interpretable; SHAP or LIME must be layered on top to explain predictions.
- Sensitive to hyperparameters: depth, width, activation function, learning rate, and regularisation strength all interact, and poor choices can prevent convergence or cause severe overfitting.
- Prone to vanishing and exploding gradients in very deep networks, though modern activations (ReLU), initialisation schemes, and batch normalisation largely mitigate this.
Frequently asked
How many hidden layers and neurons should I use?
There is no universal answer. A single hidden layer with enough neurons can approximate any continuous function in theory, but deeper networks often generalise better in practice with fewer total parameters. A common starting point is one or two hidden layers with 64–256 neurons each; then increase capacity if underfitting or add regularisation if overfitting. Use a validation set to guide these choices.
Which activation function should I choose?
ReLU (rectified linear unit) is the default for hidden layers in most modern MLPs: it is computationally cheap, avoids the vanishing gradient problem for moderate depths, and works well in practice. Sigmoid and tanh are still used in specific contexts (e.g., output layers for binary classification or gated architectures). Leaky ReLU, ELU, and GELU are reasonable alternatives when neurons die with standard ReLU.
How do I know if my MLP is overfitting?
Monitor training loss and validation loss simultaneously. If validation loss stops decreasing while training loss keeps falling, the model is overfitting. Use early stopping to halt training at the point of best validation performance. Dropout and weight decay (L2 regularisation) are standard remedies.
When should I prefer an MLP over simpler models like logistic regression or random forest?
Prefer an MLP when the input-output relationship is strongly nonlinear and interactions between features are complex, and when you have enough data for the network to learn those patterns reliably. With small datasets or when interpretability is essential, logistic regression or a regularised linear model is often safer. Random forests frequently outperform MLPs on tabular data without careful tuning.
Sources
- Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. DOI: 10.1038/323533a0 ↗
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning (Ch. 6–7). MIT Press. ISBN: 978-0-262-03561-3
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning (Ch. 5). Springer. ISBN: 978-0-387-31073-2
How to cite this page
ScholarGate. (2026, June 3). Multi-layer Perceptron (Feedforward Neural Network with Backpropagation). ScholarGate. https://scholargate.app/en/machine-learning/multi-layer-perceptron
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Logistic RegressionResearch Statistics↔ compare
- Random ForestMachine learning↔ compare
- Recurrent Neural NetworkDeep learning↔ compare
- XGBoostMachine learning↔ compare