Backpropagation and Optimization

Backpropagation efficiently computes the gradient of a network's loss with respect to its weights, and gradient-based optimizers use that gradient to train the network.

Definition

Backpropagation is an algorithm that computes the gradient of a loss function with respect to every weight in a neural network by propagating error signals backward through the layers using the chain rule; optimization then updates the weights, typically with stochastic gradient descent, to reduce the loss.

Scope

This topic covers how deep networks are trained: the backpropagation algorithm as an application of the chain rule to compute gradients layer by layer, stochastic gradient descent and its mini-batch form, momentum and adaptive learning-rate methods, and the practical challenges of vanishing and exploding gradients, learning-rate selection, and convergence on nonconvex loss surfaces.

Core questions

• How does backpropagation compute gradients efficiently?
• Why is stochastic gradient descent preferred for large datasets?
• How do momentum and adaptive methods accelerate training?
• What causes vanishing or exploding gradients and how are they mitigated?

Key theories

Backpropagation via the chain rule

By applying the chain rule from the output backward, the algorithm reuses intermediate results to compute all weight gradients in time proportional to the forward pass, making training of large networks feasible.

Stochastic gradient descent

Estimating the gradient from small random batches makes each update cheap and introduces helpful noise, enabling training on very large datasets and often improving generalization.

Adaptive and momentum methods

Momentum accumulates past gradients to smooth the descent, and adaptive methods scale the step size per parameter, both speeding convergence on the ill-conditioned loss surfaces typical of deep networks.

Clinical relevance

Backpropagation with stochastic gradient descent is the engine behind essentially all modern deep learning; understanding how gradients flow explains both why depth was historically hard to train and how innovations in activations, initialization, and optimizers made very deep networks practical.

History

Backpropagation was derived in several contexts, including Werbos's 1974 thesis, and was brought to prominence by Rumelhart, Hinton, and Williams in 1986. Stochastic gradient descent and later momentum and adaptive-learning-rate optimizers became the standard training procedures, and addressing vanishing gradients was key to training deep and recurrent networks.

Key figures

• David Rumelhart
• Geoffrey Hinton
• Ronald Williams
• Paul Werbos

Seminal works

• rumelhart1986
• goodfellow2016
• bishop2006

Frequently asked questions

What does backpropagation actually compute?

It computes the gradient of the loss with respect to every weight, that is, how much each weight should change to reduce the error. It does so efficiently by propagating error signals backward from the output layer to the input layer using the chain rule.

Why train on small batches instead of all the data at once?

Using the whole dataset for each update is expensive and unnecessary. Mini-batch stochastic gradient descent estimates the gradient from a small random sample, making each step cheap, allowing many more updates, and adding noise that can help escape poor solutions.