Restricted Boltzmann Machine (RBM)
Restricted Boltzmann Machine (RBM) — Bipartite Generative Energy Model · Also known as: RBM, Harmonium, restricted Boltzmann machine, RBM generative model, stochastic recurrent neural network
A Restricted Boltzmann Machine is a two-layer generative probabilistic model consisting of visible (observed) and hidden (latent) binary units connected by an undirected bipartite graph with no within-layer connections. Originally introduced as the 'Harmonium' by Paul Smolensky in 1986 and powerfully revived by Geoffrey Hinton and Ruslan Salakhutdinov in their landmark 2006 Science paper, RBMs became historically pivotal as the building block for greedy layer-wise pre-training of Deep Belief Networks, restarting interest in deep neural networks after years of stagnation.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
An RBM is appropriate when the goal is unsupervised representation learning from binary or real-valued data, generative modelling of a data distribution, or collaborative filtering (as in the Netflix Prize work). It is particularly suited for use as a building block in a Deep Belief Network, where greedy layer-wise RBM pre-training provides good weight initialisation before supervised fine-tuning. Inputs are assumed exchangeable; the model does not model sequential structure directly. For modern generative tasks, Variational Autoencoders or diffusion models are generally preferred; RBMs remain relevant in physics-inspired machine learning, quantum computing simulation, and as a teaching vehicle for energy-based models.
Strengths & limitations
- Tractable block Gibbs sampling due to the bipartite structure: all hidden (or visible) units can be sampled simultaneously given the other layer.
- Contrastive Divergence provides an efficient, if approximate, learning algorithm that scales to large datasets.
- Demonstrated ability to extract meaningful latent representations that generalise well, as shown in Hinton and Salakhutdinov's dimensionality reduction experiments.
- Serves as the fundamental building block of Deep Belief Networks, enabling greedy layer-wise pre-training of deep architectures.
- Naturally probabilistic: the model provides calibrated probabilities over configurations rather than point estimates.
- The partition function Z is intractable for large networks, making exact log-likelihood evaluation and exact gradient computation impossible.
- Contrastive Divergence is a biased estimator of the true gradient; CD-1 in particular does not converge to the true maximum-likelihood solution.
- Training can be sensitive to the choice of learning rate, number of hidden units, and number of CD steps.
- Binary visible units are the canonical formulation; extensions to Gaussian or other real-valued units exist but introduce additional complexity.
- Superseded for most modern generative and representation-learning tasks by Variational Autoencoders, GANs, and diffusion models.
Frequently asked
What makes an RBM 'restricted'?
The restriction is the absence of connections within the visible layer and within the hidden layer. Only connections between the two layers are allowed, forming a bipartite graph. This restriction makes inference tractable: all hidden units are conditionally independent given the visible units, and vice versa, enabling simultaneous block sampling.
How does Contrastive Divergence work and why is it approximate?
CD-k starts from a training data point, runs k steps of alternating Gibbs sampling, and uses the resulting sample as a stand-in for an exact sample from the model distribution. The exact gradient of the log-likelihood requires expectations under the true model distribution, which demands running the chain to convergence (many steps). CD-k with small k is a biased approximation, but it is computationally cheap and works well in practice.
Can RBMs handle continuous data?
The standard RBM uses binary visible and hidden units. Gaussian-visible RBMs replace the visible sigmoid with a Gaussian noise model, enabling real-valued inputs such as image pixel intensities or spectral features. The hidden units typically remain binary. Training Gaussian-visible RBMs can be less stable than the binary case and requires careful normalisation of inputs.
Are RBMs still used in modern deep learning?
Rarely as a front-line method for generation or representation learning, where VAEs, GANs, and diffusion models now dominate. However, RBMs retain active research use in physics-inspired machine learning (lattice models, phase transitions), as neural network quantum states in quantum chemistry, and as a pedagogically clear example of an energy-based probabilistic model.
Sources
- Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786), 504–507. DOI: 10.1126/science.1127647 ↗
- Hinton, G. E. (2002). Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14(8), 1771–1800. DOI: 10.1162/089976602760128018 ↗
- Smolensky, P. (1986). Information Processing in Dynamical Systems: Foundations of Harmony Theory. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel Distributed Processing, Vol. 1 (pp. 194–281). MIT Press. ISBN: 978-0-262-68053-0
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning (Ch. 20). MIT Press. ISBN: 978-0-262-03561-3
How to cite this page
ScholarGate. (2026, June 3). Restricted Boltzmann Machine (RBM) — Bipartite Generative Energy Model. ScholarGate. https://scholargate.app/en/deep-learning/restricted-boltzmann-machine
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- AutoencoderDeep learning↔ compare
- Deep Belief NetworkDeep learning↔ compare
- Variational AutoencoderDeep learning↔ compare