Bayesian Optimization — Sequential Model-Based Hyperparameter Tuning
Bayesian Optimization (Hyperparameter Tuning) · Also known as: Bayesçi Optimizasyon (Hyperparameter Tuning), surrogate-based optimization, sequential model-based optimization, SMBO
Bayesian Optimization is a sequential, model-based strategy for finding the optimum of expensive black-box functions with as few evaluations as possible. Rooted in the work of Mockus (1975) and brought to mainstream machine-learning practice by Snoek, Larochelle, and Adams (2012), it fits a probabilistic surrogate model — typically a Gaussian Process — to past observations and uses an acquisition function to decide where to probe next, balancing exploration of unknown regions with exploitation of promising ones.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+9 more
When to use it
Bayesian Optimization is the right choice when the objective function is expensive or slow to evaluate — training a large model, running a physical simulation, or conducting a laboratory experiment — and the budget is limited to tens or low hundreds of evaluations. The search space must be defined with explicit bounds for each parameter. The method handles continuous and ordinal parameters naturally; purely categorical or very high-dimensional spaces (more than roughly 20 parameters) can degrade surrogate quality and may call for extensions such as random forests as the surrogate or dimensionality-reduction preprocessing. If evaluations are cheap and parallelism is available, simpler random search or grid search may be competitive.
Strengths & limitations
- Sample-efficient: finds good configurations in far fewer evaluations than grid search or random search, which matters when each evaluation is slow or costly.
- Principled exploration–exploitation balance: the acquisition function provides a formal, interpretable criterion for where to search next.
- Model-based uncertainty: the Gaussian Process not only predicts performance but also quantifies its own confidence, enabling informed stopping decisions.
- Applicable to any black-box objective: the method requires no gradient information and no mathematical knowledge of the objective's interior.
- Scales poorly to very high-dimensional hyperparameter spaces: Gaussian Process fitting becomes computationally expensive and the surrogate loses accuracy beyond roughly 10–20 dimensions.
- Sensitive to the choice of surrogate kernel and acquisition function: poor defaults can impede convergence, especially on non-stationary or multi-modal objectives.
- Sequential by default: each evaluation must complete before the next candidate is selected, limiting parallelism unless batch-mode or asynchronous extensions are used.
- Requires bounded search spaces: parameters without explicit ranges cannot be handled directly.
Frequently asked
How is Bayesian Optimization different from grid search or random search?
Grid search evaluates a fixed lattice of configurations and random search samples uniformly at random — both ignore the results of previous trials when choosing the next point. Bayesian Optimization actively uses all past observations to build a surrogate model and selects the next point most likely to improve over the current best, making it far more sample-efficient when evaluations are expensive.
What acquisition function should I use?
Expected Improvement (EI) is the most widely used default: it measures the expected gain over the current best and tends to be reliable across problem types. Upper Confidence Bound (UCB) adds a tunable exploration bonus and is often preferred when the landscape is poorly understood. Probability of Improvement (PI) is more exploitation-heavy and can converge prematurely. Start with EI and switch to UCB if the optimizer appears to stagnate.
How many iterations are needed?
There is no universal rule, but fewer than about ten total evaluations (including initial random probes) give the surrogate too little data to be reliable. Practical deployments typically use five to ten initial random points followed by twenty to fifty sequential Bayesian steps. When the best-seen score stops improving over several consecutive iterations, the budget can be stopped.
Can Bayesian Optimization be parallelised?
Standard Bayesian Optimization is inherently sequential. Batch-mode extensions allow proposing a set of candidates simultaneously — common strategies include Local Penalization and the Kriging Believer heuristic — enabling parallel evaluation. Asynchronous variants launch a new candidate whenever a worker becomes free rather than waiting for the full batch. These extensions are available in libraries such as Optuna and BoTorch.
Sources
How to cite this page
ScholarGate. (2026, June 1). Bayesian Optimization (Hyperparameter Tuning). ScholarGate. https://scholargate.app/en/optimization/bayesian-optimization
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Neural Architecture SearchDeep learning↔ compare
- Stochastic OptimizationOptimization↔ compare