Q-Learning
Also known as: Q-learning algorithm, tabular Q-learning, off-policy TD control, Q-öğrenme
Q-learning, introduced by Christopher Watkins and Peter Dayan in 1992, is a model-free reinforcement-learning algorithm that learns the value of taking each action in each state — the Q-function — purely from experience, without a model of the environment. It is off-policy: it learns the optimal action-values while following an exploratory behaviour policy, and under standard conditions it provably converges to the optimal policy.
Key highlights
- Model-free: needs no transition/reward model, only sampled experience.
- Off-policy: learns the optimal policy while exploring with another policy.
- Provably converges to the optimal action-values under standard conditions.
- Simple, foundational, and the basis for Deep Q-Networks and many extensions.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use Q-learning for sequential decision problems framed as a Markov decision process where you can sample transitions and rewards but have no (or an unreliable) model of the dynamics — control, games, routing, scheduling, and adaptive systems. Tabular Q-learning suits small, discrete state-action spaces and converges to the optimum given sufficient exploration and a decaying step size. For large or continuous spaces the table is replaced by a function approximator (Deep Q-Networks). It assumes the Markov property and a stationary environment, can be sample-inefficient, and the max operator induces an optimistic (maximization) bias addressed by variants like Double Q-learning. When a stochastic or continuous-action policy is needed, policy-gradient methods are the alternative.
Strengths & limitations
- Model-free: needs no transition/reward model, only sampled experience.
- Off-policy: learns the optimal policy while exploring with another policy.
- Provably converges to the optimal action-values under standard conditions.
- Simple, foundational, and the basis for Deep Q-Networks and many extensions.
- Tabular form does not scale to large or continuous state/action spaces.
- Sample-inefficient; can need many episodes to converge.
- Maximization bias (overestimation) from the max operator; mitigated by Double Q-learning.
- Assumes a (stationary) Markov decision process; struggles under partial observability or drift.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
What does 'off-policy' mean in Q-learning?
It means Q-learning learns the value of the optimal policy while the agent follows a different (exploratory) behaviour policy. The update targets r + γ·max Q(s', a') — the best next action — not the action actually taken, so exploration does not bias the learned optimum.
How does Q-learning relate to Deep Q-Networks (DQN)?
DQN replaces the Q-table with a deep neural network that approximates Q(s, a), enabling Q-learning in large or continuous (e.g., pixel) state spaces. It adds stabilizing tricks — experience replay and a target network — but the core learning rule is Q-learning's temporal-difference update.
Why does Q-learning overestimate values?
The max operator in the update tends to select over-optimistic estimates, biasing Q upward, especially with noisy rewards. Double Q-learning reduces this by decoupling action selection from value evaluation using two estimators, yielding less biased and often better policies.
Sources
- 1.Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3–4), 279–292.
- 2.Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.ISBN 978-0-262-03924-6
You have read it. What now?
Cite this page
ScholarGate. (2026, June 2). Q-Learning. ScholarGate. https://scholargate.app/machine-learning/q-learning