Gap Statistic
Also known as: gap index, Tibshirani gap statistic
The Gap Statistic, developed by Tibshirani, Walther, and Hastie in 2001, is a principled statistical method for determining the optimal number of clusters in a dataset. It compares the observed within-cluster sum of squares to the expected value under a null hypothesis of no clustering structure, providing a theoretically grounded approach to cluster number selection.
Key highlights
- Theoretically grounded in statistical inference
- More objective than the Elbow Method; produces a numerical criterion
- Works well when clustering structure is moderate to strong
- Applicable to any clustering algorithm and distance metric
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use the Gap Statistic when you want a statistically principled method for selecting cluster count that accounts for the null hypothesis of no structure. It works better than the Elbow Method when the true elbow is ambiguous. However, it is computationally more expensive due to reference dataset generation and assumes uniform null distribution, which may not hold for all data types.
Strengths & limitations
- Theoretically grounded in statistical inference
- More objective than the Elbow Method; produces a numerical criterion
- Works well when clustering structure is moderate to strong
- Applicable to any clustering algorithm and distance metric
- Computationally expensive; requires generating and clustering many reference datasets
- Assumes uniform null distribution, which may not suit all data
- Can be sensitive to the number of reference datasets generated
- May perform poorly when true clusters are non-convex or have very different sizes
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How many reference datasets should I generate?
Typically 100 to 500 reference datasets are sufficient. More datasets provide more stable variance estimates but increase computation. Start with 100 and increase if results seem unstable.
What if the Gap Statistic suggests k=1, meaning no clustering?
This indicates weak or absent clustering structure in your data. You may need to re-examine your data, try different clustering algorithms, or reconsider whether clustering is appropriate for your problem.
Can I use the Gap Statistic with non-Euclidean distances?
Yes, but generating appropriate reference datasets becomes more complex. For non-Euclidean distances, you need to generate reference data that respects your distance metric or use alternative null models.
How does the Gap Statistic compare to silhouette score?
The Gap Statistic is a statistical test for selecting k, while silhouette score evaluates the quality of a given clustering. They serve complementary purposes: use Gap Statistic to choose k, then validate with silhouette score.
Sources
- 1.Tibshirani, R., Walther, G., & Hastie, T. (2001). Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(2), 411-423.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Gap Statistic. ScholarGate. https://scholargate.app/model-evaluation/gap-statistic