K-Nearest Neighbors
K-Nearest Neighbors (KNN) Classification and Regression · Also known as: KNN, K-En Yakın Komşu (KNN), nearest neighbor classifier, instance-based learning
K-Nearest Neighbors (KNN), formalized by Cover and Hart in 1967, is a non-parametric, instance-based method that classifies or predicts a new observation by looking at the k closest examples in the training data. For classification it takes a majority vote among those neighbors; for regression it averages their values.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use KNN for classification or prediction on tabular data with continuous, binary, or ordinal features and at least about 30 observations. It assumes no particular distribution, but it does require that features be normalized and that the number of features stay small relative to the sample size (p far less than n is preferred). In small or high-dimensional samples the curse of dimensionality dominates — neighbor distances become uninformative — and a parametric model such as logistic regression or Naive Bayes is the better choice.
Strengths & limitations
- Conceptually simple and easy to explain: no training-time equation, the data is the model.
- Non-parametric — makes no assumption about the underlying distribution.
- Naturally handles multi-class problems and complex, nonlinear decision boundaries.
- Works for both classification and regression with the same basic idea.
- Sensitive to feature scaling; without standardization, large-scale features dominate the distance.
- Degrades quickly in high dimensions (curse of dimensionality), where neighbor distances lose meaning.
- Every prediction requires scanning the whole training set, which is slow at prediction time on large data.
- Needs at least a moderate sample size; in small samples nearest-neighbor distances are not meaningful.
Frequently asked
How do I choose the value of k?
k should be selected with cross-validation rather than guessed — for example by trying a range of values and using the elbow method to find where error stops improving. A small k follows the data closely but is noise-sensitive, while a larger k smooths the decision boundary.
Why must I standardize the features?
KNN decides who is 'nearest' using distance, which is scale-sensitive. If one feature is on a much larger numeric scale than the others, it dominates the distance and distorts the neighbor set. Standardizing features (for example with a StandardScaler) is required so each feature contributes fairly.
Why does KNN struggle with many features?
In high-dimensional spaces all points become roughly equidistant — the curse of dimensionality — so 'nearest' neighbors are no longer meaningfully close. KNN performance drops quickly, and a parametric model such as Naive Bayes or logistic regression is usually preferable. Keeping p far smaller than n helps.
Is there a training step?
Not in the usual sense. KNN is instance-based (lazy): it simply stores the training data and does all the work at prediction time, when it computes distances to find the nearest neighbors of a new point.
Sources
- Cover, T.M. & Hart, P.E. (1967). Nearest Neighbor Pattern Classification. IEEE Transactions on Information Theory, 13(1), 21–27. DOI: 10.1109/TIT.1967.1053964 ↗
How to cite this page
ScholarGate. (2026, June 1). K-Nearest Neighbors (KNN) Classification and Regression. ScholarGate. https://scholargate.app/en/machine-learning/knn
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Decision TreeMachine learning↔ compare
- Logistic RegressionResearch Statistics↔ compare
- Naive BayesMachine learning↔ compare
- Random ForestMachine learning↔ compare
- Support Vector MachineMachine learning↔ compare