Transfer Learning
Transfer Learning (Domain Adaptation and Knowledge Transfer) · Also known as: TL, domain adaptation, fine-tuning, pre-trained model adaptation
Transfer learning is a machine learning paradigm in which knowledge gained from training a model on a source task or domain is reused to improve learning on a different but related target task or domain. It is especially powerful when labeled data for the target task is scarce, and it underlies most modern deep learning applications in computer vision, natural language processing, and beyond.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+40 more
When to use it
Transfer learning is the right choice when labeled data for the target task is limited but a related source domain with abundant labels or a large pre-trained model exists — for example, fine-tuning BERT for a domain-specific text classifier or adapting ImageNet-pretrained CNNs to medical imaging. It dramatically reduces the data and compute needed to reach strong performance. Avoid it when the source and target domains are fundamentally unrelated, because negative transfer can make performance worse than training from scratch. It is also less warranted when the target dataset is large enough to train a capable model independently.
Strengths & limitations
- Dramatically reduces labeled data requirements for the target task by reusing source representations.
- Shortens training time and compute cost compared to training large models from scratch.
- Often achieves state-of-the-art performance on small and medium target datasets.
- General-purpose pre-trained models (e.g., ResNet, BERT, GPT) provide strong starting points across many domains.
- Feature extraction mode enables use of deep representations even without GPU-heavy fine-tuning.
- Negative transfer can occur when source and target domains are too dissimilar, potentially hurting performance relative to training from scratch.
- Fine-tuning large pre-trained models requires substantial memory and, for very large models, significant GPU resources.
- The appropriate amount of the model to freeze versus fine-tune is task-dependent and requires experimentation.
- Domain shift between source and target may not be obvious and can silently degrade performance.
- Legal and ethical constraints on pre-trained models (license, training data bias) may limit use in certain applications.
Frequently asked
What is the difference between feature extraction and fine-tuning in transfer learning?
In feature extraction, the pre-trained model's weights are frozen and only a new task-specific head is trained. In fine-tuning, some or all of the pre-trained weights are also updated on the target data — typically with a small learning rate to avoid catastrophic forgetting. Feature extraction is faster and less prone to overfitting on tiny datasets; fine-tuning generally yields better performance when there is enough target data.
How do I know if transfer learning will help or hurt?
The key question is domain similarity. If the source and target distributions share meaningful structure (same modality, similar vocabulary, related visual concepts), transfer almost always helps. Train a baseline from scratch and compare: if the transferred model is not better, the domains may be too dissimilar, or you may need domain-adversarial alignment techniques.
What is negative transfer?
Negative transfer occurs when knowledge from the source domain actively hurts performance on the target task — the pre-trained representations do not match target structure and mislead optimization. It is most common when source and target are semantically or statistically dissimilar. Regularization toward the source weights (e.g., L2-SP) or using only shallower layers can mitigate it.
How much target data do I need for fine-tuning?
There is no universal threshold, but transfer learning is most valuable when target labeled examples number in the dozens to a few thousand. With very few examples (fewer than ~100), freezing most layers and only training the head (feature extraction) is safer. With thousands of examples, deeper fine-tuning is feasible and usually beneficial.
Is transfer learning applicable to tabular data, not just images and text?
Yes, though it is less plug-and-play. Approaches include pre-training neural networks on large tabular corpora from related domains, or using embeddings from foundation models as additional features. Libraries like SAINT and TabPFN explore this direction. For classic tabular tasks, gradient boosting often remains a strong competitor even without transfer.
Sources
- Pan, S. J., & Yang, Q. (2010). A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345–1359. DOI: 10.1109/TKDE.2009.191 ↗
- Bengio, Y. (2012). Deep Learning of Representations for Unsupervised and Transfer Learning. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, PMLR 27, 17–36. link ↗
How to cite this page
ScholarGate. (2026, June 3). Transfer Learning (Domain Adaptation and Knowledge Transfer). ScholarGate. https://scholargate.app/en/machine-learning/transfer-learning
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Few-shot LearningMachine learning↔ compare
- Self-supervised LearningMachine learning↔ compare
- Semi-supervised LearningMachine learning↔ compare