LoRA and PEFT
Low-Rank Adaptation and Parameter-Efficient Fine-Tuning · Also known as: LoRA ve PEFT — Parametre Verimli İnce Ayar, Low-Rank Adaptation, parameter-efficient fine-tuning, prefix tuning, prompt tuning
LoRA (Low-Rank Adaptation), introduced by Hu et al. in 2022, and the broader family of parameter-efficient fine-tuning (PEFT) methods adapt large pretrained language models to new tasks by training only a small number of extra parameters instead of every weight in the model. This makes fine-tuning possible with far less GPU memory and compute while leaving the original model largely untouched.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use LoRA and PEFT when you want to adapt a large pretrained language model to a specific text classification or prediction task but cannot afford the memory and compute of full fine-tuning. A large pretrained model must already be available, task-specific data should be prepared, and a GPU is recommended. These methods work best with a substantial task dataset; with very small datasets (below a few hundred examples) the adaptation tends to overfit and a simpler model such as random forest or gradient boosting is safer.
Strengths & limitations
- Drastically lower GPU memory and compute than full fine-tuning, since only a small set of parameters is trained.
- The original pretrained model is frozen and reused across tasks, so a single base model can be specialised many times.
- The learned low-rank update can be merged back into the weights, adding no inference-time overhead.
- Covers a family of options — LoRA, prefix tuning, prompt tuning — so the adaptation can be placed where it works best.
- Requires a large pretrained model to start from; it does not train a model from scratch.
- A GPU is recommended, and task-specific data must be prepared in advance.
- On small task datasets (below about 500 examples) the fine-tuning may not provide enough adaptation and risks overfitting.
- With very little task data (below about 50 examples) even parameter-efficient fine-tuning becomes meaningless.
Frequently asked
What is the difference between LoRA and PEFT?
PEFT (parameter-efficient fine-tuning) is the broad family of methods that adapt a large model by training only a small number of extra parameters. LoRA is one specific PEFT method that trains a low-rank update to the weights; prefix tuning and prompt tuning are other members of the same family.
How much data do I need?
These methods need a reasonably sized task dataset. Below about 500 examples the adaptation may be insufficient and prone to overfitting, and below about 50 examples parameter-efficient fine-tuning becomes meaningless — a classical model is the better choice there.
Do I need a GPU?
A GPU is recommended. Although PEFT methods use far less memory than full fine-tuning, training the added parameters on a large pretrained model still benefits substantially from GPU acceleration.
Why is the update low-rank?
The change required to adapt a pretrained model to a new task can usually be captured by a small low-rank correction rather than a dense update to every weight. Representing it as the product of two small matrices is what makes the number of trainable parameters — and the cost — drop dramatically.
Sources
- Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR. link ↗
- Lester, B. et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP. DOI: 10.18653/v1/2021.emnlp-main.243 ↗
How to cite this page
ScholarGate. (2026, June 1). Low-Rank Adaptation and Parameter-Efficient Fine-Tuning. ScholarGate. https://scholargate.app/en/deep-learning/lora-peft
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Generative Adversarial NetworkDeep learning↔ compare
- Random ForestMachine learning↔ compare
- Variational AutoencoderDeep learning↔ compare
- Vision TransformerDeep learning↔ compare
- XGBoostMachine learning↔ compare