LoRA and PEFT
Also known as: LoRA ve PEFT — Parametre Verimli İnce Ayar, Low-Rank Adaptation, parameter-efficient fine-tuning, prefix tuning, prompt tuning
LoRA (Low-Rank Adaptation), introduced by Hu et al. in 2022, and the broader family of parameter-efficient fine-tuning (PEFT) methods adapt large pretrained language models to new tasks by training only a small number of extra parameters instead of every weight in the model. This makes fine-tuning possible with far less GPU memory and compute while leaving the original model largely untouched.
Key highlights
- Drastically lower GPU memory and compute than full fine-tuning, since only a small set of parameters is trained.
- The original pretrained model is frozen and reused across tasks, so a single base model can be specialised many times.
- The learned low-rank update can be merged back into the weights, adding no inference-time overhead.
- Covers a family of options — LoRA, prefix tuning, prompt tuning — so the adaptation can be placed where it works best.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use LoRA and PEFT when you want to adapt a large pretrained language model to a specific text classification or prediction task but cannot afford the memory and compute of full fine-tuning. A large pretrained model must already be available, task-specific data should be prepared, and a GPU is recommended. These methods work best with a substantial task dataset; with very small datasets (below a few hundred examples) the adaptation tends to overfit and a simpler model such as random forest or gradient boosting is safer.
Strengths & limitations
- Drastically lower GPU memory and compute than full fine-tuning, since only a small set of parameters is trained.
- The original pretrained model is frozen and reused across tasks, so a single base model can be specialised many times.
- The learned low-rank update can be merged back into the weights, adding no inference-time overhead.
- Covers a family of options — LoRA, prefix tuning, prompt tuning — so the adaptation can be placed where it works best.
- Requires a large pretrained model to start from; it does not train a model from scratch.
- A GPU is recommended, and task-specific data must be prepared in advance.
- On small task datasets (below about 500 examples) the fine-tuning may not provide enough adaptation and risks overfitting.
- With very little task data (below about 50 examples) even parameter-efficient fine-tuning becomes meaningless.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
What is the difference between LoRA and PEFT?
PEFT (parameter-efficient fine-tuning) is the broad family of methods that adapt a large model by training only a small number of extra parameters. LoRA is one specific PEFT method that trains a low-rank update to the weights; prefix tuning and prompt tuning are other members of the same family.
How much data do I need?
These methods need a reasonably sized task dataset. Below about 500 examples the adaptation may be insufficient and prone to overfitting, and below about 50 examples parameter-efficient fine-tuning becomes meaningless — a classical model is the better choice there.
Do I need a GPU?
A GPU is recommended. Although PEFT methods use far less memory than full fine-tuning, training the added parameters on a large pretrained model still benefits substantially from GPU acceleration.
Why is the update low-rank?
The change required to adapt a pretrained model to a new task can usually be captured by a small low-rank correction rather than a dense update to every weight. Representing it as the product of two small matrices is what makes the number of trainable parameters — and the cost — drop dramatically.
Sources
- 1.Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
- 2.Lester, B. et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 1). LoRA and PEFT. ScholarGate. https://scholargate.app/deep-learning/lora-peft