Parameter-Efficient Fine-Tuning (PEFT)

Parameter-efficient fine-tuning (PEFT) is a family of methods that adapt a pretrained model to a new task by training a small number of new or selected parameters while leaving most or all of the original weights frozen. The goal is to get close to the quality of full fine-tuning at a fraction of the compute, memory and storage, and to make the result a small, swappable artifact rather than a second copy of the model.

The Problem It Solves

Full fine-tuning updates every weight, so each task produces a complete new model. Houlsby et al. framed the cost in February 2019: with many downstream tasks, "an entire new model is required for every task." As foundation models grew to billions of parameters, that became a storage problem, a GPU-memory problem and a serving problem at once. PEFT methods answer all three by keeping one shared base and attaching a small task-specific delta.

The Main Families

Method	What is trained	Reported result in the original paper	Main cost
Adapters (Houlsby et al., 2019)	Small bottleneck modules inserted between frozen layers	Within 0.4% of full fine-tuning on GLUE while adding 3.6% parameters per task (BERT)	Extra layers add inference latency
Prefix tuning (Li and Liang, 2021)	A continuous task-specific prefix that later tokens attend to as virtual tokens	Comparable to fine-tuning in full-data settings while learning 0.1% of parameters (GPT-2, BART)	Can be harder to optimize; consumes sequence positions
Prompt tuning (Lester et al., 2021)	Soft prompt embeddings prepended to the input	Matches full model tuning on T5 once models exceed billions of parameters	Lags on smaller models
BitFit (Ben-Zaken et al., 2021)	Only the bias terms	Competitive with full fine-tuning of BERT on small-to-medium data	Limited capacity on larger data
LoRA (Hu et al., 2021)	Low-rank matrices added beside frozen weight matrices	On par or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3 with up to 10,000 times fewer trainable parameters	Low rank caps how much can be learned
(IA)³ (Liu et al., 2022)	Learned vectors that rescale internal activations	Stronger accuracy than few-shot in-context learning at much lower compute; first super-human result on the RAFT benchmark with the T-Few recipe	Very few parameters, so narrow adaptation

Each row is that method's own paper on its own choice of models and benchmarks, mostly from 2019 to 2022 and on models far smaller than current large language models. The numbers show that each idea works; they are not a ranking.

Why Low-Rank Methods Won

In practice LoRA and its descendants became the default, for reasons that are structural rather than benchmark-driven. A LoRA update can be merged into the base weights, so it adds no inference latency, which the LoRA authors contrasted directly with adapters. It leaves the input sequence untouched, unlike prefix and prompt tuning. It composes with quantization: QLoRA (Dettmers et al., May 2023) trained adapters against a 4-bit base and fine-tuned a 65-billion-parameter model on a single 48 GB GPU. And it yields a file measured in megabytes that can be versioned, shared and loaded per request on one base model.

The Hugging Face PEFT library, the common reference implementation, now describes itself more broadly as a framework for "arbitrary methods of model adaption" and ships LoRA with switches for variants such as rsLoRA and DoRA. Soft-prompt methods persist mainly in research and in settings where the weights cannot be touched at all.

Trade-Offs

Capacity. A small delta can only express a small change. Biderman et al. (May 2024) found that LoRA at standard ranks substantially underperformed full fine-tuning on programming and mathematics, and that full fine-tuning learned perturbations of 10 to 100 times higher rank. The DoRA authors likewise wrote in 2024 that an accuracy gap to full fine-tuning "often" remains. Both findings concern LoRA specifically; systematic comparisons across the whole family on current models are scarce.

Forgetting. The same constraint is a benefit: Biderman et al. reported that LoRA preserved more of the base model's out-of-domain performance than full fine-tuning and more than weight decay or dropout. This makes PEFT a practical tool against the forgetting problem studied in continual learning.

Coupling. An adapter is meaningful only with the exact base model it was trained on. When the base is replaced by a new release, adapters generally have to be retrained.

Alternatives. PEFT competes not only with full fine-tuning but with changing no weights: prompt optimization, retrieval and longer context. The (IA)³ paper argued in 2022 that few-shot PEFT was both more accurate and cheaper than in-context learning, because examples are paid for once in training rather than on every request; that conclusion predates today's models and prompt caching, and should be re-tested rather than assumed.