LoRA and fine-tuning explained: when to fine-tune an LLM and how adapters work

Key points

• Fine-tuning changes how a model behaves; retrieval changes what it knows at answer time. Try a clear prompt, then retrieval, then fine-tuning.
• LoRA learns the weight update as the product of two thin matrices, B times A, with rank r, scaled by alpha over r. Only A and B are trained.
• The original paper adapted only the attention query and value matrices; the QLoRA authors found that adapters on all linear layers were needed to match full fine-tuning.
• QLoRA stores the frozen base in 4-bit NormalFloat, quantizes the quantization constants too, and pages optimizer state to absorb memory spikes.
• A merged adapter adds no inference latency. Unmerged adapters let one base model serve many fine-tunes, which systems such as S-LoRA batch together.
• A few hundred clean, varied examples often beat many thousands of noisy ones. Hold out a test set, compare against the prompted base model, and check what the model forgot.

When fine-tuning beats prompting and RAG

Fine-tuning continues training a pretrained model on your own examples so that it behaves differently: it follows one output format every time, writes in one voice, or does one narrow task better with a shorter prompt. It is a poor way to teach a model your facts. Facts change, and a model trained on last quarter's documents answers from last quarter. Retrieval puts the current documents into the prompt at answer time; see RAG explained for how that works and RAG vs fine-tuning for the choice in plain terms.

Signs that fine-tuning will pay off:

• You have tried a clear prompt with a few examples, and the model still drifts from the format or the style on a noticeable share of real inputs.
• The prompt has grown into pages of instructions and examples that you send on every call. Training those examples in can shorten every request.
• A small model trained on one task could stand in for a large general model on that task, with lower latency and less serving load.
• The task has an answer you can grade: classification, extraction, routing, a fixed output schema, a house style with clear rules.

Signs it will not: the answers depend on facts that change, you have only a handful of good examples, or nobody can yet say what a good output looks like. The usual order is prompt first, then retrieval, then fine-tuning. Keep the evaluation set from the first step. It is the only way to know whether the fine-tune helped.

Full fine-tuning vs parameter-efficient fine-tuning

Full fine-tuning updates every weight. With the Adam optimizer, training memory holds the weights, their gradients and two optimizer statistics per weight, plus activations, so it needs several times the memory of the model itself. Every fine-tuned version is also a complete copy of the model. The LoRA paper puts numbers on GPT-3 175B: a full checkpoint is 350 GB, and training it needed 1.2 TB of GPU memory.

Parameter-efficient fine-tuning, often shortened to PEFT, freezes the pretrained weights and trains a small set of new parameters. The main families are adapter layers inserted between existing layers, prefix and prompt tuning, which learn vectors placed in front of the input, and low-rank methods such as LoRA. The LoRA authors point out the costs of the first two: adapter layers run in sequence and add inference latency, and prefix tuning uses up part of the sequence length the task could have used. LoRA avoids both, which is a large part of why it became the default.

How LoRA works: a low-rank update beside a frozen weight

LoRA, from Hu and colleagues (2021), rests on a hypothesis: the change a weight matrix needs during adaptation has a low intrinsic rank, even though the matrix itself does not. So for a frozen pretrained weight W0 of shape d by k, LoRA learns the update as the product of two thin matrices, B of shape d by r and A of shape r by k, where the rank r is much smaller than d or k. The forward pass becomes h = W0 x + (alpha / r) * B A x. Only A and B receive gradients.

• Initialization: A starts random and B starts at zero, so the product BA is zero and training begins exactly at the pretrained model.
• Scaling: the update is multiplied by alpha over r. The authors note that with Adam, tuning alpha is roughly the same as tuning the learning rate, so they set alpha to the first r they tried and left it alone; the scaling reduces retuning when r changes. In the Hugging Face PEFT library these are r and lora_alpha, and use_rslora switches the scaling to alpha over the square root of r, which the docs say stabilizes adapters at higher ranks.
• Rank: on GPT-3 the authors found that a rank as small as one was enough for their WikiSQL and MultiNLI tasks when adapting query and value together, and they caution that a small r will not work for every task, for example one in a language the model was not pretrained on.
• Which matrices: the paper adapted only the attention query and value projections, for simplicity, and found that with a fixed parameter budget, adapting both beat putting all of it into one matrix at a higher rank. The QLoRA authors later could not match full fine-tuning with query and value alone, and found that adapters on all linear layers of each transformer block were required. PEFT exposes that as target_modules="all-linear".

The whole mechanism fits in plain Python. This sketch uses tiny matrices to show that the adapter starts as a no-op and that merging it into the weight gives the same output, then counts parameters at real layer sizes. We ran it; the output is in the comments.

import random

def matvec(M, v):
    return [sum(m * x for m, x in zip(row, v)) for row in M]

def rand(rows, cols):
    return [[random.gauss(0, 0.1) for _ in range(cols)] for _ in range(rows)]

random.seed(0)
d, k, r, alpha = 6, 5, 2, 4
W0 = rand(d, k)                      # frozen pretrained weight, d x k
A = rand(r, k)                       # trainable, random init, r x k
B = [[0.0] * r for _ in range(d)]    # trainable, zero init, d x r
x, s = [random.gauss(0, 1) for _ in range(k)], alpha / r

def lora_forward(x):                 # h = W0 x + (alpha / r) * B (A x)
    return [w + s * b for w, b in zip(matvec(W0, x), matvec(B, matvec(A, x)))]

print(lora_forward(x) == matvec(W0, x))  # True: with B = 0 the adapter is a no-op
B = rand(d, r)                           # pretend training has moved B
W = [[W0[i][j] + s * sum(B[i][t] * A[t][j] for t in range(r)) for j in range(k)] for i in range(d)]
print(max(abs(a - b) for a, b in zip(lora_forward(x), matvec(W, x))) < 1e-12)  # True: merged = unmerged
for d, k, r in ((4096, 4096, 8), (4096, 11008, 16)):  # real layer sizes
    print(f"{d}x{k}, r={r}: full {d * k:,} vs LoRA {r * (d + k):,} ({r * (d + k) / (d * k):.2%})")
# 4096x4096, r=8: full 16,777,216 vs LoRA 65,536 (0.39%)
# 4096x11008, r=16: full 45,088,768 vs LoRA 241,664 (0.54%)

Note that the forward pass computes A x first and then B times that, so the full d by k update is never built during training. The saving comes from there and from having no gradients or optimizer state for W0. Across a whole model the trainable share stays small. The next sketch counts LoRA parameters for a common 7B decoder shape at a few settings; again, the comments are its real output:

# Trainable LoRA parameters for a typical 7B decoder shape:
# 32 layers, hidden size 4,096, MLP size 11,008, full multi-head attention.
layers, h, m = 32, 4096, 11008
attn = {"q": (h, h), "k": (h, h), "v": (h, h), "o": (h, h)}
mlp = {"gate": (h, m), "up": (h, m), "down": (m, h)}
shapes = {**attn, **mlp}
total = layers * sum(a * b for a, b in shapes.values())

def lora_params(targets, r):
    return layers * sum(r * (shapes[t][0] + shapes[t][1]) for t in targets)

for targets, r in ((["q", "v"], 8), (list(attn), 16), (list(shapes), 16), (list(shapes), 64)):
    n = lora_params(targets, r)
    print(f"r={r:<3} {'+'.join(targets):<25} {n:>12,}  {n / total:.2%} of {total:,} block weights")
# r=8   q+v                          4,194,304  0.06% of 6,476,005,376 block weights
# r=16  q+k+v+o                     16,777,216  0.26% of 6,476,005,376 block weights
# r=16  q+k+v+o+gate+up+down        39,976,960  0.62% of 6,476,005,376 block weights
# r=64  q+k+v+o+gate+up+down       159,907,840  2.47% of 6,476,005,376 block weights

Even the largest setting here trains about 2.5 percent of the block weights. What LoRA does not shrink is the frozen base: it still has to sit in GPU memory for every forward and backward pass. That is the problem QLoRA solves.

QLoRA: fine-tuning on top of a 4-bit base model

QLoRA, from Dettmers and colleagues (2023), quantizes the frozen base model to 4 bits and backpropagates through it into LoRA adapters. Weights are stored in 4 bits and dequantized to 16-bit BFloat16 whenever they are used, so the arithmetic still happens in 16-bit. Three ideas make it work:

• 4-bit NormalFloat (NF4): a data type the authors describe as information-theoretically optimal for normally distributed data, which pretrained weights roughly are. They report better results than 4-bit integers and 4-bit floats.
• Double quantization: block-wise quantization stores a scaling constant for every block of weights, and QLoRA quantizes those constants as well. The authors report a saving of about 0.37 bits per parameter, roughly 3 GB on a 65B model.
• Paged optimizers: optimizer state lives in NVIDIA unified memory, which pages it between GPU and CPU memory automatically, so the memory spikes from a long sequence under gradient checkpointing do not crash the run.

The authors report that fine-tuning a 65B model drops from more than 780 GB of GPU memory to under 48 GB, enough for a single GPU, while matching 16-bit fine-tuning on their tasks. Their Guanaco models, trained this way, reached 99.3 percent of ChatGPT's score on the Vicuna benchmark by their measure. The costs: every step does extra dequantization work, and the adapter is trained against the quantized base. If you later merge it into a 16-bit or re-quantized copy, the result is close to, not identical with, what you evaluated, and the PEFT docs warn that unmerging on quantized models is not exact. Evaluate the exact artifact you deploy.

Merging adapters and serving many of them

With one fine-tune, merge it. W0 + (alpha / r) * B A has the same shape as W0, so the merged model runs exactly like the original, and the LoRA paper notes that this adds no inference latency by construction. In PEFT, merge_and_unload() does it; to switch tasks without reloading, you can subtract one adapter and add another. Storage changes too: the paper estimates that 100 adapted GPT-3 models need about 354 GB with LoRA, against about 35 TB as full copies.

With many fine-tunes, keep them unmerged. The base model stays in GPU memory once and each request applies its own adapter. The LoRA paper names the catch: once you merge, you cannot easily batch requests for different tasks in one forward pass. Unmerged, you can. PEFT accepts an adapter_names list so each row of an inference batch uses its own adapter, and its docs warn about the overhead as the number of different adapters in a batch grows, pointing to dedicated servers for large numbers of them.

S-LoRA, from Sheng and colleagues (2023), is one such design. It keeps all adapters in main memory and moves only the ones that running requests need onto the GPU. A single paged memory pool, which the authors call Unified Paging, holds adapter weights of different ranks next to KV cache tensors of different lengths, which limits fragmentation, and custom kernels compute the LoRA terms for a batch whose rows use different adapters. The authors report up to four times the throughput of Hugging Face PEFT and of vLLM's basic LoRA support at the time, while serving thousands of adapters. For how batching itself works on an LLM server, see how LLM inference serving works.

You can also combine adapters. PEFT's add_weighted_adapter builds a new adapter from weighted existing ones; its docs call the cat and svd methods precise and the linear method a rough approximation. Evaluate a combined adapter like any new model.

Preparing the training data

Most fine-tuning is supervised: pairs of an input and the ideal output, written in the chat format the model will see when it runs. The common file format is JSONL, one complete example per line. Three lines of an extraction task:

{"messages": [{"role": "system", "content": "Extract order_id, issue and wants as JSON."}, {"role": "user", "content": "Order 88231 came with a cracked screen. Can you send a new one?"}, {"role": "assistant", "content": "{\"order_id\": \"88231\", \"issue\": \"damaged\", \"wants\": \"replacement\"}"}]}
{"messages": [{"role": "system", "content": "Extract order_id, issue and wants as JSON."}, {"role": "user", "content": "Where is my parcel? It's order 10457."}, {"role": "assistant", "content": "{\"order_id\": \"10457\", \"issue\": \"not_delivered\", \"wants\": \"status\"}"}]}
{"messages": [{"role": "system", "content": "Extract order_id, issue and wants as JSON."}, {"role": "user", "content": "The mug I got is the wrong size."}, {"role": "assistant", "content": "{\"order_id\": \"\", \"issue\": \"wrong_item\", \"wants\": \"unknown\"}"}]}

• Match inference exactly: the same system prompt, tool definitions and output format the application will send. A model trained on one prompt and served with another behaves like neither.
• Cover the awkward cases, not just the happy path. The third line above teaches the model what to do when the order number is missing, which is where prompted models tend to make things up.
• Check how your training library computes the loss. Most train only on the assistant turns, so the model learns to answer rather than to imitate the user.
• Deduplicate, then split off a test set before you train. Split by source, such as customer or document, so near-duplicates do not leak into both sides.

Quality beats quantity, within limits. In LIMA, Zhou and colleagues (2023) fine-tuned a 65B LLaMa model on 1,000 carefully curated prompts and responses with no reinforcement learning, and in their human study its answers were equivalent to or preferred over GPT-4's in 43 percent of cases. Their ablations found that more diverse prompts and better responses helped measurably, while scaling up quantity alone might not. They frame this as the Superficial Alignment Hypothesis: a model's knowledge and capabilities come almost entirely from pretraining, and alignment mostly teaches it which formats to use with users. The QLoRA authors saw the same pattern: a 9,000-example dataset beat a 450,000-example subsample of another on chatbot performance. For a narrow task, start with a few hundred examples you have read yourself, and add more where the evaluation shows gaps.

Evaluation, overfitting and catastrophic forgetting

Evaluate on the held-out set with a measure tied to the task: exact match on extracted fields, schema validity, classification accuracy, or a rubric scored by people or by a model acting as judge. Always include the baseline the fine-tune has to beat, which is the base model with your best prompt. If it does not beat that, ship the prompt. For building evaluation sets that hold up, see how to evaluate AI agents.

Overfitting shows up as training loss that keeps falling while validation loss turns upward, and as outputs that copy training phrasing word for word. Keep the number of epochs low, save checkpoints, and pick the checkpoint by your evaluation rather than by loss. The LIMA authors found that perplexity did not track generation quality, and chose checkpoints by hand for that reason.

Catastrophic forgetting is the other risk: training hard on one narrow task can erode what the model did well before. Biderman and colleagues (2024) compared LoRA with full fine-tuning on code and math. They report that LoRA learned the new domain substantially less well under common settings, but kept more of the base model's performance on other tasks, more than weight decay or dropout did, and that full fine-tuning learned updates with a rank 10 to 100 times higher than typical LoRA settings. Practical defenses: mix some general examples into the training set, keep a regression suite of things the model must still do, such as following general instructions and refusing what it should refuse, and run it on every candidate.

Preference tuning with DPO

Supervised fine-tuning teaches by imitation: here is the input, copy this output. Preference tuning teaches by comparison: for one prompt, this response is better than that one. The classic route, reinforcement learning from human feedback, trains a separate reward model on those comparisons and then optimizes the language model against it with reinforcement learning. Direct Preference Optimization, from Rafailov and colleagues (2023), shows that the same objective can be optimized directly with a simple classification loss on the preference pairs, with no reward model and no sampling from the model during training. The authors report that it matches or beats PPO-based RLHF on sentiment control, summarization and single-turn dialogue, and is simpler to train. In practice teams run supervised fine-tuning first and DPO after it, and both work with LoRA adapters; the PEFT docs even show an SFT adapter and a DPO adapter being combined into one.

Hosted fine-tuning APIs

You do not have to run GPUs to fine-tune. Model vendors and cloud platforms offer it as a managed job: upload a JSONL file, start a job, get back a model you call by name. The offerings change often, so read the current documentation before planning around one. At the time of writing:

• OpenAI: its supervised fine-tuning guide uses JSONL in the chat format, requires at least 10 examples, and suggests starting with about 50 well-crafted ones and evaluating before adding more. It also links to preference (DPO), reinforcement and vision fine-tuning. The same guide says OpenAI is winding the fine-tuning platform down: it is closed to new users, existing eligible customers can create jobs until January 6, 2027, unless the base model shuts down earlier, and fine-tuned models stay available for inference until their base model shuts down.
• Google Vertex AI: supervised fine-tuning for several Gemini models, with an adapter size setting and a validation set of up to 5,000 examples, aimed at tasks such as classification, summarization, extractive question answering and chat.
• Amazon Bedrock: supervised fine-tuning, reinforcement fine-tuning against reward functions you define, and distillation, where a larger teacher model generates responses used to fine-tune a smaller student.

A hosted job removes the infrastructure, but you work within the provider's list of base models and settings, and a tuned model lives only as long as its base model stays available. With an open-weight base model and LoRA, the adapter is a small file you keep, version and evaluate like any other build artifact. Fine-tuned models still need guardrails around them; see LLM guardrails architecture.

If you are deciding whether a model needs fine-tuning, retrieval or just a better prompt, see our AI agent development service: the code lives in your GitHub from day one, AI-assisted work is reviewed by an engineer, and the founders are on the calls.