LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS

Introduction

LLMs become increasingly essential in natural language processing, adapting these models to specific tasks presents significant challenges due to the vast number of parameters involved.

Inspiration from Li et al. (2018a); Aghajanyan et al. (2020) which show that the learned over-parametrized models in fact reside on a low intrinsic dimension.

→ hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”

Traditional fine-tuning methods require retraining all model parameters, making it infeasible for larger models like GPT-3 with 175 billion parameters. LoRA (Low-Rank Adaptation) addresses this issue by introducing a more efficient parameter adaptation approach that significantly reduces computational costs while maintaining high performance.

Fine-tuning

Fine-tuning involves updating all parameters of a pre-trained model to adapt it to a specific downstream task. This process is computationally expensive and memory-intensive, especially for large models.

1. Forward Pass: Pass data through the model.

2. Weight Updates: Calculate updates using backpropagation.

3. Parameter Combination: Combine updated weights with base weights.

In fine-tuning, models require full-rank weight matrices, resulting in a large number of trainable parameters and high memory usage.

Example

• Fine-tine last layer (Task-specific head)

• Adapter Layers (Add new intermediate modules)

• Prefix Tuning (Tune prefixes for every layer)

Benefits of LoRA

1. Reduced Trainable Parameters: By decomposing weight matrices into smaller matrices, LoRA drastically reduces the number of trainable parameters.

2. Memory and Storage Savings: LoRA reduces GPU memory consumption by up to 3x during training. In GPT-3 175B, the VRAM requirement drops from 1.2TB to 350GB.

3. Task-Switching Efficiency: LoRA enables quick task-switching by swapping the low-rank matrices (A and B) without altering the frozen pre-trained weights.

Conventions

references to the Transformer architecture.

• $d_{model}$: input and output dimension size

• $W_q$, $W_k$, $W_v$, and $W_o$: query/key/value/output projection matrices

• $W$ or $W_0$ : pretrained weight matrix

• $∆W$: gradient update during adaptation

• Adam: model optimization

• $d_{ff_n}=4xd_{model}$

Structure

LoRA proposes that the changes in weights during model adaptation have a low "intrinsic rank." It decomposes the large weight update matrix (ΔW) into two smaller matrices (A and B):

$W_0$: Frozen, $W_0 \in \R^{d\times k}$

$A \in \R^{d\times r}$, $B \in \R^{r\times k}$

weight matrices have low intrinsic dimensionality, meaning they can be accurately represented by fewer dimensions without losing essential information.

For example, a 100x100 matrix with 10,000 elements can be replaced by two smaller matrices with a total of 1,000 elements.

• A Generalization of Full Fine-tuning: LoRA takes a step further and does not require the accumulated gradient update to weight matrices to have full-rank during adaptation. we increase the number of trainable parameters, training LoRA roughly converges to training the original model, while adapter-based methods converges to an MLP and prefix-based methods to a model that cannot take long input sequences.

• No Additional Inference Latency: Can recover $W_0$ by subtracting $BA$ and then adding a different $B'$ $A'$, a quick operation with very little memory overhead.

Implementations to Transformer

In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the number of trainable parameters.

ly adapting the attention weights for downstream tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity and parameter-efficiency.

Results

• For a large Transformer trained with Adam, we reduce that VRAM usage by up to 2/3.

  • On GPT-3 175B, we reduce the VRAM consumption during training from 1.2TB to 350GB.

  • With r = 4 and only the query and value projection matrices being adapted, the checkpoint size is reduced by roughly 10,000× (from 350GB to 35MB)

• Merging A and B into W reduces inference time but makes handling multi-task inputs harder.

  For example:

  • Task 1 uses A1, B1

  • Task 2 uses A2, B2

  If you want to process inputs for both tasks simultaneously in a single forward pass, each input sample in the batch would require different A and B values, creating a complication.

• If latency is not a concern, you can keep A and B separate and dynamically load LoRA modules for each input.

LoRA has been evaluated on several benchmarks, including:

1. GLUE Benchmark: Demonstrates competitive performance on various natural language understanding tasks.

1. E2E NLG Challenge: Outperforms other adaptation methods in natural language generation.

1. GPT-3 175B: Matches or exceeds the performance of fine-tuning across multiple datasets.

   1. Used less parameters

Key Findings

• LoRA achieves better scalability and performance with fewer trainable parameters.

• It performs well in both high-data and low-data regimes.

Questions?

1. Which weight matrices should we adapt with LoRA to maximize performance?

   1. tested different combinations of attention weight matrices (Wq, Wk, Wv, Wo) in GPT-3 175B to find the best configuration under a fixed parameter budget.

• Adapting Wq and Wv gives the best balance of performance and parameter efficiency.

1. Is the optimal adaptation matrix ∆W really low-rank?

• Low ranks (r = 1 to 4) are sufficient for achieving high accuracy.

• Increasing the rank beyond r = 8 provides diminishing returns.

1. How does ∆W correlate with the pre-trained weight matrix W?

• ∆W correlates strongly with W but focuses on less emphasized directions, which helps with task-specific adaptations.

• The amplification factor is large, meaning LoRA efficiently amplifies important features for downstream tasks.

1. Subspace Similarity Analysis

• Compared the subspaces spanned by different ranks (e.g., r = 8 and r = 64) using singular value decomposition (SVD) and calculated their overlap using Grassmann distance.

  • The top singular vectors of ∆W overlap significantly between ranks r = 8 and r = 64, indicating that most useful information is captured by the top singular vectors.

  • The low-rank adaptation matrices capture the most important directions for task-specific performance, and increasing the rank beyond a certain point does not add meaningful information.

Relationship between the adaptation matrix (∆W) and the pre-trained weight matrix (W) in a Transformer model.

1. Does ∆W correlate with W?

• Frobenius norm of W itself (||W||_F) to see how much of W's information is captured by ∆W.

1. How large is ∆W compared to W?

The amplification factor is significant.

• For r = 4, the amplification factor is about 21.5 (calculated as 6.91 / 0.32).

• For r = 64, the factor is smaller, but the trend remains.

1. What features in W are amplified by ∆W?

• low-rank adaptation matrix potentially amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model.

Takeaway

• LoRA’s low-rank updates (∆W) are effective because they amplify underemphasized features from the pre-trained weights (W), making them more relevant for specific downstream tasks.

• Even small ranks (e.g., r = 4) can significantly improve performance by targeting the right features, which is why LoRA performs well with fewer trainable parameters.

Key works