While I've heard of LoRA (Low Rank Adaptation) ever since I've been reading up on ML and LLMs, I've never really explored that side. Did not dare to. I was getting tired of the many papers on LLMs and the oversaturation of research.
Until... I was shown a LinkedIn Post on MiCA (Minor Component Analysis), which claimed to be better than LoRA. And that kind of set me off into figuring out what all these methods were about.
Now this TIL comes way later from the day I've learnt about these, but this is because I swamped myself with papers, work (loved every minute of it) and household chores like cooking (as a breather).
Enough yap, what's PEFT?¶
Parameter Efficient Fine Tuning, or PEFT in short, are a suite of techniques for fine-tuning models without training the model completely all over again.
The concept of Fine Tuning is to adapt the model better to your particular task, especially when the general training data the model was pre trained falls short. While it improves performance, you'll have to still train the model. And if you take a large enough model, then the trainable parameters would increase the computation cost exponentially. How do we reel back the computation cost while still being able to adapt the model to our needs? Enter PEFT.
There are multiple methods that achieve PEFT, but usually they work by freezing the weights of the original model and then training a small number of additional parameters on top of it.
LoRA¶
Low Rank Adaptation is one of the most popular PEFT methods. It works by decomposing the weight updates of a matrix into two lower-rank matrices. This means that instead of training the original weight matrix, we train two smaller matrices that approximate the original weight matrix.
The math: Let W be the original weight matrix of a layer. During fine-tuning, we want to learn a weight update $\Delta W$. Instead of learning $\Delta W$ directly, LoRA parameterizes it as $\Delta W = BA$, where
$$ A \in R^{r x k} $$ $$ B \in R^{d x r} $$
$r$ here is the rank of the matrix, $d$ is the dimension of the output of the matrix and $k$ is the input dimension of the matrix.
$$ x \in R^{k} $$ $$ W \in R^{d x k} $$ $$ W' = W + BA $$
The beauty of this is that the number of trainable parameters is now $r(d+k)$ instead of $dk$. So for a 1024x1024 matrix, if we take r=16, we're looking at 33,280 parameters instead of 1,048,576 parameters. That's over 90% reduction in trainable parameters, hence making your life easier when training the model.
Isn't this old news?¶
LoRA was out in 2021, and hence has been cited over 35000 times. There is a lot of research that improves upon LoRA, and one such method is MiCA, and that's what I have been reading in the past few days.