Low-Rank Adaptation
Chapter 2: Matrix — Low-Rank Factorization (also appears in Ch. 9)
From the book
Chapter 2: Matrix. In the chapter mind map this icon labels Low-Rank Factorization: LoRA & Attention $QK^\top$. The discussion below is excerpted and lightly edited from § Definition: Low-Rank Factorization in Mathematics for AI and Machine Learning. Related material also appears in Chapter 9 (Low-Rank Adaptation (LoRA): $W = W_0 + BA$).
A matrix $W \in \mathbb{R}^{m \times n}$ has a rank-$r$ factorization if
where $\mathbf u_i, \mathbf v_i$ are the columns of $U, V$; the sum is the outer-product view of matrix multiplication (View 4). By the rank of a product, $\mathrm{rank}(W) \le r$. Conversely, every $W$ of rank $r$ has such a factorization: let the columns of $U$ be a basis of $C(W)$; each column of $W$ is $U\mathbf c_j$ for some $\mathbf c_j \in \mathbb{R}^r$, and $V^\top = [\mathbf c_0, \ldots, \mathbf c_{n-1}]$. When $r \ll \min(m, n)$ this is a low-rank factorization: it stores $r(m + n)$ numbers instead of $mn$.
What this drawing shows
What you see. Shows a full update approximated by thin low-rank factors, matching the LoRA idea of efficient parameter adaptation.
In the mind map. Chapter 2 — Low-Rank Factorization: LoRA & Attention. See From the book above for definitions, figures, and worked examples.
Also appears in Ch. 9 (Low-Rank Adaptation (LoRA)).
Where to read next
Read the full definitions, figures, and worked examples in Chapter 2: Matrix — see the mind-map node Low-Rank Factorization: LoRA & Attention.
This concept is also referenced in Chapter 9 (Low-Rank Adaptation (LoRA)).
