Eigendecomposition

Chapter 6: Geometric Transformation and Eigen Decomposition. In the chapter mind map this icon labels Spectral Decomp $A=V\Lambda V^{-1}$. The discussion below is excerpted and lightly edited from § Spectral Decomposition in Mathematics for AI and Machine Learning.

The spectral decomposition expresses a diagonalizable matrix $A = V\Lambda V^{-1} \in \mathbb{R}^{N \times N}$ as a weighted sum of rank-1 matrices formed from its eigenvectors:

where the right eigenvectors $A\mathbf v_n = \lambda_n\mathbf v_n$ are the columns of $V$ and the left eigenvectors $\mathbf w_n^\top A = \lambda_n \mathbf w_n^\top$ (multiplying $A$ from the left) are the rows of $V^{-1}$, by row $n$ of $V^{-1}A = \Lambda V^{-1}$; $W = (V^{-1})^\top$ collects them as columns.

Each term $\lambda_n \mathbf v_n \mathbf w_n^\top$ is a rank-1 matrix that represents the contribution of the $n$-th eigenmode to the matrix $A$. The spectral decomposition reveals how $A$ acts on vectors: it projects onto each eigenvector direction, scales by the corresponding eigenvalue, and recombines the results.

Special case for symmetric matrices: When $A$ is symmetric, $\mathbf w_n = \mathbf v_n$ (left and right eigenvectors coincide), and the spectral decomposition simplifies to:

where the eigenvectors $\mathbf v_n$ form an orthonormal basis (proved in the matrix chapter). This form is particularly important in Principal Component Analysis (PCA), where symmetric covariance matrices are decomposed into principal directions.