Fréchet Derivative

Chapter 11: Matrix Calculus. In the chapter mind map this icon labels Fréchet Derivative. The discussion below is excerpted and lightly edited from § Fréchet Derivative in Mathematics for AI and Machine Learning.

For matrix-valued functions $f : \mathbb{R}^{M \times N} \to \mathbb{R}^{P \times Q}$, we have seen that the Jacobian can be represented as a $(PQ) \times (MN)$ matrix via vectorization. However, there is an alternative and often more elegant viewpoint: treating the derivative as a linear operator acting directly on matrices, without vectorization.

The Fréchet derivative of a matrix-valued function $f$ at a point $X$ is a linear operator $f'(X)$ that maps a perturbation matrix to the first-order change in $f$. It is not a new kind of derivative: the scalar derivative, the gradient and the Jacobian are all its coordinate forms, and the name only stresses that it is defined as a linear map rather than as an array of partial derivatives. The definition needs only a norm on each space, so it applies unchanged to scalars in $\mathbb{R}$, vectors in $\mathbb{R}^N$, matrices in $\mathbb{R}^{M \times N}$ and functions in spaces such as $L^2(\Omega)$ or $C([0, 1])$; in finite dimensions every norm gives the same derivative, and in function spaces $f'(X)$ must also be a bounded linear map. It satisfies:

where $\Delta \in \mathbb{R}^{M \times N}$ is a perturbation matrix, and $o(\|\Delta\|)$ denotes terms that vanish faster than $\|\Delta\|$ as $\|\Delta\| \to 0$. The notation $f'(X)[\Delta]$ means applying the linear operator $f'(X)$ to the matrix $\Delta$. When the context is clear (i.e., there is only one matrix variable), we use $\Delta$ without a subscript. When multiple matrices are involved and clarity is needed, we use subscripts such as $\Delta_X$, $\Delta_A$, etc., to indicate which matrix is being perturbed. This notation is consistent with the perturbation notation used in the matrix chapter for rank-1 perturbation analysis (e.g., backward error analysis where $A + \Delta_A = QR$)[^frechet-notation].

Relationship to Vectorization: The Fréchet derivative and the vectorized Jacobian are equivalent representations. The vectorized Jacobian $\frac{\partial \mathrm{vec}(f(X))}{\partial \mathrm{vec}(X)} \in \mathbb{R}^{(PQ) \times (MN)}$ is the matrix representation of the linear operator $f'(X)$ with respect to the standard basis after vectorization. However, the operator viewpoint often provides more insight into the structure of derivatives, especially for functions like matrix inverses, matrix exponentials, and other matrix operations.