What Is a Fréchet Derivative?

Let $U$ and $V$ be Banach spaces (complete normed vector spaces). The Fréchet derivative of a function $f:U \to V$ at $X\in U$ is a linear mapping $L:U\to V$ such that

$\notag f(X+E) - f(X) - L(X,E) = o(\|E\|)$

for all $E\in U$ . The notation $L(X,E)$ should be read as “the Fréchet derivative of $f$ at $X$ in the direction $E$ ”. The Fréchet derivative may not exist, but if it does exist then it is unique. When $U = V = \mathbb{R}$ , the Fréchet derivative is just the usual derivative of a scalar function: $L(x,e) = f'(x)e$ .

As a simple example, consider $U = V = \mathbb{R}^{n\times n}$ and $f(X) = X^2$ . From the expansion

$f(X+E) - f(X) = XE + EX + E^2$

we deduce that $L(X,E) = XE + EX$ , the first order part of the expansion. If $X$ commutes with $E$ then $L_X(E) = 2XE = 2EX$ .

More generally, it can be shown that if $f$ has the power series expansion $f(x) = \sum_{i=0}^\infty a_i x^i$ with radius of convergence $r$ then for $X,E\in\mathbb{R}^{n\times n}$ with $\|X\| < r$ , the Fréchet derivative is

$L(X,E) = \displaystyle\sum_{i=1}^\infty a_i \displaystyle\sum_{j=1}^i X^{j-1} E X^{i-j}.$

An explicit formula for the Fréchet derivative of the matrix exponential, $f(A) = \mathrm{e}^A$ , is

$L(A,E) = \displaystyle\int_0^1 \mathrm{e}^{A(1-s)} E \mathrm{e}^{As} \, ds.$

Like the scalar derivative, the Fréchet derivative satisfies sum and product rules: if $g$ and $h$ are Fréchet differentiable at $A$ then

$\notag \begin{alignedat}{2} f &= \alpha g + \beta h &&\;\Rightarrow\; L_f(A,E) = \alpha L_g(A,E) + \beta L_h(A,E),\\ f &= gh &&\;\Rightarrow\; L_f(A,E) = L_g(A,E) h(A) + g(A) L_h(A,E). \end{alignedat}$

A key requirement of the definition of Fréchet derivative is that $L(X,E)$ must satisfy the defining equation for all $E$ . This is what makes the Fréchet derivative different from the Gâteaux derivative (or directional derivative), which is the mapping $G:U \to V$ given by

$\notag G(X,E) = \lim_{t\to0} \displaystyle\frac{f(X+t E)-f(X)}{t} = \frac{\mathrm{d}}{\mathrm{dt}}\Bigm|_{t=0} f(X + tE).$

Here, the limit only needs to exist in the particular direction $E$ . If the Fréchet derivative exists at $X$ then it is equal to the Gâteaux derivative, but the converse is not true.

A natural definition of condition number of $f$ is

$\mathrm{cond}(f,X) = \lim_{\epsilon\to0} \displaystyle\sup_{\|\Delta X\| \le \epsilon \|X\|} \displaystyle\frac{\|f(X+\Delta X) - f(X)\|}{\epsilon\|f(X)\|},$

and it can be shown that $\mathrm{cond}$ is given in terms of the Fréchet derivative by

$\mathrm{cond}(f,X) = \displaystyle\frac{\|L(X)\| \|X\|}{\|f(X)\|},$

where

$\|L(X)\| = \sup_{Z\ne0}\displaystyle\frac{\|L(X,Z)\|}{\|Z\|}.$

For matrix functions, the Fréchet derivative has a number of interesting properties, one of which is that the eigenvalues of $L(X)$ are the divided differences

$\notag f[\lambda_i,\lambda_j] = \begin{cases} \dfrac{ f(\lambda_i)-f(\lambda_j) }{\lambda_i - \lambda_j}, & \lambda_i\ne\lambda_j, \\ f'(\lambda_i), & \lambda_i=\lambda_j, \end{cases}$

for $1\le i,j \le n$ , where the $\lambda_i$ are the eigenvalues of $X$ . We can check this formula in the case $F(X) = X^2$ . Let $(\lambda,u)$ be an eigenpair of $X$ and $(\mu,v)$ an eigenpair of $X^T$ , so that $Xu = \lambda u$ and $X^Tv = \mu v$ , and let $E = uv^T$ . Then

$L(X,E) = XE + EX = Xuv^T + uv^TX = (\lambda + \mu) uv^T.$

So $uv^T$ is an eigenvector of $L(X)$ with eigenvalue $\lambda + \mu$ . But $f[\lambda,\mu] = (\lambda^2-\mu^2)/(\lambda - \mu) = \lambda + \mu$ (whether or not $\lambda$ and $\mu$ are distinct).

References

This is a minimal set of references, which contain further useful references within.

Kendall Atkinson and Weimin Han, Theoretical Numerical Analysis: A Functional Analysis Framework, Springer-Verlag, New York, 2009. (Section 5.3).
Nicholas J. Higham, Functions of Matrices: Theory and Computation, Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2008. (Chapter 3).
James Ortega and Werner Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, Society for Industrial and Applied Mathematics, Philadelphia, PA, 2000. (Section 3.1).

What Is a Fréchet Derivative?

References

Related Blog Posts

Leave a Reply Cancel reply

References

Related Blog Posts

Share this:

Related

Leave a Reply Cancel reply