What Is a Fréchet Derivative?

Let U and V be Banach spaces (complete normed vector spaces). The Fréchet derivative of a function f:U \to V at X\in U is a linear mapping L:U\to V such that

\notag       f(X+E) - f(X) - L(X,E) = o(\|E\|)

for all E\in U. The notation L(X,E) should be read as “the Fréchet derivative of f at X in the direction E”. The Fréchet derivative may not exist, but if it does exist then it is unique. When U = V = \mathbb{R}, the Fréchet derivative is just the usual derivative of a scalar function: L(x,e) = f'(x)e.

As a simple example, consider U = V = \mathbb{R}^{n\times n} and f(X) = X^2. From the expansion

f(X+E) - f(X) = XE + EX + E^2

we deduce that L(X,E) = XE + EX, the first order part of the expansion. If X commutes with E then L_X(E) = 2XE = 2EX.

More generally, it can be shown that if f has the power series expansion f(x) = \sum_{i=0}^\infty a_i x^i with radius of convergence r then for X,E\in\mathbb{R}^{n\times n} with \|X\| < r, the Fréchet derivative is

L(X,E) = \displaystyle\sum_{i=1}^\infty a_i             \displaystyle\sum_{j=1}^i X^{j-1} E X^{i-j}.

An explicit formula for the Fréchet derivative of the matrix exponential, f(A) = \mathrm{e}^A, is

L(A,E) = \displaystyle\int_0^1 \mathrm{e}^{A(1-s)} E \mathrm{e}^{As} \, ds.

Like the scalar derivative, the Fréchet derivative satisfies sum and product rules: if g and h are Fréchet differentiable at A then

\notag  \begin{alignedat}{2}    f &= \alpha g + \beta h  &&\;\Rightarrow\;   L_f(A,E) = \alpha L_g(A,E) + \beta L_h(A,E),\\   f &= gh  &&\;\Rightarrow\; L_f(A,E) = L_g(A,E) h(A) + g(A) L_h(A,E). \end{alignedat}

A key requirement of the definition of Fréchet derivative is that L(X,E) must satisfy the defining equation for all E. This is what makes the Fréchet derivative different from the Gâteaux derivative (or directional derivative), which is the mapping G:U \to V given by

\notag       G(X,E) = \lim_{t\to0} \displaystyle\frac{f(X+t E)-f(X)}{t}              = \frac{\mathrm{d}}{\mathrm{dt}}\Bigm|_{t=0} f(X + tE).

Here, the limit only needs to exist in the particular direction E. If the Fréchet derivative exists at X then it is equal to the Gâteaux derivative, but the converse is not true.

A natural definition of condition number of f is

\mathrm{cond}(f,X) = \lim_{\epsilon\to0}                \displaystyle\sup_{\|\Delta X\| \le \epsilon \|X\|}                \displaystyle\frac{\|f(X+\Delta X) - f(X)\|}{\epsilon\|f(X)\|},

and it can be shown that \mathrm{cond} is given in terms of the Fréchet derivative by

\mathrm{cond}(f,X) = \displaystyle\frac{\|L(X)\| \|X\|}{\|f(X)\|},


\|L(X)\| = \sup_{Z\ne0}\displaystyle\frac{\|L(X,Z)\|}{\|Z\|}.

For matrix functions, the Fréchet derivative has a number of interesting properties, one of which is that the eigenvalues of L(X) are the divided differences

\notag    f[\lambda_i,\lambda_j] = \begin{cases}    \dfrac{ f(\lambda_i)-f(\lambda_j) }{\lambda_i - \lambda_j},    & \lambda_i\ne\lambda_j, \\    f'(\lambda_i), & \lambda_i=\lambda_j,    \end{cases}

for 1\le i,j \le n, where the \lambda_i are the eigenvalues of X. We can check this formula in the case F(X) = X^2. Let (\lambda,u) be an eigenpair of X and (\mu,v) an eigenpair of X^T, so that Xu = \lambda u and X^Tv = \mu v, and let E = uv^T. Then

L(X,E) = XE + EX = Xuv^T + uv^TX = (\lambda + \mu) uv^T.

So uv^T is an eigenvector of L(X) with eigenvalue \lambda + \mu. But f[\lambda,\mu] = (\lambda^2-\mu^2)/(\lambda - \mu) = \lambda + \mu (whether or not \lambda and \mu are distinct).


This is a minimal set of references, which contain further useful references within.

Related Blog Posts

This article is part of the “What Is” series, available from https://nhigham.com/category/what-is and in PDF form from the GitHub repository https://github.com/higham/what-is.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s