02-08-gradient-dynamics

#eth/cil/theory

Gradient Flow

Can we gain analytic insights into the gradient dynamics?

Use ODEs to study the gradient flow = gradient dynamics in the limit of small step sizes

Assume $u = v$ , balanced initialization and they evolve identically.
We look at $x = u v = u^{2}$ :

\begin{aligned} \frac{d u}{d t} & = - v (u v - a) \\ \frac{d x}{d t} & = \frac{d u^{2}}{d t} = - 2 u v (u v - a) = - 2 x (x - a) \end{aligned}

This ODE can be solved, yields:

x (t) = \frac{a e^{2 a t}}{e^{2 a t} - 1 + (a / c)} = a + \frac{a c - a^{2}}{c e^{2 a t} + a - c}

4-archive/cil/theory/old/assets/ode-solution.png