SDE Continuous Optimization Limit
Chapter 19: Stochastic Differential Equations — SDEs as Limits of Optimization Algorithms
From the book
Chapter 19: Stochastic Differential Equations. In the chapter mind map this icon labels SDEs as Limits of Optim Algos. The discussion below is excerpted and lightly edited from § SDEs as Limits of Optimization Algorithms in Mathematics for AI and Machine Learning.
The previous section started from algorithms with exact gradients and added the noise by hand. In stochastic optimization the gradient itself is random, and that mini-batch noise plays the role of the diffusion term, so the same SDE tools describe where noisy algorithms go and which solutions they settle in.
What this drawing shows
What you see. Green discrete optimization steps fixed; purple continuous SDE path grows from the last iterate toward the continuous-time limit.
In the mind map. Chapter 19 — SDEs as Limits of Optim Algos. See From the book above for definitions, figures, and worked examples.
Where to read next
Read the full definitions, figures, and worked examples in Chapter 19: Stochastic Differential Equations — see the mind-map node SDEs as Limits of Optim Algos.
