A birds-eye view of optimization algorithms

Outline

• 1. Introduction to optimization algorithms.
• 2. A unifying principle: Surrogate Optimization.
• 3 Gradient descent.
• 4. Newton's method.
• 5. Quasi-Newton.
• 6. Exploting Structure: The Stochastic Gradient Method.
• 7. Software: scipy.optimize.

Credits. This material was created by Fabian Pedregosa for an invied lecture at McGill University. Source code can be found here. The template and the visualizations are modified from the distill article How momentum really works. Some parts of the knowing your problem section are based on the scipy lecture notes.