While the most common accelerated methods like Polyak and Nesterov incorporate a momentum term, a little known fact is that simple gradient descent –no momentum– can achieve the same rate
through only a well-chosen sequence of step-sizes. In this post we'll derive this method and through simulations discuss its practical …
I've seen things you people wouldn't believe.
Valleys sculpted by trigonometric functions.
Rates on fire off the shoulder of divergence.
Beams glitter in the dark near the Polyak gate.
All those landscapes will be lost in time, like tears in rain. Time to halt.
We can tighten the analysis of momentum methods through Chebyshev polynomials of the first and second kind. Following this connection, we'll derive one of the most iconic methods in optimization: Polyak momentum.
There's a fascinating link between minimization of quadratic functions and polynomials. A link
deep and allows to phrase optimization problems in the language of polynomials and vice versa.
Using this connection, we can tap into centuries of research in the theory of polynomials and
shed new light on …
A naive implementation of the logistic regression loss can results in numerical indeterminacy even for moderate values. This post takes a closer look into the source of these instabilities and discusses more robust Python implementations.
This blog post extends the convergence theory from the first part of these notes on the
Frank-Wolfe (FW) algorithm with convergence guarantees on the primal-dual gap which generalize
and strengthen the convergence guarantees obtained in the first part.
The main contribution is to develop a parallel (fully asynchronous, no locks) variant of the SAGA algorighm. This is a stochastic variance-reduced method for general optimization, specially adapted for problems …