Keep the gradient flowing

Optimization Nuggets: Stochastic Polyak Step-size, Part 2

Sun, 19 Nov 2023 00:00:00 +0100

This blog post discusses the convergence rate of the Stochastic Gradient Descent with Stochastic Polyak Step-size (SGD-SPS) algorithm for minimizing a finite sum objective. Building upon the proof of the previous post, we show that the convergence rate can be improved to O(1/t) under the additional assumption that …

Optimization Nuggets: Stochastic Polyak Step-size

Fabian Pedregosa — Fri, 29 Sep 2023 00:00:00 +0200

The stochastic Polyak step-size (SPS) is a practical variant of the Polyak step-size for stochastic optimization. In this blog post, we'll discuss the algorithm and provide a simple analysis for convex objectives with bounded gradients.

On the Convergence of the Unadjusted Langevin Algorithm

Wed, 14 Jun 2023 00:00:00 +0200

The Langevin algorithm is a simple and powerful method to sample from a probability distribution. It's a key ingredient of some machine learning methods such as diffusion models and differentially private learning. In this post, I'll derive a simple convergence analysis of this method in the special case when the …

The Russian Roulette: An Unbiased Estimator of the Limit

Fabian Pedregosa — Sat, 15 Oct 2022 00:00:00 +0200

The idea for what was later called Monte Carlo method occurred to me when I was playing solitaire during my illness.

Stanislaw Ulam, Adventures of a Mathematician

The Russian Roulette offers a simple way to construct an unbiased estimator for the limit of a sequence. It allows for example to …

Notes on the Frank-Wolfe Algorithm, Part III: backtracking line-search

Fabian Pedregosa — Fri, 26 Aug 2022 00:00:00 +0200

Backtracking step-size strategies (also known as adaptive step-size or approximate line-search) that set the step-size based on a sufficient decrease condition are the standard way to set the step-size on gradient descent and quasi-Newton methods. However, these techniques are much less common for Frank-Wolfe-like algorithms. In this blog post I …

On the Link Between Optimization and Polynomials, Part 5

Fri, 27 May 2022 00:00:00 +0200

Six: All of this has happened before.
Baltar: But the question remains, does all of this have to happen again?
Six: This time I bet no.
Baltar: You know, I've never known you to play the optimist. Why the change of heart?
Six: Mathematics. Law of averages. Let a complex …

Optimization Nuggets: Implicit Bias of Gradient-based Methods

Fabian Pedregosa — Mon, 10 Jan 2022 00:00:00 +0100

When an optimization problem has multiple global minima, different algorithms can find different solutions, a phenomenon often referred to as the implicit bias of optimization algorithms. In this post we'll characterize the implicit bias of gradient-based methods on a class of regression problems that includes linear least squares and Huber …

Optimization Nuggets: Exponential Convergence of SGD

Fabian Pedregosa — Wed, 15 Dec 2021 00:00:00 +0100

This is the first of a series of blog posts on short and beautiful proofs in optimization (let me know what you think in the comments!). For this first post in the series I'll show that stochastic gradient descent (SGD) converges exponentially fast to a neighborhood of the solution.

On the Link Between Optimization and Polynomials, Part 4

Fabian Pedregosa — Tue, 13 Apr 2021 00:00:00 +0200

While the most common accelerated methods like Polyak and Nesterov incorporate a momentum term, a little known fact is that simple gradient descent –no momentum– can achieve the same rate through only a well-chosen sequence of step-sizes. In this post we'll derive this method and through simulations discuss its practical …

On the Link Between Optimization and Polynomials, Part 3

Fabian Pedregosa — Tue, 02 Mar 2021 00:00:00 +0100

I've seen things you people wouldn't believe.
Valleys sculpted by trigonometric functions.
Rates on fire off the shoulder of divergence.
Beams glitter in the dark near the Polyak gate.
All those landscapes will be lost in time, like tears in rain.
Time to halt.

A momentum optimizer *

On the Link Between Optimization and Polynomials, Part 2

Fabian Pedregosa — Mon, 21 Dec 2020 00:00:00 +0100

We can tighten the analysis of gradient descent with momentum through a cobination of Chebyshev polynomials of the first and second kind. Following this connection, we'll derive one of the most iconic methods in optimization: Polyak momentum.

On the Link Between Polynomials and Optimization, Part 1

Fabian Pedregosa — Tue, 07 Apr 2020 00:00:00 +0200

There's a fascinating link between minimization of quadratic functions and polynomials. A link that goes deep and allows to phrase optimization problems in the language of polynomials and vice versa. Using this connection, we can tap into centuries of research in the theory of polynomials and shed new light on …

How to Evaluate the Logistic Loss and not NaN trying

Fri, 27 Sep 2019 00:00:00 +0200

A naive implementation of the logistic regression loss can results in numerical indeterminacy even for moderate values. This post takes a closer look into the source of these instabilities and discusses more robust Python implementations.

Notes on the Frank-Wolfe Algorithm, Part II: A Primal-dual Analysis

Fabian Pedregosa — Sat, 17 Nov 2018 00:00:00 +0100

This blog post extends the convergence theory from the first part of these notes on the Frank-Wolfe (FW) algorithm with convergence guarantees on the primal-dual gap which generalize and strengthen the convergence guarantees obtained in the first part.

Three Operator Splitting

Fabian Pedregosa — Thu, 06 Sep 2018 00:00:00 +0200

I discuss a recently proposed optimization algorithm: the Davis-Yin three operator splitting.

Notes on the Frank-Wolfe Algorithm, Part I

Fabian Pedregosa — Wed, 21 Mar 2018 00:00:00 +0100

This blog post is the first in a series discussing different theoretical and practical aspects of the Frank-Wolfe algorithm.

Optimization inequalities cheatsheet

Fabian Pedregosa — Wed, 11 Jan 2017 00:00:00 +0100

Most proofs in optimization consist in using inequalities for a particular function class in some creative way. This is a cheatsheet with inequalities that I use most often. It considers class of functions that are convex, strongly convex and $L$-smooth.

A fully asynchronous variant of the SAGA algorithm

Fabian Pedregosa — Wed, 12 Oct 2016 00:00:00 +0200

My friend Rémi Leblond has recently uploaded to ArXiv our preprint on an asynchronous version of the SAGA optimization algorithm.

The main contribution is to develop a parallel (fully asynchronous, no locks) variant of the SAGA algorighm. This is a stochastic variance-reduced method for general optimization, specially adapted for problems …

Hyperparameter optimization with approximate gradient

Fabian Pedregosa — Wed, 25 May 2016 00:00:00 +0200

TL;DR: I describe a method for hyperparameter optimization by gradient descent.

Most machine …

Lightning v0.1

Fabian Pedregosa — Fri, 25 Mar 2016 00:00:00 +0100

Announce: first public release of lightning!, a library for large-scale linear classification, regression and ranking in Python. The library was started a couple of years ago by Mathieu Blondel who also contributed the vast majority of source code. I joined recently its development and decided it was about time for …

scikit-learn-contrib, an umbrella for scikit-learn related projects.

Fabian Pedregosa — Sun, 06 Mar 2016 00:00:00 +0100

Together with other scikit-learn developers we've created an umbrella organization for scikit-learn-related projects named scikit-learn-contrib. The idea is for this organization to host projects that are deemed too specific or too experimental to be included in the scikit-learn codebase but still offer an API which is compatible with scikit-learn and …

SAGA algorithm in the lightning library

Fabian Pedregosa — Mon, 22 Feb 2016 00:00:00 +0100

Recently I've implemented, together with Arnaud Rachez, the SAGA[¹] algorithm in the lightning machine learning library (which by the way, has been recently moved to the new scikit-learn-contrib project). The lightning library uses the same API as scikit-learn but is particularly adapted to online learning. As for the SAGA …

On the consistency of ordinal regression methods

Fabian Pedregosa — Fri, 09 Oct 2015 00:00:00 +0200

My latests work (with Francis Bach and Alexandre Gramfort) is on the consistency of ordinal regression methods. It has the wildly imaginative …

Holdout cross-validation generator

Fabian Pedregosa — Thu, 20 Aug 2015 00:00:00 +0200

Cross-validation iterators in scikit-learn are simply generator objects, that is, Python objects that implement the __iter__ method and that for each call to this method return (or more precisely, yield) the indices or a boolean mask for the train and test set. Hence, implementing new cross-validation iterators that behave as …

IPython/Jupyter notebook gallery

Fabian Pedregosa — Tue, 21 Apr 2015 00:00:00 +0200

Due to lack of time and interest, I'm no longer maintaining this project. Feel free to grab the sources from https://github.com/fabianp/nbgallery and fork the project.

TL;DR I created a gallery for IPython/Jupyter notebooks. Check it out :-)

A couple of months ago I put online …

PyData Paris - April 2015

Fabian Pedregosa — Tue, 07 Apr 2015 00:00:00 +0200

Last Friday was PyData Paris, in words of the organizers, ''a gathering of users and developers of data analysis tools in Python''.

The organizers did a great job in putting together and the event started already with a full room for Gael's keynote

My take-away message from the talks is …

Data-driven hemodynamic response function estimation

Fabian Pedregosa — Fri, 05 Dec 2014 00:00:00 +0100

My latest research paper[¹] deals with the estimation of the hemodynamic response function (HRF) from fMRI data.

This is an important topic since the knowledge of a hemodynamic response function is what makes it possible to extract the brain activation maps that are used in most of the impressive …

Plot memory usage as a function of time

Fabian Pedregosa — Fri, 07 Nov 2014 00:00:00 +0100

:og_image: http://fa.bianp.net/blog/images/2014/mprof_example.png

One of the lesser known features of the memory_profiler package is its ability to plot memory consumption as a function of time. This was implemented by my friend Philippe Gervais, previously a colleague at INRIA and now at Google.

With …

Surrogate Loss Functions in Machine Learning

Fabian Pedregosa — Fri, 20 Jun 2014 00:00:00 +0200

TL; DR These are some notes on calibration of surrogate loss functions in the context of machine learning. But mostly it is …

Different ways to get memory consumption or lessons learned from ``memory_profiler``

Fabian Pedregosa — Thu, 25 Jul 2013 00:00:00 +0200

As part of the development of memory_profiler I've tried several ways to get memory usage of a program from within Python. In this post I'll describe the different alternatives I've tested.

The psutil library

psutil is a python library that provides an interface for retrieving information on running processes. It …

Numerical optimizers for Logistic Regression

Fabian Pedregosa — Mon, 20 May 2013 00:00:00 +0200

In this post I compar several implementations of Logistic Regression. The task was to implement a Logistic Regression model using standard optimization …

Logistic Ordinal Regression

Fabian Pedregosa — Thu, 02 May 2013 00:00:00 +0200

TL;DR: I've implemented a logistic ordinal regression or proportional odds model. Here is the Python code

The logistic ordinal regression model …

Isotonic Regression

Fabian Pedregosa — Tue, 16 Apr 2013 00:00:00 +0200

My latest contribution for scikit-learn is an implementation of the isotonic regression model that I coded with Nelle Varoquaux and Alexandre Gramfort …

Householder matrices

Fabian Pedregosa — Sat, 30 Mar 2013 00:00:00 +0100

Householder matrices are square matrices of the form

$$ P = I - \beta v v^T$$

where $\beta$ is a scalar and $v$ is …

Loss Functions for Ordinal regression

Fabian Pedregosa — Wed, 27 Feb 2013 00:00:00 +0100

** Note: this post contains a fair amount of LaTeX, if you don't visualize the math correctly come to its original location **

In …

Memory plots with memory_profiler

Fabian Pedregosa — Fri, 04 Jan 2013 00:00:00 +0100

Besides performing a line-by-line analysis of memory consumption, memory_profiler exposes some functions that allow to retrieve the memory consumption of a function in real-time, allowing e.g. to visualize the memory consumption of a given function over time.

The function to be used is memory_usage. The first argument specifies what …

Singular Value Decomposition in SciPy

Fabian Pedregosa — Sat, 08 Dec 2012 00:00:00 +0100

SciPy contains two methods to compute the singular value decomposition (SVD) of a matrix: scipy.linalg.svd and scipy.sparse.linalg.svds. In this post I'll compare both methods for the task of computing the full SVD of a large dense matrix.

The first method, scipy.linalg.svd, is perhaps …

Learning to rank with scikit-learn: the pairwise transform

Fabian Pedregosa — Tue, 23 Oct 2012 00:00:00 +0200

This tutorial introduces the concept of pairwise preference used in most ranking problems. I'll use scikit-learn and for learning and matplotlib for …

line-by-line memory usage of a Python program

Fabian Pedregosa — Tue, 24 Apr 2012 07:04:00 +0200

My newest project is a Python library for monitoring memory consumption of arbitrary process, and one of its most useful features is the line-by-line analysis of memory usage for Python code. I wrote a basic prototype six months ago after being surprised by the lack of related tools. I wanted …

Low rank approximation

Fabian Pedregosa — Sun, 06 Nov 2011 12:05:00 +0100

A little experiment to see what low rank approximation looks like. These are the best rank-k approximations (in the Frobenius norm) to the a natural image for increasing values of k and an original image of rank 512.

Python code can be found here. GIF animation made using ImageMagic's convert …

qr_multiply function in scipy.linalg

Fabian Pedregosa — Fri, 14 Oct 2011 16:44:00 +0200

In scipy's development version there's a new function closely related to the QR-decomposition of a matrix and to the least-squares solution of a linear system. What this function does is to compute the QR-decomposition of a matrix and then multiply the resulting orthogonal factor by another arbitrary matrix. In pseudocode …

scikit-learn 0.9

Fabian Pedregosa — Sun, 02 Oct 2011 11:19:00 +0200

Last week we released a new version of scikit-learn. The Changelog is particularly impressive, yet personally this release is important for other reasons. This will probably be my last release as a paid engineer. I'm starting a PhD next month, and although I plan to continue contributing to the project …

Reworked example gallery for scikit-learn

Fabian Pedregosa — Sun, 04 Sep 2011 20:09:00 +0200

I've been working lately in improving the scikit-learn example gallery to show also a small thumbnail of the plotted result. Here is what the gallery looks like now:

And the real thing should be already displayed in the development-documentation. The next thing is to add a static image to those …

scikit-learn’s EuroScipy 2011 coding sprint -- day two

Fabian Pedregosa — Thu, 25 Aug 2011 00:33:00 +0200

Today's coding sprint was a bit more crowded, with some notable scipy hackers such as Ralph Gommers, Stefan van der Walt, David Cournapeau or Fernando Perez from Ipython joining in. On what got done: - We merged Jake's new BallTree code. This is a pure Cython implementation of a nearest-neighbor …

scikit-learn EuroScipy 2011 coding sprint -- day one

Fabian Pedregosa — Tue, 23 Aug 2011 21:38:00 +0200

As a warm-up for the upcoming EuroScipy-conference, some of the scikit-learn developers decided to gather and work together for a couple of days. Today was the first day and there was only a handfull of us, as the real kickoff is expected tomorrow. Some interesting coding happened, although most of …

Ridge regression path

Fabian Pedregosa — Tue, 12 Jul 2011 09:21:00 +0200

Ridge coefficients for multiple values of the regularization parameter can be elegantly computed by updating the thin SVD decomposition of the design matrix:

import numpy as np
from scipy import linalg
def ridge(A, b, alphas):
    """
    Return coefficients for regularized least squares

         min ||A x - b||^2 + alpha ||x||^2 …

LLE comes in different flavours

Fabian Pedregosa — Thu, 30 Jun 2011 16:22:00 +0200

I haven't worked in the manifold module since last time, yet thanks to Jake VanderPlas there are some cool features I can talk about. First of, the ARPACK backend is finally working and gives factor one speedup over the lobcpg + PyAMG approach. The key is to use ARPACK's shift-invert mode …

Manifold learning in scikit-learn

Fabian Pedregosa — Tue, 07 Jun 2011 09:19:00 +0200

The manifold module in scikit-learn is slowly progressing: the locally linear embedding implementation was finally merged along with some documentation. At about the same time but in a different timezone, Jake VanderPlas began coding other manifold learning methods and back in Paris Olivier Grisel made my digits example a lot …

Handwritten digits and Locally Linear Embedding

Fabian Pedregosa — Wed, 04 May 2011 10:46:00 +0200

I decided to test my new Locally Linear Embedding (LLE) implementation against a real dataset. At first I didn't think this would turn out very well, since LLE seems to be somewhat fragile, yielding largely different results for small differences in parameters such as number of neighbors or tolerance, but …

Low-level routines for Support Vector Machines

Fabian Pedregosa — Wed, 27 Apr 2011 15:27:00 +0200

I've been working lately in improving the low-level API of the libsvm bindings in scikit-learn. The goal is to provide an API that encourages an efficient use of these libraries for expert users. These are methods that have lower overhead than the object-oriented interface as they are closer to the …

new get_blas_funcs in scipy.linalg

Fabian Pedregosa — Sat, 23 Apr 2011 18:24:00 +0200

Today got merged some changes I made to function scipy.linalg.get_blas_funcs(). The main enhacement is that get_blas_funcs() now also accepts a single string as input parameter and a dtype, so that fetching the BLAS function for a specific type becomes more natural. For example, fetching the gemm routine for …

Locally linear embedding and sparse eigensolvers

Fabian Pedregosa — Thu, 21 Apr 2011 14:28:00 +0200

I've been working for some time on implementing a locally linear embedding algorithm for the upcoming manifold module in scikit-learn. While several implementations of this algorithm exist in Python, as far as I know none of them is able to use a sparse eigensolver in the last step of the …

scikits.learn is now part of pythonxy

Fabian Pedregosa — Wed, 20 Apr 2011 13:48:00 +0200

The guys behind pythonxy have been kind enough to add the latest scikit-learn as an additional plugin for their distribution. Having scikit-learn being in both pythonxy and EPD will hopefully make it easier to use for Windows users. For now I will continue to make windows precompiled binaries, but pythonxy …

Least squares with equality constrain

Fabian Pedregosa — Thu, 14 Apr 2011 10:02:00 +0200

The following algorithm computes the Least squares solution || Ax - b|| subject to the equality constrain Bx = d. It's a classic algorithm that can be implemented only using a QR decomposition and a least squares solver. This implementation uses numpy and scipy. It makes use of the new linalg.solve_triangular function …

A profiler for Python extensions

Fabian Pedregosa — Wed, 06 Apr 2011 14:02:00 +0200

Profiling Python extensions has not been a pleasant experience for me, so I made my own package to do the job. Existing alternatives were either hard to use, forcing you to recompile with custom flags like gprofile or desperately slow like valgrind/callgrind. The package I'll talk about is called …

scikit-learn coding sprint in Paris

Fabian Pedregosa — Sat, 02 Apr 2011 12:07:00 +0200

Yesterday was the scikit-learn coding sprint in Paris. It was great to meet with old developers (Vincent Michel) and new ones: some of whom I was already familiar with from the mailing list while others came just to say hi and get familiar with the code. It was really great …

py3k in scikit-learn

Fabian Pedregosa — Mon, 28 Mar 2011 15:23:00 +0200

One thing I'd really like to see done in this Friday's scikit-learn sprint is to have full support for Python 3. There's a branch were the hard word has been done (porting C extensions, automatic 2to3 conversion, etc.), although joblib still has some bugs and no one has attempted to …

Computing the vector norm

Fabian Pedregosa — Tue, 15 Feb 2011 10:31:00 +0100

Update: a fast and stable norm was added to scipy.linalg in August 2011 and will be available in scipy 0.10 Last week I discussed with Gael how we should compute the euclidean norm of a vector a using SciPy. Two approaches suggest themselves, either calling scipy.linalg.norm …

Smells like hacker spirit

Fabian Pedregosa — Fri, 11 Feb 2011 09:50:00 +0100

I was last weekend in FOSDEM presenting scikits.learn (here are the slides I used at the Data Analytics Devroom). Kudos to Olivier Grisel and all the people who organized such a fun and authentic meeting!

New examples in scikits.learn 0.6

Fabian Pedregosa — Fri, 31 Dec 2010 13:55:00 +0100

Latest release of scikits.learn comes with an awesome collection of examples. These are some of my favorites:

Faces recognition

This example by Olivier Grisel, downloads a 58MB faces dataset from Labeled Faces in the Wild, and is able to perform PCA for feature extraction and SVC for classification, yielding …

Weighted samples for SVMs

Fabian Pedregosa — Mon, 29 Nov 2010 13:20:00 +0100

Based on the work of libsvm-dense by Ming-Wei Chang, Hsuan-Tien Lin, Ming-Hen Tsai, Chia-Hua Ho and Hsiang-Fu Yu I patched the libsvm distribution shipped with scikits.learn to allow setting weights for individual instances. The motivation behind this is to be able force a classifier to focus its attention in …

Coming soon ...

Fabian Pedregosa — Wed, 24 Nov 2010 10:39:00 +0100

Highlights for this release: * New stochastic gradient descent module by Peter Prettenhofer * Improved svm module: memory efficiency, automatic class weights. * Wrap for liblinear's Multi-class SVC (option multi_class in LinearSVC) * New features and performance improvements of text feature extraction. * Improved sparse matrix support, both in main classes (GridSearch) as in sparse …

memory efficient bindigs for libsvm

Fabian Pedregosa — Fri, 19 Nov 2010 15:08:00 +0100

scikits.learn.svm now uses LibSVM-dense instead of LibSVM for some support vector machine related algorithms when input is a dense matrix. As a result most of the copies associated with argument passing are avoided, giving 50% less memory footprint and several times less than the python bindings that ship …

solve triangular matrices using scipy.linalg

Fabian Pedregosa — Sat, 30 Oct 2010 01:13:00 +0200

For some time now I've been missing a function in scipy that exploits the triangular structure of a matrix to efficiently solve the associated system, so I decided to implement it by binding the LAPACK method "trtrs", which also checks for singularities and is capable handling several right-hand sides. Contrary …

LARS algorithm

Fabian Pedregosa — Thu, 30 Sep 2010 16:01:00 +0200

I've been working lately with Alexandre Gramfort coding the LARS algorithm in scikits.learn. This algorithm computes the solution to several general linear models used in machine learning: LAR, Lasso, Elasticnet and Forward Stagewise. Unlike the implementation by coordinate descent, the LARS algorithm gives the full coefficient path along the …

Second scikits.learn coding sprint

Fabian Pedregosa — Sun, 12 Sep 2010 22:31:00 +0200

Las week took place in Paris the second scikits.learn sprint. It was two days of insane activity (115 commits, 6 branches, 33 coffees) in which we did a lot of work, both implementing new algorithms and fixing or improving old ones. This includes: * sparse version of Lasso by coordinate …

Support for sparse matrices in scikits.learn

Fabian Pedregosa — Mon, 23 Aug 2010 17:47:00 +0200

I recently added support for sparse matrices (as defined in scipy.sparse) in some classifiers of scikits.learn. In those classes, the fit method will perform the algorithm without converting to a dense representation and will also store parameters in an efficient format. Right now, the only classese that implements …

Flags to debug python C extensions.

Fabian Pedregosa — Wed, 18 Aug 2010 13:40:00 +0200

I often find myself debugging python C extensions from gdb, but usually some variables are hidden because aggressive optimizations that distutils sets by default. What I did not know, is that you can prevent those optimizations by passing flags -O0 -fno-inline to gcc in keyword extra_compile_args (note: this will only …

July in Paris

Fabian Pedregosa — Fri, 30 Jul 2010 00:11:00 +0200

One of the best things of spending summer in Paris: its parcs (here, with friends @ Parc Montsouris).

Support Vector machines with custom kernels using scikits.learn

Fabian Pedregosa — Thu, 27 May 2010 10:42:00 +0200

It is now possible (using the development version as of may 2010) to use Support Vector Machines with custom kernels in scikits.learn. How to use it couldn't be more simple: you just pass a callable (the kernel) to the class constructor). For example, a linear kernel would be implemented …

Howto link against system-wide BLAS library using numpy.distutils

Fabian Pedregosa — Thu, 22 Apr 2010 14:28:00 +0200

If your numpy installation uses system-wide BLAS libraries (this will most likely be the case unless you installed it through prebuilt windows binaries), you can retrieve this information at compile time to link python modules to BLAS. The function get_info in numpy.distutils.system_info will return a dictionary that contains …

scikits.learn 0.2 release

Fabian Pedregosa — Mon, 22 Mar 2010 11:37:00 +0100

Today I released a new version of the scikits.learn library for machine learning. This new release includes the new libsvm bindings, Jake VanderPlas' BallTree algorithm for *fast* nearest neighbor queries in high dimension, etc. Here is the official announcement. As usual, it can be downloaded from sourceforge or from …

Plot the maximum margin hyperplane with scikits.learn

Fabian Pedregosa — Wed, 17 Mar 2010 12:24:00 +0100

Suppose some given data points each belong to one of two classes, and the goal is to decide which class a new data point will be in. In the case of support vector machines, a data point is viewed as a p-dimensional vector (2-dimensional in this example), and we want …

Fast bindings for LibSVM in scikits.learn

Fabian Pedregosa — Tue, 09 Mar 2010 15:49:00 +0100

LibSVM is a C++ library that implements several Support Vector Machine algorithms that are commonly used in machine learning. It is a fast library that has no dependencies and most machine learning frameworks bind it in some way or another. LibSVM comes with a Python interface written in swig, but …

scikits.learn coding sprint in Paris

Fabian Pedregosa — Thu, 04 Mar 2010 11:25:00 +0100

Yesterday we had an extremely productive coding sprint for the scikits.learn. The idea was to put people with common interests in a room and make them work in a single codebase. Alexandre Gramfort and Olivier Grisel worked on GLMNet, Bertrand Thirion and Gaël Varoquaux worked on univariate feature selection …

Scikit-learn 0.1

Fabian Pedregosa — Mon, 01 Feb 2010 15:32:00 +0100

Today I released the first public version of Scikit-Learn (release notes). It's a python module implementing some machine learning algorithms, and it's shaping quite good.

For this release I did not want to do any incompatible changes, so most of them are just bug fixes and updates. For the next …

scikit-learn project on sourceforge

Fabian Pedregosa — Thu, 07 Jan 2010 15:17:00 +0100

This week we created a sourceforge project to host our development of scikit-learn. Although the project already had a directory in scipy's repo, we needed more flexibility in the user management and in the mailing list creation, so we opted for SourceForge. To be honest, after using git and Google …

After holidays

Fabian Pedregosa — Tue, 05 Jan 2010 10:56:00 +0100

New job, new code, new city, new colleagues. Feels something like this:

Winter in Paris is not funny

Fabian Pedregosa — Tue, 22 Dec 2009 19:36:00 +0100

This week I arrived to the place where I will be working the following two years: Neurospin.

It's a research center located 20 km from Paris, and so far things are going smoothly: the place is beautiful, work is great and food is excellent. Well OK, I do miss some …

Last days in Granada

Fabian Pedregosa — Tue, 15 Dec 2009 23:42:00 +0100

Nice thing about winter in Granada is, that even in the coldest days, the sky is always blue.

Learning, Machine Learning

Fabian Pedregosa — Tue, 15 Dec 2009 23:34:00 +0100

My new job is about managing an open source package for machine learning in Python. I've had some experience with Python now, but I am a total newbie in the field of machine learning, so my first task will be to find a good reference book in the subject and …

Moving to Paris!

Fabian Pedregosa — Sat, 12 Dec 2009 02:07:00 +0100

I'm extremely glad that finally I am moving to Paris to work as part of the INRIA crew. I'll be working with Gael Varoquaux and his team in an extremely cool Python related project (more to come on this in the following weeks). Granada has been a great place for …

Summer of Code is over

Fabian Pedregosa — Sat, 05 Sep 2009 12:21:00 +0200

Google Summer of Code program is officially over. It has been four months of intense work, exciting benchmarks and patch reviewing. It was a huge pleasure working with you guys! As for the project, I implemented a complete logic module and then an assumption system for sympy (sympy.logic, sympy …

Speed improvements for ask() (sympy.queries.ask)

Fabian Pedregosa — Thu, 20 Aug 2009 00:36:00 +0200

I managed to overcome the overhead in ask() that arises when converting between symbol and integer representation of sentences in conjunctive normal. The result went beyond what I expected. The test suite for the query module got 10x times faster in my laptop. From 26 seconds, it descended to an …

Logic module (sympy.logic): improving speed

Fabian Pedregosa — Tue, 18 Aug 2009 23:35:00 +0200

Today I've been doing some speed improvements for the logic module. More precisely, I implemented an efficient internal representation for clauses in conjunctive normal form. In practice this means a huge performance boost for all problems that make use the function satisfiable() or dpll_satisfiable(). For example, test_dimacs.py has moved …

Refine module

Fabian Pedregosa — Mon, 17 Aug 2009 19:20:00 +0200

This commit introduced a new module in sympy: the refine module. The purpose of this module is to simplify expressions when they are bound to assumptions. For example, if you know that x>0, then you can simplify abs(x) to x. This code was traditionally embedded into the core …

Query module - finally in trunk

Fabian Pedregosa — Mon, 10 Aug 2009 21:51:00 +0200

The query module is finally in the main SymPy repository. I made substantial changes since last post, most of them at the user interface level (thanks to Vinzent and Mateusz for many insightful comments). Main function is ask(), which replaces the old expression.is_* syntax. You can ask many things …

django, change language settings dynamically

Fabian Pedregosa — Fri, 07 Aug 2009 16:12:00 +0200

After some failed attempts, I just found how to change the language settings dynamically in django, and I thought it could be useful to someone. Just use function activate() from django.utils.translation. For example: [cc lang="python"] from django.utils.translation import activate activate('es-ES') [/cc] will change global …

can we merge now, pleeease ?

Fabian Pedregosa — Tue, 21 Jul 2009 22:36:00 +0200

Three months after I began to write sympy.queries, I feel it's about time to include it in sympy's trunk, so today I sent for review 4 patches that implement the complete query module. It's been a lot of fun, but it has also caused me some headaches ... specially last …

Refine module, proof of concept

Fabian Pedregosa — Thu, 09 Jul 2009 02:49:00 +0200

The 0.6.5 release of SymPy is taking longer than expected because some bugs in the testing framework, so my query module is not merged into trunk (yet). In the meantime, I am implementing a refine module (very little code is available yet). The refine module implements a refine …

Preparing a new release

Fabian Pedregosa — Tue, 30 Jun 2009 08:57:00 +0200

Last days I've been busy preparing the first public beta of SymPy 0.6.5. Most of the time was spent solving a bug that made documentation tests fail under python2.4, but now that this is solved, I hope that by the end of the week we could have …

Efficient DPLL algorithm

Fabian Pedregosa — Sun, 28 Jun 2009 18:16:00 +0200

Background: DPLL is the algorithm behind SymPy's implementation of logic.inference.satisfiable After reading the original papers by Davis & Putnam [1], I managed to implement a more efficient version of the DPLL algorithm. It is 10x times faster on medium-sized problems (40 variables), and solves some wrong result bugs [2 …

Queries and performance

Fabian Pedregosa — Tue, 23 Jun 2009 00:02:00 +0200

After some hacking on the queries module, I finally got it right without the limitations of past versions. You can check it out from my repo http://fa.bianp.net/git/sympy.git, branch master. It now relies even more on logic.inference.satisfiable(), which is just an implementation of …

Reading CNF files

Fabian Pedregosa — Sat, 20 Jun 2009 16:27:00 +0200

The DIMACS CNF file format is used to define a Boolean expression, written in conjunctive normal form, that may be used as an example of the satisfiability problem. The new logic module (sympy.logic) can read the content of a cnf file and transform it into a boolean expression suitable …

Logic module merged

Fabian Pedregosa — Fri, 19 Jun 2009 12:23:00 +0200

Yesterday I finally merged the logic module in sympy's official master branch, and should be released together with SymPy 0.6.5. Next thing to do: profile the code and write some docs before the release.

The boolean satisfiability problem

Fabian Pedregosa — Mon, 15 Jun 2009 06:00:00 +0200

Most annoying problem in my implementation of the query system is that it will not solve implications if the implicates are far away from each other. For instance, if the graph of known facts is something like this

Integer ----> Rational --> Real --> Complex
  ^  ^
  |  |
  |   -------
  |         |
Prime      Even
  ^
  |
  |
MersennePrime

Then it will not know …

Initial implementation of the query system

Fabian Pedregosa — Fri, 12 Jun 2009 06:36:00 +0200

I sent some patches to sympy-patches with an initial implementation of the query system. You can check it out by pulling from my branch: git pull http://fa.bianp.net/git/sympy.git master into your sympy repo. Some examples of what you can do (sample isympy session): In [1 …

Assumption system and automatic theorem proving. Should I be learning LISP ?

Fabian Pedregosa — Wed, 03 Jun 2009 13:50:00 +0200

This is the third time I attempt to write the assumption system. Other attempts could be described as me following the rule: “For any complex problem, there is always a solution that is simple, clear, and wrong.” My first attempt (although better than the current assumption system) did use very …

Homenaje a Antonio Vega en La Percha

Fabian Pedregosa — Mon, 01 Jun 2009 15:34:00 +0200

El pasado jueves estuvimos en La Percha tocando algunas canciones de Antonio Vega. El vídeo se lo ha currado mi padre mezclando el sonido del directo con una grabación que hicimos en casa de Migue

LOS ESCLAVOS: homenaje a Antonio Vega from Felipe Pedregosa on Vimeo.

Fun with the new Logic module

Fabian Pedregosa — Sun, 31 May 2009 23:16:00 +0200

The logic module is slowly becoming useful. This week I managed to get some basic inference in propositional logic working. This should be enough for the assumption sysmtem (although having first-order inference would be cool). You can pull from my branch: `` git pull http://fa.bianp.net/git/sympy.git …