the

making-of



Fabian Pedregosa,

Data Science Meetup

Sheffield, April 2014

Outline

1. Why scikit-learn
2. Project Vision
3. Key Features
4. Current Development

1

Who

PARIETAL (INRIA). Computer Science lab in France applying machine learning to understand brain function.
Mixed research facility with cognitive scientists and medical staff (Machine Learning as a tool). Already invested in Python (MayaVi, NiPy, BrainVisa, etc.)

2

Day-to-day technicalities

Private repo with buggy code.
Slow code.
Lead scientist leaves
I don't understand the code I've written a year ago. Code duplication, undocumented code, etc.
$\implies$ invest time and resources to release quality code

3

Project conception

January 2010: Started working for PARIETAL.
Main Python libraries at that point:
Scikit-learn was a project started by David Cournapeau for the Google Summer of Code 2007. No public release yet.

4

Project Vision

Generic machine learning library.

Implementation of classical models (not the latest NIPS paper).
Right trade-off between efficiency and maintainability: extensive use of NumPy and SciPy. Cython for low-level optimization. Whenever possible optimize algorithms, not for loops (avoid C/C++)
Domain agnostic, benefit from cross-domain interaction.

5

Project Vision

Easy to use machine learning library.

Easy setup, works out of the box (installing software sucks)
Easy to debug. As little magic as possible.
Extensive documentation (software that is not documented is software that does not exist)

6

Project Vision

Community project

Goal: project that would live when my contract expired
Decision process open to all developers.
Communicate: mailing list, blog, twitter, etc.
Permissive license (BSD).

7

The beginning

Started working in January 2010.
0.1. February 2010. KNN + ANN + SVM + manifold. Badly organized code base. It compiled and the tests passed. 1 developer.
...
0.6. December 2010. Custom bindings for LibSVM + Stochastic gradient descent + sparse models. Examples in the website. 13 active developers (4 from INRIA)

8

The Example Gallery

Easier to learn from well tailored examples than from an API reference (like matplotlib)
Auto-generated examples to ensure they keep functional

9

Community

Everything is discussed on mailing list or github.
Each pull request needs 2 reviews
~300 emails/month mailing list.
Standard in Community projects but not in projects run by institutions.

10

Welcome Newcomers

Precise guidelines for contributors.
Easy issues.
Take seriously every single contribution

11

The API

Estimator: object that implements .fit(X [, y]) and .predict(X)
Data format: NumPy array
Later added .transform(X), .partial_fit(X, y)

12

JMLR paper

Volunteers in academia also need citations!
JMLR OSS track with 13 authors
~500 citations

13

But also ...

Late 2010, we had to change the name of the package ...
... breaking everyone's code in the process
Several refactoring also broke people's code

Highlighting the need for more controlled code review

14

Scikit-learn today

Maintainer: Andreas Mueller (amazon)
Last release: 0.14 (87 contributors, 8 INRIA)
Funding from INRIA (1 paid engineer), Google (7 summer of code) and others.

15

Key features

Consistent API

Estimator (.fit, .predict), generalized to include multi-target, ranking, etc.
Other projects are adopting the scikit-learn API
making it easier to compare algorithms/models across libraries.

16

Key features

Comprehensive

Clustering, Covariance Estimators, Matrix Decomposition, Ensemble Methods, Feature Extraction, Feature Selection, Gaussian Processes, Isotonic regression, Kernel Approximation, Semi-Supervised Learning, Linear Discriminant Analysis, Generalized Linear Models, Manifold Learning, Gaussian Mixture Models, Multiclass and multilabel classification, Naive Bayes, Nearest Neighbors, Neural network models, Cross decomposition (PLS), Quadratic Discriminant Analysis, Random projections, Support Vector Machines, Decision Trees

17

Key features

Healthy community-driven project

credit: F. Perez, A. Meurer

18

Key features

Healthy community-driven project

credit: F. Perez, A. Meurer

19

Key features

Fast (most methods, most usercases)

credit: Gilles Louppe

Profiling and benchmarking is integrated into the code review process

20

Key features

Effortless parallelization

Several methods accept a n_jobs keyword
Joblib: great for embarrassingly parallel loops and NumPy arrays
The right grain of parallelism?
Too fine $\implies$ overhead
Too coarse $\implies$ memory shortage

21

Key features

Documentation

demo

22

Big Data

When data cannot fit in memory
estimator.partial_fit(X, y)
Load chunks of the data and fit the model incrementally
Works in k-means, SGD

23

Work in Progress

Big Data / Large-scale / Out-of-core learning $\implies$ partial_fit on more models
More efficient model selection (warm restarts)
Data sharing in parallelization
Test, test, test!