line-by-line memory usage of a Python program
My newest project is a Python library for monitoring memory consumption of arbitrary process, and one of its most useful features is the line-by-line analysis of memory usage for Python code. I wrote a basic prototype six months ago after being surprised by the lack of related tools. I wanted to plot memory consumption of a couple of Python functions but did not find a python module to do the job. I came to the conclusion that there is no standard way to get the memory usage of the Python interpreter from within Python, so I resorted to reading for from /proc/$PID/statm. From there on I realized that one the fetching of memory is done, making a line-by-line report wouldn't be hard. Back to today. I've been using the line-by-line memory monitoring to diagnose poor memory management (hidden temporaries, unused allocation, etc.) for some time. It seems to work on two different computers, so full of confidence as I am, I'll write a blog post about it ...
How to use it?
The easiest way to get it is to install from the Python Package Index:
$ easy_install -U memory_profiler # pip install -U memory_profiler
but other options include fetching the latests from github or dropping it on your current working directory or somewhere else on your PYTHONPATH since it consist of a single file. Then next step is to write some python code to profile. It can be just about any function, but for the purpose of this blog post I'll create a function `my_func()` with mostly memory allocations and save it to a file named example.py:
@profile
def my_func():
a = [1] * (10 ** 6)
b = [2] * (2 * 10 ** 7)
del b
return a
if __name__ == '__main__':
my_func()
Note that I've decorated the function with @profile. This tells the profiler to look into function my_func and gather the memory consumption for each line.
Wake up the cookie monster
To start profiling and output the result to stdout, run the script as usual and append the options -m memory_profiler -l -v to the python interpreter:
$ python -m memory_profiler example.py
Filename: example.py
Line # Mem usage Increment Line Contents
================================================
2 @profile
3 8.00 MB 0.00 MB def my_func():
4 15.00 MB 7.00 MB a = [1] * (10 ** 6)
5 168.00 MB 153.00 MB b = [2] * (2 * 10 ** 7)
6 15.00 MB -153.00 MB del b
7 15.00 MB 0.00 MB return a
voilá! Each line is prefixed by the memory usage in MB of the Python interpreter after that line has been executed.