line-by-line memory usage of a Python program
My newest project is a Python library for monitoring memory consumption of arbitrary process, and one of its most useful features is the line-by-line analysis of memory usage for Python code. I wrote a basic prototype six months ago after being surprised by the lack of related tools. I wanted to plot memory consumption of a couple of Python functions but did not find a python module to do the job. I came to the conclusion that there is no standard way to get the memory usage of the Python interpreter from within Python, so I resorted to reading for from /proc/$PID/statm. From there on I realized that one the fetching of memory is done, making a line-by-line report wouldn't be hard. Back to today. I've been using the line-by-line memory monitoring to diagnose poor memory management (hidden temporaries, unused allocation, etc.) for some time. It seems to work on two different computers, so full of confidence as I am, I'll write a blog post about it ...
How to use it?
The easiest way to get it is to install from the Python Package Index:
$ easy_install -U memory_profiler # pip install -U memory_profiler
but other options include fetching the latests from github or dropping it on your current working directory or somewhere else on your PYTHONPATH since it consist of a single file. Then next step is to write some python code to profile. It can be just about any function, but for the purpose of this blog post I'll create a function `my_func()` with mostly memory allocations and save it to a file named example.py:
@profile def my_func(): a =  * (10 ** 6) b =  * (2 * 10 ** 7) del b return a if __name__ == '__main__': my_func()
Note that I've decorated the function with @profile. This tells the profiler to look into function my_func and gather the memory consumption for each line.