Benchmarking and Profiling Python

notes.md

High Performance Python

Overview

Performance?

• Memory
• CPU/GPU
• I/O

Improve:

• Follow best practices
• Know your tool (i.e. Python)
• Algorithms

There is a point where performance enhancements will only have as bottleneck the algorithms, heuristics, assumptions you make.

Profiling and Benchmarking

Python

timeit module and magic in Jupyter Notebooks.

• $ python -m cProfile -s cumtime module_to_profile.py
• $ python -m cProfile -o output.pstats module_to_profile.py - create an output.pstats file to be used elsewhere.

Libraries

• $ pip install line_profiler - https://github.com/rkern/line_profiler
• $ pip install gprof2dot - https://github.com/jrfonseca/gprof2dot
• $ pip install graphviz - http://www.graphviz.org/
• $ pip install pyprof2calltree - https://github.com/pwaller/pyprof2calltree
• $ pip install pyinstrument - https://github.com/joerick/pyinstrument
• $ pip install snakeviz - https://github.com/jiffyclub/snakeviz

Using the Libraries

gprof2dot

Use the output.pstats that is created using Python's cProfile module to create a graphviz compatible dot file.

$ python -m gprof2dot -f pstats output.pstats >> stats.dot

You can use graphviz from the command line to create a diagram of the analysis by dot -Tpng -o output.png stats.dot.

QCacheGrind

$ pyprof2calltree -i output.pstats -o filename.grind - and you can use https://sourceforge.net/projects/qcachegrindwin/ to open generated filename.grind to explore and perform profiling.

SnakeViz

$ snakeviz output.pstats - input is the cProfile file output.