Performance profiling resources for Python

Where to start and what not to miss when interested in profiling python apps. Target is to manage profiling web apps based on python frameworks (e.g. flask) right in production, but also many use-cases of basic python profiling methods (regardless of web apps aim) are listed in order to better understand the possible drawbacks of tracing production apps. The main issue considered about production is not causing overheads to avoid negative effects on services, which is also important when comparing performance.

Basic python profiling tools:

• py-spy - Py-Spy top-like sampling profiler (low overhead, non-interuptive, flame graphs)
• line_profiler - Line-by-line profiling.
• memory_profiler - Monitor Memory usage of Python code.
• memprof - Memory profiler for Python
• profiling - An interactive Python profiler.
• vprof - Visual Python profiler.
• scalene - a high-performance CPU and memory profiler for Python.

Performance analysis:

• A guide to analyzing Python performance - Very informative article written by SE at Flickr
• Computation and scaling - Computation on multicore processors and scaling issues/tooling

Memory leaks use-cases:

• Tracing python memory leaks #PDB
• Hunting memory leaks in Python #PDB
• Python object graphs #GC
• Diagnosing Memory "Leaks" in Python #Objgraph #Heapy #GDB

Production profiling use-cases and tools:

• nylas-perftools - one of the best profiling toolsets for python that I tried so far, with very little CPU overhead (they litteraly say its negligible). It was actually designed for profiling in production, when its developers were in need to find a new ways to optimizate their wast Nylas platform and it also features flame graphs introduced by Brendan Gregg, these are well known as a very good technique for visualisation of stack profiles. Read more about their use-case: https://www.nylas.com/blog/performance/ #Flask
• Profiling a Werkzeug (flask) app #Flask
• Deploying and Monitoring Python Web Apps with uWSGI - there is a way to look at a current stack trace using very simple setting in uwsgi app config, and the affect to performance of the application itself seems to be minimal. This may be interesting for those running multiple workers (separate stats socket is used for every worker). Other debugging examples and tools are stated in the article too. #Tracebacker

Issues

• Remember to search for advices if having issues running these in docker (e.g. py-spy)