Profiling memory and CPU usage

Introduction

During the years I've developed the w3af framework I've found myself in two different situations when dealing with performance:

1. I know module / plugin X has issues, but I'm unaware of where the issue is
2. I want to know the most CPU / memory / network consuming areas of w3af

The first question is easier to answer, and usually is done using line_profiler as follows:

pip install line_profiler

# Decorate the methods you're interested in using @profile

kernprof -o w3af_console.lprof -v -l w3af_console -s scripts/some-script.w3af

# Let the framework run for N minutes
# Ctrl+c
# Analyze the results

To get an answer to the second question more complicated profiling code is required. The rest of this wiki page explains how to do it.

Installing requirements

# Install meliae
sudo apt-get install cython
wget https://launchpad.net/meliae/trunk/0.4/+download/meliae-0.4.0.tar.gz
tar -zxpvf meliae-0.4.0.tar.gz
cd meliae-0.4.0/
sudo python setup.py install

# Install yappi + RunSnakeRun
sudo pip install yappi --upgrade
sudo pip install RunSnakeRun --upgrade

Profiling a w3af run

Manual analysis:

W3AF_CPU_PROFILING=1 ./w3af_console -s scripts/test.w3af

Automated profiling information collection can be done using the collector tool.

Profiling a nosetests run

python -m cProfile -o profile.out `which nosetests` -v w3af/plugins/tests/crawl/test_phishtank.py:TestPhishtank.test_xml_parsing
runsnake profile.out

Analysis

Manual analysis:

runsnake /tmp/yappi-w3af.cpu
runsnakemem /tmp/meliae-w3af.memory

Automated analysis using the w3af-performance-analysis tool.

Related source code

The profiling instrumentation is embedded into w3af itself. It was the best way to allow me to collect this information in cases where users wanted to help me debug memory usage issues.

The source code is pretty simple to understand.

Documentation

There's documentation on how w3af uses memory which you might want to read.