Skip to content

Developer's Guide

Jump to bottom Edit New page
Andres Riancho edited this page Aug 10, 2015 · 39 revisions

Introduction

This page is the most important source of information for w3af developers and contributors. This documents has links to the development process, code conventions, tips and tricks, etc.

There are basically two different types of w3af developers:

  • Code Hackers: Advanced users that might identify bugs or potential improvements and simply modify a couple of files to apply their change. These users might be new to git, w3af and python. We recommend they use the Contributing 101 guide until you feel comfortable with the process.

  • Developers: Advanced users which are interested in enhancing the framework in more complex ways, add new features, refactor sections of code, etc. We recommend you follow this document in order to setup your development environment.

Building your development environment

IDE

Each developer can choose his own IDE to work with w3af, but we recommend PyCharm.

Extra tools

pylint is a code verifier for Python. All pull requests, patches, etc. which are submitted to us via github 'must be reviewed by pylint first'. In all systems with pip installed you can install get the latest pylint by running:

sudo pip install pylint --upgrade

We recommend you use pip instead of apt-get install pylint because you'll get a more recent version of the tool. And then run the following command:

pylint --additional-builtins=_ -E plugins/audit/new_plugin.py

w3af is a rather complex piece of software and pylint raises several false positives if run without custom plugins and configuration settings. If you want to run pylint against the whole w3af code base I recommend using test_pylint.py

Unit testing

Nosetests is the best unittest runner we've found, it's extensible using plugins and fits all of our needs. We're also using the python mock library in some tests and a set of plugins to enhance nose's output and provide code coverage. Installation of these tools is straight forward:

pip install -r w3af/tests/requirements.txt

Once again, please note that we use mock for writing our unittests that require mocked objects, keep that in mind when writing new unittests!

You can run all of w3af's unittests with the following command:

cd w3af/
nosetests --with-yanc --with-doctest --doctest-result-variable=_abc_ \
          --doctest-tests --exclude-dir-file=exclude_dirs.txt \
          --with-sneazr-pynotify --with-timer

This is the content of exclude_dirs.txt:

core/ui/gui/
plugins/discovery/oHalberd

Alternatively, you can download the nose.cfg file from our w3af-qa repository, save it to the project root and run the tests using this shorter syntax:

# You only need to run this once
wget https://raw.github.com/andresriancho/w3af-qa/master/nose/nose.cfg

# And then
nosetests --config=nose.cfg

In order to run only the smoke tests you can run:

nosetests --config=nose.cfg -a smoke core/data/parsers/

And if you want to only run the test cases that failed in the previous run, which is perfect for running until all cases in a directory pass and then run all the tests again:

nosetests --config=nose.cfg --failed --with-id core/data/parsers/

TDD freak? Then you'll want to install the randomize plugin (read this) and run your tests with --randomize.

Running the test suite

Our test suite requires you to run several test applications, all of them are dockerized and are started by running:

cd w3af/tests/
docker-compose up -d

Once all the containers are running the test suite needs to know where to find the services:

cd w3af/
w3af/tests/add-test-routes.sh

Continuous integration

We're using CircleCI for continuous integration. CircleCI provides free builds for open source projects, to access our builds just create a CI user (free) with your GitHub account.

Debugging

One of the neatest tricks for debugging w3af is to run this command in a console:

HTTP_PROXY=127.0.0.1:8080 nosetests --config=nose.cfg plugins/tests/attack/test_rfi.py

Which will route all HTTP traffic through a local proxy (burp, ZAP, etc.) which will allow you to view a very clear log of what w3af is sending to the wire.

Code conventions

PEP8 MUST be used for all new code. Also, if you're working on old code and see any non-PEP8 code snippets, update it!

Dependencies and third party libraries

Keep in mind that we don't usually add new dependencies to the project, and if we do we need to verify the following:

  • The license is GPL compatible
  • The library is stable and maintained
  • Unless there is no other way around it, or the performance of the C library is much better, we recommend using pure-python libraries.
  • Since this changes the packaging of w3af, please ask in w3af-develop about adding a new dependency
  • If a new dependency is added you need to check for correct installation at the dependency check module. There is one file that checks w3af-wide dependencies and another that checks the dependencies for the GTK user interface. Make sure you modify the corresponding file.

SDLC

Our software development life cycle uses A successful Git branching model (also available here in PDF format) and git-flow.

Getting started

For the best introduction to get started with git flow, please read Jeff Kreeftmeijer's blog post:

http://jeffkreeftmeijer.com/2010/why-arent-you-using-git-flow/

Or have a look at one of these screen casts:

Forking

Create a fork by clicking here. Clone w3af's fork from your github user URL and call it w3af_<user>. Make sure you replace both instances of <user> in the next line:

git clone [email protected]:<user>/w3af.git w3af_<user>

Keep your repository updated

If you want to contribute with the w3af project the best way is to follow the fork and pull-request methodology. If you're not a developer this basically means you'll create a copy of w3af's repository, apply your changes, and then send your changes back to us using a pull-request (as explained here)

After you fork our repository we'll continue to work on it, so your copy will get outdated. In order to keep your repository updated, please follow these steps.

Set the remote upstream repository to w3af's main repo

git remote add upstream [email protected]:andresriancho/w3af.git

Retrieve the updates from the upstream repository

git fetch upstream

Make sure that you know what you're getting from upstream and what is in your master

git log upstream/master ^master
git log ^upstream/master master

If everything is good, you can merge upstream into your clone

git merge upstream/master

Now you're ready to update your fork

git push origin master

Sending pull requests

Before you submit a pull request please make sure:

  • You're submitting it against the development branch.
  • Provide a good description of what your doing and why.
  • Provide tests that cover your changes.
  • Run the unittests locally first and make sure they all PASS.

Branches

Only contributors with write access to the w3af repository can create branches. We encourage the creation of branches for any feature development using git flow. Please keep your branch updated with the development branch!

GIT tricks

Git extensions

git-extras provides many interesting shortcuts for the developer. Some of them are:

  • git-local-commits: List all commits on the local branch that have not yet been sent to origin.
  • git-undo: Remove the latest commit
  • git-changelog: Generate a changelog file
  • git-release: Release a new version of the software

git-flow provides very helpful extensions for our SDLC.

.gitconfig

.gitconfig is your friend, here are some aliases we recommend (please make sure you collapse long lines into one line after copy+paste):

[alias]
lg = log --graph --all --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) 
     %C(white)%s%C(reset) %C(bold white)— %an%C(reset)%C(bold yellow)%d%C(reset)' --abbrev-commit
     --date=relative

After that git lg should show you an easy to read, graphical output.

Git commands

  • git rev-parse HEAD: returns the latest commit
  • git reset --hard <commit-id>: Moves the current working copy to commit-id and removes all childs from that commit from the index (git log)
  • git for-each-ref: Show all commit-ids for the latest commit in each tag, branch, etc.

Releasing a new version of w3af

Releasing a new version of w3af is much more than just building the new "bz2 package"! For instructions about the technical and community tasks around a release, please visit our shipping a new release page.

circle.yml

circle.yml is the configuration file for CircleCI, most of the steps are well documented and are run in the order which appear in the yml file. If at any point you're in doubt about how to run w3af tests just check how we're doing it there. This wiki might become outdated but the circle.yml file wont!

Add a custom footer
Add a custom sidebar
Clone this wiki locally