Developer's Guide
This page is the most important source of information for w3af developers and contributors. This documents has links to the development process, code conventions, tips and tricks, etc.
There are basically two different types of w3af developers:
-
Code Hackers: Advanced users that might identify bugs or potential improvements and simply modify a couple of files to apply their change. These users might be new to git, w3af and python. We recommend they use the Contributing 101 guide until you feel comfortable with the process.
-
Developers: Advanced users which are interested in enhancing the framework in more complex ways, add new features, refactor sections of code, etc. We recommend you follow this document in order to setup your development environment.
Each developer can choose his own IDE to work with w3af, but we're using Eclipse with PyDev for framework development. Please follow the steps in the Eclipse, PyDev and EGit setup to get your Eclipse environment working with w3af in order to have code completion for w3af classes, references (ctrl + click over a class), fast code search with (ctrl+shift+R) and all the other Eclipse goodies.
pylint is a code verifier for Python. All pull requests, patches, etc. which are submitted to us via github 'must be reviewed by pylint first'. In all systems with pip installed you can install get the latest pylint by running:
sudo pip install pylint --upgrade
We recommend you use pip instead of apt-get install pylint because you'll get a more recent version of the tool. And then run the following command:
pylint --additional-builtins=_ -E plugins/audit/new_module_you_developed.py
Nosetests is the best unittest runner we've found, it's extensible using plugins and fits all of our needs. We're also using the python mock library in some tests and a set of plugins to enhance nose's output and provide code coverage. Installation of these tools is straight forward:
sudo pip install mock
sudo pip install pylint
sudo pip install nose
sudo pip install nose-exclude
sudo pip install yanc
sudo pip install coverage
sudo pip install sneazr-pynotify
sudo pip install -e git+git://github.com/andresriancho/nose-timer.git#egg=nose-timer
sudo pip install psutil
Once again, please note that we use mock for writing our unittests that require mocked objects, keep that in mind when writing new unittests!
You can run all of w3af's unittests with the following command:
cd w3af/
nosetests --with-yanc --with-doctest --doctest-result-variable=_abc_ --doctest-tests \
--exclude-dir-file=exclude_dirs.txt --with-sneazr-pynotify --with-timer
This is the content of exclude_dirs.txt:
core/ui/gui/
plugins/discovery/oHalberd
Alternatively, you can download the nose.cfg file from our w3af-qa repository, save it to the project root and run the tests using this shorter syntax:
wget https://raw.github.com/andresriancho/w3af-qa/master/nose/nose.cfg # You only need to run this once
nosetests --config=nose.cfg
If you want to measure test coverage, you can also use nosetests:
nosetests --config=nose.cfg --with-coverage --cover-package=core.data.parsers core/data/parsers/
In order to run only the smoke tests you can run:
nosetests --config=nose.cfg -a smoke core/data/parsers/
And if you want to only run the test cases that failed in the previous run, which is perfect for running until all cases in a directory pass and then run all the tests again:
nosetests --config=nose.cfg --failed --with-id core/data/parsers/
TDD freak? Then you'll want to install the randomize plugin (read this) and run your tests with --randomize.
We're using Travis-CI for continuous integration. Travis' configuration is in our own repository under the .travis.yml file. If you're interested in knowing more about this configuration please take a look at Travis CI: Building a Python project document.
Our current build status is: [
PEP8 MUST be used for all new code. Also, if you're working on old code and see any non-PEP8 code snippets, update it!
Keep in mind that we don't usually add new dependencies to the project, and if we do we need to verify the following:
- The license is GPL compatible
- The library is stable and maintained
- Unless there is no other way around it, we recommend using pure-python libraries.
- Since this changes the packaging of w3af, please ask in w3af-develop about adding a new dependency
- If a new dependency is added you need to check for correct installation at the dependency_check.py files. There is one file that checks w3af-wide dependencies and another that checks the dependencies for the GTK user interface. Make sure you modify the corresponding file.
Our software development life cycle uses A successful Git branching model (also available here in PDF format) and git-flow.
For the best introduction to get started with git flow, please read Jeff Kreeftmeijer's blog post:
http://jeffkreeftmeijer.com/2010/why-arent-you-using-git-flow/
Or have a look at one of these screen casts:
- How to use a scalable Git branching model called git-flow (by Build a Module)
- A short introduction to git-flow (by Mark Derricutt)
- On the path with git-flow (by Dave Bock)
Create a fork by clicking here. Clone w3af's fork from your github user URL and call it w3af_<user>. Make sure you replace both instances of <user> in the next line:
git clone [email protected]:<user>/w3af.git w3af_<user>
If you want to contribute with the w3af project the best way is to follow the fork and pull-request methodology. If you're not a developer this basically means you'll create a copy of w3af's repository, apply your changes, and then send your changes back to us using a pull-request (as explained here)
After you fork our repository we'll continue to work on it, so your copy will get outdated. In order to keep your repository updated, please follow these steps.
Set the remote upstream repository to w3af's main repo
git remote add upstream [email protected]:andresriancho/w3af.git
Retrieve the updates from the upstream repository
git fetch upstream
Make sure that you know what you're getting from upstream and what is in your master
git log upstream/master ^master
git log ^upstream/master master
If everything is good, you can merge upstream into your clone
git merge upstream/master
Now you're ready to update your fork
git push origin master
Before you submit a pull request please make sure:
- You're submitting it against the
developmentbranch. - Provide a good description of what your doing and why.
- Provide tests that cover your changes.
- Run the unittests locally first and make sure they all PASS.
Only contributors with write access to the w3af repository can create branches. We encourage the creation of branches for any feature development using git flow. Please keep your branch updated with the development branch!
git-extras provides many interesting shortcuts for the developer. Some of them are:
-
git-local-commits: List all commits on the local branch that have not yet been sent to origin. -
git-undo: Remove the latest commit -
git-changelog: Generate a changelog file -
git-release: Release a new version of the software
git-flow provides very helpful extensions for our SDLC.
.gitconfig is your friend, here are some aliases we recommend (please make sure you collapse long lines into one line after copy+paste):
[alias]
lg = log --graph --all --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset)
%C(white)%s%C(reset) %C(bold white)— %an%C(reset)%C(bold yellow)%d%C(reset)' --abbrev-commit
--date=relative
After that git lg should show you an easy to read, graphical output.
-
git rev-parse HEAD: returns the latest commit -
git reset --hard <commit-id>: Moves the current working copy to commit-id and removes all childs from that commit from the index (git log) -
git for-each-ref: Show all commit-ids for the latest commit in each tag, branch, etc.
Releasing a new version of w3af is much more than just building the new "bz2 package"! For instructions about the technical and community tasks around a release, please visit our shipping a new release page.