mnp: Mininet Packaging Tools Docs

Introduction

mnp, available here, is a command-line tool developed during Google Summer of Code 2013 with these purposes:

• Help Mininet users download a Mininet modules from a sharing website, including the dependencies with as few commands as possible.
• Help Mininet module developers package, upload, and share their Mininet modules on a sharing website.
• Make it easy to do some basic querying to the sharing website from the command line.

The sharing website, a modified PyPI clone with some non-standard behaviour, is developed with djangopypi2, an existing PyPI clone, as a base during Google Summer of Code 2013 as well.

mnp Commands

• download: Download specified packages
• upload: Package and upload the package from the current directory
• list: List all available packages
• search: Search for packages
• docs: Retrieve the documentation of a package
• info: Print the essential information (author, version, summary etc.) of a package

Other Features

• GitHub authentication support for upload command

Implementation

The commands supported by mnp are implemented internally by reusing existing tools as much as possible.

Commands

download

Implemented by calling pip from the command line.

Original command to download a package from a custom PyPI located on http://localhost:8000/simple/ with fallback on original PyPI: pip install -i http://localhost:8000/simple/ --extra-index-url https://pypi.python.org/simple/ SomePackage

upload

Implemented by calling setup.py from the command line which in turn calls setuptools which in turn calls distutils in some cases.

Original command to upload a package to a mirror called mininet: python setup.py register -r mininet sdist upload -r mininet

list, search, docs, info

Implemented by making an XML-RPC request to the sharing website. list and docs are standard PyPI XML-RPC API while docs and info are non-standard.

Others

GitHub Authentication Support for upload Command

Implementing the support for GitHub authentication is the most technically challenging part of the development of mnp.

After trying out several approaches, I arrived at the conclusion that the most elegant way to implement GitHub authentication is by implementing additional commands to setup.py by creating additional cmdclass.

By reading through the source code of both setuptools and distutils for both register and upload commands, I managed to find the proper methods to override in order to read in users' specified GitHub username from .pypirc. If the GitHub password is not specified in .pypirc, then the user will be asked for it when the command is executed. Once GitHub username and password is obtained, the authentication mechanism will be used to authenticate to the sharing website.

Implementing it this way, however, means that users who wanted to use GitHub account to submit their packages must always import and specify the cmdclass-es in their setup.py files. This is very inconvenient as the setup.py files are quite complex enough as it is, so I decided to monkey-patch the setup.py to patch the setup() function to include new commands, github_register and github_upload which use the cmdclass-es written to support GitHub authentication.

This monkey-patch mechanism can then be easily activated simply by writing import mnp.patch at the top of the setup.py files. The written packages must also depend on mnp in order to be able to import the patch in the first place, but the syntax to achieve this is much friendlier than the syntax to declare a new cmdclass.

With support for GitHub authentication in-place, mnp upload reads the .pypirc and see if the username and password are both specified for the chosen upload destination. If so, then simply use that as the authentication information and execute python setup.py register -r <destination> sdist upload -r <destination> on the command-line.

If any of username and password is not specified in .pypirc, then mnp will see if github_username is specified. If so, then it either reads the github_password attribute from the .pypirc as well or asks for the GitHub password interactively on the command-line. Once done, it then executes python setup.py github_register -r <destination> sdist github_upload -r <destination> on the command-line. This command will effectively call the cmdclass-es written because of the monkey-patching mechanism.

GitHub Account Authentication Mechanism

Given the GitHub username and password, mnp will do these:

• It will send a request to GitHub users API to convert the username and password combination to a user id.
• It will get the list of authorization tokens associated with the account via the GitHub OAuth API. From the list of authorization tokens, mnp will find the one token associated with the sharing website by finding the one with the client_id property equal to the application id of the sharing website.

Once both the user id and the authorization token are retrieved, the user id is used as the username while the authorization token is hashed and then used as the password to authenticate to the sharing website via command line.

Note that for all this to work, the user must have used GitHub login at least once before on the sharing website (i.e. not via command-line) so it has a chance to store the authorization token of the user. I think this is a reasonable requirement in order to use this feature.

Modification Needed on Sharing Website

The sharing website also needs to be modified in order to support both local account (i.e.: account registered on the sharing website itself) and GitHub authentication for upload command. I modified it to initially use the username and password to authenticate as local account first. If it fails to authenticate as a local account, then it checks if it is a GitHub account.

Unit Testing

The commands implemented by mnp either make a call on command-line or make an XML-RPC request. In order to be able to write unit tests for this kind of functions, I make use of the mock library.

For commands calling command-line, I use mock to replace the command-line calls with a mock object and simply checks that the command-line calls are called with the correct parameters.

For commands making XML-RPC request, I use mock to replace the destination XML-RPC server with a mock object and then checks if it is called with a correct parameter. Some of these commands also print out messages to the command line which can be tested by using mock to replace the stdout object by some kind of string buffer and then simply checking that some expected values exist inside the string buffer.

I currently have no idea how to write unit tests for the monkey-patching to support GitHub authentication.