-
Notifications
You must be signed in to change notification settings - Fork 109
3 MapReduce for Python
The Python MapReduce library can be used for complete map-shuffle-reduce pipelines only. It does not have the ability to run a map-only job.
The App Engine adaptation of Google's MapReduce model is optimized for the needs of the App Engine environment, where resource quota management is a key consideration. This release of the MapReduce API provides the following features and capabilities:
- Automatic sharding for faster execution, allowing you to use as many workers as you need to get your results faster
- Standard data input readers for iterating over blob and datastore data.
- Standard output writers
- Status pages to let you see how your jobs are running
- Processing rate limiting to slow down your mapper functions and space out the work, helping you avoid exceeding your resource quotas
To build the library and the demo application, run the provided build script as follows from within the python sub-directory of the source tree:
./build.sh build_demo
The python MapReduce code uses Pypi for dependency management. The build script runs the pip commands for you. If you prefer, you may run pip directly using the following command:
pip install -r src/requirements.txt -t src/