Skip to content

BigJob and Clouds

Jump to bottom
drelu edited this page Aug 12, 2012 · 15 revisions

Google Compute Engine and Google Storage

Requirements:

  • Google API Client - This package is installed automatically with the BigJob package.

For debugging purposes, the Google command line tools are helpful:

OAUTH Authentication

  1. Create a client key via the Google API console: https://code.google.com/apis/console/

Custom Image creation

BigJob will not be able to run on the standard Ubuntu image used by GCE. To installed the required dependency create a custom image (see Google documentation):

  1. Start an image and log in

  2. Install requirements:

     sudo apt-get install gcc build-essential checkinstall python-all-dev

  3. Create and upload image

     $ sudo python /usr/share/imagebundle/image_bundle.py   -r / -o /tmp/ --log_file=/tmp/bigjob-image-creation.log
 $ gsutil config
 $ gsutil mb gs://bigjob-image
 $ gsutil cp /tmp/<unique-hash>.image.tar.gz gs://bigjob-image/
 $ gcutil addimage bigjob-image  "http://commondatastorage.googleapis.com/bigjob-image/<unique-hash.image.tar.gz" --  project_id="<project_id"

For strange reason GCE and GS use different kinds of project ids.

BigJob Configuration

  1. Configure your GCE project settings in the GCE/BigJob job plugin:

     """
 AN OAUTH2 Client Id must be created at the Google API console at:
 
 https://code.google.com/apis/console/

 => API Access

 More information with respect to OAUTH: https://developers.google.com/compute/docs/api/how- tos/authorization
 """
 OAUTH2_CLIENT_ID='1004462711324-55akehip32m59u6omdfrt9s8u8ehb0hm.apps.googleusercontent.com'
 OAUTH2_CLIENT_SECRET='EIMML1W7anu0XijVghws0DY-'


 GCE_PROJECT_ID='bigjob-pilot'
 USER_ID='luckow'
 SSH_KEY_FILE='/Users/luckow/.ssh/google_compute_engine'
 
 """
 Google Compute Engine currently provides a default image with Ubuntu 12.04
 To use BJ, a custom image containing gcc and build essentials needs to be
 created.

 $ apt-get update
 $ apt-get install gcc python-all-dev 

 """
 GCE_IMAGE_URL="https://www.googleapis.com/compute/v1beta12/projects/bigjob-pilot/images/bigjob-image"

  2. You can use the following resource URL to start a BJ on a GCE instance:

     gce+ssh://api.google.com

  3. Check out the cloud example.

Amazon EC2 and S3

Requirements:

  • Boto - This package is installed automatically with the BigJob package.

Custom Images:

Similar to GCE, an image containing the necessary dependencies must be created.

  1. Start an image and log in (best to use the [AWS Admin Console] (https://console.aws.amazon.com/ec2))

  2. Install requirements:

     sudo apt-get install gcc build-essential checkinstall python-all-dev

  3. Create Image using [Admin Console] (https://console.aws.amazon.com/ec2). Right click on instance and the select "Create Image".

BigJob Configuration

  1. Configure your Amazon project settings in the EC2/BigJob job plugin:

     ###############################################################################
 # EC2 General
 EC2_SSH_PRIVATE_KEY_FILE="/Users/luckow/.ec2/id-lsu-keypair"
 EC2_INSTANCE_TYPE="t1.micro"
 EC2_PLACEMENT_GROUP=None
 EC2_SECURITY_GROUP="default"

 # VM/Image specific configurations
 EC2_AMI_ID="ami-c7943cae" #  ami-82fa58eb official Amazon Ubuntu 12.04 LTS (requires dev tools installation)
 EC2_USERNAME="ubuntu"
 EC2_KEYNAME="lsu-keypair"

 # Authentication
 # Please use ~/.boto file to configure your security credentials (if possible)
 # see http://boto.readthedocs.org/en/latest/boto_config_tut.html
 # 
 # [Credentials]
 # aws_access_key_id = <your access key>
 # aws_secret_access_key = <your secret key>

  2. You can now launch a BJ on an EC2 instance using ec2+ssh://<EC2 Endpoint> as a resource URL.

  3. Check out the cloud example.

Clone this wiki locally