Home

storm-deploy makes it dead-simple to launch Storm clusters on AWS. It is built on top of jclouds and pallet. After you follow the instructions in this tutorial, you will be able to provision, configure, and install a fully functional Storm cluster with just one command:

lein run :deploy --start --name mycluster

You can then stop a cluster like this:

lein run :deploy --stop --name mycluster

Setup instructions

1. Install leiningen. All you have to do is download this script, place it on your PATH, and make it executable.

2. Clone storm-deploy using git

3. Run lein deps

4. Create a ~/.pallet/config.clj file that looks like the following (and fill in the blanks). This provides the deploy with the credentials necessary to launch and configure instances on AWS.

(defpallet
  :services
  {
   :default {
             :blobstore-provider "aws-s3"
             :provider "aws-ec2"
             :environment {:user {:username "storm"
                                  :private-key-path "$YOUR_PRIVATE_KEY_PATH$"
                                  :public-key-path "$YOUR_PUBLIC_KEY_PATH$"}
                           :aws-user-id "$YOUR_USER_ID$"}
             :identity "$YOUR_AWS_ACCESS_KEY$"
             :credential "$YOUR_AWD_ACCESS_KEY_SECRET$"
             :jclouds.regions "$YOUR_AWS_REGION$"
             }
    })

5. Configure your cluster by editing conf/clusters.yaml. You can change the number of zookeeper nodes or supervisor nodes by editing zookeeper.count or supervisor.count, respectively. You can launch spot instances for supervisor nodes by setting supervisor.spot.price. The other properties should be self-explanatory.

6. (optional) Place any custom configurations for your Storm cluster by editing conf/storm.yaml. For example, you may change timeouts, register custom serializations, or put in other configurations you want available to your topologies.

Launching clusters

Run this command:

lein run :deploy --start --name mycluster

The --name parameter names your cluster so that you can attach to it or stop it later. If you omit --name, it will default to "dev". The deploy sets up Zookeeper, sets up Nimbus, launches the Storm UI on port 8080 on Nimbus, sets up the Supervisors, sets configurations appropriately, sets the appropriate permissions for the security groups, and attaches your machine to the cluster (see below for more information on attaching).