Welcome to hackathon 2015!
Download the kickoff deck for more information on the business problem, the hackathon agenda, the data, and more! Find it in the data folder:
We have a Hadoop cluster with one master and four slaves. The slaves have sixteen cores, 15 1TB data drives, and 128GB of RAM. You will have ssh access to the slaves.
Please spread yourselves out across the machines.
You can obtain a username and login information from one of the Tresata representatives.
Edit your hosts file (/etc/hosts) and add a line to associate the ip to the hackathon hostname:
<ip-address> hackaton-master.datachambers.com
Ssh into a server where you can access the retail data stored on the hackathon Hadoop cluster.
> ssh <username>@<ip-address>
and enter the password you were given.
We made Hive, Spark, and pySpark command-line interfaces available, and included a tool to compile and run simple Scalding scripts on-the-fly.
Give Hive a whirl and run a sample query:
> hive
Try pasting the following query into the hive command-line interface:
hive> show tables;
OK
hackathon
Time taken: 0.034 seconds, Fetched: 1 row(s)
hive> select * from hackathon limit 10;
This will return all the fields for the first ten items in the 'hackathon' table.
If you'd like to create a file from the command, you can use a create table command:
hive> create table test row format delimited fields terminated by '|' stored as textfile as select * from default.hackathon limit 10;
You can then extract the table from the hive warehouse for a table named abc:
hadoop fs -text /user/hive/warehouse/abc/*.snappy > textfile
Now give the Spark-shell a test:
> spark-shell --num-executors 4 --executor-cores 1 --executor-memory 1G
Read in the data and run a simple query that calculates the number of purchases for each upc in the sample data:
val dataRDD = sc.textFile("/data/customer_sample")
val upcs = dataRDD.map(_.split("\\|")(12))
val upcCounts = upcs.map(upc => (upc, 1)).reduceByKey((a, b) => a + b)
upcCounts.take(10)
Note that for your "production" run on the full dataset you might want to increase resources used on the cluster:
--num-executors 4 --executor-cores 4 --executor-memory 4G
Keep in mind that a spark-shell takes up these resources on the cluster even when you do not use them so please do not keep a spark-shell with "production" resources open unused.
You can also do the same query using a python version of the Spark shell.
> pyspark --num-executors 4 --executor-cores 1 --executor-memory 1G
dataRDD = sc.textFile("/data/customer_sample")
upcs = dataRDD.map(lambda line: line.split('|')[12])
upcCounts = upcs.map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)
upcCounts.take(10)
Note that for your "production" run on the full dataset you might want to increase resources used on the cluster:
--num-executors 4 --executor-cores 4 --executor-memory 4G
Keep in mind that a pyspark takes up these resources on the cluster even when you do not use them so please do not keep a pyspark shell (interpreter) with "production" resources open unused.
In addition to the Hive and Spark shells, we're also packaging Eval-tool, a tool to compile and run Scalding scripts without having to create a project. If you create a file called test.scala with the following contents:
import com.twitter.scalding._
import com.tresata.scalding.Dsl._
import com.tresata.scalding.util.ScaldingUtil
(args: Args) => {
new Job(args) {
ScaldingUtil.sourceFromArg(args("input"))
.groupBy('upc) { _.size }
.write(ScaldingUtil.sourceFromArg(args("output")))
}
}
you can run a query on the data set sample from the command-line:
> eval-tool test.scala --hdfs --input bsv%/data/customer_sample --output bsv%upc_counts
This will generate a bar-separated file called 'upc_counts' in your HDFS home directory, containing the upc numbers along with their total counts.
To access your HDFS location, you need to use hadoop fs commands (some reference: http://www.folkstalk.com/2013/09/hadoop-fs-shell-command-example-tutorial.html). For example, to take a look at your home directory on HDFS, use
> hadoop fs -ls
or
> hadoop fs -ls /user/username
You can find the data on HDFS in the /data folder, which contains the full data set with/without headers, as well as a 1% sample by customer (so all transactions for 1% of customers) with/without headers.
/data/customer_sample
/data/customer_sample_no_header
/data/full
/data/full_no_header
In hive the following tables are available:
hackathon: full data set
customer_sample: the 1% sample by customer
2c_ht_raw: full data set in faster ORC file format, created by Dave
The sample files are also available on the local file system:
/home/shared/customer_sample.bsv
/home/shared/customer_sample_no_header.bsv
/home/shared/tiny_customer_sample.bsv