Skip to content

Latest commit

 

History

History
71 lines (61 loc) · 3.46 KB

README.md

File metadata and controls

71 lines (61 loc) · 3.46 KB
Raw

tpch-datagen-as-hive-query

This are set of UDFs and queries that you can use with Hive to use TPCH datagen in parrellel on hadoop cluster. You can deploy to azure using :

##How to use with Hive CLI

  1. Clone this repo.

    git clone https://github.com/dharmeshkakadia/tpch-datagen-as-hive-query/ && cd tpch-datagen-as-hive-query

  2. Run TPCHDataGen.hql with settings.hql file and set the required config variables.

    hive -i settings.hql -f TPCHDataGen.hql -hiveconf SCALE=10 -hiveconf PARTS=10 -hiveconf LOCATION=/HiveTPCH/ -hiveconf TPCHBIN=resources

    Here, SCALE is a scale factor for TPCH, PARTS is a number of task to use for datagen (parrellelization), LOCATION is the directory where the data will be stored on HDFS, TPCHBIN is where the resources are found. You can specify specific settings in settings.hql file.

  3. Now you can create tables on the generated data.

    hive -i settings.hql -f ddl/createAllExternalTables.hql -hiveconf LOCATION=/HiveTPCH/ -hiveconf DBNAME=tpch

    Generate ORC tables and analyze

    hive -i settings.hql -f ddl/createAllORCTables.hql -hiveconf ORCDBNAME=tpch_orc -hiveconf SOURCE=tpch -hiveconf LOCATION=/HiveTPCHOrc/ 
hive -i settings.hql -f ddl/analyze.hql -hiveconf ORCDBNAME=tpch_orc

  4. Run the queries !

    hive -database tpch_orc -i settings.hql -f queries/tpch_query1.hql

##How to use with Beeline CLI

  1. Clone this repo.

    git clone https://github.com/dharmeshkakadia/tpch-datagen-as-hive-query/ && cd tpch-datagen-as-hive-query

  2. Upload the resources to DFS.

    hdfs dfs -copyFromLocal resoruces /tmp

  3. Run TPCHDataGen.hql with settings.hql file and set the required config variables.

    beeline -u "jdbc:hive2://`hostname -f`:10001/;transportMode=http" -n "" -p "" -i settings.hql -f TPCHDataGen.hql -hiveconf SCALE=10 -hiveconf PARTS=10 -hiveconf LOCATION=/HiveTPCH/ -hiveconf TPCHBIN=`grep -A 1 "fs.defaultFS" /etc/hadoop/conf/core-site.xml | grep -o "wasb[^<]*"`/tmp/resources

    Here, SCALE is a scale factor for TPCH, PARTS is a number of task to use for datagen (parrellelization), LOCATION is the directory where the data will be stored on HDFS, TPCHBIN is where the resources are uploaded on step 2. You can specify specific settings in settings.hql file.

  4. Now you can create tables on the generated data.

    beeline -u "jdbc:hive2://`hostname -f`:10001/;transportMode=http" -n "" -p "" -i settings.hql -f ddl/createAllExternalTables.hql -hiveconf LOCATION=/HiveTPCH/ -hiveconf DBNAME=tpch

    Generate ORC tables and analyze

    beeline -u "jdbc:hive2://`hostname -f`:10001/;transportMode=http" -n "" -p "" -i settings.hql -f ddl/createAllORCTables.hql -hiveconf ORCDBNAME=tpch_orc -hiveconf SOURCE=tpch 
beeline -u "jdbc:hive2://`hostname -f`:10001/;transportMode=http" -n "" -p "" -i settings.hql -f ddl/analyze.hql -hiveconf ORCDBNAME=tpch_orc

  5. Run the queries !

    beeline -u "jdbc:hive2://`hostname -f`:10001/tpch_orc;transportMode=http" -n "" -p "" -i settings.hql -f queries/tpch_query1.hql