build errors due to dependencies

[mpkmtv]

I'm new to scala and spark. Got overwhelmed with build errors due to unresolved dependencies like :

[warn] Note: Unresolved dependencies path:
[error] sbt.librarymanagement.ResolveException: Error downloading com.databricks:sbt-databricks;sbtVersion=1.0;scalaVersion=2.12:0.1.3
[error] Not found
[error] Not found
[error] not found: https://repo1.maven.org/maven2/com/databricks/sbt-databricks_2.12_1.0/0.1.3/sbt-databricks-0.1.3.pom
[error] not found: https://dl.bintray.com/spark-packages/maven/com/databricks/sbt-databricks_2.12_1.0/0.1.3/sbt-databricks-0.1.3.pom
[error] not found: https://oss.sonatype.org/content/repositories/releases/com/databricks/sbt-databricks_2.12_1.0/0.1.3/sbt-databricks-0.1.3.pom
[error] not found: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.databricks/sbt-databricks/scala_2.12/sbt_1.0/0.1.3/ivys/ivy.xml
[error] not found: https://repo.typesafe.com/typesafe/ivy-releases/com.databricks/sbt-databricks/scala_2.12/sbt_1.0/0.1.3/ivys/ivy.xml
[error] Error downloading com.jsuereth:sbt-pgp;sbtVersion=1.0;scalaVersion=2.12:1.0.0
[`

Any help ?

[hitachi-ben]

@mpkmtv, did you ever figure it out? Having the same issue.

[wangyum]

You can use pre-build jars.
spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar.tgz

tar -zxf spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar.tgz

java --add-opens=java.base/sun.nio.ch=ALL-UNNAMED -Djava.io.tmpdir=/hadoop1/tmp,/hadoop2/tmp -Dspark.local.dir=/hadoop1/tmp,/hadoop2/tmp -cp /home/hadoop/spark-3.5.3/spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar:/home/hadoop/spark-3.5.3/spark-3.5.3-bin-hadoop3/jars/*:/home/hadoop/spark-3.5.3/scopt_2.12-4.0.1.jar  com.databricks.spark.sql.perf.tpcds.GenTPCDSData --dsdgenDir /home/hadoop/spark-3.5.3/tpcds-kit/tools -s 5000 -l hdfs://myhdfs:9000/user/hive/warehouse/tpcds5t -f parquet

After generating the data, you can run benchmark through:

git clone https://github.com/wangyum/spark.git -b run-benchmark && cd spark
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark --data-location hdfs://myhdfs:9000/user/hive/warehouse/tpcds5t --cbo"