Skip to content

Latest commit

 

History

History
65 lines (50 loc) · 2.3 KB

DEVELOPMENT.md

File metadata and controls

65 lines (50 loc) · 2.3 KB
Raw

Comet Development Guide

Project Layout

├── common     <- common Java/Scala code
├── conf       <- configuration files
├── core       <- core native code, in Rust
├── spark      <- Spark integration

Development Setup

  1. Make sure JAVA_HOME is set and point to JDK 11 installation.
  2. Install Rust toolchain. The easiest way is to use rustup.

Build & Test

A few common commands are specified in project's Makefile:

  • make: compile the entire project, but don't run tests
  • make test: compile the project and run tests in both Rust and Java side.
  • make release: compile the project and creates a release build. This is useful when you want to test Comet local installation in another project such as Spark.
  • make clean: clean up the workspace
  • bin/comet-spark-shell -d . -o spark/target/ run Comet spark shell for V1 datasources
  • bin/comet-spark-shell -d . -o spark/target/ --conf spark.sql.sources.useV1SourceList="" run Comet spark shell for V2 datasources

Benchmark

There's a make command to run micro benchmarks in the repo. For instance:

make benchmark-org.apache.spark.sql.benchmark.CometReadBenchmark

To run TPC-H or TPC-DS micro benchmarks, please follow the instructions in the respective source code, e.g., CometTPCHQueryBenchmark.

Debugging

Comet is a multi-language project with native code written in Rust and JVM code written in Java and Scala. It is possible to debug both native and JVM code concurrently as described in the DEBUGGING guide