| layout | title | nav_order | parent |
|---|---|---|---|
page |
Build Parameters for Velox Backend |
4 |
Getting-Started |
Please set them via --, e.g. --build_type=Release.
| Parameters | Description | Default value |
|---|---|---|
| build_type | Gluten build type(for arrow/velox/cpp), CMAKE_BUILD_TYPE | Release |
| build_tests | build test code in cpp folder and arrow | OFF |
| build_benchmarks | build benchmark code in cpp folder and arrow | OFF |
| build_jemalloc | build with jemalloc | ON |
| build_protobuf | build protobuf lib | ON |
| enable_qat | enable QAT for shuffle data de/compression | OFF |
| enable_iaa | enable IAA for shuffle data de/compression | OFF |
| enable_hbm | enable HBM allocator | OFF |
| enable_s3 | build with s3 lib | OFF |
| enable_hdfs | build with hdfs lib | OFF |
| enable_ep_cache | enable caching for external project build (Arrow and Velox) | OFF |
Please set them via --, e.g., --enable_custom_codec=ON.
| Parameters | Description | Default value |
|---|---|---|
| enable_custom_codec | Apply patch to plugin custom codec (used by QAT/IAA) in Arrow cpp IPC module. | OFF |
Please set them via --, e.g., --arrow_home=/YOUR/PATH.
| Parameters | Description | Default value |
|---|---|---|
| arrow_home | Arrow build path | GLUTEN_DIR/ep/build-arrow/build |
| build_type | ARROW build type, CMAKE_BUILD_TYPE | Release |
| build_tests | Build arrow with -DARROW_JSON=ON | OFF |
Please set them via --, e.g., --velox_home=/YOUR/PATH.
| Parameters | Description | Default value |
|---|---|---|
| velox_home | Velox build path | GLUTEN_DIR/ep/build-velox/build/velox_ep |
| build_type | Velox build type, CMAKE_BUILD_TYPE | Release |
| enable_s3 | Build Velox with -DENABLE_S3 | OFF |
| enable_hdfs | Build Velox with -DENABLE_HDFS | OFF |
| build_protobuf | build protobuf from source | ON |
| run_setup_script | Run Velox setup script before build | ON |
To build different backends, there are 3 parameters can be set via -P for mvn.
| Parameters | Description | Activation state by default |
|---|---|---|
| backends-velox | Add -Pbackends-velox in maven command to compile the JVM part of Velox backend. | disabled |
| backends-clickhouse | Add -Pbackends-clickhouse in maven command to compile the JVM part of ClickHouse backend. | disabled |
The gluten jar's name pattern is gluten-<backend_type>-bundle-spark<sparkbundle.version>_<scala.binary.version>-<os.detected.release>_<os.detected.release.version>-<project.version>.jar.
| Spark Version | sparkbundle.version | scala.binary.version |
|---|---|---|
| 3.2.2 | 3.2 | 2.12 |
| 3.3.1 | 3.3 | 2.12 |
The velox backend and the clickhouse backend support both spark-3.2.2 and spark-3.3.1.