Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Gluten CI #6848

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
+63 −29
Open

Fix Gluten CI #6848

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a7eacb9
Fix Gluten CI
wForget Dec 9, 2024
100be1c
fix
wForget Dec 11, 2024
0f4d2f5
fix
wForget Dec 11, 2024
e4af6fc
fix
wForget Dec 11, 2024
8c3e480
fix
wForget Dec 11, 2024
c433330
fix
wForget Dec 11, 2024
b9117e9
fix test
wForget Dec 11, 2024
91f3e0d
fix test
wForget Dec 11, 2024
839b9f2
clean
wForget Dec 11, 2024
fc83db6
Update .github/workflows/gluten.yml
wForget Dec 11, 2024
848cbe3
pr trigger
wForget Dec 12, 2024
e2e3277
Revert "pr trigger"
wForget Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 33 additions & 21 deletions .github/workflows/gluten.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,6 @@ jobs:
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Update and Upgrade
run: sudo apt-get update && sudo apt-get upgrade -y
- name: Install dependencies
run: |
sudo apt-get install -y software-properties-common
sudo apt-get install -y libunwind-dev build-essential cmake libssl-dev libre2-dev libcurl4-openssl-dev clang lldb lld libz-dev git ninja-build uuid-dev
- name: Setup JDK 8
uses: actions/setup-java@v4
with:
Expand All @@ -55,20 +49,45 @@ jobs:
id: gluten-cache
uses: actions/cache@v4
with:
path: gluten/package/target/
path: incubator-gluten/package/target/
key: gluten_package_${{ steps.date.outputs.date }}
- name: Build gluten project
- name: Get Ccache
id: gluten-ccache-cache
uses: actions/cache@v4
with:
path: incubator-gluten/.ccache
key: ccache-centos7-release-default-${{github.sha}}
restore-keys: |
ccache-centos7-release-default
- name: Build Gluten velox third party
run: |
if [[ "${{ steps.gluten-cache.outputs.cache-hit }}" != 'true' ]]; then
git clone https://github.com/oap-project/gluten.git
cd gluten
./dev/buildbundle-veloxbe.sh
git clone https://github.com/apache/incubator-gluten.git
wForget marked this conversation as resolved.
Show resolved Hide resolved
cd incubator-gluten
docker run -v $GITHUB_WORKSPACE/incubator-gluten:/work -w /work apache/gluten:vcpkg-centos-7 bash -c "
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How hard would it be if we just built it from scratch?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This docker image includes the gluten/velox dependencies, reducing the time to download and build dependencies.

df -a
cd /work
export CCACHE_DIR=/work/.ccache
bash dev/ci-velox-buildstatic-centos-7.sh
ccache -s
mkdir -p /work/.m2/repository/org/apache/arrow/
cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
"
mkdir -p ~/.m2/repository/org/apache/arrow
cp -r $GITHUB_WORKSPACE/incubator-gluten/.m2/repository/org/apache/arrow/* ~/.m2/repository/org/apache/arrow/
mvn clean package -Pbackends-velox -Pceleborn -Puniffle -Pspark-3.5 -DskipTests
fi
- uses: actions/cache@v4
if: steps.gluten-cache.outputs.cache-hit != 'true'
with:
path: gluten/package/target/
path: incubator-gluten/package/target/
key: gluten_package_${{ steps.date.outputs.date }}
- name: Save Ccache
uses: actions/cache@v4
if: steps.gluten-ccache-cache.outputs.cache-hit != 'true'
with:
path: incubator-gluten/.ccache/
key: ccache-centos7-release-default-${{github.sha}}

gluten-it:
name: Gluten Integration TPC-H/DS Test
Expand All @@ -77,20 +96,13 @@ jobs:
strategy:
fail-fast: false
matrix:
spark: [ '3.4', '3.3' ]
spark: [ '3.5' ]
steps:
- uses: actions/checkout@v4
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Update and Upgrade
run: sudo apt-get update && sudo apt-get upgrade -y
- name: Install dependencies
run: |
sudo apt-get install -y software-properties-common
sudo apt-get install -y libunwind-dev build-essential cmake libssl-dev libre2-dev libcurl4-openssl-dev clang lldb lld libz-dev git ninja-build uuid-dev
sudo apt-get install -y libsnappy-dev libthrift-dev libboost-all-dev libgflags-dev libgoogle-glog-dev
- name: Cache Engine Archives
uses: ./.github/actions/cache-engine-archives
- name: Get gluten cache date
Expand All @@ -100,7 +112,7 @@ jobs:
id: gluten-cache
uses: actions/cache@v4
with:
path: gluten/package/target/
path: incubator-gluten/package/target/
key: gluten_package_${{ steps.date.outputs.date }}
- name: Setup JDK 8
uses: actions/setup-java@v4
Expand Down
29 changes: 26 additions & 3 deletions integration-tests/kyuubi-gluten-it/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,11 @@
<url>https://kyuubi.apache.org/</url>

<properties>
<gluten.version>1.2.0-SNAPSHOT</gluten.version>
<gluten.artifact.name>gluten-velox-bundle-spark${spark.binary.version}_${scala.binary.version}-ubuntu_22.04</gluten.artifact.name>
<gluten.version>1.3.0-SNAPSHOT</gluten.version>
<os.detected.release>ubuntu</os.detected.release>
<os.detected.release.version>24.04</os.detected.release.version>
<os.detected.arch>x86_64</os.detected.arch>
<gluten.artifact.name>gluten-velox-bundle-spark${spark.binary.version}_${scala.binary.version}-${os.detected.release}_${os.detected.release.version}_${os.detected.arch}</gluten.artifact.name>
</properties>

<dependencies>
Expand All @@ -40,7 +43,7 @@
<artifactId>${gluten.artifact.name}</artifactId>
<version>${gluten.version}</version>
<scope>system</scope>
<systemPath>${project.basedir}/../../gluten/package/target/${gluten.artifact.name}_x86_64-${gluten.version}.jar</systemPath>
<systemPath>${project.basedir}/../../incubator-gluten/package/target/${gluten.artifact.name}-${gluten.version}.jar</systemPath>
</dependency>

<dependency>
Expand All @@ -57,6 +60,14 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-util-scala_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-connector-common_${scala.binary.version}</artifactId>
Expand Down Expand Up @@ -88,6 +99,18 @@
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>jakarta.servlet</groupId>
<artifactId>jakarta.servlet-api</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
3 changes: 1 addition & 2 deletions ...ation-tests/kyuubi-gluten-it/src/test/scala/org/apache/kyuubi/it/gluten/GlutenSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,7 @@ class GlutenSuite extends KyuubiFunSuite with GlutenSuiteMixin {
test("KYUUBI #5467: test gluten plan") {
withSparkSession(SparkSession.builder.config(sparkConf).getOrCreate()) { spark =>
val plan = spark.sql("explain SELECT 1").head().getString(0)
assert(plan.contains("VeloxColumnarToRowExec") && plan.contains(
"VeloxColumnarToRowExec") && plan.contains("RowToVeloxColumnar"))
assert(plan.contains("VeloxColumnarToRow") && plan.contains("RowToVeloxColumnar"))
}
}
}
4 changes: 2 additions & 2 deletions ...uubi-gluten-it/src/test/scala/org/apache/kyuubi/it/gluten/tpch/GlutenTPCHQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ import org.apache.kyuubi.tags.GlutenTest
@Slow
@GlutenTest
class GlutenTPCHQuerySuite extends KyuubiFunSuite with GlutenSuiteMixin {
// TODO: Fix the inconsistency in q9 results.
val queries: Set[String] = (1 to 22).map(i => s"q$i").toSet - "q9"
// TODO: Fix the inconsistency in q3/q9 results.
val queries: Set[String] = (1 to 22).map(i => s"q$i").toSet -- Seq("q3", "q9")

lazy val sparkConf: SparkConf = {
val glutenConf = new SparkConf().setMaster("local[*]")
Expand Down
2 changes: 1 addition & 1 deletion kyuubi-common/src/test/scala/org/apache/kyuubi/GlutenSuiteMixin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ trait GlutenSuiteMixin {
}

protected def extraConfigs: Map[String, String] = Map(
"spark.plugins" -> "io.glutenproject.GlutenPlugin",
"spark.plugins" -> "org.apache.gluten.GlutenPlugin",
"spark.memory.offHeap.size" -> "4g",
"spark.memory.offHeap.enabled" -> "true",
"spark.shuffle.manager" -> "org.apache.spark.shuffle.sort.ColumnarShuffleManager",
Expand Down
Loading