feat: Enable Spark SQL tests for Spark 3.5.1 (apache#603)

* Enable Spark SQL tests for Spark 3.5.1 * fix conflicts * finish creating diff * improve docs, update slf4j version * improve docs * fix diff * fix diff * address feedback * add IgnoreComet to patch * update test * remove unused imports * ignore 2 failing tests * scalastyle * ignore failing parquet metadata tests * add links to tracking issue
himadripal · Jul 12, 2024 · be6f4d8 · be6f4d8
1 parent 806b269
commit be6f4d8
Show file tree

Hide file tree

Showing 5 changed files with 2,947 additions and 2 deletions.
diff --git a/.github/workflows/spark_sql_test.yml b/.github/workflows/spark_sql_test.yml
@@ -45,7 +45,7 @@ jobs:
       matrix:
         os: [ubuntu-latest]
         java-version: [11]
-        spark-version: [{short: '3.4', full: '3.4.3'}]
+        spark-version: [{short: '3.4', full: '3.4.3'}, {short: '3.5', full: '3.5.1'}]
         module:
           - {name: "catalyst", args1: "catalyst/test", args2: ""}
           - {name: "sql/core-1", args1: "", args2: sql/testOnly * -- -l org.apache.spark.tags.ExtendedSQLTest -l org.apache.spark.tags.SlowSQLTest}
@@ -75,6 +75,7 @@ jobs:
       - name: Run Spark tests
         run: |
           cd apache-spark
+          rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups
           ENABLE_COMET=true build/sbt ${{ matrix.module.args1 }} "${{ matrix.module.args2 }}"
         env:
           LC_ALL: "C.UTF-8"