build: Add Spark SQL test pipeline with ANSI mode enabled #321
Conversation
| + val v = System.getenv("ENABLE_COMET_ANSI_MODE") | ||
| + v != null && v.toBoolean |
There was a problem hiding this comment.
I think this could be simplified as:
| + val v = System.getenv("ENABLE_COMET_ANSI_MODE") | |
| + v != null && v.toBoolean | |
| + System.getenv("ENABLE_COMET_ANSI_MODE").exists(_.toBoolean) |
There was a problem hiding this comment.
Compiler didn't like the suggested change and I didn't quite understand why. Also, the way it is in the PR, matches the pattern followed by similar other code in the diff, so leaving it as it is.
|
Hmm, I think currently this is going to have failed pipeline, do you want to have this in CI? Or you want to hold it here until we have ANSI mode support to make this passed? |
I think we should exclude this from ci but it should be available to run manually so we can run this after every improvement to ANSI support. Let me investigate how to make it manual/on demand. |
|
@liang-chi made this manual run only. Thanks for pointing that out! |
|
@viirya @andygrove, perhaps we can merge this? |
|
Merged. Thanks. |
|
Thank you @viirya @andygrove |
* build: Add Spark SQL test pipeline with ANSI mode enabled * add ENABLE_COMET_ANSI_MODE to actual run of tests * fix diff and rat exclusion * fix diff * Make workflow manual run only * fix diff
Closes #320 .
Rationale for this change
Setup a test pipeline to run Spark sql tests with ANSI modeenabled. This is part of the effort to reach full ANSI mode support
What changes are included in this PR?
Adds a new manual pipeline to run Spark SQL tests with ANSOI mode enabled.
Sets the environment variable
ENABLE_COMET_ANSI_MODE=truebefore running the tests.How are these changes tested?
Manually run spark tests on command line with Spark 3.4.2.