Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Make ANSI fallback more granular #509

Closed
wants to merge 21 commits into from
Closed

chore: Make ANSI fallback more granular #509

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
84f6f70
make ANSI fallback more granular
andygrove Jun 3, 2024
5ab3ee0
Add CometEvalMode
andygrove Jun 3, 2024
e377e44
Add CometEvalMode enum
andygrove Jun 3, 2024
cccee7d
remove old code
andygrove Jun 3, 2024
e0bddd6
shim
andygrove Jun 3, 2024
338ba6d
more shim work
andygrove Jun 3, 2024
54032df
more shim work
andygrove Jun 3, 2024
97959a9
lint and spark 4 shim
andygrove Jun 3, 2024
bf8ed7f
formatting
andygrove Jun 4, 2024
bc72ba2
Merge remote-tracking branch 'apache/main' into ansi-fallback
andygrove Jun 4, 2024
e7d648d
more ansi fallback
andygrove Jun 4, 2024
60de09c
format
andygrove Jun 4, 2024
4d43485
refine
andygrove Jun 4, 2024
992ad20
simplify PR
andygrove Jun 4, 2024
9688cc6
address feedback
andygrove Jun 5, 2024
901a622
fall back for more expressions in ANSI mode
andygrove Jun 6, 2024
23f5996
stop disabling ANSI mode in tests
andygrove Jun 6, 2024
9f30b4c
fix 3.3 shim
andygrove Jun 6, 2024
ba21817
Enable Comet ANSI mode by default
andygrove Jun 6, 2024
fda7b30
use default Comet ANSI mode
andygrove Jun 7, 2024
1bb6d04
remove COMET_ANSI_MODE_ENABLED
andygrove Jun 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 1 addition & 12 deletions spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._

import org.apache.comet.CometConf._
import org.apache.comet.CometSparkSessionExtensions.{createMessage, getCometShuffleNotEnabledReason, isANSIEnabled, isCometBroadCastForceEnabled, isCometEnabled, isCometExecEnabled, isCometJVMShuffleMode, isCometNativeShuffleMode, isCometOperatorEnabled, isCometScan, isCometScanEnabled, isCometShuffleEnabled, isSchemaSupported, isSpark34Plus, shouldApplyRowToColumnar, withInfo, withInfos}
import org.apache.comet.CometSparkSessionExtensions.{createMessage, getCometShuffleNotEnabledReason, isCometBroadCastForceEnabled, isCometEnabled, isCometExecEnabled, isCometJVMShuffleMode, isCometNativeShuffleMode, isCometOperatorEnabled, isCometScan, isCometScanEnabled, isCometShuffleEnabled, isSchemaSupported, isSpark34Plus, shouldApplyRowToColumnar, withInfo, withInfos}
import org.apache.comet.parquet.{CometParquetScan, SupportsComet}
import org.apache.comet.serde.OperatorOuterClass.Operator
import org.apache.comet.serde.QueryPlanSerde
Expand Down Expand Up @@ -712,17 +712,6 @@ class CometSparkSessionExtensions
}

override def apply(plan: SparkPlan): SparkPlan = {
// DataFusion doesn't have ANSI mode. For now we just disable CometExec if ANSI mode is
// enabled.
if (isANSIEnabled(conf)) {
if (COMET_ANSI_MODE_ENABLED.get()) {
logWarning("Using Comet's experimental support for ANSI mode.")
} else {
logInfo("Comet extension disabled for ANSI mode")
return plan
}
}
Comment on lines -717 to -724
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Spark 3.x, I feel we still need this protection in case users enable ANSI mode as well as native execution.
If I understand correctly we have not set up CI yet with ANSI enabled.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This current check will make all plans fall back to Spark if ANSI is enabled, so no native plans will run, unless COMET_ANSI_MODE_ENABLED is enabled.

The changes in this PR mean that we still have the same check but only if the plan actually contains any expressions that would be affected by enabling ANSI support and we still fall back to Spark by default unless COMET_ANSI_MODE_ENABLED is enabled.

The main risk with this PR is if we don't have the complete list of expressions that support ANSI mode.

Copy link
Member Author

@andygrove andygrove Jun 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly we have not set up CI yet with ANSI enabled.

Well, we have the Spark 4 tests where ANSI is enabled and that is catching issues for sure.

I noticed that we were explicitly disabling ANSI mode in CometTestBase and I have removed that so we use the default ANSI mode for whatever Spark version we are running against. This should mean that all of our unit tests will now run with ANSI enabled when running against Spark 4+. Let's see how many issues this finds 😰

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly we have not set up CI yet with ANSI enabled.

It would be good to add a CI pipeline for Spark 3.4 with ANSI enabled.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am close to finish my PR to enable the spark tests with Spark 4.0 I.e. ANSI enabled, and I am disabling expressions that failed those tests with ANSI enabled.

I would propose to hold this part of change for now. Perhaps after we fix all Spark 4.0 issues, we can backport the learning to Spark 3.4

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I have moved this to draft for now.


// We shouldn't transform Spark query plan if Comet is disabled.
if (!isCometEnabled(conf)) return plan

Expand Down
6 changes: 3 additions & 3 deletions spark/src/main/scala/org/apache/comet/GenerateDocs.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import scala.io.Source

import org.apache.spark.sql.catalyst.expressions.Cast

import org.apache.comet.expressions.{CometCast, Compatible, Incompatible}
import org.apache.comet.expressions.{CometCast, CometEvalMode, Compatible, Incompatible}

/**
* Utility for generating markdown documentation from the configs.
Expand Down Expand Up @@ -72,7 +72,7 @@ object GenerateDocs {
if (Cast.canCast(fromType, toType) && fromType != toType) {
val fromTypeName = fromType.typeName.replace("(10,2)", "")
val toTypeName = toType.typeName.replace("(10,2)", "")
CometCast.isSupported(fromType, toType, None, "LEGACY") match {
CometCast.isSupported(fromType, toType, None, CometEvalMode.LEGACY) match {
case Compatible(notes) =>
val notesStr = notes.getOrElse("").trim
w.write(s"| $fromTypeName | $toTypeName | $notesStr |\n".getBytes)
Expand All @@ -89,7 +89,7 @@ object GenerateDocs {
if (Cast.canCast(fromType, toType) && fromType != toType) {
val fromTypeName = fromType.typeName.replace("(10,2)", "")
val toTypeName = toType.typeName.replace("(10,2)", "")
CometCast.isSupported(fromType, toType, None, "LEGACY") match {
CometCast.isSupported(fromType, toType, None, CometEvalMode.LEGACY) match {
case Incompatible(notes) =>
val notesStr = notes.getOrElse("").trim
w.write(s"| $fromTypeName | $toTypeName | $notesStr |\n".getBytes)
Expand Down
4 changes: 2 additions & 2 deletions spark/src/main/scala/org/apache/comet/expressions/CometCast.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ object CometCast {
fromType: DataType,
toType: DataType,
timeZoneId: Option[String],
evalMode: String): SupportLevel = {
evalMode: CometEvalMode.Value): SupportLevel = {

if (fromType == toType) {
return Compatible()
Expand Down Expand Up @@ -102,7 +102,7 @@ object CometCast {
private def canCastFromString(
toType: DataType,
timeZoneId: Option[String],
evalMode: String): SupportLevel = {
evalMode: CometEvalMode.Value): SupportLevel = {
toType match {
case DataTypes.BooleanType =>
Compatible()
Expand Down
42 changes: 42 additions & 0 deletions spark/src/main/scala/org/apache/comet/expressions/CometEvalMode.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.comet.expressions

/**
* We cannot reference Spark's EvalMode directly because the package is different between Spark
* versions, so we copy it here.
*
* Expression evaluation modes.
* - LEGACY: the default evaluation mode, which is compliant to Hive SQL.
* - ANSI: a evaluation mode which is compliant to ANSI SQL standard.
* - TRY: a evaluation mode for `try_*` functions. It is identical to ANSI evaluation mode
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is conf.ansiEnabled ignored for try_* ? Maybe the comment can clarify.

* except for returning null result on errors.
*/
object CometEvalMode extends Enumeration {
val LEGACY, ANSI, TRY = Value

def fromBoolean(ansiEnabled: Boolean): Value = if (ansiEnabled) {
ANSI
} else {
LEGACY
}

def fromString(str: String): CometEvalMode.Value = CometEvalMode.withName(str)
}
80 changes: 45 additions & 35 deletions spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@

package org.apache.comet.serde

import java.util.Locale

import scala.collection.JavaConverters._

import org.apache.spark.internal.Logging
Expand All @@ -45,7 +43,7 @@ import org.apache.spark.unsafe.types.UTF8String

import org.apache.comet.CometConf
import org.apache.comet.CometSparkSessionExtensions.{isCometOperatorEnabled, isCometScan, isSpark32, isSpark34Plus, withInfo}
import org.apache.comet.expressions.{CometCast, Compatible, Incompatible, Unsupported}
import org.apache.comet.expressions.{CometCast, CometEvalMode, Compatible, Incompatible, Unsupported}
import org.apache.comet.serde.ExprOuterClass.{AggExpr, DataType => ProtoDataType, Expr, ScalarFunc}
import org.apache.comet.serde.ExprOuterClass.DataType.{DataTypeInfo, DecimalInfo, ListInfo, MapInfo, StructInfo}
import org.apache.comet.serde.OperatorOuterClass.{AggregateMode => CometAggregateMode, JoinType, Operator}
Expand Down Expand Up @@ -578,6 +576,15 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde with CometExprShim
}
}

def evalModeToProto(evalMode: CometEvalMode.Value): ExprOuterClass.EvalMode = {
evalMode match {
case CometEvalMode.LEGACY => ExprOuterClass.EvalMode.LEGACY
case CometEvalMode.TRY => ExprOuterClass.EvalMode.TRY
case CometEvalMode.ANSI => ExprOuterClass.EvalMode.ANSI
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a 'catch all' case for when someone tries to change CometEvalMode and things don't work as planned?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I have added this.

case _ => throw new IllegalStateException(s"Invalid evalMode $evalMode")
}
}

/**
* Convert a Spark expression to protobuf.
*
Expand All @@ -590,18 +597,6 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde with CometExprShim
* @return
* The protobuf representation of the expression, or None if the expression is not supported
*/

def stringToEvalMode(evalModeStr: String): ExprOuterClass.EvalMode =
evalModeStr.toUpperCase(Locale.ROOT) match {
case "LEGACY" => ExprOuterClass.EvalMode.LEGACY
case "TRY" => ExprOuterClass.EvalMode.TRY
case "ANSI" => ExprOuterClass.EvalMode.ANSI
case invalid =>
throw new IllegalArgumentException(
s"Invalid eval mode '$invalid' "
) // Assuming we want to catch errors strictly
}

def exprToProto(
expr: Expression,
input: Seq[Attribute],
Expand All @@ -610,15 +605,14 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde with CometExprShim
timeZoneId: Option[String],
dt: DataType,
childExpr: Option[Expr],
evalMode: String): Option[Expr] = {
evalMode: CometEvalMode.Value): Option[Expr] = {
val dataType = serializeDataType(dt)
val evalModeEnum = stringToEvalMode(evalMode) // Convert string to enum

if (childExpr.isDefined && dataType.isDefined) {
val castBuilder = ExprOuterClass.Cast.newBuilder()
castBuilder.setChild(childExpr.get)
castBuilder.setDatatype(dataType.get)
castBuilder.setEvalMode(evalModeEnum) // Set the enum in protobuf
castBuilder.setEvalMode(evalModeToProto(evalMode))

val timeZone = timeZoneId.getOrElse("UTC")
castBuilder.setTimezone(timeZone)
Expand Down Expand Up @@ -646,26 +640,26 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde with CometExprShim
inputs: Seq[Attribute],
dt: DataType,
timeZoneId: Option[String],
actualEvalModeStr: String): Option[Expr] = {
evalMode: CometEvalMode.Value): Option[Expr] = {

val childExpr = exprToProtoInternal(child, inputs)
if (childExpr.isDefined) {
val castSupport =
CometCast.isSupported(child.dataType, dt, timeZoneId, actualEvalModeStr)
CometCast.isSupported(child.dataType, dt, timeZoneId, evalMode)

def getIncompatMessage(reason: Option[String]): String =
"Comet does not guarantee correct results for cast " +
s"from ${child.dataType} to $dt " +
s"with timezone $timeZoneId and evalMode $actualEvalModeStr" +
s"with timezone $timeZoneId and evalMode $evalMode" +
reason.map(str => s" ($str)").getOrElse("")

castSupport match {
case Compatible(_) =>
castToProto(timeZoneId, dt, childExpr, actualEvalModeStr)
castToProto(timeZoneId, dt, childExpr, evalMode)
case Incompatible(reason) =>
if (CometConf.COMET_CAST_ALLOW_INCOMPATIBLE.get()) {
logWarning(getIncompatMessage(reason))
castToProto(timeZoneId, dt, childExpr, actualEvalModeStr)
castToProto(timeZoneId, dt, childExpr, evalMode)
} else {
withInfo(
expr,
Expand All @@ -677,7 +671,7 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde with CometExprShim
withInfo(
expr,
s"Unsupported cast from ${child.dataType} to $dt " +
s"with timezone $timeZoneId and evalMode $actualEvalModeStr")
s"with timezone $timeZoneId and evalMode $evalMode")
None
}
} else {
Expand All @@ -701,17 +695,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde with CometExprShim

case UnaryExpression(child) if expr.prettyName == "trycast" =>
val timeZoneId = SQLConf.get.sessionLocalTimeZone
handleCast(child, inputs, expr.dataType, Some(timeZoneId), "TRY")
handleCast(child, inputs, expr.dataType, Some(timeZoneId), CometEvalMode.TRY)

case Cast(child, dt, timeZoneId, evalMode) =>
val evalModeStr = if (evalMode.isInstanceOf[Boolean]) {
// Spark 3.2 & 3.3 has ansiEnabled boolean
if (evalMode.asInstanceOf[Boolean]) "ANSI" else "LEGACY"
} else {
// Spark 3.4+ has EvalMode enum with values LEGACY, ANSI, and TRY
evalMode.toString
}
handleCast(child, inputs, dt, timeZoneId, evalModeStr)
case c @ Cast(child, dt, timeZoneId, _) =>
handleCast(child, inputs, dt, timeZoneId, evalMode(c))

case expr if isUnsupportedAnsiExpr(expr) && !CometConf.COMET_ANSI_MODE_ENABLED.get() =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel !CometConf.COMET_ANSI_MODE_ENABLED.get() condition needs to be removed.

When COMET_ANSI_MODE_ENABLED=true, we still need to fallback to Spark if the expression is not ANSI compatible

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, let's remove this. It was intended as as short term workaround but I agree that we don't really need it now that we have this fallback logic.

withInfo(
expr,
"ANSI mode not supported. " +
s"Set ${CometConf.COMET_ANSI_MODE_ENABLED.key}=true to enable it anyway.")
None

case add @ Add(left, right, _) if supportedDataType(left.dataType) =>
val leftExpr = exprToProtoInternal(left, inputs)
Expand Down Expand Up @@ -2006,7 +2000,7 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde with CometExprShim
// TODO: Remove this once we have new DataFusion release which includes
// the fix: https://github.com/apache/arrow-datafusion/pull/9459
if (childExpr.isDefined) {
castToProto(None, a.dataType, childExpr, "LEGACY")
castToProto(None, a.dataType, childExpr, CometEvalMode.LEGACY)
} else {
withInfo(expr, a.children: _*)
None
Expand Down Expand Up @@ -2207,6 +2201,22 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde with CometExprShim
exprToProtoInternal(newExpr, input)
}

/** These expressions are not compatible with Spark in ANSI mode yet */
private def isUnsupportedAnsiExpr(expr: Expression): Boolean = expr match {
case expr: Add => evalMode(expr) == CometEvalMode.ANSI
case expr: Subtract => evalMode(expr) == CometEvalMode.ANSI
case expr: Multiply => evalMode(expr) == CometEvalMode.ANSI
case expr: Divide => evalMode(expr) == CometEvalMode.ANSI
case expr: IntegralDivide => evalMode(expr) == CometEvalMode.ANSI
case expr: Remainder => evalMode(expr) == CometEvalMode.ANSI
case expr: Pmod => evalMode(expr) == CometEvalMode.ANSI
case expr: Round => evalMode(expr) == CometEvalMode.ANSI
case expr: BRound => evalMode(expr) == CometEvalMode.ANSI
case expr: Sum => evalMode(expr) == CometEvalMode.ANSI
case expr: Average => evalMode(expr) == CometEvalMode.ANSI
case _ => false
}

def scalarExprToProtoWithReturnType(
funcName: String,
returnType: DataType,
Expand Down
35 changes: 35 additions & 0 deletions spark/src/main/spark-3.2/org/apache/comet/shims/CometExprShim.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@
*/
package org.apache.comet.shims

import org.apache.comet.expressions.CometEvalMode
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate.{Average, Sum}

/**
* `CometExprShim` acts as a shim for for parsing expressions from different Spark versions.
Expand All @@ -30,4 +32,37 @@ trait CometExprShim {
def unhexSerde(unhex: Unhex): (Expression, Expression) = {
(unhex.child, Literal(false))
}

def evalMode(expr: Add): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Subtract): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Multiply): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Divide): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: IntegralDivide): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Remainder): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Pmod): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Sum): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Average): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(c: Cast): CometEvalMode.Value = CometEvalMode.fromBoolean(c.ansiEnabled)

def evalMode(r: Round): CometEvalMode.Value = CometEvalMode.LEGACY

def evalMode(r: BRound): CometEvalMode.Value = CometEvalMode.LEGACY
}
35 changes: 35 additions & 0 deletions spark/src/main/spark-3.3/org/apache/comet/shims/CometExprShim.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@
*/
package org.apache.comet.shims

import org.apache.comet.expressions.CometEvalMode
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate.{Average, Sum}

/**
* `CometExprShim` acts as a shim for for parsing expressions from different Spark versions.
Expand All @@ -30,4 +32,37 @@ trait CometExprShim {
def unhexSerde(unhex: Unhex): (Expression, Expression) = {
(unhex.child, Literal(false))
}

def evalMode(expr: Add): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Subtract): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Multiply): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Divide): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: IntegralDivide): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Remainder): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Pmod): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Sum): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(expr: Average): CometEvalMode.Value =
CometEvalMode.fromBoolean(expr.failOnError)

def evalMode(c: Cast): CometEvalMode.Value = CometEvalMode.fromBoolean(c.ansiEnabled)

def evalMode(r: Round): CometEvalMode.Value = CometEvalMode.LEGACY

def evalMode(r: BRound): CometEvalMode.Value = CometEvalMode.LEGACY
}
Loading
Loading