From 7c6405add933299c8f3fdb60bd1f049e8173a1c6 Mon Sep 17 00:00:00 2001 From: Kazuyuki Tanimura Date: Fri, 27 Sep 2024 13:15:18 -0700 Subject: [PATCH] chore: fix compatibility guide --- docs/templates/compatibility-template.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/templates/compatibility-template.md b/docs/templates/compatibility-template.md index 137870a7e..f6a725ac6 100644 --- a/docs/templates/compatibility-template.md +++ b/docs/templates/compatibility-template.md @@ -38,6 +38,14 @@ Comet uses the Rust regexp crate for evaluating regular expressions, and this ha regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`. +## Floating number comparison + +Spark normalizes NaN and zero for floating point numbers for several cases. See `NormalizeFloatingNumbers` optimization rule in Spark. +However, one exception is comparison. Spark does not normalize NaN and zero when comparing values +because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison +functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)). +So Comet will add additional normalization expression of NaN and zero for comparison. + ## Cast Cast operations in Comet fall into three levels of support: