feat: Add extended explain info to Comet plan (apache#255)

* feat: Add extended explain info to Comet plan

Requires Spark 4.0.0 for the explain info to be visible in Spark UI.
(see: https://issues.apache.org/jira/browse/SPARK-47289)

* spotless apply

* Address review comments

* fix ci

* Add one more explanation

* address review comments

* fix formatting after rebase

dev/ensure-jars-have-correct-contents.sh

@@ -78,6 +78,8 @@ allowed_expr+="|^org/apache/spark/shuffle/sort/CometShuffleExternalSorter.*$"
 allowed_expr+="|^org/apache/spark/shuffle/sort/RowPartition.class$"
 allowed_expr+="|^org/apache/spark/shuffle/comet/.*$"
 allowed_expr+="|^org/apache/spark/sql/$"
+# allow ExplainPlanGenerator trait since it may not be available in older Spark versions
+allowed_expr+="|^org/apache/spark/sql/ExtendedExplainGenerator.*$"
 allowed_expr+="|^org/apache/spark/CometPlugin.class$"
 allowed_expr+="|^org/apache/spark/CometDriverPlugin.*$"
 allowed_expr+="|^org/apache/spark/CometTaskMemoryManager.class$"

pom.xml

@@ -959,6 +959,9 @@ under the License.
                         <ignoreClass>javax.annotation.meta.TypeQualifierValidator</ignoreClass>
                         <!-- this class is not properly excluded from comet-spark right now -->
                         <ignoreClass>org.apache.parquet.filter2.predicate.SparkFilterApi</ignoreClass>
+                        <!-- we explicitly include a duplicate to allow older versions of Spark -->
+                        <!-- this can be removed once we no longer support spark 3.x  -->
+                        <ignoreClass>org.apache.spark.sql.ExtendedExplainGenerator</ignoreClass>
                       </ignoreClasses>
                     </dependency>
                     <dependency>