Skip to content

Commit

Permalink
revert plan changes
Browse files Browse the repository at this point in the history
  • Loading branch information
@andygrove
andygrove committed Aug 20, 2024
1 parent 897dd84 commit 9193d83
Show file tree
Hide file tree
Showing 6 changed files with 573 additions and 695 deletions.
158 changes: 73 additions & 85 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q17/explain.txt
View file
Original file line number Diff line number Diff line change
@@ -1,48 +1,46 @@
== Physical Plan ==
TakeOrderedAndProject (44)
+- * HashAggregate (43)
+- * ColumnarToRow (42)
+- CometColumnarExchange (41)
+- RowToColumnar (40)
+- * HashAggregate (39)
+- * ColumnarToRow (38)
+- CometProject (37)
+- CometBroadcastHashJoin (36)
:- CometProject (32)
: +- CometBroadcastHashJoin (31)
: :- CometProject (27)
: : +- CometBroadcastHashJoin (26)
: : :- CometProject (24)
: : : +- CometBroadcastHashJoin (23)
: : : :- CometProject (18)
: : : : +- CometBroadcastHashJoin (17)
: : : : :- CometProject (12)
: : : : : +- CometBroadcastHashJoin (11)
: : : : : :- CometProject (7)
: : : : : : +- CometBroadcastHashJoin (6)
: : : : : : :- CometFilter (2)
: : : : : : : +- CometScan parquet spark_catalog.default.store_sales (1)
: : : : : : +- CometBroadcastExchange (5)
: : : : : : +- CometFilter (4)
: : : : : : +- CometScan parquet spark_catalog.default.store_returns (3)
: : : : : +- CometBroadcastExchange (10)
: : : : : +- CometFilter (9)
: : : : : +- CometScan parquet spark_catalog.default.catalog_sales (8)
: : : : +- CometBroadcastExchange (16)
: : : : +- CometProject (15)
: : : : +- CometFilter (14)
: : : : +- CometScan parquet spark_catalog.default.date_dim (13)
: : : +- CometBroadcastExchange (22)
: : : +- CometProject (21)
: : : +- CometFilter (20)
: : : +- CometScan parquet spark_catalog.default.date_dim (19)
: : +- ReusedExchange (25)
: +- CometBroadcastExchange (30)
: +- CometFilter (29)
: +- CometScan parquet spark_catalog.default.store (28)
+- CometBroadcastExchange (35)
+- CometFilter (34)
+- CometScan parquet spark_catalog.default.item (33)
* ColumnarToRow (42)
+- CometTakeOrderedAndProject (41)
+- CometHashAggregate (40)
+- CometColumnarExchange (39)
+- CometHashAggregate (38)
+- CometProject (37)
+- CometBroadcastHashJoin (36)
:- CometProject (32)
: +- CometBroadcastHashJoin (31)
: :- CometProject (27)
: : +- CometBroadcastHashJoin (26)
: : :- CometProject (24)
: : : +- CometBroadcastHashJoin (23)
: : : :- CometProject (18)
: : : : +- CometBroadcastHashJoin (17)
: : : : :- CometProject (12)
: : : : : +- CometBroadcastHashJoin (11)
: : : : : :- CometProject (7)
: : : : : : +- CometBroadcastHashJoin (6)
: : : : : : :- CometFilter (2)
: : : : : : : +- CometScan parquet spark_catalog.default.store_sales (1)
: : : : : : +- CometBroadcastExchange (5)
: : : : : : +- CometFilter (4)
: : : : : : +- CometScan parquet spark_catalog.default.store_returns (3)
: : : : : +- CometBroadcastExchange (10)
: : : : : +- CometFilter (9)
: : : : : +- CometScan parquet spark_catalog.default.catalog_sales (8)
: : : : +- CometBroadcastExchange (16)
: : : : +- CometProject (15)
: : : : +- CometFilter (14)
: : : : +- CometScan parquet spark_catalog.default.date_dim (13)
: : : +- CometBroadcastExchange (22)
: : : +- CometProject (21)
: : : +- CometFilter (20)
: : : +- CometScan parquet spark_catalog.default.date_dim (19)
: : +- ReusedExchange (25)
: +- CometBroadcastExchange (30)
: +- CometFilter (29)
: +- CometScan parquet spark_catalog.default.store (28)
+- CometBroadcastExchange (35)
+- CometFilter (34)
+- CometScan parquet spark_catalog.default.item (33)


(1) Scan parquet spark_catalog.default.store_sales
Expand Down Expand Up @@ -223,96 +221,86 @@ Arguments: [ss_item_sk#1], [i_item_sk#26], Inner, BuildRight
Input [8]: [ss_item_sk#1, ss_quantity#5, sr_return_quantity#11, cs_quantity#16, s_state#25, i_item_sk#26, i_item_id#27, i_item_desc#28]
Arguments: [ss_quantity#5, sr_return_quantity#11, cs_quantity#16, s_state#25, i_item_id#27, i_item_desc#28], [ss_quantity#5, sr_return_quantity#11, cs_quantity#16, s_state#25, i_item_id#27, i_item_desc#28]

(38) ColumnarToRow [codegen id : 1]
Input [6]: [ss_quantity#5, sr_return_quantity#11, cs_quantity#16, s_state#25, i_item_id#27, i_item_desc#28]

(39) HashAggregate [codegen id : 1]
(38) CometHashAggregate
Input [6]: [ss_quantity#5, sr_return_quantity#11, cs_quantity#16, s_state#25, i_item_id#27, i_item_desc#28]
Keys [3]: [i_item_id#27, i_item_desc#28, s_state#25]
Functions [9]: [partial_count(ss_quantity#5), partial_avg(ss_quantity#5), partial_stddev_samp(cast(ss_quantity#5 as double)), partial_count(sr_return_quantity#11), partial_avg(sr_return_quantity#11), partial_stddev_samp(cast(sr_return_quantity#11 as double)), partial_count(cs_quantity#16), partial_avg(cs_quantity#16), partial_stddev_samp(cast(cs_quantity#16 as double))]
Aggregate Attributes [18]: [count#29, sum#30, count#31, n#32, avg#33, m2#34, count#35, sum#36, count#37, n#38, avg#39, m2#40, count#41, sum#42, count#43, n#44, avg#45, m2#46]
Results [21]: [i_item_id#27, i_item_desc#28, s_state#25, count#47, sum#48, count#49, n#50, avg#51, m2#52, count#53, sum#54, count#55, n#56, avg#57, m2#58, count#59, sum#60, count#61, n#62, avg#63, m2#64]

(40) RowToColumnar
Input [21]: [i_item_id#27, i_item_desc#28, s_state#25, count#47, sum#48, count#49, n#50, avg#51, m2#52, count#53, sum#54, count#55, n#56, avg#57, m2#58, count#59, sum#60, count#61, n#62, avg#63, m2#64]

(41) CometColumnarExchange
Input [21]: [i_item_id#27, i_item_desc#28, s_state#25, count#47, sum#48, count#49, n#50, avg#51, m2#52, count#53, sum#54, count#55, n#56, avg#57, m2#58, count#59, sum#60, count#61, n#62, avg#63, m2#64]
(39) CometColumnarExchange
Input [21]: [i_item_id#27, i_item_desc#28, s_state#25, count#29, sum#30, count#31, n#32, avg#33, m2#34, count#35, sum#36, count#37, n#38, avg#39, m2#40, count#41, sum#42, count#43, n#44, avg#45, m2#46]
Arguments: hashpartitioning(i_item_id#27, i_item_desc#28, s_state#25, 5), ENSURE_REQUIREMENTS, CometColumnarShuffle, [plan_id=1]

(42) ColumnarToRow [codegen id : 2]
Input [21]: [i_item_id#27, i_item_desc#28, s_state#25, count#47, sum#48, count#49, n#50, avg#51, m2#52, count#53, sum#54, count#55, n#56, avg#57, m2#58, count#59, sum#60, count#61, n#62, avg#63, m2#64]

(43) HashAggregate [codegen id : 2]
Input [21]: [i_item_id#27, i_item_desc#28, s_state#25, count#47, sum#48, count#49, n#50, avg#51, m2#52, count#53, sum#54, count#55, n#56, avg#57, m2#58, count#59, sum#60, count#61, n#62, avg#63, m2#64]
(40) CometHashAggregate
Input [21]: [i_item_id#27, i_item_desc#28, s_state#25, count#29, sum#30, count#31, n#32, avg#33, m2#34, count#35, sum#36, count#37, n#38, avg#39, m2#40, count#41, sum#42, count#43, n#44, avg#45, m2#46]
Keys [3]: [i_item_id#27, i_item_desc#28, s_state#25]
Functions [9]: [count(ss_quantity#5), avg(ss_quantity#5), stddev_samp(cast(ss_quantity#5 as double)), count(sr_return_quantity#11), avg(sr_return_quantity#11), stddev_samp(cast(sr_return_quantity#11 as double)), count(cs_quantity#16), avg(cs_quantity#16), stddev_samp(cast(cs_quantity#16 as double))]
Aggregate Attributes [9]: [count(ss_quantity#5)#65, avg(ss_quantity#5)#66, stddev_samp(cast(ss_quantity#5 as double))#67, count(sr_return_quantity#11)#68, avg(sr_return_quantity#11)#69, stddev_samp(cast(sr_return_quantity#11 as double))#70, count(cs_quantity#16)#71, avg(cs_quantity#16)#72, stddev_samp(cast(cs_quantity#16 as double))#73]
Results [15]: [i_item_id#27, i_item_desc#28, s_state#25, count(ss_quantity#5)#65 AS store_sales_quantitycount#74, avg(ss_quantity#5)#66 AS store_sales_quantityave#75, stddev_samp(cast(ss_quantity#5 as double))#67 AS store_sales_quantitystdev#76, (stddev_samp(cast(ss_quantity#5 as double))#67 / avg(ss_quantity#5)#66) AS store_sales_quantitycov#77, count(sr_return_quantity#11)#68 AS as_store_returns_quantitycount#78, avg(sr_return_quantity#11)#69 AS as_store_returns_quantityave#79, stddev_samp(cast(sr_return_quantity#11 as double))#70 AS as_store_returns_quantitystdev#80, (stddev_samp(cast(sr_return_quantity#11 as double))#70 / avg(sr_return_quantity#11)#69) AS store_returns_quantitycov#81, count(cs_quantity#16)#71 AS catalog_sales_quantitycount#82, avg(cs_quantity#16)#72 AS catalog_sales_quantityave#83, (stddev_samp(cast(cs_quantity#16 as double))#73 / avg(cs_quantity#16)#72) AS catalog_sales_quantitystdev#84, (stddev_samp(cast(cs_quantity#16 as double))#73 / avg(cs_quantity#16)#72) AS catalog_sales_quantitycov#85]

(44) TakeOrderedAndProject
Input [15]: [i_item_id#27, i_item_desc#28, s_state#25, store_sales_quantitycount#74, store_sales_quantityave#75, store_sales_quantitystdev#76, store_sales_quantitycov#77, as_store_returns_quantitycount#78, as_store_returns_quantityave#79, as_store_returns_quantitystdev#80, store_returns_quantitycov#81, catalog_sales_quantitycount#82, catalog_sales_quantityave#83, catalog_sales_quantitystdev#84, catalog_sales_quantitycov#85]
Arguments: 100, [i_item_id#27 ASC NULLS FIRST, i_item_desc#28 ASC NULLS FIRST, s_state#25 ASC NULLS FIRST], [i_item_id#27, i_item_desc#28, s_state#25, store_sales_quantitycount#74, store_sales_quantityave#75, store_sales_quantitystdev#76, store_sales_quantitycov#77, as_store_returns_quantitycount#78, as_store_returns_quantityave#79, as_store_returns_quantitystdev#80, store_returns_quantitycov#81, catalog_sales_quantitycount#82, catalog_sales_quantityave#83, catalog_sales_quantitystdev#84, catalog_sales_quantitycov#85]
(41) CometTakeOrderedAndProject
Input [15]: [i_item_id#27, i_item_desc#28, s_state#25, store_sales_quantitycount#47, store_sales_quantityave#48, store_sales_quantitystdev#49, store_sales_quantitycov#50, as_store_returns_quantitycount#51, as_store_returns_quantityave#52, as_store_returns_quantitystdev#53, store_returns_quantitycov#54, catalog_sales_quantitycount#55, catalog_sales_quantityave#56, catalog_sales_quantitystdev#57, catalog_sales_quantitycov#58]
Arguments: TakeOrderedAndProject(limit=100, orderBy=[i_item_id#27 ASC NULLS FIRST,i_item_desc#28 ASC NULLS FIRST,s_state#25 ASC NULLS FIRST], output=[i_item_id#27,i_item_desc#28,s_state#25,store_sales_quantitycount#47,store_sales_quantityave#48,store_sales_quantitystdev#49,store_sales_quantitycov#50,as_store_returns_quantitycount#51,as_store_returns_quantityave#52,as_store_returns_quantitystdev#53,store_returns_quantitycov#54,catalog_sales_quantitycount#55,catalog_sales_quantityave#56,catalog_sales_quantitystdev#57,catalog_sales_quantitycov#58]), [i_item_id#27, i_item_desc#28, s_state#25, store_sales_quantitycount#47, store_sales_quantityave#48, store_sales_quantitystdev#49, store_sales_quantitycov#50, as_store_returns_quantitycount#51, as_store_returns_quantityave#52, as_store_returns_quantitystdev#53, store_returns_quantitycov#54, catalog_sales_quantitycount#55, catalog_sales_quantityave#56, catalog_sales_quantitystdev#57, catalog_sales_quantitycov#58], 100, [i_item_id#27 ASC NULLS FIRST, i_item_desc#28 ASC NULLS FIRST, s_state#25 ASC NULLS FIRST], [i_item_id#27, i_item_desc#28, s_state#25, store_sales_quantitycount#47, store_sales_quantityave#48, store_sales_quantitystdev#49, store_sales_quantitycov#50, as_store_returns_quantitycount#51, as_store_returns_quantityave#52, as_store_returns_quantitystdev#53, store_returns_quantitycov#54, catalog_sales_quantitycount#55, catalog_sales_quantityave#56, catalog_sales_quantitystdev#57, catalog_sales_quantitycov#58]

(42) ColumnarToRow [codegen id : 1]
Input [15]: [i_item_id#27, i_item_desc#28, s_state#25, store_sales_quantitycount#47, store_sales_quantityave#48, store_sales_quantitystdev#49, store_sales_quantitycov#50, as_store_returns_quantitycount#51, as_store_returns_quantityave#52, as_store_returns_quantitystdev#53, store_returns_quantitycov#54, catalog_sales_quantitycount#55, catalog_sales_quantityave#56, catalog_sales_quantitystdev#57, catalog_sales_quantitycov#58]

===== Subqueries =====

Subquery:1 Hosting operator id = 1 Hosting Expression = ss_sold_date_sk#6 IN dynamicpruning#7
BroadcastExchange (49)
+- * ColumnarToRow (48)
+- CometProject (47)
+- CometFilter (46)
+- CometScan parquet spark_catalog.default.date_dim (45)
BroadcastExchange (47)
+- * ColumnarToRow (46)
+- CometProject (45)
+- CometFilter (44)
+- CometScan parquet spark_catalog.default.date_dim (43)


(45) Scan parquet spark_catalog.default.date_dim
(43) Scan parquet spark_catalog.default.date_dim
Output [2]: [d_date_sk#19, d_quarter_name#20]
Batched: true
Location [not included in comparison]/{warehouse_dir}/date_dim]
PushedFilters: [IsNotNull(d_quarter_name), EqualTo(d_quarter_name,2001Q1), IsNotNull(d_date_sk)]
ReadSchema: struct<d_date_sk:int,d_quarter_name:string>

(46) CometFilter
(44) CometFilter
Input [2]: [d_date_sk#19, d_quarter_name#20]
Condition : ((isnotnull(d_quarter_name#20) AND (d_quarter_name#20 = 2001Q1)) AND isnotnull(d_date_sk#19))

(47) CometProject
(45) CometProject
Input [2]: [d_date_sk#19, d_quarter_name#20]
Arguments: [d_date_sk#19], [d_date_sk#19]

(48) ColumnarToRow [codegen id : 1]
(46) ColumnarToRow [codegen id : 1]
Input [1]: [d_date_sk#19]

(49) BroadcastExchange
(47) BroadcastExchange
Input [1]: [d_date_sk#19]
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=2]

Subquery:2 Hosting operator id = 3 Hosting Expression = sr_returned_date_sk#12 IN dynamicpruning#13
BroadcastExchange (54)
+- * ColumnarToRow (53)
+- CometProject (52)
+- CometFilter (51)
+- CometScan parquet spark_catalog.default.date_dim (50)
BroadcastExchange (52)
+- * ColumnarToRow (51)
+- CometProject (50)
+- CometFilter (49)
+- CometScan parquet spark_catalog.default.date_dim (48)


(50) Scan parquet spark_catalog.default.date_dim
(48) Scan parquet spark_catalog.default.date_dim
Output [2]: [d_date_sk#21, d_quarter_name#22]
Batched: true
Location [not included in comparison]/{warehouse_dir}/date_dim]
PushedFilters: [In(d_quarter_name, [2001Q1,2001Q2,2001Q3]), IsNotNull(d_date_sk)]
ReadSchema: struct<d_date_sk:int,d_quarter_name:string>

(51) CometFilter
(49) CometFilter
Input [2]: [d_date_sk#21, d_quarter_name#22]
Condition : (d_quarter_name#22 IN (2001Q1,2001Q2,2001Q3) AND isnotnull(d_date_sk#21))

(52) CometProject
(50) CometProject
Input [2]: [d_date_sk#21, d_quarter_name#22]
Arguments: [d_date_sk#21], [d_date_sk#21]

(53) ColumnarToRow [codegen id : 1]
(51) ColumnarToRow [codegen id : 1]
Input [1]: [d_date_sk#21]

(54) BroadcastExchange
(52) BroadcastExchange
Input [1]: [d_date_sk#21]
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=3]

Expand Down
Loading

0 comments on commit 9193d83

Please sign in to comment.