Skip to content

Commit

Permalink
Update plan stability
Browse files Browse the repository at this point in the history
  • Loading branch information
@viirya
viirya committed May 5, 2024
1 parent b37070d commit 557b753
Show file tree
Hide file tree
Showing 268 changed files with 25,029 additions and 28,380 deletions.
192 changes: 84 additions & 108 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q1/explain.txt
View file
Original file line number Diff line number Diff line change
@@ -1,50 +1,44 @@
== Physical Plan ==
TakeOrderedAndProject (46)
+- * Project (45)
+- * BroadcastHashJoin Inner BuildRight (44)
:- * Project (39)
: +- * BroadcastHashJoin Inner BuildRight (38)
: :- * Project (32)
: : +- * BroadcastHashJoin Inner BuildRight (31)
: : :- * Filter (12)
: : : +- * HashAggregate (11)
: : : +- * ColumnarToRow (10)
: : : +- CometColumnarExchange (9)
: : : +- RowToColumnar (8)
: : : +- * HashAggregate (7)
: : : +- * Project (6)
: : : +- * BroadcastHashJoin Inner BuildRight (5)
: : : :- * ColumnarToRow (3)
: : : : +- CometFilter (2)
: : : : +- CometScan parquet spark_catalog.default.store_returns (1)
: : : +- ReusedExchange (4)
: : +- BroadcastExchange (30)
: : +- * Filter (29)
: : +- * HashAggregate (28)
: : +- * ColumnarToRow (27)
: : +- CometColumnarExchange (26)
: : +- RowToColumnar (25)
: : +- * HashAggregate (24)
: : +- * HashAggregate (23)
: : +- * ColumnarToRow (22)
: : +- CometColumnarExchange (21)
: : +- RowToColumnar (20)
: : +- * HashAggregate (19)
: : +- * Project (18)
: : +- * BroadcastHashJoin Inner BuildRight (17)
: : :- * ColumnarToRow (15)
: : : +- CometFilter (14)
: : : +- CometScan parquet spark_catalog.default.store_returns (13)
: : +- ReusedExchange (16)
: +- BroadcastExchange (37)
: +- * ColumnarToRow (36)
: +- CometProject (35)
: +- CometFilter (34)
: +- CometScan parquet spark_catalog.default.store (33)
+- BroadcastExchange (43)
+- * ColumnarToRow (42)
+- CometFilter (41)
+- CometScan parquet spark_catalog.default.customer (40)
TakeOrderedAndProject (40)
+- * Project (39)
+- * BroadcastHashJoin Inner BuildRight (38)
:- * Project (33)
: +- * BroadcastHashJoin Inner BuildRight (32)
: :- * Project (26)
: : +- * BroadcastHashJoin Inner BuildRight (25)
: : :- * Filter (10)
: : : +- * HashAggregate (9)
: : : +- Exchange (8)
: : : +- * HashAggregate (7)
: : : +- * Project (6)
: : : +- * BroadcastHashJoin Inner BuildRight (5)
: : : :- * ColumnarToRow (3)
: : : : +- CometFilter (2)
: : : : +- CometScan parquet spark_catalog.default.store_returns (1)
: : : +- ReusedExchange (4)
: : +- BroadcastExchange (24)
: : +- * Filter (23)
: : +- * HashAggregate (22)
: : +- Exchange (21)
: : +- * HashAggregate (20)
: : +- * HashAggregate (19)
: : +- Exchange (18)
: : +- * HashAggregate (17)
: : +- * Project (16)
: : +- * BroadcastHashJoin Inner BuildRight (15)
: : :- * ColumnarToRow (13)
: : : +- CometFilter (12)
: : : +- CometScan parquet spark_catalog.default.store_returns (11)
: : +- ReusedExchange (14)
: +- BroadcastExchange (31)
: +- * ColumnarToRow (30)
: +- CometProject (29)
: +- CometFilter (28)
: +- CometScan parquet spark_catalog.default.store (27)
+- BroadcastExchange (37)
+- * ColumnarToRow (36)
+- CometFilter (35)
+- CometScan parquet spark_catalog.default.customer (34)


(unknown) Scan parquet spark_catalog.default.store_returns
Expand All @@ -62,7 +56,7 @@ Condition : (isnotnull(sr_store_sk#2) AND isnotnull(sr_customer_sk#1))
(3) ColumnarToRow [codegen id : 2]
Input [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]

(4) ReusedExchange [Reuses operator id: 51]
(4) ReusedExchange [Reuses operator id: 45]
Output [1]: [d_date_sk#6]

(5) BroadcastHashJoin [codegen id : 2]
Expand All @@ -82,24 +76,18 @@ Functions [1]: [partial_sum(UnscaledValue(sr_return_amt#3))]
Aggregate Attributes [1]: [sum#7]
Results [3]: [sr_customer_sk#1, sr_store_sk#2, sum#8]

(8) RowToColumnar
(8) Exchange
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#8]
Arguments: hashpartitioning(sr_customer_sk#1, sr_store_sk#2, 5), ENSURE_REQUIREMENTS, [plan_id=1]

(9) CometColumnarExchange
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#8]
Arguments: hashpartitioning(sr_customer_sk#1, sr_store_sk#2, 5), ENSURE_REQUIREMENTS, CometColumnarShuffle, [plan_id=1]

(10) ColumnarToRow [codegen id : 9]
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#8]

(11) HashAggregate [codegen id : 9]
(9) HashAggregate [codegen id : 9]
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#8]
Keys [2]: [sr_customer_sk#1, sr_store_sk#2]
Functions [1]: [sum(UnscaledValue(sr_return_amt#3))]
Aggregate Attributes [1]: [sum(UnscaledValue(sr_return_amt#3))#9]
Results [3]: [sr_customer_sk#1 AS ctr_customer_sk#10, sr_store_sk#2 AS ctr_store_sk#11, MakeDecimal(sum(UnscaledValue(sr_return_amt#3))#9,17,2) AS ctr_total_return#12]

(12) Filter [codegen id : 9]
(10) Filter [codegen id : 9]
Input [3]: [ctr_customer_sk#10, ctr_store_sk#11, ctr_total_return#12]
Condition : isnotnull(ctr_total_return#12)

Expand All @@ -111,89 +99,77 @@ PartitionFilters: [isnotnull(sr_returned_date_sk#4), dynamicpruningexpression(sr
PushedFilters: [IsNotNull(sr_store_sk)]
ReadSchema: struct<sr_customer_sk:int,sr_store_sk:int,sr_return_amt:decimal(7,2)>

(14) CometFilter
(12) CometFilter
Input [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]
Condition : isnotnull(sr_store_sk#2)

(15) ColumnarToRow [codegen id : 4]
(13) ColumnarToRow [codegen id : 4]
Input [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]

(16) ReusedExchange [Reuses operator id: 51]
(14) ReusedExchange [Reuses operator id: 45]
Output [1]: [d_date_sk#6]

(17) BroadcastHashJoin [codegen id : 4]
(15) BroadcastHashJoin [codegen id : 4]
Left keys [1]: [sr_returned_date_sk#4]
Right keys [1]: [d_date_sk#6]
Join type: Inner
Join condition: None

(18) Project [codegen id : 4]
(16) Project [codegen id : 4]
Output [3]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3]
Input [5]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4, d_date_sk#6]

(19) HashAggregate [codegen id : 4]
(17) HashAggregate [codegen id : 4]
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3]
Keys [2]: [sr_customer_sk#1, sr_store_sk#2]
Functions [1]: [partial_sum(UnscaledValue(sr_return_amt#3))]
Aggregate Attributes [1]: [sum#14]
Results [3]: [sr_customer_sk#1, sr_store_sk#2, sum#15]

(20) RowToColumnar
(18) Exchange
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#15]
Arguments: hashpartitioning(sr_customer_sk#1, sr_store_sk#2, 5), ENSURE_REQUIREMENTS, [plan_id=2]

(21) CometColumnarExchange
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#15]
Arguments: hashpartitioning(sr_customer_sk#1, sr_store_sk#2, 5), ENSURE_REQUIREMENTS, CometColumnarShuffle, [plan_id=2]

(22) ColumnarToRow [codegen id : 5]
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#15]

(23) HashAggregate [codegen id : 5]
(19) HashAggregate [codegen id : 5]
Input [3]: [sr_customer_sk#1, sr_store_sk#2, sum#15]
Keys [2]: [sr_customer_sk#1, sr_store_sk#2]
Functions [1]: [sum(UnscaledValue(sr_return_amt#3))]
Aggregate Attributes [1]: [sum(UnscaledValue(sr_return_amt#3))#9]
Results [2]: [sr_store_sk#2 AS ctr_store_sk#11, MakeDecimal(sum(UnscaledValue(sr_return_amt#3))#9,17,2) AS ctr_total_return#12]

(24) HashAggregate [codegen id : 5]
(20) HashAggregate [codegen id : 5]
Input [2]: [ctr_store_sk#11, ctr_total_return#12]
Keys [1]: [ctr_store_sk#11]
Functions [1]: [partial_avg(ctr_total_return#12)]
Aggregate Attributes [2]: [sum#16, count#17]
Results [3]: [ctr_store_sk#11, sum#18, count#19]

(25) RowToColumnar
Input [3]: [ctr_store_sk#11, sum#18, count#19]

(26) CometColumnarExchange
Input [3]: [ctr_store_sk#11, sum#18, count#19]
Arguments: hashpartitioning(ctr_store_sk#11, 5), ENSURE_REQUIREMENTS, CometColumnarShuffle, [plan_id=3]

(27) ColumnarToRow [codegen id : 6]
(21) Exchange
Input [3]: [ctr_store_sk#11, sum#18, count#19]
Arguments: hashpartitioning(ctr_store_sk#11, 5), ENSURE_REQUIREMENTS, [plan_id=3]

(28) HashAggregate [codegen id : 6]
(22) HashAggregate [codegen id : 6]
Input [3]: [ctr_store_sk#11, sum#18, count#19]
Keys [1]: [ctr_store_sk#11]
Functions [1]: [avg(ctr_total_return#12)]
Aggregate Attributes [1]: [avg(ctr_total_return#12)#20]
Results [2]: [(avg(ctr_total_return#12)#20 * 1.2) AS (avg(ctr_total_return) * 1.2)#21, ctr_store_sk#11 AS ctr_store_sk#11#22]

(29) Filter [codegen id : 6]
(23) Filter [codegen id : 6]
Input [2]: [(avg(ctr_total_return) * 1.2)#21, ctr_store_sk#11#22]
Condition : isnotnull((avg(ctr_total_return) * 1.2)#21)

(30) BroadcastExchange
(24) BroadcastExchange
Input [2]: [(avg(ctr_total_return) * 1.2)#21, ctr_store_sk#11#22]
Arguments: HashedRelationBroadcastMode(List(cast(input[1, int, true] as bigint)),false), [plan_id=4]

(31) BroadcastHashJoin [codegen id : 9]
(25) BroadcastHashJoin [codegen id : 9]
Left keys [1]: [ctr_store_sk#11]
Right keys [1]: [ctr_store_sk#11#22]
Join type: Inner
Join condition: (cast(ctr_total_return#12 as decimal(24,7)) > (avg(ctr_total_return) * 1.2)#21)

(32) Project [codegen id : 9]
(26) Project [codegen id : 9]
Output [2]: [ctr_customer_sk#10, ctr_store_sk#11]
Input [5]: [ctr_customer_sk#10, ctr_store_sk#11, ctr_total_return#12, (avg(ctr_total_return) * 1.2)#21, ctr_store_sk#11#22]

Expand All @@ -204,28 +180,28 @@ Location [not included in comparison]/{warehouse_dir}/store]
PushedFilters: [IsNotNull(s_state), EqualTo(s_state,TN), IsNotNull(s_store_sk)]
ReadSchema: struct<s_store_sk:int,s_state:string>

(34) CometFilter
(28) CometFilter
Input [2]: [s_store_sk#23, s_state#24]
Condition : ((isnotnull(s_state#24) AND (s_state#24 = TN)) AND isnotnull(s_store_sk#23))

(35) CometProject
(29) CometProject
Input [2]: [s_store_sk#23, s_state#24]
Arguments: [s_store_sk#23], [s_store_sk#23]

(36) ColumnarToRow [codegen id : 7]
(30) ColumnarToRow [codegen id : 7]
Input [1]: [s_store_sk#23]

(37) BroadcastExchange
(31) BroadcastExchange
Input [1]: [s_store_sk#23]
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=5]

(38) BroadcastHashJoin [codegen id : 9]
(32) BroadcastHashJoin [codegen id : 9]
Left keys [1]: [ctr_store_sk#11]
Right keys [1]: [s_store_sk#23]
Join type: Inner
Join condition: None

(39) Project [codegen id : 9]
(33) Project [codegen id : 9]
Output [1]: [ctr_customer_sk#10]
Input [3]: [ctr_customer_sk#10, ctr_store_sk#11, s_store_sk#23]

Expand All @@ -236,39 +212,39 @@ Location [not included in comparison]/{warehouse_dir}/customer]
PushedFilters: [IsNotNull(c_customer_sk)]
ReadSchema: struct<c_customer_sk:int,c_customer_id:string>

(41) CometFilter
(35) CometFilter
Input [2]: [c_customer_sk#25, c_customer_id#26]
Condition : isnotnull(c_customer_sk#25)

(42) ColumnarToRow [codegen id : 8]
(36) ColumnarToRow [codegen id : 8]
Input [2]: [c_customer_sk#25, c_customer_id#26]

(43) BroadcastExchange
(37) BroadcastExchange
Input [2]: [c_customer_sk#25, c_customer_id#26]
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=6]

(44) BroadcastHashJoin [codegen id : 9]
(38) BroadcastHashJoin [codegen id : 9]
Left keys [1]: [ctr_customer_sk#10]
Right keys [1]: [c_customer_sk#25]
Join type: Inner
Join condition: None

(45) Project [codegen id : 9]
(39) Project [codegen id : 9]
Output [1]: [c_customer_id#26]
Input [3]: [ctr_customer_sk#10, c_customer_sk#25, c_customer_id#26]

(46) TakeOrderedAndProject
(40) TakeOrderedAndProject
Input [1]: [c_customer_id#26]
Arguments: 100, [c_customer_id#26 ASC NULLS FIRST], [c_customer_id#26]

===== Subqueries =====

Subquery:1 Hosting operator id = 1 Hosting Expression = sr_returned_date_sk#4 IN dynamicpruning#5
BroadcastExchange (51)
+- * ColumnarToRow (50)
+- CometProject (49)
+- CometFilter (48)
+- CometScan parquet spark_catalog.default.date_dim (47)
BroadcastExchange (45)
+- * ColumnarToRow (44)
+- CometProject (43)
+- CometFilter (42)
+- CometScan parquet spark_catalog.default.date_dim (41)


(unknown) Scan parquet spark_catalog.default.date_dim
Expand All @@ -278,21 +254,21 @@ Location [not included in comparison]/{warehouse_dir}/date_dim]
PushedFilters: [IsNotNull(d_year), EqualTo(d_year,2000), IsNotNull(d_date_sk)]
ReadSchema: struct<d_date_sk:int,d_year:int>

(48) CometFilter
(42) CometFilter
Input [2]: [d_date_sk#6, d_year#27]
Condition : ((isnotnull(d_year#27) AND (d_year#27 = 2000)) AND isnotnull(d_date_sk#6))

(49) CometProject
(43) CometProject
Input [2]: [d_date_sk#6, d_year#27]
Arguments: [d_date_sk#6], [d_date_sk#6]

(50) ColumnarToRow [codegen id : 1]
(44) ColumnarToRow [codegen id : 1]
Input [1]: [d_date_sk#6]

(51) BroadcastExchange
(45) BroadcastExchange
Input [1]: [d_date_sk#6]
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=7]

Subquery:2 Hosting operator id = 13 Hosting Expression = sr_returned_date_sk#4 IN dynamicpruning#5
Subquery:2 Hosting operator id = 11 Hosting Expression = sr_returned_date_sk#4 IN dynamicpruning#5


Loading

0 comments on commit 557b753

Please sign in to comment.