feat: partition by left most cluster key when building merge into filter #13547

SkyFan2002 · 2023-11-02T07:50:07Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

For example:

create table target (insert_time datetime,id,...) CLUSTER BY(to_yyyymmdd(insert_time), id);

merge into target as t1 using (select * from source) as t2 on t1.insert_time = t2.insert_time and t1.id = t2.id and... when not 

matched then insert *;

Then left most cluster key of target table, to_yyyymmdd(insert_time), will be used as partition key of source table when

collecting statistics, by:

to_yyyymmdd(target.insert_time) is projected to to_yyyymmdd(source.insert_time), which is inferred from join

condition t1.insert_time = t2.insert_time.
Rewrite logical plan of source table to query the statistics,like this:

     select min(id),max(id)... from source group by to_yyyymmdd(insert_time);

Construct Filter for target table, min,max from different partitions will be concatenated by OR, like this:

    [min_group1,max_group1] U [min_group2,max_group2] U ...

Push down the Filter to target table.

close #13567

This change is

vercel · 2023-11-02T07:50:12Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
databend	⬜️ Ignored (Inspect)	Visit Preview		Nov 3, 2023 1:42pm

# Conflicts: # src/query/service/src/interpreters/interpreter_merge_into.rs

SkyFan2002 · 2023-11-02T13:51:11Z

Tested in standalone mode for 3000 times:

[2023-11-02T13:41:59Z INFO  test_replace_recluster] ==========================
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ====verify table state====
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ==========================
[2023-11-02T13:41:59Z INFO  test_replace_recluster]                            
[2023-11-02T13:41:59Z INFO  test_replace_recluster]                            
[2023-11-02T13:41:59Z INFO  test_replace_recluster] number of successfully executed merge-into statements : 3000
[2023-11-02T13:41:59Z INFO  test_replace_recluster]                            
[2023-11-02T13:41:59Z INFO  test_replace_recluster]                            
[2023-11-02T13:41:59Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements
[2023-11-02T13:41:59Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements: client 3000000, server 3000000
[2023-11-02T13:41:59Z INFO  test_replace_recluster] CHECK: distinct ids: client 3000, server 3000
[2023-11-02T13:41:59Z INFO  test_replace_recluster] CHECK: value of correlated column
[2023-11-02T13:41:59Z INFO  test_replace_recluster] CHECK: full table scanning
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ===========================
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ======     PASSED      ====
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ===========================
[2023-11-02T13:41:59Z INFO  test_replace_recluster]                            
[2023-11-02T13:41:59Z INFO  test_replace_recluster]                            
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ========METRICS============
[2023-11-02T13:41:59Z INFO  test_replace_recluster] fuse_commit_mutation_unresolvable_conflict_total : 1930.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 49366253064.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 62336.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":8392.0},{"less_than":50.0,"count":8392.0},{"less_than":100.0,"count":8392.0},{"less_than":250.0,"count":8392.0},{"less_than":500.0,"count":8392.0},{"less_than":1000.0,"count":8392.0},{"less_than":2500.0,"count":8392.0},{"less_than":5000.0,"count":8392.0},{"less_than":10000.0,"count":8392.0},{"less_than":20000.0,"count":8392.0},{"less_than":30000.0,"count":8392.0},{"less_than":60000.0,"count":8392.0},{"less_than":300000.0,"count":8392.0},{"less_than":600000.0,"count":8392.0},{"less_than":1800000.0,"count":8392.0},{"less_than":null,"count":8392.0}]
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 8392.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 11392.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 4284000.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3003.0},{"less_than":100.0,"count":3006.0},{"less_than":250.0,"count":3015.0},{"less_than":500.0,"count":3035.0},{"less_than":1000.0,"count":3052.0},{"less_than":2500.0,"count":3428.0},{"less_than":5000.0,"count":3428.0},{"less_than":10000.0,"count":3428.0},{"less_than":20000.0,"count":3428.0},{"less_than":30000.0,"count":3428.0},{"less_than":60000.0,"count":3428.0},{"less_than":300000.0,"count":3428.0},{"less_than":600000.0,"count":3428.0},{"less_than":1800000.0,"count":3428.0},{"less_than":null,"count":3428.0}]
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 3428.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 519160.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 79.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 77002.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":8392.0},{"less_than":50.0,"count":8392.0},{"less_than":100.0,"count":8392.0},{"less_than":250.0,"count":8392.0},{"less_than":500.0,"count":8392.0},{"less_than":1000.0,"count":8392.0},{"less_than":2500.0,"count":8392.0},{"less_than":5000.0,"count":8392.0},{"less_than":10000.0,"count":8392.0},{"less_than":20000.0,"count":8392.0},{"less_than":30000.0,"count":8392.0},{"less_than":60000.0,"count":8392.0},{"less_than":300000.0,"count":8392.0},{"less_than":600000.0,"count":8392.0},{"less_than":1800000.0,"count":8392.0},{"less_than":null,"count":8392.0}]
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 8392.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 1427.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 1284000.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3000.0},{"less_than":100.0,"count":3000.0},{"less_than":250.0,"count":3000.0},{"less_than":500.0,"count":3000.0},{"less_than":1000.0,"count":3000.0},{"less_than":2500.0,"count":3000.0},{"less_than":5000.0,"count":3000.0},{"less_than":10000.0,"count":3000.0},{"less_than":20000.0,"count":3000.0},{"less_than":30000.0,"count":3000.0},{"less_than":60000.0,"count":3000.0},{"less_than":300000.0,"count":3000.0},{"less_than":600000.0,"count":3000.0},{"less_than":1800000.0,"count":3000.0},{"less_than":null,"count":3000.0}]
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_count : 3000.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_sum : 0.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 1747.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 473177895.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":11392.0},{"less_than":50.0,"count":11392.0},{"less_than":100.0,"count":11392.0},{"less_than":250.0,"count":11392.0},{"less_than":500.0,"count":11392.0},{"less_than":1000.0,"count":11392.0},{"less_than":2500.0,"count":11392.0},{"less_than":5000.0,"count":11392.0},{"less_than":10000.0,"count":11392.0},{"less_than":20000.0,"count":11392.0},{"less_than":30000.0,"count":11392.0},{"less_than":60000.0,"count":11392.0},{"less_than":300000.0,"count":11392.0},{"less_than":600000.0,"count":11392.0},{"less_than":1800000.0,"count":11392.0},{"less_than":null,"count":11392.0}]
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 11392.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 1.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 3000000.0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ===========================
[2023-11-02T13:41:59Z INFO  test_replace_recluster]                            
[2023-11-02T13:41:59Z INFO  test_replace_recluster]                            
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ======CLUSTERING INFO======
[2023-11-02T13:41:59Z INFO  test_replace_recluster] cluster_key : (to_yyyymmdd(insert_time), id)
[2023-11-02T13:41:59Z INFO  test_replace_recluster] block_count: 14
[2023-11-02T13:41:59Z INFO  test_replace_recluster] constant_block_count: 0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] unclustered_block_count: 0
[2023-11-02T13:41:59Z INFO  test_replace_recluster] average_overlaps: 8.4286
[2023-11-02T13:41:59Z INFO  test_replace_recluster] average_depth: 6.7143
[2023-11-02T13:41:59Z INFO  test_replace_recluster] block_depth_histogram: {"00003":1,"00007":13}
[2023-11-02T13:41:59Z INFO  test_replace_recluster] ===========================

SkyFan2002 · 2023-11-03T15:12:03Z

Tested in cluster mode for 3000 times using https://github.com/JackTan25/test-scripts/tree/distributed_test

[2023-11-03T15:07:29Z INFO  test_replace_recluster] ==========================
[2023-11-03T15:07:29Z INFO  test_replace_recluster] ====verify table state====
[2023-11-03T15:07:29Z INFO  test_replace_recluster] ==========================
[2023-11-03T15:07:29Z INFO  test_replace_recluster]                            
[2023-11-03T15:07:29Z INFO  test_replace_recluster]                            
[2023-11-03T15:07:29Z INFO  test_replace_recluster] number of successfully executed merge-into statements : 3000
[2023-11-03T15:07:29Z INFO  test_replace_recluster]                            
[2023-11-03T15:07:29Z INFO  test_replace_recluster]                            
[2023-11-03T15:07:29Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements
[2023-11-03T15:07:29Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements: client 3000000, server 3000000
[2023-11-03T15:07:29Z INFO  test_replace_recluster] CHECK: distinct ids: client 3000, server 3000
[2023-11-03T15:07:29Z INFO  test_replace_recluster] CHECK: value of correlated column
[2023-11-03T15:07:29Z INFO  test_replace_recluster] CHECK: full table scanning
[2023-11-03T15:07:30Z INFO  test_replace_recluster] ===========================
[2023-11-03T15:07:30Z INFO  test_replace_recluster] ======     PASSED      ====
[2023-11-03T15:07:30Z INFO  test_replace_recluster] ===========================
[2023-11-03T15:07:30Z INFO  test_replace_recluster]                            
[2023-11-03T15:07:30Z INFO  test_replace_recluster]                            
[2023-11-03T15:07:30Z INFO  test_replace_recluster] ========METRICS============
[2023-11-03T15:07:30Z INFO  test_replace_recluster] fuse_commit_mutation_unresolvable_conflict_total : 427.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 29682519802.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 4718640462.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 14477894136.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 28435.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 4419.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 13330.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":4752.0},{"less_than":50.0,"count":4752.0},{"less_than":100.0,"count":4752.0},{"less_than":250.0,"count":4752.0},{"less_than":500.0,"count":4752.0},{"less_than":1000.0,"count":4752.0},{"less_than":2500.0,"count":4752.0},{"less_than":5000.0,"count":4752.0},{"less_than":10000.0,"count":4752.0},{"less_than":20000.0,"count":4752.0},{"less_than":30000.0,"count":4752.0},{"less_than":60000.0,"count":4752.0},{"less_than":300000.0,"count":4752.0},{"less_than":600000.0,"count":4752.0},{"less_than":1800000.0,"count":4752.0},{"less_than":null,"count":4752.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":780.0},{"less_than":50.0,"count":780.0},{"less_than":100.0,"count":780.0},{"less_than":250.0,"count":780.0},{"less_than":500.0,"count":780.0},{"less_than":1000.0,"count":780.0},{"less_than":2500.0,"count":780.0},{"less_than":5000.0,"count":780.0},{"less_than":10000.0,"count":780.0},{"less_than":20000.0,"count":780.0},{"less_than":30000.0,"count":780.0},{"less_than":60000.0,"count":780.0},{"less_than":300000.0,"count":780.0},{"less_than":600000.0,"count":780.0},{"less_than":1800000.0,"count":780.0},{"less_than":null,"count":780.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":2716.0},{"less_than":50.0,"count":2716.0},{"less_than":100.0,"count":2716.0},{"less_than":250.0,"count":2716.0},{"less_than":500.0,"count":2716.0},{"less_than":1000.0,"count":2716.0},{"less_than":2500.0,"count":2716.0},{"less_than":5000.0,"count":2716.0},{"less_than":10000.0,"count":2716.0},{"less_than":20000.0,"count":2716.0},{"less_than":30000.0,"count":2716.0},{"less_than":60000.0,"count":2716.0},{"less_than":300000.0,"count":2716.0},{"less_than":600000.0,"count":2716.0},{"less_than":1800000.0,"count":2716.0},{"less_than":null,"count":2716.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 4752.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 780.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 2716.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 7.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 4752.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 780.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 5716.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 377329.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 155708.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 3750963.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":3157.0},{"less_than":50.0,"count":3158.0},{"less_than":100.0,"count":3158.0},{"less_than":250.0,"count":3158.0},{"less_than":500.0,"count":3160.0},{"less_than":1000.0,"count":3162.0},{"less_than":2500.0,"count":3427.0},{"less_than":5000.0,"count":3427.0},{"less_than":10000.0,"count":3427.0},{"less_than":20000.0,"count":3427.0},{"less_than":30000.0,"count":3427.0},{"less_than":60000.0,"count":3427.0},{"less_than":300000.0,"count":3427.0},{"less_than":600000.0,"count":3427.0},{"less_than":1800000.0,"count":3427.0},{"less_than":null,"count":3427.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":3288.0},{"less_than":50.0,"count":3290.0},{"less_than":100.0,"count":3292.0},{"less_than":250.0,"count":3295.0},{"less_than":500.0,"count":3304.0},{"less_than":1000.0,"count":3323.0},{"less_than":2500.0,"count":3428.0},{"less_than":5000.0,"count":3428.0},{"less_than":10000.0,"count":3428.0},{"less_than":20000.0,"count":3428.0},{"less_than":30000.0,"count":3428.0},{"less_than":60000.0,"count":3428.0},{"less_than":300000.0,"count":3428.0},{"less_than":600000.0,"count":3428.0},{"less_than":1800000.0,"count":3428.0},{"less_than":null,"count":3428.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":3071.0},{"less_than":50.0,"count":3071.0},{"less_than":100.0,"count":3072.0},{"less_than":250.0,"count":3074.0},{"less_than":500.0,"count":3074.0},{"less_than":1000.0,"count":3074.0},{"less_than":2500.0,"count":3427.0},{"less_than":5000.0,"count":3427.0},{"less_than":10000.0,"count":3427.0},{"less_than":20000.0,"count":3427.0},{"less_than":30000.0,"count":3427.0},{"less_than":60000.0,"count":3427.0},{"less_than":300000.0,"count":3427.0},{"less_than":600000.0,"count":3427.0},{"less_than":1800000.0,"count":3427.0},{"less_than":null,"count":3427.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 3427.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 3428.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 3427.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 402496.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 181128.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 557974.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 427.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 427000.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":4752.0},{"less_than":50.0,"count":4752.0},{"less_than":100.0,"count":4752.0},{"less_than":250.0,"count":4752.0},{"less_than":500.0,"count":4752.0},{"less_than":1000.0,"count":4752.0},{"less_than":2500.0,"count":4752.0},{"less_than":5000.0,"count":4752.0},{"less_than":10000.0,"count":4752.0},{"less_than":20000.0,"count":4752.0},{"less_than":30000.0,"count":4752.0},{"less_than":60000.0,"count":4752.0},{"less_than":300000.0,"count":4752.0},{"less_than":600000.0,"count":4752.0},{"less_than":1800000.0,"count":4752.0},{"less_than":null,"count":4752.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":780.0},{"less_than":50.0,"count":780.0},{"less_than":100.0,"count":780.0},{"less_than":250.0,"count":780.0},{"less_than":500.0,"count":780.0},{"less_than":1000.0,"count":780.0},{"less_than":2500.0,"count":780.0},{"less_than":5000.0,"count":780.0},{"less_than":10000.0,"count":780.0},{"less_than":20000.0,"count":780.0},{"less_than":30000.0,"count":780.0},{"less_than":60000.0,"count":780.0},{"less_than":300000.0,"count":780.0},{"less_than":600000.0,"count":780.0},{"less_than":1800000.0,"count":780.0},{"less_than":null,"count":780.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":2716.0},{"less_than":50.0,"count":2716.0},{"less_than":100.0,"count":2716.0},{"less_than":250.0,"count":2716.0},{"less_than":500.0,"count":2716.0},{"less_than":1000.0,"count":2716.0},{"less_than":2500.0,"count":2716.0},{"less_than":5000.0,"count":2716.0},{"less_than":10000.0,"count":2716.0},{"less_than":20000.0,"count":2716.0},{"less_than":30000.0,"count":2716.0},{"less_than":60000.0,"count":2716.0},{"less_than":300000.0,"count":2716.0},{"less_than":600000.0,"count":2716.0},{"less_than":1800000.0,"count":2716.0},{"less_than":null,"count":2716.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 4752.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 780.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 2716.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 67.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 210.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 1436.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 377329.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 155708.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 750963.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3000.0},{"less_than":100.0,"count":3000.0},{"less_than":250.0,"count":3000.0},{"less_than":500.0,"count":3000.0},{"less_than":1000.0,"count":3000.0},{"less_than":2500.0,"count":3000.0},{"less_than":5000.0,"count":3000.0},{"less_than":10000.0,"count":3000.0},{"less_than":20000.0,"count":3000.0},{"less_than":30000.0,"count":3000.0},{"less_than":60000.0,"count":3000.0},{"less_than":300000.0,"count":3000.0},{"less_than":600000.0,"count":3000.0},{"less_than":1800000.0,"count":3000.0},{"less_than":null,"count":3000.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_count : 3000.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_sum : 0.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 793.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 145.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 356.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 292821824.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 46637630.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 142232546.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":8180.0},{"less_than":50.0,"count":8180.0},{"less_than":100.0,"count":8180.0},{"less_than":250.0,"count":8180.0},{"less_than":500.0,"count":8180.0},{"less_than":1000.0,"count":8180.0},{"less_than":2500.0,"count":8180.0},{"less_than":5000.0,"count":8180.0},{"less_than":10000.0,"count":8180.0},{"less_than":20000.0,"count":8180.0},{"less_than":30000.0,"count":8180.0},{"less_than":60000.0,"count":8180.0},{"less_than":300000.0,"count":8180.0},{"less_than":600000.0,"count":8180.0},{"less_than":1800000.0,"count":8180.0},{"less_than":null,"count":8180.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":4208.0},{"less_than":50.0,"count":4208.0},{"less_than":100.0,"count":4208.0},{"less_than":250.0,"count":4208.0},{"less_than":500.0,"count":4208.0},{"less_than":1000.0,"count":4208.0},{"less_than":2500.0,"count":4208.0},{"less_than":5000.0,"count":4208.0},{"less_than":10000.0,"count":4208.0},{"less_than":20000.0,"count":4208.0},{"less_than":30000.0,"count":4208.0},{"less_than":60000.0,"count":4208.0},{"less_than":300000.0,"count":4208.0},{"less_than":600000.0,"count":4208.0},{"less_than":1800000.0,"count":4208.0},{"less_than":null,"count":4208.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":6052.0},{"less_than":50.0,"count":6052.0},{"less_than":100.0,"count":6052.0},{"less_than":250.0,"count":6052.0},{"less_than":500.0,"count":6052.0},{"less_than":1000.0,"count":6052.0},{"less_than":2500.0,"count":6052.0},{"less_than":5000.0,"count":6052.0},{"less_than":10000.0,"count":6052.0},{"less_than":20000.0,"count":6052.0},{"less_than":30000.0,"count":6052.0},{"less_than":60000.0,"count":6052.0},{"less_than":300000.0,"count":6052.0},{"less_than":600000.0,"count":6052.0},{"less_than":1800000.0,"count":6052.0},{"less_than":null,"count":6052.0}]
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 8180.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 4208.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 6052.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 7.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 3533037.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 3906671.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 4128292.0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] ===========================
[2023-11-03T15:07:30Z INFO  test_replace_recluster]                            
[2023-11-03T15:07:30Z INFO  test_replace_recluster]                            
[2023-11-03T15:07:30Z INFO  test_replace_recluster] ======CLUSTERING INFO======
[2023-11-03T15:07:30Z INFO  test_replace_recluster] cluster_key : (to_yyyymmdd(insert_time), id)
[2023-11-03T15:07:30Z INFO  test_replace_recluster] block_count: 10
[2023-11-03T15:07:30Z INFO  test_replace_recluster] constant_block_count: 0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] unclustered_block_count: 0
[2023-11-03T15:07:30Z INFO  test_replace_recluster] average_overlaps: 6
[2023-11-03T15:07:30Z INFO  test_replace_recluster] average_depth: 5
[2023-11-03T15:07:30Z INFO  test_replace_recluster] block_depth_histogram: {"00005":10}
[2023-11-03T15:07:30Z INFO  test_replace_recluster] ===========================

BohuTANG · 2023-11-03T15:23:41Z

Ready for review?

JackTan25 · 2023-11-03T15:36:50Z

The test is not over, but this pr is merged , we also need to test delete, and the most important cloud performance test. cc @BohuTANG merge too fast.

dantengsky · 2023-11-03T15:47:47Z

... merge too fast.

@JackTan25 my fault, I put it into the merge queue...

Let's wait for the result of stress test with deletion scenarios, if anything goes wrong, revert it

BohuTANG · 2023-11-03T15:56:31Z

Can we merge the basic long run(@dantengsky 's) with merge into long run(JackTan25/distributed_test)? Run all of them each time.

dantengsky · 2023-11-03T16:03:07Z

Can we merge the basic long run(@dantengsky 's) with merge into long run(JackTan25/distributed_test)? Run all of them each time.

The test result that @SkyFan2002 given in #13547 (comment) is based on the "merged" version (long run script, adapts to "merge-into" scenario).

Later, @JackTan25 improved the test script to cover the "deletion" operation of the merge-into statement.

Currently, @SkyFan2002 is stress testing this PR again, using the test script of improved version.

SkyFan2002 · 2023-11-03T16:18:31Z

Tested in cluster mode for 3000 times using: https://github.com/JackTan25/test-scripts/tree/test_delete

[2023-11-03T16:16:32Z INFO  test_replace_recluster] ==========================
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ====verify table state====
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ==========================
[2023-11-03T16:16:32Z INFO  test_replace_recluster]                            
[2023-11-03T16:16:32Z INFO  test_replace_recluster]                            
[2023-11-03T16:16:32Z INFO  test_replace_recluster] number of successfully executed merge-into statements : 3000
[2023-11-03T16:16:32Z INFO  test_replace_recluster]                            
[2023-11-03T16:16:32Z INFO  test_replace_recluster]                            
[2023-11-03T16:16:32Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements
[2023-11-03T16:16:32Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements: client 3000000, server 0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] CHECK: full table scanning
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ===========================
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ======     PASSED      ====
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ===========================
[2023-11-03T16:16:32Z INFO  test_replace_recluster]                            
[2023-11-03T16:16:32Z INFO  test_replace_recluster]                            
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ========METRICS============
[2023-11-03T16:16:32Z INFO  test_replace_recluster] fuse_commit_mutation_unresolvable_conflict_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 300200636.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 3000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":0.0},{"less_than":50.0,"count":0.0},{"less_than":100.0,"count":0.0},{"less_than":250.0,"count":0.0},{"less_than":500.0,"count":0.0},{"less_than":1000.0,"count":0.0},{"less_than":2500.0,"count":0.0},{"less_than":5000.0,"count":0.0},{"less_than":10000.0,"count":0.0},{"less_than":20000.0,"count":0.0},{"less_than":30000.0,"count":0.0},{"less_than":60000.0,"count":0.0},{"less_than":300000.0,"count":0.0},{"less_than":600000.0,"count":0.0},{"less_than":1800000.0,"count":0.0},{"less_than":null,"count":0.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3000.0},{"less_than":100.0,"count":3000.0},{"less_than":250.0,"count":3000.0},{"less_than":500.0,"count":3000.0},{"less_than":1000.0,"count":3000.0},{"less_than":2500.0,"count":3000.0},{"less_than":5000.0,"count":3000.0},{"less_than":10000.0,"count":3000.0},{"less_than":20000.0,"count":3000.0},{"less_than":30000.0,"count":3000.0},{"less_than":60000.0,"count":3000.0},{"less_than":300000.0,"count":3000.0},{"less_than":600000.0,"count":3000.0},{"less_than":1800000.0,"count":3000.0},{"less_than":null,"count":3000.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":0.0},{"less_than":50.0,"count":0.0},{"less_than":100.0,"count":0.0},{"less_than":250.0,"count":0.0},{"less_than":500.0,"count":0.0},{"less_than":1000.0,"count":0.0},{"less_than":2500.0,"count":0.0},{"less_than":5000.0,"count":0.0},{"less_than":10000.0,"count":0.0},{"less_than":20000.0,"count":0.0},{"less_than":30000.0,"count":0.0},{"less_than":60000.0,"count":0.0},{"less_than":300000.0,"count":0.0},{"less_than":600000.0,"count":0.0},{"less_than":1800000.0,"count":0.0},{"less_than":null,"count":0.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 3000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 3000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 3000000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 6000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 6000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 6000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 10416.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 3000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 3000000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":0.0},{"less_than":50.0,"count":0.0},{"less_than":100.0,"count":0.0},{"less_than":250.0,"count":0.0},{"less_than":500.0,"count":0.0},{"less_than":1000.0,"count":0.0},{"less_than":2500.0,"count":0.0},{"less_than":5000.0,"count":0.0},{"less_than":10000.0,"count":0.0},{"less_than":20000.0,"count":0.0},{"less_than":30000.0,"count":0.0},{"less_than":60000.0,"count":0.0},{"less_than":300000.0,"count":0.0},{"less_than":600000.0,"count":0.0},{"less_than":1800000.0,"count":0.0},{"less_than":null,"count":0.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":0.0},{"less_than":50.0,"count":0.0},{"less_than":100.0,"count":0.0},{"less_than":250.0,"count":0.0},{"less_than":500.0,"count":0.0},{"less_than":1000.0,"count":0.0},{"less_than":2500.0,"count":0.0},{"less_than":5000.0,"count":0.0},{"less_than":10000.0,"count":0.0},{"less_than":20000.0,"count":0.0},{"less_than":30000.0,"count":0.0},{"less_than":60000.0,"count":0.0},{"less_than":300000.0,"count":0.0},{"less_than":600000.0,"count":0.0},{"less_than":1800000.0,"count":0.0},{"less_than":null,"count":0.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":0.0},{"less_than":50.0,"count":0.0},{"less_than":100.0,"count":0.0},{"less_than":250.0,"count":0.0},{"less_than":500.0,"count":0.0},{"less_than":1000.0,"count":0.0},{"less_than":2500.0,"count":0.0},{"less_than":5000.0,"count":0.0},{"less_than":10000.0,"count":0.0},{"less_than":20000.0,"count":0.0},{"less_than":30000.0,"count":0.0},{"less_than":60000.0,"count":0.0},{"less_than":300000.0,"count":0.0},{"less_than":600000.0,"count":0.0},{"less_than":1800000.0,"count":0.0},{"less_than":null,"count":0.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 3000000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3000.0},{"less_than":100.0,"count":3000.0},{"less_than":250.0,"count":3000.0},{"less_than":500.0,"count":3000.0},{"less_than":1000.0,"count":3000.0},{"less_than":2500.0,"count":3000.0},{"less_than":5000.0,"count":3000.0},{"less_than":10000.0,"count":3000.0},{"less_than":20000.0,"count":3000.0},{"less_than":30000.0,"count":3000.0},{"less_than":60000.0,"count":3000.0},{"less_than":300000.0,"count":3000.0},{"less_than":600000.0,"count":3000.0},{"less_than":1800000.0,"count":3000.0},{"less_than":null,"count":3000.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_count : 3000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 6000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 6000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 6000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 6000000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 6000000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 3000000.0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ===========================
[2023-11-03T16:16:32Z INFO  test_replace_recluster]                            
[2023-11-03T16:16:32Z INFO  test_replace_recluster]                            
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ======CLUSTERING INFO======
[2023-11-03T16:16:32Z INFO  test_replace_recluster] cluster_key : (to_yyyymmdd(insert_time), id)
[2023-11-03T16:16:32Z INFO  test_replace_recluster] block_count: 0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] constant_block_count: 0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] unclustered_block_count: 0
[2023-11-03T16:16:32Z INFO  test_replace_recluster] average_overlaps: NaN
[2023-11-03T16:16:32Z INFO  test_replace_recluster] average_depth: NaN
[2023-11-03T16:16:32Z INFO  test_replace_recluster] block_depth_histogram: {}
[2023-11-03T16:16:32Z INFO  test_replace_recluster] ===========================

…ter (databendlabs#13547) * chore: support collect statistics of multi join expr in merge into * partition by first join condition * fix filter * update * fix clippy * fix * fix bind * fix and add log * fix stackoverflow

SkyFan2002 added 8 commits October 31, 2023 21:28

chore: support collect statistics of multi join expr in merge into

73b8f43

partition by first join condition

257d8d8

fix filter

e819b40

Merge branch 'main' into merge_filter

1f96752

Merge branch 'main' into merge_filter

6a0ced9

update

d48cc82

fix clippy

29af9c5

fix

5ade648

github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Nov 2, 2023

Merge remote-tracking branch 'upstream/main' into merge_filter

c0a6fd6

# Conflicts: # src/query/service/src/interpreters/interpreter_merge_into.rs

vercel bot deployed to Preview November 2, 2023 07:58 View deployment

SkyFan2002 added 2 commits November 2, 2023 20:41

fix bind

021cff5

Merge remote-tracking branch 'upstream/main' into merge_filter

df0fb5d

# Conflicts: # src/query/service/src/interpreters/interpreter_merge_into.rs

SkyFan2002 added 4 commits November 2, 2023 21:57

Merge branch 'main' into merge_filter

55866f9

Merge branch 'main' into merge_filter

385cc6a

fix and add log

1771825

Merge branch 'main' into merge_filter

ee37117

vercel bot deployed to Preview November 3, 2023 08:20 View deployment

fix stackoverflow

0e7c698

SkyFan2002 requested review from JackTan25 and dantengsky November 3, 2023 10:02

SkyFan2002 marked this pull request as ready for review November 3, 2023 10:03

SkyFan2002 marked this pull request as draft November 3, 2023 11:41

Merge branch 'main' into merge_filter

3b1c1f4

dantengsky approved these changes Nov 3, 2023

View reviewed changes

SkyFan2002 marked this pull request as ready for review November 3, 2023 15:28

BohuTANG merged commit 33051f0 into databendlabs:main Nov 3, 2023
64 checks passed

JackTan25 added the ci-cloud Build docker image for cloud test label Nov 3, 2023

SkyFan2002 mentioned this pull request Nov 7, 2023

fix: cluster key in merge into filter #13609

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: partition by left most cluster key when building merge into filter #13547

feat: partition by left most cluster key when building merge into filter #13547

SkyFan2002 commented Nov 2, 2023 •

edited

Loading

vercel bot commented Nov 2, 2023 •

edited

Loading

SkyFan2002 commented Nov 2, 2023

SkyFan2002 commented Nov 3, 2023

BohuTANG commented Nov 3, 2023

JackTan25 commented Nov 3, 2023

dantengsky commented Nov 3, 2023

BohuTANG commented Nov 3, 2023

dantengsky commented Nov 3, 2023

SkyFan2002 commented Nov 3, 2023

feat: partition by left most cluster key when building merge into filter #13547

feat: partition by left most cluster key when building merge into filter #13547

Conversation

SkyFan2002 commented Nov 2, 2023 • edited Loading

Summary

vercel bot commented Nov 2, 2023 • edited Loading

SkyFan2002 commented Nov 2, 2023

SkyFan2002 commented Nov 3, 2023

BohuTANG commented Nov 3, 2023

JackTan25 commented Nov 3, 2023

dantengsky commented Nov 3, 2023

BohuTANG commented Nov 3, 2023

dantengsky commented Nov 3, 2023

SkyFan2002 commented Nov 3, 2023

SkyFan2002 commented Nov 2, 2023 •

edited

Loading

vercel bot commented Nov 2, 2023 •

edited

Loading