Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ch-benchmark-pg-cdc data verification failed in daily chaos mesh test #15434

Closed
xuefengze opened this issue Mar 5, 2024 · 2 comments
Closed
Labels
found-by-chaos-mesh type/bug Something isn't working
Milestone

Comments

@xuefengze
Copy link
Contributor

================================================================================
chaos-mesh Result
================================================================================
Result               FAIL                
Pipeline Message     Nightly ch-benchmark-pg-cdc
Namespace            longcmkf-20240304-190318
TestBed              medium-arm-all-affinity
RW Version           nightly-20240304    
Test Start time      2024-03-04 19:07:29 
Test End time        2024-03-04 19:34:24 
Test Queries         q1,q2,q3,q4,q5,q6,q7,q8,q9,q10,q11,q12,q13,q14,q15,q17,q18,q19,q20,q21,q22
Grafana Metric       https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=longcmkf-20240304-190318&from=1709579249000&to=1709580864000
Grafana Logs         https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=longcmkf-20240304-190318&from=1709579249000&to=1709580864000
Memory Dumps         https://s3.console.aws.amazon.com/s3/buckets/test-useast1-mgmt-bucket-archiver?region=us-east-1&bucketType=general&prefix=k8s/longcmkf-20240304-190318/&showversions=false
Buildkite Job        https://buildkite.com/risingwave-test/chaos-mesh/builds/637

Data verification fails:

ch_benchmark_q3 table/mv data checksum inconsistent
customer table/mv data checksum inconsistent
new_order table/mv data checksum inconsistent
orders table/mv data checksum inconsistent
stock table/mv data checksum inconsistent
================================================================================
chaos-mesh Result
================================================================================
Result               FAIL                
Pipeline Message     Nightly ch-benchmark-pg-cdc
Namespace            longcmkf-20240304-193627
TestBed              medium-arm-all-affinity
RW Version           nightly-20240304    
Test Start time      2024-03-04 19:38:51 
Test End time        2024-03-04 21:38:31 
Test Queries         q1,q2,q3,q4,q5,q6,q7,q8,q9,q10,q11,q12,q13,q14,q15,q17,q18,q19,q20,q21,q22
Grafana Metric       https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=longcmkf-20240304-193627&from=1709581131000&to=1709588311000
Grafana Logs         https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=longcmkf-20240304-193627&from=1709581131000&to=1709588311000
Memory Dumps         https://s3.console.aws.amazon.com/s3/buckets/test-useast1-mgmt-bucket-archiver?region=us-east-1&bucketType=general&prefix=k8s/longcmkf-20240304-193627/&showversions=false
Buildkite Job        https://buildkite.com/risingwave-test/chaos-mesh/builds/637

Data verification fails:

ch_benchmark_q9 table/mv data checksum inconsistent
customer table/mv data checksum inconsistent
history table/mv data checksum inconsistent
new_order table/mv data checksum inconsistent
orders table/mv data checksum inconsistent
order_line table/mv data checksum inconsistent
stock table/mv data checksum inconsistent
revenue1 table/mv data checksum inconsistent
================================================================================
chaos-mesh Result
================================================================================
Result               FAIL                
Pipeline Message     Nightly ch-benchmark-pg-cdc
Namespace            longcmkf-20240304-214005
TestBed              medium-arm-all-affinity
RW Version           nightly-20240304    
Test Start time      2024-03-04 21:42:20 
Test End time        2024-03-04 22:17:39 
Test Queries         q1,q2,q3,q4,q5,q6,q7,q8,q9,q10,q11,q12,q13,q14,q15,q17,q18,q19,q20,q21,q22
Grafana Metric       https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=longcmkf-20240304-214005&from=1709588540000&to=1709590659000
Grafana Logs         https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=longcmkf-20240304-214005&from=1709588540000&to=1709590659000
Memory Dumps         https://s3.console.aws.amazon.com/s3/buckets/test-useast1-mgmt-bucket-archiver?region=us-east-1&bucketType=general&prefix=k8s/longcmkf-20240304-214005/&showversions=false
Buildkite Job        https://buildkite.com/risingwave-test/chaos-mesh/builds/637

Data verification fails:

ch_benchmark_q3 table/mv data checksum inconsistent
customer table/mv data checksum inconsistent
new_order table/mv data checksum inconsistent
orders table/mv data checksum inconsistent
stock table/mv data checksum inconsistent
@xuefengze xuefengze added type/bug Something isn't working found-by-chaos-mesh labels Mar 5, 2024
@github-actions github-actions bot added this to the release-1.8 milestone Mar 5, 2024
@xuefengze
Copy link
Contributor Author

xuefengze commented Mar 5, 2024

All failed cases have logs similar to the one below:

2024-03-04T22:13:16.929620005Z  WARN risingwave_connector_node: Cannot seek to the last known offset '1/B7FD0710' on replication slot 'orders_slot'. Error from server: 'ERROR: cannot advance replication slot to 1/B7FD0710, minimum is 1/BFE32748': org.postgresql.util.PSQLException: ERROR: cannot advance replication slot to 1/B7FD0710, minimum is 1/BFE32748

link #15141

@fuyufjh
Copy link
Member

fuyufjh commented Mar 12, 2024

All failed cases have logs similar to the one below:

2024-03-04T22:13:16.929620005Z  WARN risingwave_connector_node: Cannot seek to the last known offset '1/B7FD0710' on replication slot 'orders_slot'. Error from server: 'ERROR: cannot advance replication slot to 1/B7FD0710, minimum is 1/BFE32748': org.postgresql.util.PSQLException: ERROR: cannot advance replication slot to 1/B7FD0710, minimum is 1/BFE32748

link #15141

Thanks for identifying the problem! @xuefengze I tend to keep one issue for one problem in order to be organized. So let's close this issue as duplicated.

@fuyufjh fuyufjh closed this as not planned Won't fix, can't repro, duplicate, stale Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
found-by-chaos-mesh type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants