PG CDC data checksums inconsistent

[xuefengze]

https://buildkite.com/risingwave-test/chaos-mesh/builds/484, data verification failed with q16.

rw version: nightly-20240204
TESTBED: medium-arm-all-affinity
Grafana Logs: https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&from=now-3h&to=now&var-data_source=PE59595AED52CF917&var-namespace=longcmkf-20240205-025235&var-pod=benchmark-risingwave-compute-c-0&var-search=

step:

• start ch_benchmark_pg_cdc
• inject network faults to risingwave
• after a few minutes, check to see if the data is consistent between RW and PG

chaos-mesh:

• loss between compute and frontend
• corrupt between compute and meta
• delay between meta and frontend
• set bandwidth limit to compute

[xuefengze]

q16 failed without chaos-mesh.
https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/200

[cyliu0]

Reproduced in ch-benchmark-pg-cdc-20240206-040048. The source tables are synced completed. Something wrong with the query.

https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/203#018d7c92-00f0-4993-81ee-26288f847171

bin/qa consistency compare --upstream-driver postgres --upstream-port 54321 --upstream-user postgres --upstream-password postgres --upstream-database-name postgres --downstream-port 45678 --downstream-user root --downstream-password "" --downstream-database-name dev -t ch_benchmark_q16
{
    "consistent": false,
    "table-compare-results": [
        {
            "consistent": false,
            "table-checksums": [
                {
                    "url": "postgres://postgres:postgres@localhost:54321/postgres",
                    "table-name": "ch_benchmark_q16",
                    "table-checksum": 7521164819788754079,
                    "table-rows": 99977
                },
                {
                    "url": "postgres://root:@localhost:45678/dev",
                    "table-name": "ch_benchmark_q16",
                    "table-checksum": -7032245099579614466,
                    "table-rows": 99977
                }
            ]
        }
    ]
}

bin/qa consistency compare --upstream-driver postgres --upstream-port 54321 --upstream-user postgres --upstream-password postgres --upstream-database-name postgres --downstream-port 45678 --downstream-user root --downstream-password "" --downstream-database-name dev -t stock,item                                                         
{
    "consistent": true,
    "table-compare-results": [
        {
            "consistent": true,
            "table-checksums": [
                {
                    "url": "postgres://postgres:postgres@localhost:54321/postgres",
                    "table-name": "stock",
                    "table-checksum": 3121076962066288589,
                    "table-rows": 100000
                },
                {
                    "url": "postgres://root:@localhost:45678/dev",
                    "table-name": "stock",
                    "table-checksum": 3121076962066288589,
                    "table-rows": 100000
                }
            ]
        },
        {
            "consistent": true,
            "table-checksums": [
                {
                    "url": "postgres://postgres:postgres@localhost:54321/postgres",
                    "table-name": "item",
                    "table-checksum": -7200303760742608541,
                    "table-rows": 100000
                },
                {
                    "url": "postgres://root:@localhost:45678/dev",
                    "table-name": "item",
                    "table-checksum": -7200303760742608541,
                    "table-rows": 100000
                }
            ]
        }
    ]
}

[fuyufjh]

Might be the same reason of #15057. Feel free to reopen if it recurs.