2024-02-07 nexmark performance degradation

[lmatz]

https://risingwave-labs.slack.com/archives/C04R6R5236C/p1707350402322639

http://metabase.risingwave-cloud.xyz/dashboard/241-nexmark-blackhole-1cn-anti-affinity-rw-avg-source-throughput?namespace=daily
http://metabase.risingwave-cloud.xyz/question/10799-nexmark-anti-affinity-rw-avg-source-throughput-geometric

[st1page]

nightly-20240207

• 85f002347c65b3a955ff110146629f1d06890b48 fix(scale): move reschedule_lock to ScaleController & use universal scale_controller (#15037)
• 05b84b449c8c129f3ed0662de15ae1f80abb37e1 fix(sql-backend): change stream node type from json to binary to fix stack overflow (#15040)
• d50cc770253f74d14ce39f25f0f946ec2726ff9e feat(Sink): Enhanced integration with StarRocks (#14363)
• 1a231cdc1fd724bba7c4bf547e66c04256fee2cf chore(deps): Bump tempfile from 3.9.0 to 3.10.0 (#15041)
• 90df1a7abe1e7f4b84b5953aedc4613a35cfef86 feat(open telemetry): report rw version in telemetry (#13490)
• 8313b0a941af019df4715f45fee217914c65e8b2 chore(bench): increase key count (#15002)
• 873549172133a029fcf299683f3daaccaa946549 fix(integration_tests): separate test for arrangement backfill cancel with high barrier latency (#15011)
• 8e3c5265263100f70aedac909b4fbd7fb6bd6933 feat(optimizer): support PullUpCorrelatedPredicateAggRule (#15026)
• d5edfe6e5924257c4656922d32554399d9f28a0a refactor(error): clean-up direct error formatting (part 5, done) (#14956)
• 7a4389f6d6853ceef6d7d810e077a779a871d8f0 chore(deps): Bump parse-display from 0.8.2 to 0.9.0 (#15012)

[st1page]

Looks like the block cache's slight difference makes the join's performance worse(q4). the cache ops metrics in the picture is the table 1032's cache miss ops, which is the join's right table and the executor cache miss rate is 100% forever so the block cache miss can affect the performace significantly

[lmatz]

It reminds me that the join actor match duration per second often increases a lot when scaling up #14448

[st1page]

It could be introduced by d5edfe6 #14956

http://metabase.risingwave-cloud.xyz/question/1136-nexmark-q4-blackhole-medium-1cn-avg-source-output-rows-per-second-rows-s-history-thtb-117?start_date=2024-02-05
http://metabase.risingwave-cloud.xyz/question/1204-nexmark-q5-blackhole-medium-1cn-avg-source-output-rows-per-second-rows-s-history-thtb-118?start_date=2024-02-05

[st1page]

looks today(20240208) result comes back... but I do not know why https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240208

[lmatz]

Is it possible that the compaction is not deterministic and affects the block miss rate?

[lmatz]

can it be the same problem as in #15169, aka the failure task in compaction? @st1page

[st1page]

Looks like the block cache's slight difference makes the join's performance worse(q4). the cache ops metrics in the picture is the table 1032's cache miss ops, which is the join's right table and the executor cache miss rate is 100% forever so the block cache miss can affect the performace significantly

I checked q4 and q5 quickly and did not find the failure task in compaction in metrics and there is no significant difference of the SST cound panel. So I think not

[st1page]

Because we have not seen this degradation anymore, can we close the issue?

[lmatz]

I close it now, but the reason of fluctuation is unknown