feat: enable shared source in session variable by default, and add cluster-level config to disable

[xxchan]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

User interface

• Can use session variable streaming_use_shared_source to control whether to use shared feature:

# change the config in the current session, default true
SET streaming_use_shared_source=[true|false];

# change the default value of the session variable in the cluster
# (the current session is not affected)
ALTER SYSTEM SET streaming_use_shared_source=[true|false];

• There’s also a cluster config stream_enable_shared_source ([streaming.developer] in risingwave.toml). Set it to false can completely disable this feature in the cluster.

What's changed

The config is similar to streaming_use_arrangement_backfill (session) and stream_enable_arrangement_backfill (cluster)

For more discussions see https://risingwave-labs.slack.com/archives/C06AL46R10S/p1729051215981299

• Rename session var enable_shared_source to streaming_use_shared_source, to be consistent with streaming_use_arrangement_backfill
• Make the session var defaults to true
• Add config stream_enable_shared_source in [streaming.developer] in risingwave.toml, defaults to true. When this is false, the feature is disabled in the cluster (to re-enable, need to update toml and restart); When it's true, respect session variable.
  • TODO: cloud, similar to https://github.com/risingwavelabs/risingwave-cloud/pull/7664/ We will not have special handling on cloud
• Changes to tests:
  • testing using ALTER SOURCE need to use non-shared source, as altering shared source is not supported yet
  • rate_limit_source_kafka.slt: alter source_rate_limit do not affect source backfill.
• Tracking issue for shared source #16003

Benchmarks

Discussed with @lmatz , we will first disable shared source in daily benchmark pipeline. After fixing things like metrics collection, we add a new weekly benchmark for shared source.

We will need to tweak benchmark pipelines after enabling shared source, otherwise they may fail. Click to see details

Nexmark

• nexmark failed, need fix;

Nexmark first creates a Kafka topic and fill data, then builds an MV. This will result in the entire benchmark process being backfill.

1. there is a slight change in metrics; it doesn't have source throughput, but only source backfill throughput, so it will fail.
2. the performance during backfill might be worse than the source

We can see backfill performance is indeed worse than source:

Longevity

It didn't fail, but the performance behavior will look different.

We have enable_create_mv_nexmark_table = true by default. So we have 3 source executors (bid/auction/person are MVs) -> 1 source executor + 3 backfill executors.

Previously the 3 sources have individual consumption progress, but now they share consumption progress. This is like "unified topic" in nexmark benchmark.

• longevity:
  • grafana; Compare with prev daily run
• We can also try enable_create_mv_nexmark_table = false

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

Need release note & doc for the whole shared source feature: https://www.notion.so/risingwave-labs/Doc-for-Shared-Source-10ef0d69cb768078b2a5e05bfa5d4807

[xxchan]

• refactor(meta): generalize replace table to replace job #18935
• feat: enable shared source in session variable by default, and add cluster-level config to disable #18749 👈
• feat: support background DDL for MV on shared source #18919 : 1 other dependent PR (#18924 )
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @xxchan and the rest of your teammates on Graphite

[BugenZhao]

Currently session variables have different styles:

#13265

[fuyufjh]

LGTM.

Shall we cherry-pick to release 2.1?

[fuyufjh]

This doesn't follow the standard pattern of a session/global variable, particularly, the "cluster-level config" should be SystemParams instead of RwConfig. However, as the configuration is not user-facing but only for us, it's acceptable to me.

[xxchan]

standard pattern of a session/global variable

FYI actually session variable can also be ALTER SYSTEM to be changed globally.

Just add this extra gate in case it's needed.

[xxchan]

Shall we cherry-pick to release 2.1?

Yes, it's planned

[xxchan]

Will merge this after changing benchmark pipeline https://linear.app/risingwave-labs/issue/PERF-163