feat(udf): support always_retry_on_network_error config for udf functions

[kwannoel]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Closes #15137.

The main issue is that when UDF call fails, the function call returns null instead.
If the function call updates a key, e.g. for join or group by, it is possible for the state to be inconsistent.

As such we provide the option to define a udf function which will always retry on network errors, since we cannot tolerate the UDF failing non-deterministically in such cases.

Here are the changes:

1. Introduce a new CreateFunctionWithOptions.
2. Since it's an AST object, we provide option for each of its fields. This is to make it compatible to Display (otherwise no WITH options, or only some options set, we still display all CreateFunctionWithOptions. See tests for examples). This also preserves the AST structure.
3. Flatten the CreateFunctionWithOptions into parameters to Function, UserDefinedFunction definitions.
4. Initialize the udf expr with always_retry_on_network_error.
5. Add tests for display of CreateFunctionWithOptions.
6. Add tests to test MV contents when there's no retry, vs when there's retry.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Users can now do:

CREATE FUNCTION ... WITH ( always_retry_on_network_error = true );

This means network errors will always be retried for function calls of that function.

Note that the entire stream graph will be blocked when UDF server goes offline, until it is back online.

[xxchan]

qq: What's the relationship with #15171?

[kwannoel]

qq: What's the relationship with #15171?

This PR makes it configurable as an with_option.

In the other PR, it just hardcodes it to always retry. Reason being I want to introduce as little code as possible to the user's cluster to avoid breakage, and their UDFs must always retry.

[kwannoel]

Can @xxchan and @wangrunji0408 PTAL at this? I want to include this in 1.7. Since it's a breaking change, so users can upgrade their UDF functions ASAP, since they have to recreate all dependent MVs.

[kwannoel]

Can review the code first, there's some separate failure due to some migration issues.

[xxchan]

Generally LGTM. I'm not sure what's the best way, but I think offering this option doesn't hurt.

[xxchan]

Should we put all disable_retry_count related stuff in else branch?

[kwannoel]

Hmm no difference to me? Seems to add more nesting.

[xxchan]

I think mixing them together can lead to confusion. Why do we still update disable_retry_count when always_retry_on_network_error?

If nesting doesn't look good, we can add a sth like call_with_ disable_retry_count...

[kwannoel]

Ah I didn't get what you meant originally. Now I do. Updated it.

[wangrunji0408]

LGTM!