[ci/cd] Split Test pipeline according to runtime-benchmarks feature

[bkontur]

Summary

Based on original issue from @gilescope related to the monorepo:

Currently the tests have started to be run with the runtime-benchmarks feature enabled ( E.g. https://github.com/paritytech/polkadot-sdk/blob/aabed6757e26e21ce06dd59946a17b72f349185e/.gitlab/pipeline/test.yml#L37 ). This was done so that the autogenerated tests that the benchmark macros create get run (but it seems that we already have a short-benchmark CI job... - do the additional tests give us anything extra? EDIT: the short-benchmarks only test 3 relay chains so not comprehensive enough).

However, this is inherently dangerous as not everything will be in a production like state when it is being tested.

For safety's sake we should be running without the runtime-benchmarks feature enabled. Assuming the CI system can take it, we should set up another job that runs the tests with the runtime-benchmarks feature enabled. (if we did not do this we would catch any problems later when the weights are rerun for the runtime release builds).

Proposed solution

Split test pipeline to the two pipelines:

• one without runtime-benchmarks feature
• one with runtime-benchmarks feature

[bkontur]

bot help

[gilescope]

I’m confused, pretty sure it was split into two already?

…

On Fri, 1 Dec 2023 at 17:53, Wei Tang ***@***.***> wrote: ***@***.**** approved this pull request. — Reply to this email directly, view it on GitHub <#48 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGEJCBSF7TOYLC6RCXKU7LYHIKRRAVCNFSM6AAAAAA5KSH6SGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTONRQGMYTEOJWGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bkontur]

I’m confused, pretty sure it was split into two already?

yes, it was, but only for polkadot-sdk repo, not for polkadot-fellows repo

[bkchr]

/merge

[fellowship-merge-bot]

Enabled auto-merge in Pull Request

Available commands

• /merge: Enables auto-merge for Pull Request
• /merge cancel: Cancels auto-merge for Pull Request
• /merge help: Shows this menu

For more information see the documentation