Switch Bicep Recipe deletion logic from sequential to parallel and ad…

…d retries (#6256)

# Description

Continuation of #6069

* Changing Recipe BicepDriver deletion logic to be parallel with retries
* Changing Recipe GCResources to use Driver delete

## Type of change

<!--

Please select **one** of the following options that describes your
change and delete the others. Clearly identifying the type of change you
are making will help us review your PR faster, and is used in authoring
release notes.

If you are making a bug fix or functionality change to Radius and do not
have an associated issue link please create one now.

-->

- This pull request fixes a bug in Radius and has an approved issue
(issue link required).

<!--

Please update the following to link the associated issue. This is
required for some kinds of changes (see above).

-->

Fixes: #5294

## Auto-generated summary

<!--
GitHub Copilot for docs will auto-generate a summary of the PR
-->

<!--
copilot:all
-->
### <samp>🤖 Generated by Copilot at 33c5342</samp>

### Summary
🚀🧪🛠️

<!--
1. 🚀 - This emoji represents the performance and scalability
improvements for the worker server and the bicep driver, as well as the
use of the armresources package and the AWS credentials client.
2. 🧪 - This emoji represents the addition of unit tests for the sdk
package and the bicep driver, as well as the use of gomock and time
packages for testing.
3. 🛠️ - This emoji represents the bug fixes and enhancements for the
configmaps, the documentation, the factory, the database timeouts, and
the error handling.
-->
This pull request updates various files in the radius project to use the
NewClientOptions function with a nil parameter, improve the bicep
driver, increase the worker server concurrency, and fix the radius
secret. It also adds a new dependency, updates the documentation, and
adds unit tests.

> _This pull request has many changes_
> _To improve the worker ranges_
> _It adds some retry_
> _And merges structs neatly_
> _And updates the client options for stages_

### Walkthrough
* Increase the maximum operation concurrency for the worker server from
3 to 10 in various configuration files and update the documentation
accordingly
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-5b57e9314a18299dc316a8cfec855ae859078a6b3baaf8a9d9205f4d1e26e665L30-R30),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-a9312f9d13becb76eeb2cba6b9a0f78cc182ae1c203e3f3db5899f3aec534940L40-R40),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-cdbd007d23572be9425f931432181ce0b9468a548a9c28999d0943277a424c7bL36-R36),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-9976f6e926619046569a25ece5f0cd16adfe3413b6334097c6f736e83cda58caL30-R30),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-4df28288cb6f2d9e8e1c520e2a82d7382058acaeebae8882c92b1634882efc43L40-R40),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-58d6d96c3a9a74e33fc72e9a458a3e9f4c463287d906985725fddb0398081126L40-R40),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-58d6d96c3a9a74e33fc72e9a458a3e9f4c463287d906985725fddb0398081126L86-R86),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-c5a6e900bac29ec26476b731e62862ee9afb8f9f67225da0aa8fd1d052f8183fL99-R99),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-c5a6e900bac29ec26476b731e62862ee9afb8f9f67225da0aa8fd1d052f8183fL212-R212))
* Add a new dependency to the `go.mod` file to use the mergo package for
merging structs in the `sdk` package
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R98))
* Add a new parameter to the `NewClientOptions` function in the
`pipeline.go` file to allow passing a custom `clientOptions` struct and
implement the logic for merging it with the default options using the
mergo package
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-9817c1a3a0f4fe01748698c7f17bcce4346f3baebdbb4e731fa2bd9cd54a4113R22),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-9817c1a3a0f4fe01748698c7f17bcce4346f3baebdbb4e731fa2bd9cd54a4113L40-R48),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-9817c1a3a0f4fe01748698c7f17bcce4346f3baebdbb4e731fa2bd9cd54a4113R70-R80))
* Add unit tests for the `NewClientOptions` function in the
`pipeline_test.go` file
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-15e4a018a66c77b12fbf73c7d29edd2fbfa87807fd0b65475ee29caf5bf8c3e9R1-R103))
* Add a new parameter to the `NewClientOptions` function calls in
various packages to pass a nil `clientOptions` struct, which will use
the default options for creating different clients
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-d4132acf041f5a0725d02f79003aca9ad44bc2829ec159cb704b927f13d7b916L71-R71),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-d4132acf041f5a0725d02f79003aca9ad44bc2829ec159cb704b927f13d7b916L134-R134),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-d4132acf041f5a0725d02f79003aca9ad44bc2829ec159cb704b927f13d7b916L187-R187),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-d4132acf041f5a0725d02f79003aca9ad44bc2829ec159cb704b927f13d7b916L209-R209),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-5d6b976ca1401aa4b4ea6fdaf8142700ada0d6ddd9690bb7eed8797f91d33466L117-R122),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-7ea25eedbaa97be92839b34529ce6c828e50f74b8845eaefc02832c23cd6d833L61-R61),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-2779f210632b96e2b7717b9cfae7f667d7f8f5d4053a67d9e796c00d39e34ce1L80-R80),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-ba263c7aa6158e7fdb51e32b1e42473d51fe6dbba4cd83b78bd0291d0465d629L171-R180),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-33881a33475ba368a26aee51ed3c9840f6af46e31f5535106506e6303729f6e3L43-R43))
* Increase the timeout values for async create or update and delete
operations for Mongo and SQL databases in the `datastoresrp` package to
accommodate the longer time required for these operations in AWS
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-59a31913465fb9882e5a2daa03bb09e235cee7c6b313dbf62f11369427b3069dL28-R30),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-59a31913465fb9882e5a2daa03bb09e235cee7c6b313dbf62f11369427b3069dL43-R45))
* Add a new import, constants, and a helper function to the
`resourceclient.go` file in the `portableresources` package to use the
`armresourcesv1` package for defining error codes and details for AWS
operations and checking if an error is retryable
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-ba263c7aa6158e7fdb51e32b1e42473d51fe6dbba4cd83b78bd0291d0465d629R26),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-ba263c7aa6158e7fdb51e32b1e42473d51fe6dbba4cd83b78bd0291d0465d629R45-R51),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-ba263c7aa6158e7fdb51e32b1e42473d51fe6dbba4cd83b78bd0291d0465d629R269-R272))
* Add a new field, constant, and import to the `bicep.go` file in the
`recipes` package to pass a default retry config to the bicep driver and
use the time package for defining the retry delay
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0L27-R29),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0R51-R52),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0R63),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0R71))
* Add a new file `retry.go` to the `recipes` package to define the
`RetryConfig` type and the `NewDefaultRetryConfig` function for the
bicep driver
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-50fbb436c8d21092902b891817b78b2c53d6a2f34710380a61c89c9d7d00b6b6R1-R41))
* Refactor the logic for deleting output resources in the `bicep.go`
file in the `recipes` package to use goroutines, errgroup, and retry
logic based on the retry config and the retryable error codes
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0L157-R219))
* Update the `Mode` field in the `DeploymentProperties` struct and the
type of the `resources` parameter in the `prepareRecipeResponse`
function in the `bicep.go` file in the `recipes` package to use the
`armresources` package instead of the `deployments` package
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0L119-R124),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0L289-R326))
* Refactor the logic for deleting obsolete output resources in the
`Execute` function of the `bicep.go` file in the `recipes` package to
use the `Delete` function instead of the `deleteGCOutputResources`
function and convert the resource IDs to output resources with parsed
IDs
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0L147-R166))
* Remove the `deleteGCOutputResources` function and the test cases for
it from the `bicep.go` and `bicep_test.go` files in the `recipes`
package, since they are no longer used after the refactoring of the
`Delete` function
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-f12e37706fa75e9e04b8dfa01a45cf9bf2a0463e1962605b2420d0cb343922a0L342-L355),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-a0a208a111e71713a21b404fdb3b31fd13f668e5716ab4331584027cc3d98debL483-R518))
* Update the `Delete` function calls in the test cases in the
`bicep_test.go` file in the `recipes` package to use `gomock.Any()`
instead of `ctx` for the first argument, since the context is not
relevant for the mock expectation
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-a0a208a111e71713a21b404fdb3b31fd13f668e5716ab4331584027cc3d98debL393-R397),
[link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-a0a208a111e71713a21b404fdb3b31fd13f668e5716ab4331584027cc3d98debL422-R426))
* Add a new field to the `setupDeleteInputs` function call in the
`bicep_test.go` file in the `recipes` package to pass a retry config to
the bicep driver test
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-a0a208a111e71713a21b404fdb3b31fd13f668e5716ab4331584027cc3d98debR362-R365))
* Add a new import to the `bicep_test.go` file in the `recipes` package
to use the time package for defining the retry delay in the bicep driver
test
([link](https://github.com/radius-project/radius/pull/6256/files?diff=unified&w=0#diff-a0a208a111e71713a21b404fdb3b31fd13f668e5716ab4331584027cc3d98debL20-R22))

---------

Co-authored-by: Karishma Chawla <[email protected]>

cmd/applications-rp/portableresource-dev.yaml

@@ -35,5 +35,8 @@ ucp:
 logging:
   level: "info"
   json: false
+bicep:
+  deleteRetryCount: 20
+  deleteRetryDelaySeconds: 60
 terraform:
   path: "/terraform"

cmd/applications-rp/portableresource-self-hosted.yaml

@@ -47,5 +47,8 @@ ucp:
 logging:
   level: "info"
   json: false
+bicep:
+  deleteRetryCount: 20
+  deleteRetryDelaySeconds: 60
 terraform:
   path: "/tmp"

cmd/applications-rp/radius-cloud.yaml

@@ -49,5 +49,8 @@ ucp:
 logging:
   level: "info"
   json: false
+bicep:
+  deleteRetryCount: 20
+  deleteRetryDelaySeconds: 60
 terraform:
   path: "/terraform"

cmd/applications-rp/radius-dev.yaml

@@ -36,5 +36,8 @@ ucp:
 logging:
   level: "info"
   json: false
+bicep:
+  deleteRetryCount: 20
+  deleteRetryDelaySeconds: 60
 terraform:
   path: "/terraform"

cmd/applications-rp/radius-self-hosted.yaml

@@ -52,5 +52,8 @@ tracerProvider:
   serviceName: "applications.core"
   zipkin:
     url: "http://localhost:9411/api/v2/spans"
+bicep:
+  deleteRetryCount: 20
+  deleteRetryDelaySeconds: 60
 terraform:
   path: "/tmp"

deploy/Chart/templates/rp/configmaps.yaml

@@ -51,6 +51,9 @@ data:
       zipkin: 
         url: {{ .Values.global.zipkin.url }}
     {{- end }}
+    bicep:
+      deleteRetryCount: 20
+      deleteRetryDelaySeconds: 60
     terraform:
       path: "/terraform"
 
@@ -98,5 +101,8 @@ data:
       zipkin: 
         url: {{ .Values.global.zipkin.url }}
     {{- end }}
+    bicep:
+      deleteRetryCount: 20
+      deleteRetryDelaySeconds: 60
     terraform:
       path: "/terraform"

deploy/Chart/values.yaml

@@ -45,5 +45,8 @@ rp:
       # limit is higher for applications-rp because the Terraform execution
       # can spike memory usage.
       memory: "500Mi"
+  bicep:
+    deleteRetryCount: 20
+    deleteRetryDelaySeconds: 60
   terraform:
     path: "/terraform"

go.mod

@@ -11,7 +11,6 @@ require (
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armsubscriptions v1.2.0
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/servicebus/armservicebus/v2 v2.0.0-beta.3
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.3.0
-	github.com/Azure/go-autorest/autorest/to v0.4.0
 	github.com/Azure/secrets-store-csi-driver-provider-azure v1.4.1
 	github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d
 	github.com/agnivade/levenshtein v1.1.1
@@ -104,7 +103,6 @@ require (
 	github.com/AdaLogics/go-fuzz-headers v0.0.0-20230106234847-43070de90fa1 // indirect
 	github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect
 	github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
-	github.com/Azure/go-autorest v14.2.0+incompatible // indirect
 	github.com/AzureAD/microsoft-authentication-library-for-go v1.0.0 // indirect
 	github.com/BurntSushi/toml v1.2.1 // indirect
 	github.com/MakeNowJust/heredoc v1.0.0 // indirect

go.sum

@@ -72,10 +72,6 @@ github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.3.0
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.3.0/go.mod h1:+OgGVo0Httq7N5oayfvaLQ/Jq+2gJdqfp++Hyyl7Tws=
 github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8=
 github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
-github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs=
-github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24=
-github.com/Azure/go-autorest/autorest/to v0.4.0 h1:oXVqrxakqqV1UZdSazDOPOLvOIz+XA683u8EctwboHk=
-github.com/Azure/go-autorest/autorest/to v0.4.0/go.mod h1:fE8iZBn7LQR7zH/9XU2NcPR4o9jEImooCeWJcYV/zLE=
 github.com/Azure/secrets-store-csi-driver-provider-azure v1.4.1 h1:24/mZ06Uzu8EkdgJj8zcy5C3Ipsg1doM62hx0WPscNU=
 github.com/Azure/secrets-store-csi-driver-provider-azure v1.4.1/go.mod h1:xUXKV8vOut59vIrFhyEY+4PgiK2LXkP10BtI+2y8VXM=
 github.com/AzureAD/microsoft-authentication-library-for-go v1.0.0 h1:OBhqkivkhkMqLPymWEppkm7vgPQY2XsHoEkaMQ0AdZY=

pkg/armrpc/hostoptions/providerconfig.go

@@ -41,6 +41,7 @@ type ProviderConfig struct {
 	ProfilerProvider profilerprovider.ProfilerProviderOptions `yaml:"profilerProvider"`
 	UCP              config.UCPOptions                        `yaml:"ucp"`
 	Logging          ucplog.LoggingOptions                    `yaml:"logging"`
+	Bicep            BicepOptions                             `yaml:"bicep,omitempty"`
 	Terraform        TerraformOptions                         `yaml:"terraform,omitempty"`
 
 	// FeatureFlags includes the list of feature flags.
@@ -70,6 +71,14 @@ type WorkerServerOptions struct {
 	MaxOperationRetryCount *int `yaml:"maxOperationRetryCount,omitempty"`
 }
 
+// BicepOptions includes options required for bicep execution.
+type BicepOptions struct {
+	// DeleteRetryCount is the number of times to retry the request.
+	DeleteRetryCount string `yaml:"deleteRetryCount,omitempty"`
+	// DeleteRetryDelaySeconds is the delay between retries in seconds.
+	DeleteRetryDelaySeconds string `yaml:"deleteRetryDelaySeconds,omitempty"`
+}
+
 // TerraformOptions includes options required for terraform execution.
 type TerraformOptions struct {
 	// Path is the path to the directory mounted to the container where terraform can be installed and executed.

pkg/recipes/controllerconfig/config.go

@@ -17,6 +17,8 @@ limitations under the License.
 package controllerconfig
 
 import (
+	"strconv"
+
 	"github.com/radius-project/radius/pkg/armrpc/hostoptions"
 	aztoken "github.com/radius-project/radius/pkg/azure/tokencredentials"
 	"github.com/radius-project/radius/pkg/kubeutil"
@@ -69,11 +71,29 @@ func New(options hostoptions.HostOptions) (*RecipeControllerConfig, error) {
 		return nil, err
 	}
 
+	bicepDeleteRetryCount, err := strconv.Atoi(options.Config.Bicep.DeleteRetryCount)
+	if err != nil {
+		return nil, err
+	}
+
+	bicepDeleteRetryDeleteSeconds, err := strconv.Atoi(options.Config.Bicep.DeleteRetryDelaySeconds)
+	if err != nil {
+		return nil, err
+	}
+
 	cfg.ConfigLoader = configloader.NewEnvironmentLoader(clientOptions)
 	cfg.Engine = engine.NewEngine(engine.Options{
 		ConfigurationLoader: cfg.ConfigLoader,
 		Drivers: map[string]driver.Driver{
-			recipes.TemplateKindBicep: driver.NewBicepDriver(clientOptions, cfg.DeploymentEngineClient, cfg.ResourceClient),
+			recipes.TemplateKindBicep: driver.NewBicepDriver(
+				clientOptions,
+				cfg.DeploymentEngineClient,
+				cfg.ResourceClient,
+				driver.BicepOptions{
+					DeleteRetryCount:        bicepDeleteRetryCount,
+					DeleteRetryDelaySeconds: bicepDeleteRetryDeleteSeconds,
+				},
+			),
 			recipes.TemplateKindTerraform: driver.NewTerraformDriver(options.UCPConnection, provider.NewSecretProvider(options.Config.SecretProvider),
 				driver.TerraformOptions{
 					Path: options.Config.Terraform.Path,

pkg/recipes/driver/bicep.go

@@ -24,8 +24,10 @@ import (
 
 	"github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
 	"github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime"
+	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources"
+	"github.com/radius-project/radius/pkg/to"
+	"golang.org/x/sync/errgroup"
 
-	deployments "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources"
 	"github.com/go-logr/logr"
 	"github.com/radius-project/radius/pkg/metrics"
 	"github.com/radius-project/radius/pkg/portableresources/datamodel"
@@ -52,19 +54,26 @@ const (
 
 var _ Driver = (*bicepDriver)(nil)
 
-// NewBicepDriver creates a new bicep driver instance with the given ARM client options, deployment client and resource client.
-func NewBicepDriver(armOptions *arm.ClientOptions, deploymentClient *clients.ResourceDeploymentsClient, client processors.ResourceClient) Driver {
+// NewBicepDriver creates a new bicep driver instance with the given ARM client options, deployment client, resource client, and options.
+func NewBicepDriver(armOptions *arm.ClientOptions, deploymentClient *clients.ResourceDeploymentsClient, client processors.ResourceClient, options BicepOptions) Driver {
 	return &bicepDriver{
 		ArmClientOptions: armOptions,
 		DeploymentClient: deploymentClient,
 		ResourceClient:   client,
+		options:          options,
 	}
 }
 
+type BicepOptions struct {
+	DeleteRetryCount        int
+	DeleteRetryDelaySeconds int
+}
+
 type bicepDriver struct {
 	ArmClientOptions *arm.ClientOptions
 	DeploymentClient *clients.ResourceDeploymentsClient
 	ResourceClient   processors.ResourceClient
+	options          BicepOptions
 }
 
 // Execute fetches recipe contents from container registry, creates a deployment ID, a recipe context parameter, recipe parameters,
@@ -117,7 +126,7 @@ func (d *bicepDriver) Execute(ctx context.Context, opts ExecuteOptions) (*recipe
 		ctx,
 		clients.Deployment{
 			Properties: &clients.DeploymentProperties{
-				Mode:           deployments.DeploymentModeIncremental,
+				Mode:           armresources.DeploymentModeIncremental,
 				ProviderConfig: &providerConfig,
 				Parameters:     parameters,
 				Template:       recipeData,
@@ -145,10 +154,15 @@ func (d *bicepDriver) Execute(ctx context.Context, opts ExecuteOptions) (*recipe
 	// as bicep does not take care of automatically deleting the unused resources.
 	// Identify the output resources that are no longer relevant to the recipe.
 	garbageCollectionStartTime := time.Now()
-	diff := d.getGCOutputResources(recipeResponse.Resources, opts.PrevState)
+	diff, err := d.getGCOutputResources(recipeResponse.Resources, opts.PrevState)
+	if err != nil {
+		return nil, err
+	}
 
 	// Deleting obsolete output resources.
-	err = d.deleteGCOutputResources(ctx, diff)
+	err = d.Delete(ctx, DeleteOptions{
+		OutputResources: diff,
+	})
 	if err != nil {
 		metrics.DefaultRecipeEngineMetrics.RecordRecipeGarbageCollectionDuration(ctx, garbageCollectionStartTime,
 			metrics.NewRecipeAttributes(metrics.RecipeEngineOperationGC, opts.Recipe.Name, &opts.Definition, metrics.FailedOperationState))
@@ -159,32 +173,60 @@ func (d *bicepDriver) Execute(ctx context.Context, opts ExecuteOptions) (*recipe
 	return recipeResponse, nil
 }
 
-// Delete deletes output resources in reverse dependency order, logging each resource deleted and skipping any
-// resources that are not managed by Radius. It returns an error if any of the resources fail to delete.
+// Delete deletes all of the output resources that are marked as managed by Radius.
+// It will create a goroutine for each resource to be deleted and wait for them to finish,
+// retrying if necessary.
+// We don't have context on the dependency ordering here, so we need to try to delete them
+// all in parallel. Since some resources may depend on others, we may need to retry.
 func (d *bicepDriver) Delete(ctx context.Context, opts DeleteOptions) error {
 	logger := ucplog.FromContextOrDiscard(ctx)
 
-	orderedOutputResources, err := rpv1.OrderOutputResources(opts.OutputResources)
-	if err != nil {
-		return recipes.NewRecipeError(recipes.RecipeDeletionFailed, err.Error(), "", recipes.GetRecipeErrorDetails(err))
-	}
+	// Create a waitgroup to track the deletion of each output resource
+	g, groupCtx := errgroup.WithContext(ctx)
 
-	// Loop over each output resource and delete in reverse dependency order
-	for i := len(orderedOutputResources) - 1; i >= 0; i-- {
-		outputResource := orderedOutputResources[i]
+	for i := range opts.OutputResources {
+		outputResource := opts.OutputResources[i]
 
-		resourceType := outputResource.GetResourceType()
-		logger.Info(fmt.Sprintf("Deleting output resource: %v, LocalID: %s, resource type: %s\n", outputResource.ID.String(), outputResource.LocalID, resourceType.Type))
-		if outputResource.RadiusManaged == nil || !*outputResource.RadiusManaged {
-			continue
-		}
+		// Create a goroutine that handles the deletion of one resource
+		g.Go(func() error {
+			id := outputResource.ID.String()
+			logger.V(ucplog.LevelInfo).Info(fmt.Sprintf("Deleting output resource: %v, LocalID: %s, resource type: %s\n", outputResource.ID, outputResource.LocalID, outputResource.GetResourceType()))
 
-		err = d.ResourceClient.Delete(ctx, outputResource.ID.String())
-		if err != nil {
-			return recipes.NewRecipeError(recipes.RecipeDeletionFailed, err.Error(), "", recipes.GetRecipeErrorDetails(err))
-		}
-		logger.Info(fmt.Sprintf("Deleted output resource: %q", outputResource.ID.String()), ucplog.LogFieldTargetResourceID, outputResource.ID.String())
+			// If the resource is not managed by Radius, skip the deletion
+			if outputResource.RadiusManaged == nil || !*outputResource.RadiusManaged {
+				logger.Info(fmt.Sprintf("Skipping deletion of output resource: %q, not managed by Radius", id))
+				return nil
+			}
+
+			var err error
+			for attempt := 0; attempt <= d.options.DeleteRetryCount; attempt++ {
+				logger.WithValues("attempt", attempt)
+				ctx := logr.NewContext(groupCtx, logger)
+				logger.V(ucplog.LevelDebug).Info("beginning attempt")
+
+				err = d.ResourceClient.Delete(ctx, id)
+				if err != nil {
+					if attempt <= d.options.DeleteRetryCount {
+						logger.V(ucplog.LevelInfo).Error(err, "attempt failed", "delay", d.options.DeleteRetryDelaySeconds)
+						time.Sleep(time.Duration(d.options.DeleteRetryDelaySeconds) * time.Second)
+						continue
+					}
 
+					return recipes.NewRecipeError(recipes.RecipeDeletionFailed, err.Error(), "", recipes.GetRecipeErrorDetails(err))
+				}
+
+				// If the err is nil, then the resource is deleted successfully
+				logger.V(ucplog.LevelInfo).Info(fmt.Sprintf("Deleted output resource: %q", id))
+				return nil
+			}
+
+			deletionErr := fmt.Errorf("failed to delete resource after %d attempt(s), last error: %s", d.options.DeleteRetryCount+1, err.Error())
+			return recipes.NewRecipeError(recipes.RecipeDeletionFailed, deletionErr.Error(), "", recipes.GetRecipeErrorDetails(deletionErr))
+		})
+	}
+
+	if err := g.Wait(); err != nil {
+		return err
 	}
 
 	return nil
@@ -291,7 +333,7 @@ func newProviderConfig(resourceGroup string, envProviders coredm.Providers) clie
 
 // prepareRecipeResponse populates the recipe response from parsing the deployment output 'result' object and the
 // resources created by the template.
-func (d *bicepDriver) prepareRecipeResponse(outputs any, resources []*deployments.ResourceReference) (*recipes.RecipeOutput, error) {
+func (d *bicepDriver) prepareRecipeResponse(outputs any, resources []*armresources.ResourceReference) (*recipes.RecipeOutput, error) {
 	// We populate the recipe response from the 'result' output (if set)
 	// and the resources created by the template.
 	//
@@ -323,38 +365,32 @@ func (d *bicepDriver) prepareRecipeResponse(outputs any, resources []*deployment
 }
 
 // getGCOutputResources [GC stands for Garbage Collection] compares two slices of resource ids and
-// returns a slice of resource ids that contains the elements that are in the "previous" slice but not in the "current".
-func (d *bicepDriver) getGCOutputResources(current []string, previous []string) []string {
+// returns a slice of OutputResources that contains the elements that are in the "previous" slice but not in the "current".
+func (d *bicepDriver) getGCOutputResources(current []string, previous []string) ([]rpv1.OutputResource, error) {
 	// We can easily determine which resources have changed via a brute-force search comparing IDs.
 	// The lists of resources we work with are small, so this is fine.
-	diff := []string{}
-	for _, prevResource := range previous {
+	diff := []rpv1.OutputResource{}
+	for _, prevResourceId := range previous {
 		found := false
-		for _, currentResource := range current {
-			if prevResource == currentResource {
+		for _, currentResourceId := range current {
+			if prevResourceId == currentResourceId {
 				found = true
 				break
 			}
 		}
 
 		if !found {
-			diff = append(diff, prevResource)
-		}
-	}
-
-	return diff
-}
+			id, err := resources.Parse(prevResourceId)
+			if err != nil {
+				return nil, recipes.NewRecipeError(recipes.RecipeGarbageCollectionFailed, err.Error(), recipes_util.ExecutionError, nil)
+			}
 
-func (d *bicepDriver) deleteGCOutputResources(ctx context.Context, diff []string) error {
-	logger := ucplog.FromContextOrDiscard(ctx)
-	for _, resource := range diff {
-		logger.Info(fmt.Sprintf("Deleting output resource: %s", resource), ucplog.LogFieldTargetResourceID, resource)
-		err := d.ResourceClient.Delete(ctx, resource)
-		if err != nil {
-			return err
+			diff = append(diff, rpv1.OutputResource{
+				ID:            id,
+				RadiusManaged: to.Ptr(true),
+			})
 		}
-		logger.Info(fmt.Sprintf("Deleted output resource: %s", resource), ucplog.LogFieldTargetResourceID, resource)
 	}
 
-	return nil
+	return diff, nil
 }