addons: move the fetching of cephfsid only for non-provider mode

[rewantsoni]

Restricting to the fetching of cephFSID for when the MirrorPeer is not for StorageClients
Requeue if MirrorPeer is in Phase: ExchangingSecret

[rewantsoni]

Testing:
Tokenexchange agent no longer tries to fetch the cephfsid for provider mode

{"level":"info","ts":1734424165.4503007,"caller":"controller/controller.go:175","msg":"Starting EventSource","controller":"s3secret_controller","source":"kind source: *v1alpha1.ObjectBucketClaim"}
{"level":"info","ts":1734424165.4503536,"caller":"controller/controller.go:183","msg":"Starting Controller","controller":"s3secret_controller"}
{"level":"info","ts":1734424165.5472872,"caller":"controller/controller.go:217","msg":"Starting workers","controller":"mirrorpeer","controllerGroup":"multicluster.odf.openshift.io","controllerKind":"MirrorPeer","worker count":1}
{"level":"info","ts":1734424165.5476346,"msg":"Running MirrorPeer reconciler on spoke cluster","controller":"MirrorPeerReconciler","MirrorPeer":"/mirrorpeer-5kvv5"}
{"level":"info","ts":1734424165.5521386,"caller":"controller/controller.go:217","msg":"Starting workers","controller":"greensecret_controller","worker count":1}
{"level":"info","ts":1734424165.5521545,"caller":"controller/controller.go:217","msg":"Starting workers","controller":"s3secret_controller","worker count":1}
{"level":"info","ts":1734424165.5523129,"msg":"Reconciling OBC","controller":"S3SecretReconciler","OBC":"openshift-storage/odrbucket-2299ecfe64da875db721bd"}
{"level":"info","ts":1734424165.5548131,"msg":"MirrorPeer has client reference?","controller":"MirrorPeerReconciler","MirrorPeer":"/mirrorpeer-5kvv5","True/False":true}
{"level":"info","ts":1734424165.5605562,"msg":"Creating S3 buckets","controller":"MirrorPeerReconciler","MirrorPeer":"/mirrorpeer-5kvv5"}
{"level":"info","ts":1734424165.5643506,"msg":"ObjectBucketClaim odrbucket-2299ecfe64da875db721bd was unchanged in namespace openshift-storage","controller":"MirrorPeerReconciler"}
{"level":"info","ts":1734424165.5643795,"msg":"Trying to unmarshal onboarding token.","controller":"MirrorPeerReconciler","MirrorPeer":"/mirrorpeer-5kvv5"}
{"level":"info","ts":1734424165.5644326,"msg":"Successfully unmarshalled onboarding ticket","controller":"MirrorPeerReconciler","MirrorPeer":"/mirrorpeer-5kvv5","ticketData":{"id":"37a6449d-a54f-4d8c-bfcb-0f0bcb271bac","expirationDate":"1734596888","subjectRole":"ocs-peer","storageCluster":"4d4e88f9-fc78-4438-9db1-86c0550ea33c"}}
{"level":"info","ts":1734424165.5644586,"msg":"Onboarding token has not expired yet. Not renewing it.","controller":"MirrorPeerReconciler","MirrorPeer":"/mirrorpeer-5kvv5","Token":"f0363be3-b36c-4982-be37-6416535c8a29","ExpirationDate":1734596888}
{"level":"info","ts":1734424166.1645212,"msg":"Found StorageCluster for provider","controller":"S3SecretReconciler","Provider":"ibm-baremetal3","StorageClusterName":"ocs-storagecluster","StorageCluster Namespace":"openshift-storage"}
{"level":"info","ts":1734424166.2920365,"msg":"Secret already exists on hub cluster, attempting to update","controller":"S3SecretReconciler","secret":"0e34ab2956ef17f55d3fe99876da88c55c4c1aa","namespace":"ibm-baremetal3"}
{"level":"info","ts":1734424166.2955313,"msg":"Successfully updated existing secret on hub cluster","controller":"S3SecretReconciler","secret":"0e34ab2956ef17f55d3fe99876da88c55c4c1aa","namespace":"ibm-baremetal3"}
{"level":"info","ts":1734424166.2955647,"msg":"Successfully reconciled OBC and synced Blue Secret","controller":"S3SecretReconciler","OBC":"openshift-storage/odrbucket-2299ecfe64da875db721bd"}



[rewantsoni qe] k get mirrorpeers.multicluster.odf.openshift.io mirrorpeer-5kvv5 -oyaml | yq '.status'
phase: ExchangedSecret

[rewantsoni]

/assign @umangachapagain @vbnrh

[rewantsoni]

/cherrypick release-4.18

[openshift-cherrypick-robot]

@rewantsoni: once the present PR merges, I will cherry-pick it on top of release-4.18 in a new PR and assign it to you.

In response to this:

/cherrypick release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vbnrh]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rewantsoni, vbnrh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vbnrh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vbnrh]

/hold for @umangachapagain

[vbnrh]

/hold cancel

[openshift-cherrypick-robot]

@rewantsoni: new pull request created: #249

In response to this:

/cherrypick release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.