Add automated testing of AWS templates, rework existing e2e testing to use kubeclient

[squizzi]

This PR:

• Reworks the existing e2e tests so that we no longer use iterations of exec.Command and instead use a new KubeClient type that implements kubernetes.Interface
• Adds new Makefile targets make dev-aws-nuke and make cloudnuke to purge AWS resources affiliated with a desired deployment name, whether that's the test created one or the one created by config/dev/deployment.yaml
• Utilizes this new Makefile target to purge deployed AWS resources during teardown.
• Adds a series of provider-agnostic test validators to verify creation and deletion; which can be used to validate a provider deployed cluster, such as the newly added AWS tests.

Closes: #211
See: #201

[squizzi]

Any thoughts on what the timeout should be when validating resources coming online from a provider? I've seen AWS take up to 20 min sometimes to deploy resources, I currently have it set to 30 min with 10 second polling.

[squizzi]

👋 this can be reviewed, but do note the aws-hosted-cp template is still not functional so for now I'm skipping that test while I continue to debug it for inclusion. This way the provider-agnostic verification code can be potentially committed and this is no longer blocking other provider testing initiatives.

[squizzi]

I don't know what's going on with the AWS key/secret that IT added, for whatever reason clusterawsadm is barfing when it tries to encode the credentials but make test-e2e works locally for me.

Edit: womp womp

Run echo $(aws sts get-caller-identity)

Unable to locate credentials. You can configure credentials by running "aws configure".

---

Local run of make test-e2e (with some lines omitted for brevity):

$ make test-e2e
make test-e2e
KIND_CLUSTER_NAME="hmc-test" KIND_VERSION=v0.23.0 go test ./test/e2e/ -v -ginkgo.v -timeout=2h
=== RUN   TestE2E
  Starting hmc suite
Running Suite: e2e suite - /home/squizzi/go/src/github.com/Mirantis/hmc/test/e2e
================================================================================
Random Seed: 1725033619

Will run 3 of 3 specs
------------------------------
controller Operator should run successfully
/home/squizzi/go/src/github.com/Mirantis/hmc/test/e2e/e2e_test.go:59
  STEP: building and deploying the controller-manager @ 08/30/24 09:00:19.688
  running: make test-apply
  running: ./bin/clusterawsadm bootstrap credentials encode-as-profile
  STEP: validating that the hmc-controller and capi provider controllers are running @ 08/30/24 09:01:49.489
  Controller pod validation failed: expected 1 hmc-controller-manager controller pod, got 2
  Controller pod validation failed: expected 1 hmc-controller-manager controller pod, got 2
  Controller pod validation failed: expected 1 cluster-api controller pod, got 0
  Controller pod validation failed: expected 1 cluster-api controller pod, got 0
  Controller pod validation failed: expected 1 hmc-controller-manager controller pod, got 2
  Controller pod validation failed: expected 1 cluster-api controller pod, got 0
  Controller pod validation failed: expected 1 cluster-api controller pod, got 0
  Controller pod validation failed: controller pod: capi-controller-manager-7f49f575d4-r7dkh in Pending status
  Controller pod validation failed: expected 1 infrastructure-aws controller pod, got 0
  Controller pod validation failed: expected 1 infrastructure-aws controller pod, got 0
  Controller pod validation failed: controller pod: capa-controller-manager-bc5cbb7b7-mpf74 in Pending status
  Controller pod validation failed: expected 1 infrastructure-azure controller pod, got 0
  Controller pod validation failed: expected 1 infrastructure-azure controller pod, got 0
  Controller pod validation failed: expected 1 infrastructure-azure controller pod, got 0
  Controller pod validation failed: controller pod: capz-controller-manager-58677bc6fd-wcm2b in Pending status
• [239.989 seconds]
------------------------------
controller AWS Templates should work with an AWS provider and aws-standalone-cp template
/home/squizzi/go/src/github.com/Mirantis/hmc/test/e2e/e2e_test.go:136
  STEP: ensuring AWS credentials are set @ 08/30/24 09:04:19.677
  STEP: creating a Deployment @ 08/30/24 09:04:19.68
  Generated cluster name: "1a5b1a9c-e2e-test"
  running: ./bin/clusterawsadm ami list --os=ubuntu-20.04 -o json
  STEP: waiting for infrastructure providers to deploy successfully @ 08/30/24 09:04:21.233
  [clusters] validation error: failed to get clusters 1a5b1a9c-e2e-test: clusters.cluster.x-k8s.io "1a5b1a9c-e2e-test" not found
  [clusters] validation error: Cluster 1a5b1a9c-e2e-test is not ready with conditions:
  [clusters] validation error: Cluster 1a5b1a9c-e2e-test is not ready with conditions:
  Type: ControlPlaneReady, Status: False, Reason: WaitingForControlPlane, Message: 
  Type: ControlPlaneInitialized, Status: False, Reason: WaitingForControlPlaneProviderInitialized, Message: Waiting for control plane provider to indicate the control plane has been initialized
  Type: Ready, Status: False, Reason: WaitingForControlPlane, Message: 
  [clusters] validation error: Cluster 1a5b1a9c-e2e-test is not ready with conditions:
  Type: ControlPlaneReady, Status: False, Reason: WaitingForControlPlane, Message: 
  Type: ControlPlaneInitialized, Status: False, Reason: WaitingForControlPlaneProviderInitialized, Message: Waiting for control plane provider to indicate the control plane has been initialized
  Type: Ready, Status: False, Reason: WaitingForControlPlane, Message: 
  [clusters] validation succeeded
  [machines] validation succeeded
  [control-planes] validation succeeded
  [csi-driver] validation error: 1a5b1a9c-e2e-test-csi-test-pvc PersistentVolume not yet 'Bound', current phase: "Pending"
  [csi-driver] validation error: 1a5b1a9c-e2e-test-csi-test-pvc PersistentVolume not yet 'Bound', current phase: "Pending"
  [csi-driver] validation error: 1a5b1a9c-e2e-test-csi-test-pvc PersistentVolume not yet 'Bound', current phase: "Pending"
  [csi-driver] validation succeeded
  [ccm] validation error: loadbalancer-1a5b1a9c-e2e-test Service does not yet have an external hostname
  [ccm] validation error: loadbalancer-1a5b1a9c-e2e-test Service does not yet have an external hostname
  [ccm] validation succeeded

Can you take another look? Outside of controller-runtime being used as the client I think I've addressed all your concerns. We can always change the client up later but it would be a non-trivial rework of this PR to modify it now and this is blocking a few people at this point.

[squizzi]

I need to update the Deployment types to ManagedCluster as well for the dynamic client, missed that.

[a13x5]

@squizzi Yes the choose of client is not that important. The rest looks good.