Merge pull request #113 from canonical/KU-1226/integration-test-for-d…

…isableDefaultCNI-flag Integration test for disable default cni flag
canonical · Sep 9, 2024 · eab4b8b · eab4b8b
2 parents 7f6bc64 + f313dcc
commit eab4b8b
Show file tree

Hide file tree

Showing 6 changed files with 602 additions and 308 deletions.
diff --git a/README.md b/README.md
@@ -68,7 +68,7 @@ make run
 
 As soon as the bootstrap and control-plane controllers are up and running you can apply the cluster manifests describing the desired specs of the cluster you want to provision. Each machine is associated with a MicroK8sConfig through which you can set the cluster's properties. Please review  the available options in [the respective definitions file](./apis/v1beta1/microk8sconfig_types.go). You may also find useful the example manifests found under the [examples](./examples/) directory. Note that the configuration structure followed is similar to the the one of kubeadm, in the MicroK8sConfig you will find a CLusterConfiguration and an InitConfiguration sections. When targeting a specific infrastructure you should be aware of which ports are used by MicroK8s and allow them in the network security groups on your deployment.
 
-Two workload cluster templates are available under the [templates](./templates/) folder, which are actively used to validate releases:
+Two workload cluster templates are available under the [templates](./templates) folder, which are actively used to validate releases:
 - [AWS](./templates/cluster-template-aws.yaml), using the [AWS Infrastructure Provider](https://github.com/kubernetes-sigs/cluster-api-provider-aws)
 - [OpenStack](./templates/cluster-template-openstack.yaml), using the [OpenStack Infrastructure Provider](https://github.com/kubernetes-sigs/cluster-api-provider-openstack)
 

diff --git a/integration/README.md b/integration/README.md
@@ -4,40 +4,10 @@
 
 The integration/e2e tests have the following prerequisites:
 
-  * an environment variable `CLUSTER_MANIFEST_FILE` pointing to the cluster manifest. Cluster manifests can be produced with the help of the templates found under `templates`. For example:
-  ```
-    export AWS_REGION=us-east-1
-    export AWS_SSH_KEY_NAME=capi
-    export CONTROL_PLANE_MACHINE_COUNT=3
-    export WORKER_MACHINE_COUNT=3
-    export AWS_CREATE_BASTION=false
-    export AWS_PUBLIC_IP=false
-    export AWS_CONTROL_PLANE_MACHINE_FLAVOR=t3.large
-    export AWS_NODE_MACHINE_FLAVOR=t3.large
-    export CLUSTER_NAME=test-ci-cluster
-    clusterctl generate cluster ${CLUSTER_NAME} --from "templates/cluster-template-aws.yaml" --kubernetes-version 1.25.0 > cluster.yaml
-    export CLUSTER_MANIFEST_FILE=$PWD/cluster.yaml
-  ```
-
-  *  Additional environment variables when testing cluster upgrades:
-  ```
-    export CAPI_UPGRADE_VERSION=v1.26.0
-    export CAPI_UPGRADE_MD_NAME=${CLUSTER_NAME}-md-0
-    export CAPI_UPGRADE_MD_TYPE=machinedeployments.cluster.x-k8s.io
-    export CAPI_UPGRADE_CP_NAME=${CLUSTER_NAME}-control-plane
-    export CAPI_UPGRADE_CP_TYPE=microk8scontrolplanes.controlplane.cluster.x-k8s.io
-
-    # Change the control plane and worker machine count to desired values for in-place upgrades tests and create a new cluster manifest.
-    CONTROL_PLANE_MACHINE_COUNT=1
-    WORKER_MACHINE_COUNT=1
-    clusterctl generate cluster ${CLUSTER_NAME} --from "templates/cluster-template-aws.yaml" --kubernetes-version 1.25.0 > cluster-inplace.yaml
-    export CLUSTER_INPLACE_MANIFEST_FILE=$PWD/cluster-inplace.yaml
-
-  ```
-
-  * `clusterctl` available in the PATH
-
-  * `kubectl` available in the PATH
+  * make sure to have ssh key in aws `capi`in `us-east-1 region` if you do not have key refer
+    to CAPI on [AWS prerequisites documentation](https://cluster-api-aws.sigs.k8s.io/topics/using-clusterawsadm-to-fulfill-prerequisites#ssh-key-pair)
+
+  * local testing requires the following to be available in the PATH: `clusterctl`, `kubectl`, `helm`
 
   * a management cluster initialised via `clusterctl` with the infrastructure targeted as well as the version of the MicroK8s providers we want to be tested
 
@@ -67,7 +37,7 @@ microk8s config > ~/.kube/config
 
 #### Initialize infrastructure provider
 
-Visit [here](https://cluster-api.sigs.k8s.io/user/quick-start.html#initialization-for-common-providers) for a list of common infrasturture providers.
+Visit [here](https://cluster-api.sigs.k8s.io/user/quick-start.html#initialization-for-common-providers) for a list of common infrastructure providers.
 
 ```bash
   clusterctl init --infrastructure <infra> --bootstrap - --control-plane -
@@ -83,7 +53,7 @@ docker push <username>/capi-bootstrap-provider-microk8s:<tag>
 sed "s,docker.io/cdkbot/capi-bootstrap-provider-microk8s:latest,docker.io/<username>/capi-bootstrap-provider-microk8s:<tag>," -i bootstrap-components.yaml
 ```
 
-Similarly for control-plane provider
+Similarly, for control-plane provider
 ```bash
 cd control-plane
 docker build -t <username>/capi-control-plane-provider-microk8s:<tag> .
@@ -96,6 +66,9 @@ sed "s,docker.io/cdkbot/capi-control-plane-provider-microk8s:latest,docker.io/<u
 ```bash
 kubectl apply -f bootstrap/bootstrap-components.yaml -f control-plane/control-plane-components.yaml
 ```
+### Cluster definitions for e2e
+
+Cluster definitions are stored in the [`manifests`](./cluster-manifests) directory.
 
 #### Trigger the e2e tests
 

diff --git a/integration/cluster-manifests/cluster-disable-default-cni.yaml b/integration/cluster-manifests/cluster-disable-default-cni.yaml
@@ -0,0 +1,117 @@
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: Cluster
+metadata:
+  name: test-ci-cluster
+  namespace: default
+spec:
+  controlPlaneRef:
+    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+    kind: MicroK8sControlPlane
+    name: test-ci-cluster-control-plane
+  infrastructureRef:
+    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+    kind: AWSCluster
+    name: test-ci-cluster
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+kind: AWSCluster
+metadata:
+  name: test-ci-cluster
+  namespace: default
+spec:
+  bastion:
+    enabled: false
+  region: us-east-1
+  sshKeyName: capi
+---
+apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+kind: MicroK8sControlPlane
+metadata:
+  name: test-ci-cluster-control-plane
+  namespace: default
+spec:
+  controlPlaneConfig:
+    clusterConfiguration:
+      portCompatibilityRemap: true
+    initConfiguration:
+      IPinIP: true
+      addons:
+      - dns
+      - ingress
+      confinement: classic
+      disableDefaultCNI: true
+      joinTokenTTLInSecs: 900000
+      riskLevel: stable
+  machineTemplate:
+    infrastructureTemplate:
+      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+      kind: AWSMachineTemplate
+      name: test-ci-cluster-control-plane
+  replicas: 1
+  upgradeStrategy: SmartUpgrade
+  version: v1.27.0
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+kind: AWSMachineTemplate
+metadata:
+  name: test-ci-cluster-control-plane
+  namespace: default
+spec:
+  template:
+    spec:
+      iamInstanceProfile: control-plane.cluster-api-provider-aws.sigs.k8s.io
+      instanceType: t3.large
+      publicIP: false
+      sshKeyName: capi
+---
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: MachineDeployment
+metadata:
+  name: test-ci-cluster-md-0
+  namespace: default
+spec:
+  clusterName: test-ci-cluster
+  replicas: 1
+  selector:
+    matchLabels: null
+  template:
+    spec:
+      bootstrap:
+        configRef:
+          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+          kind: MicroK8sConfigTemplate
+          name: test-ci-cluster-md-0
+      clusterName: test-ci-cluster
+      infrastructureRef:
+        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+        kind: AWSMachineTemplate
+        name: test-ci-cluster-md-0
+      version: 1.27.0
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+kind: AWSMachineTemplate
+metadata:
+  name: test-ci-cluster-md-0
+  namespace: default
+spec:
+  template:
+    spec:
+      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
+      instanceType: t3.large
+      publicIP: false
+      sshKeyName: capi
+---
+apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+kind: MicroK8sConfigTemplate
+metadata:
+  name: test-ci-cluster-md-0
+  namespace: default
+spec:
+  template:
+    spec:
+      clusterConfiguration:
+        portCompatibilityRemap: true
+      initConfiguration:
+        confinement: classic
+        disableDefaultCNI: true
+        riskLevel: stable
diff --git a/integration/cluster-manifests/cluster-inplace.yaml b/integration/cluster-manifests/cluster-inplace.yaml
@@ -0,0 +1,115 @@
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: Cluster
+metadata:
+  name: test-ci-cluster
+  namespace: default
+spec:
+  controlPlaneRef:
+    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+    kind: MicroK8sControlPlane
+    name: test-ci-cluster-control-plane
+  infrastructureRef:
+    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+    kind: AWSCluster
+    name: test-ci-cluster
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+kind: AWSCluster
+metadata:
+  name: test-ci-cluster
+  namespace: default
+spec:
+  bastion:
+    enabled: false
+  region: us-east-1
+  sshKeyName: capi
+---
+apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+kind: MicroK8sControlPlane
+metadata:
+  name: test-ci-cluster-control-plane
+  namespace: default
+spec:
+  controlPlaneConfig:
+    clusterConfiguration:
+      portCompatibilityRemap: true
+    initConfiguration:
+      IPinIP: true
+      addons:
+      - dns
+      - ingress
+      confinement: classic
+      joinTokenTTLInSecs: 900000
+      riskLevel: stable
+  machineTemplate:
+    infrastructureTemplate:
+      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+      kind: AWSMachineTemplate
+      name: test-ci-cluster-control-plane
+  replicas: 1
+  upgradeStrategy: SmartUpgrade
+  version: v1.27.0
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+kind: AWSMachineTemplate
+metadata:
+  name: test-ci-cluster-control-plane
+  namespace: default
+spec:
+  template:
+    spec:
+      iamInstanceProfile: control-plane.cluster-api-provider-aws.sigs.k8s.io
+      instanceType: t3.large
+      publicIP: false
+      sshKeyName: capi
+---
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: MachineDeployment
+metadata:
+  name: test-ci-cluster-md-0
+  namespace: default
+spec:
+  clusterName: test-ci-cluster
+  replicas: 1
+  selector:
+    matchLabels: null
+  template:
+    spec:
+      bootstrap:
+        configRef:
+          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+          kind: MicroK8sConfigTemplate
+          name: test-ci-cluster-md-0
+      clusterName: test-ci-cluster
+      infrastructureRef:
+        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+        kind: AWSMachineTemplate
+        name: test-ci-cluster-md-0
+      version: 1.27.0
+---
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
+kind: AWSMachineTemplate
+metadata:
+  name: test-ci-cluster-md-0
+  namespace: default
+spec:
+  template:
+    spec:
+      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
+      instanceType: t3.large
+      publicIP: false
+      sshKeyName: capi
+---
+apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+kind: MicroK8sConfigTemplate
+metadata:
+  name: test-ci-cluster-md-0
+  namespace: default
+spec:
+  template:
+    spec:
+      clusterConfiguration:
+        portCompatibilityRemap: true
+      initConfiguration:
+        confinement: classic
+        riskLevel: stable