Make 1.7/stable work in an airgap environment

[DnPlas]

Bug Description

The goal of this task is to make the CKF 1.7/stable work in airgap environments, which means that the CKF deployment should work on an environment without internet access.

This issue will help tracking all the required changes on individual charms, as well as ensuring when integrated as a bundle, everything can be deployed and tested.

What needs to be done?

• Make sure images are configurable for all charms in the track/<versio>
• Make sure all charms are promoted to <version>/edge and <version>/stable
• Test the bundle in an airgap environment (steps here Extend tests and automation scripts for airgapped installations #682)
• Replicate this structure in the releases/1.7/edge directory

State of charms

	Status	State of the CI
canonical/admission-webhook-operator	canonical/admission-webhook-operator#113	Green
canonical/argo-operators	canonical/argo-operators#136	Green
canonical/dex-auth-operator	canonical/dex-auth-operator#167	Green
canonical/istio-operators	canonical/istio-operators#332	Green
canonical/katib-operators	canonical/katib-operators#144	Green
canonical/kfp-operators	canonical/kfp-operators#335	Green
canonical/knative-operators	canonical/knative-operators#159	Green
canonical/kserve-operators	canonical/kserve-operators#179	Green
canonical/kubeflow-dashboard-operator	canonical/kubeflow-dashboard-operator#165	Green
canonical/kubeflow-profiles-operator	canonical/kubeflow-profiles-operator#152	Green
canonical/kubeflow-roles-operator	canonical/kubeflow-roles-operator#72	Green
canonical/kubeflow-tensorboards-operator	canonical/kubeflow-tensorboards-operator#94	Failing to publish
canonical/kubeflow-volumes-operator	canonical/kubeflow-volumes-operator#116	Green
canonical/metacontroller-operator	canonical/metacontroller-operator#90	Green
canonical/minio-operator	canonical/minio-operator#149	Green
canonical/notebook-operators	canonical/notebook-operators#307	Green
canonical/oidc-gatekeeper-operator	canonical/oidc-gatekeeper-operator#113	Green
canonical/seldon-core-operator	canonical/seldon-core-operator#212	Green
canonical/training-operator	canonical/training-operator#141	Green

Potential issues

#688
#690

[orfeas-k]

Looks like knative-operator telemetry image is not configurable at the moment neither in main nor track/1.8 canonical/knative-operators#161.

[DnPlas]

Looks like knative-operator telemetry image is not configurable at the moment neither in main nor track/1.8 canonical/knative-operators#161.

These two patches should fix that:

• feat: make otel-collector image configurable knative-operators#162
• feat: make otel-collector image configurable knative-operators#163

[orfeas-k]

Adding the note here as well that during #709, the following image 172.17.0.2:5000/knative-releases/knative.dev/serving/cmd/default-domain:2b3e4074f951dd6b6fa5018047e97a5cb10b6a6ecbb554ed77c9a8dc01011f3b was the only image (between the ones produced by the scripts) that wasn't used in the charms' configuration. We could not find a container that needs that and this is not configured in latest/edge bundle-airgap.yaml neither. This can be OK since we could fetch an image we don't need in our script. We should though keep it in the back of our minds in case we see something's off with knative-serving's deployments.
Note that I also do not see it anywhere in the serving's manifests.

[NohaIhab]

sent a fix for kubeflow dashboard CI in canonical/kubeflow-dashboard-operator#169

[DnPlas]

Current status

Charms

All charms have been updated and are now published to their /edge tracks in Charmhub. In the case of kubeflow-tensorboards-operator the version in track/1.7 should be compatible with arigap environments.

• Waiting for knative-operators to be published to 1.8/edge
• Waiting for kubeflow-dashboard to be published to 1.7/edge

Bundle and scripts

#709 is introducing the bundle definition for 1.7/edge airgap, but pending things to check on that PR are:

1. The knative-operator should include the otel-collector image
2. We have to confirm that the charm names are accurate to what is downloaded by juju in both bundle-airgap.yaml and podspec_script.sh

Testing

Not started, we can start testing after charms are published to version/edge and we update the bundle PR with the mentioned changes.
@orfeas-k to help execute these tests.

MLFLow integration

Not started, but we need to do a similar effort in the MLFlow bundle: https://github.com/canonical/mlflow-operator/blob/main/releases/2.1/edge/mlflow/bundle.yaml.

• TBC - version of MLFlow, 2.1?
• We have to add a bundle definition for airgap
• We have a get-images script
• TBC - that MLFlow images are configurable

Observability integration

Not started, TBC.

[DnPlas]

Based on a discussion with @kimwnasptd , we decided to add a "MLOps" bundle definition in this repository to help integrate with MLFlow and other charms required for a deployment with CKF and MLFlow. Please make sure you take a look at #712.

[DnPlas]

Based on a discussion with @kimwnasptd , we decided to add a "MLOps" bundle definition in this repository to help integrate with MLFlow and other charms required for a deployment with CKF and MLFlow. Please make sure you take a look at #712.

This effort is not going to be considered for this effort.

[orfeas-k]

Deployed the bundle using files from PR #709 and everything went to green without issues.

Regarding the UATs, we cannot run them at the moment since they need connection to download some extra images that may not be present in the cluster at the moment and also to pip install stuff (similar reason we could not run them in self-hosted runners). Once @NohaIhab checks that UATs run successfully in a non-airgapped 1.7/edge cluster, then we could check "Test the bundle in an airgap environment" and consider this issue task.

[orfeas-k]

I forgot to mention that during testing:

• we noticed this issue istio-pilot goes to Error if the charms related via ingress-auth are not ready istio-operators#336 which is unrelated to airgap
• we observed the same issue with activator deployment in knative-serving namespace that we had also seen in airgap latest/edge Provide bundle definitions for airgapped installations #680 (comment)

[orfeas-k]

I think we could close this @DnPlas

[DnPlas]

Closing as all the required work has been completed. Thanks everyone!