OpenShift Compatibility for Jenkins Operator

[syanpriyajot]

Describe the solution you'd like
Since v0.6.0, it was mentioned that Jenkins Operator is not compatible with OpenShift.
https://github.com/jenkinsci/kubernetes-operator/releases/tag/v0.6.0

• May I know why is the compatibility with OpenShift removed?
• Can we please make this operator compatible with OpenShift?

Describe alternatives you've considered
Is there working documentation for OpenShift using which I can test the compatibility of latest release on OpenShift?
Or should exactly the same documentation as for Kubernetes work on OpenShift as well?

@brokenpip3

[brokenpip3]

Generally speaking I don't think we have something specific that will let the operator to not run correctly in openshift.
I believe that in the past the team supported the jenkins version shipped by redhat which is different from the mainline jenkins lts that we use in our operator, so probably the removal was related to that.

Coming back to ocp: I belive that we may have some issue if we do not run the jenkins pod with an scc that will grants the anyuid, or the specific jenkins image id to be set as running user of the container. I'm not sure, we need to test it to understand if my guess will be true. But apart of this I do not expect any other critical issue in running vs ocp.

Now the real problem is: testing.
Minishift is stuck at ocp 3.11, we need to find a way to test it via a github actions in ocp 4.x.
Will try to build something after a small spring break

[kumadee]

Generally speaking I don't think we have something specific that will let the operator to not run correctly in openshift. I believe that in the past the team supported the jenkins version shipped by redhat which is different from the mainline jenkins lts that we use in our operator, so probably the removal was related to that.

Coming back to ocp: I belive that we may have some issue if we do not run the jenkins pod with an scc that will grants the anyuid, or the specific jenkins image id to be set as running user of the container. I'm not sure, we need to test it to understand if my guess will be true. But apart of this I do not expect any other critical issue in running vs ocp.

Now the real problem is: testing. Minishift is stuck at ocp 3.11, we need to find a way to test it via a github actions in ocp 4.x. Will try to build something after a small spring break

@brokenpip3 I don't think RedHat will support minishift for ocp 4.x any longer. They have moved on to https://github.com/crc-org/crc for ocp 4.x, due to lot of changes in installation.

[brokenpip3]

yes I saw that but the crc will require 16gb of memory to run and the github runner cannot handle more than 7gb (and we do no have any free credits here to use the enterprise runners).
So we need to find a way to test the installation in openshift, if someone has other ideas I can take a look on ocp 4.x and fix all the issues but I cannot run it freely anywhere :(

[kumadee]

I followed the issue crc-org/crc#557 and it led to this issue crc-org/crc#2507, which aims to reduce the resource requirements of OCP 4.8. I think we are blocked until this issue has some meaningful progress.

There were some experimental changes done by crc team crc-org/snc#241 to do this but doesn't look ready for our use here.

[brokenpip3]

If someone can try to deploy it in openshift and send here the logs, decribe, evetns will be great, I can take a look and try a fix if needed (90% will be a scc issue)

[akilada]

@brokenpip3
I have tried to implement jenkins operator v0.60 on openshift 4.11. However, the jenkins instance pod keeps getting terminated.

The following are my deployment steps:

• Followed https://jenkinsci.github.io/kubernetes-operator/docs/getting-started/v0.6.x/installing-the-operator/. I used the install using YAML.
• Followed https://jenkinsci.github.io/kubernetes-operator/docs/getting-started/v0.6.x/deploying-jenkins/ to deploy jenkins instance.
• Also, I have created AnyUID security context for the jenkins instance.

However, the jenkins instance pod keeps getting terminated soon after initiation. The following are the logs from jenkins operator pod.

jenkins-operator-v0.60.log
events.log
jenkins-master-describe.log
jenkins-operator-describe.log

[brokenpip3]

@akilada could you please try by using the latest version (0.8.0)?
the 0.6.0 it's a very old version and contains issues that make impossible to run it everywhere

[BeastHook]

@brokenpip3
I have tried it with the latest operator version on a 4.13 Openshift cluster.
The master pod keeps terminating.

I used the example CR in your doc.

Here is the log of the terminated pod:
jenkins-example-jenkins-jenkins-master.log

The rest looks similar to @akilada logs

[jonesbusy]

This is similar to deployment of the helm chart on Openshift/OKD

You just need to adapt some environment vars for the plugin manager tool to download to a writtable location

Like jenkinsci/helm-charts#506 (comment)

[brokenpip3]

in our case it's slightly better you can choose to:

• change the env to a specific tmp dir and add a tmp dir in the volume/volumemounts
• change the env to something like /tmp/dir as per @jonesbusy example
• mount /usr/share/jenkins/ as tempdir

In any cases we should change our helmchart, or even better the code to always use /tmp or a specific emptydir to download the plugins, both in openshift or not openshift scenario.
Also I wish we can harden by default our securitycontext (with readonlyfs for instance) but in this period I do not have the amount of free time to add these changes, will try in the near future. In the mean time any PR is welcomed :)

[BeastHook]

That worked out great.
I had to correct some plugins to higher versions because I use a newer LTS version of Jenkins.
The container in the pod comes up to the startup of Jenkins and terminates without any further action after the log line "Jenkins is fully up and running".

jenkins-example-jenkins-master .log

[brokenpip3]

Can you also attach the operator log?

[brokenpip3]

Also kubernetes events if possible

[BeastHook]

Atm the Pod keeps getting a different problem:
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://ftp.halifax.rwth-aachen.de:443: Network is unreachable

Operator log:
jenkins-operator-6cdbd9f879-52nkg-jenkins-operator.log

I dont have a eventrouter on the Openshift running.
The events view shows:

[brokenpip3]

to me seems that kubelet that is terminating the pod

I dont have a eventrouter on the Openshift running.

you just need to run oc get ev -n <namespace or kubectl

also this part of the code is something I never touched, and will be hard to have a testing env, its seems kind of broken, I don't think we should handle the ocp route for the user, like we do with the ingress

2024-04-19T12:36:46.121Z DEBUG controller-jenkins Route API is available. Now creating route. {"cr": "msg"}
2024-04-19T12:36:46.132Z DEBUG controller-jenkins Jenkins Route is present {"cr": "msg"}

[BeastHook]

Good morning,

this is what i get. not really meaningful...
Eventlog_jenkins.txt

Which parameter can I use to increase the log level?

The created route looks fine for me.

[brokenpip3]

the issue is this one:

Jenkins master pod restarted by operator:; Jenkins pod security context has changed; Security context has changed

I don't know who is changing the security context of the pod.
Right now it seems that we have:

• an issue with the plugin download directory that is not declared as emptyDir in the manifest
• the operator that is trying all the time to create a route and say that is not present
• a securitycontext change that will restart the pod all the time

I wish I can take a look but since I can't afford a ocp cluster it will be very hard to have a proper support for openshift 4.x.

[brokenpip3]

Good news: there may be a company that will provide us some ocp compute time to verify and implement the openshift compatibility, will share more news as soon I will have more info