Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deploy vpa-admission-controller failed in kind cluster #6126

Closed
yf4n opened this issue Sep 22, 2023 · 6 comments
Closed

deploy vpa-admission-controller failed in kind cluster #6126

yf4n opened this issue Sep 22, 2023 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@yf4n
Copy link

yf4n commented Sep 22, 2023

Which component are you using?:
vertical-pod-autoscaler

What version of the component are you using?:
use tag cluster-autoscaler-1.28.0

Component version:
vpa-admission-controller: 0.14.0

What k8s version are you using (kubectl version)?:

kubectl version Output
$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

What environment is this in?:

Linux workstation 6.5.4-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 21 Sep 2023 11:06:39 +0000 x86_64 GNU/Linux

deploy cluster with kind. kind image: kindest/node: v1.27.3

OpenSSL 3.1.3 19 Sep 2023 (Library: OpenSSL 3.1.3 19 Sep 2023)
What did you expect to happen?:

I want to deploy vpa components, but vpa-admission-controller not working.

Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    8m29s                 default-scheduler  Successfully assigned kube-system/vpa-admission-controller-9b8db6df-krqdg to kind-worker
  Warning  FailedMount  113s (x3 over 6m26s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[tls-certs], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
  Warning  FailedMount  15s (x12 over 8m29s)  kubelet            MountVolume.SetUp failed for volume "tls-certs" : secret "vpa-tls-certs" not found

I clone repo and checkout tag cluster-autoscaler-1.28.0, run script /vertical-pod-autoscaler/hack/vpa-up.sh to deploy vpa. The script generate certs for admission controller failed with error Error adding extensions defined via -addext. I paste log below

[work@workstation hack]$ sh vpa-up.sh
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictioner-binding created
serviceaccount/vpa-admission-controller created
serviceaccount/vpa-recommender created
serviceaccount/vpa-updater created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
deployment.apps/vpa-updater created
deployment.apps/vpa-recommender created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Error adding extensions defined via -addext
40F75A36787F0000:error:0580008C:x509 certificate routines:X509at_add1_attr:duplicate attribute:crypto/x509/x509_att.c:86:
deployment.apps/vpa-admission-controller created
service/vpa-webhook created

What happened instead?:

How to reproduce it (as minimally and precisely as possible):

just deploy with vpa-up.sh

Anything else we need to know?:

all of these is running in virtual machine.

@yf4n yf4n added the kind/bug Categorizes issue or PR as related to a bug. label Sep 22, 2023
@yf4n
Copy link
Author

yf4n commented Sep 22, 2023

it seems script vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh failed at L54, so secret not be create

openssl req -new -key ${TMP_DIR}/serverKey.pem -out ${TMP_DIR}/server.csr -subj "/CN=vpa-webhook.kube-system.svc" -config ${TMP_DIR}/server.conf -addext "subjectAltName = DNS:vpa-webhook.kube-system.svc"
openssl x509 -req -in ${TMP_DIR}/server.csr -CA ${TMP_DIR}/caCert.pem -CAkey ${TMP_DIR}/caKey.pem -CAcreateserial -out ${TMP_DIR}/serverCert.pem -days 100000 -extensions SAN -extensions v3_req -extfile ${TMP_DIR}/server.conf
echo "Uploading certs to the cluster."
kubectl create secret --namespace=kube-system generic vpa-tls-certs --from-file=${TMP_DIR}/caKey.pem --from-file=${TMP_DIR}/caCert.pem --from-file=${TMP_DIR}/serverKey.pem --from-file=${TMP_DIR}/serverCert.pem
.

@rascalking
Copy link

If anyone else hits this before a fix lands, it seems like the problem is that the subjectAltName = DNS:vpa-webhook.kube-system.svc attribute is being specified both on the command line (as @f4nd0y mentioned) and in the server.conf.

The workaround that worked for me was removing -addext "subjectAltName = DNS:vpa-webhook.kube-system.svc" from the command line.

@yf4n
Copy link
Author

yf4n commented Oct 7, 2023

#6149 fixed this issus.

@yf4n yf4n closed this as completed Oct 7, 2023
@mrsymlove
Copy link

mrsymlove commented Oct 20, 2023

The workaround that worked for me was removing -addext "subjectAltName = DNS:vpa-webhook.kube-system.svc" from the command line.

Despite the issue is fixed on head, you will need this workaround if you have to install an older VPA version (in my case 0.11) due to the kubernetes version compatibility.

@tom10271
Copy link

tom10271 commented Jan 19, 2024

Not 100% sure what is going on but I was trying to install the VPA today with latest master branch and result in fail. In autoscaler/vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh there are -addext "subjectAltName = DNS:${CN_BASE}_ca".

I can install everything after removing the -addext part.

I am using openssl 3.x and -addext exist in ~1.1 only
https://www.openssl.org/docs/man1.1.1/man1/req.html

@yf4n
Copy link
Author

yf4n commented Jan 26, 2024

Not 100% sure what is going on but I was trying to install the VPA today with latest master branch and result in fail. In autoscaler/vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh there are -addext "subjectAltName = DNS:${CN_BASE}_ca".

I can install everything after removing the -addext part.

I am using openssl 3.x and -addext exist in ~1.1 only https://www.openssl.org/docs/man1.1.1/man1/req.html

subjectAltName has been set in server.conf, some version of openssl will throw a error.

you can remove -addext "subjectAltName=xxx" or upgrade openssl version. it can work with alpine/openssl:latest image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

4 participants