Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

caa-provisioner-cli -action=install has odd dependency on cmctl #2207

Open
gcoon151 opened this issue Dec 11, 2024 · 12 comments
Open

caa-provisioner-cli -action=install has odd dependency on cmctl #2207

gcoon151 opened this issue Dec 11, 2024 · 12 comments
Labels
bug Something isn't working

Comments

@gcoon151
Copy link

gcoon151 commented Dec 11, 2024

Describe the bug

I'll look at finding/fixing also but I wanted to capture a note about something odd. I'm on Mac OS Sonoma 14.6.1 following recently updated instructions to deploy on OCP in IBM cloud. It got to here and died:

validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
# Wait for service to be up
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager condition met
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager-webhook --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager-webhook condition met
# Wait for few seconds for the cert-manager API to be ready
# otherwise you'll hit the error "x509: certificate signed by unknown authority"
# Best is to use cmctl - https://cert-manager.io/docs/installation/kubectl/#2-optional-wait-for-cert-manager-webhook-to-be-ready
./cmctl check api --wait=2m
bash: ./cmctl: cannot execute binary file
make: *** [deploy-cert-manager] Error 126 
FATA[0061] exit status 2   

I hadn't installed or even realized at that pointed I would need cmctl. brew install cmctl && cmctl check api --wait=2m
The cert-manager API is ready
Assuming it had completed like the output said, I rerun ./caa-provisioner-cli -action=install
where it again fails same way

validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
# Wait for service to be up
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager condition met
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager-webhook --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager-webhook condition met
# Wait for few seconds for the cert-manager API to be ready
# otherwise you'll hit the error "x509: certificate signed by unknown authority"
# Best is to use cmctl - https://cert-manager.io/docs/installation/kubectl/#2-optional-wait-for-cert-manager-webhook-to-be-ready
./cmctl check api --wait=2m
bash: ./cmctl: cannot execute binary file
make: *** [deploy-cert-manager] Error 126 
FATA[0061] exit status 2    

Is it trying to run cmctl from literally whatever directory I'm in? I'll help chase this down when I have a moment. I just didn't want to forget to open an issue about this.

How to reproduce

I was going through these instructions https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/ibmcloud/ROKS_SETUP.md and got all the way to
( ~/gits/cloud-api-adaptor/src/cloud-api-adaptor/test/tools/) $ ./caa-provisioner-cli -action=install

CoCo version information

main from this repo.

What TEE are you seeing the problem on

Tdx

Failing command and relevant log output

( ~/gits/cloud-api-adaptor/src/cloud-api-adaptor/test/tools/) $  ./caa-provisioner-cli -action=install

...

validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
# Wait for service to be up
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager condition met
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager-webhook --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager-webhook condition met
# Wait for few seconds for the cert-manager API to be ready
# otherwise you'll hit the error "x509: certificate signed by unknown authority"
# Best is to use cmctl - https://cert-manager.io/docs/installation/kubectl/#2-optional-wait-for-cert-manager-webhook-to-be-ready
./cmctl check api --wait=2m
bash: ./cmctl: cannot execute binary file
make: *** [deploy-cert-manager] Error 126 
FATA[0061] exit status 2
@gcoon151 gcoon151 added the bug Something isn't working label Dec 11, 2024
@stevenhorsman
Copy link
Member

I think this is because the curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_linux_${ARCH} assumes you are running linux, not mac, I'm guessing you might need darwin_arm64 or something instead?

@gcoon151
Copy link
Author

I think you might be right. I think Frank ran into similar and extended arch to support what the target cluster would have as arch. Perhaps we need to have CLUSTER_ARCH and CLIENT_ARCH. I'll look more in there on mine too and see if I can confirm this and come up with a fix PR for you.

@gcoon151
Copy link
Author

I actually just noticed this which might help

make BUILTIN_CLOUD_PROVIDERS="ibmcloud" all
GOOS=darwin GOARCH=arm64 CGO_ENABLED=0 go build -tags=ibmcloud -o caa-provisioner-cli ./provisioner-cli/

Seems like those vars could be reused for cmdctl?

@gcoon151
Copy link
Author

gcoon151 commented Dec 12, 2024

I have another option in the works also of just examining the curl we're using to get cmctl and just match that platform. Let me test out both and I'll send over a PR on this to fix this. Something like

curl --version | awk 'NR==1 && match($0, /\(.*\)/) {
    platform=substr($0, RSTART+1, RLENGTH-2); 
    split(platform, a, "-"); 
    if (length(a) >= 3) { 
        os=a[3]; arch=a[1]; 
        if (arch == "x86_64") arch="amd64"; 
        else if (arch ~ /^(arm64|aarch64)$/) arch="arm64"; 
        if (os ~ /^(linux|darwin|windows)$/) print os "_" arch; 
        else print "unsupported"; 
    } else print "unsupported";
}'

This output generates based on the curl string something like linux_amd64 which can be assigned to a variable CLIENT_ARCH? and then get the right file. Just saving the thought here for my own notes. I'll PR this up as a suggestion.

@stevenhorsman
Copy link
Member

Can you do just do something like:

curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_$(uname | tr '[:upper:]' '[:lower:]')_${ARCH}

to select the correct os for the download? It seems much simpler than your curl approach?

@gcoon151
Copy link
Author

ARCH has to be defined by something and right now I think we're only letting you define the target platform for the podvm image which doesn't match the orchestrator/laptop that's instantiating caa. The core problem is my laptop is ARM so I need cmctl for ARM but I'm deploying to amd64 which is what I pass the caa-provisioner-cli

@stevenhorsman
Copy link
Member

ARCH is defined here:

ARCH ?= $(subst x86_64,amd64,$(shell uname -m))
and is using uname -m, so we match the machine it is running on?

@gcoon151
Copy link
Author

I'm rebuilding the env today where I have some tdx hosts. When I get to that part I'll examine what my ARCH is set to and dig deeper on why it didn't get the right thing. I thought it was defined from the flag Frank added to specify arch.

@gcoon151
Copy link
Author

I just noticed earlier, you said we grab curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_linux_${ARCH} (note "linux") when I look here https://github.com/cert-manager/cmctl/releases/ the one I need is darwin_arm64. uname -m only gives you the arm64. I think the problem might be a little further left in the URL where we presume linux ... which you can also get from the curl if we decide to do that.

Golang defines OS and ARCH. Seems this Makefile only defines ARCH and assumes linux.

@gcoon151
Copy link
Author

Oh I should add one reason I went with the curl specifics to identify OS/ARCH is I tested what the uname output would look like if someone ran this code in Ubuntu on WSL on Windows. Obviously not supported today but that's what prompted me to consider a wider method of defining these.

@stevenhorsman
Copy link
Member

I just noticed earlier, you said we grab curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_linux_${ARCH} (note "linux") when I look here https://github.com/cert-manager/cmctl/releases/ the one I need is darwin_arm64. uname -m only gives you the arm64. I think the problem might be a little further left in the URL where we presume linux ... which you can also get from the curl if we decide to do that.

Yes, this is why I have $(uname | tr '[:upper:]' '[:lower:]') in my suggested fix:

$ uname | tr '[:upper:]' '[:lower:]'
darwin

@gcoon151
Copy link
Author

I missed that and confirmed on ubuntu on wsl which sent me down this path that'd return linux also so that would be a good approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants