[BUG] error logging into registry: fork/exec /root/.hauler/cosign: operation not permitted

[ertimas]

Environmental Info:

Linux rke2-server 5.15.0-209.161.7.2.el8uek.x86_64 #2 SMP Tue Aug 20 10:44:07 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Hauler Version:

Describe the Bug:

hauler login fails, where cosign login/ctr login/crictl login succeed.

Steps to Reproduce:

1. Create VM with Oracle Linux 8, with a single network interface, connected to the internet
2. curl -sfL https://get.hauler.dev | bash
3. hauler login regsitry1.dso.mil -u <my username> -p <my password>

Expected Behavior:

Receive a login succeeded messaeg.

Actual Behavior:

Here is a screenshot demonstrating hauler failing to login (left terminal) while ctr is able to login and pull an image (right terminal).

Here is an example of using cosign, which was installed along with hauler, to login to the target repository.

Additional Context:

As of ~3 weeks ago I had another VM and was able to install hauler via the command listed above, login to the registry, and add images to my image store. That is to say-- this is new behavior.

[zackbradys]

Hey @ertimas, in your first screenshot, on the left terminal, it looks like you have a typo in the command/flag and that's probably why hauler failed to accept it. You have -password instead of --password or -p.

Please try again and let us know!

[ertimas]

Ah-- I failed to post the correct screenshot

Sorry @zackbradys please see this one

[zackbradys]

Ah that would do it! Unfortunately this is more a limitation than a bug and shouldn't be a new issue to v1.1.0...

Basically hauler is unable to open multiple connections to the cosign binary. If you have multiple concurrent processes running with hauler, then one of them fail. If you stop any other processes using hauler and try the command again, it should work for you. Feel free to check with something like ps aux | grep hauler

[zackbradys]

We're actively looking at how we leverage cosign, since we don't use much of it, and should be addressed relatively soon... our recommendation is try not to run multiple concurrent processes at the same time...

Feel free to add any thoughts to Discussion #358

[ertimas]

So Hauler Fileserver and Hauler Registry can't be run at the same time @zackbradys?

[zackbradys]

No the registry and fileserver can be run at the same time... It is only anything that leverages the cosign binary. I will get a list together and update the docs here -> https://docs.hauler.dev/docs/known-limits#limitations

[ertimas]

Running "multiple concurrent processes" stemmed from the implementation in Hauler it seems since cosign ran fine on its own and I simply ran hauler login to generate the error.

[dweomer]

Hi @ertimas, @zackbradys and I are discussing this on a call. Initially, @zackbradys was thinking you were running into the error originally described in #211 (since captured in #361) but it looks more likely that the problem for you is #249. We are working on a short-term work-around for this embodied in #354

[ertimas]

@dweomer thank you! I'll try #354

[ertimas]

Meant to close

[ertimas]

Just an FYI-- turns out fapolicy was preventing cosign from running... My bad team. Added a rule and it worked immediately

echo "allow perm=executable uid=<user> : path=<user_home/.hauler/cosign" >> /etc/fapolicyd/rules.d/80-cosign-rule.rules