podman run fails to start a container with OCI permission denied #24100
Labels
kind/bug
Categorizes issue or PR as related to a bug.
Comments
|
I am not able to reproduce locally. Any special reason for using Can you please share the output of |
|
Hi @giuseppe Interesting that you can't reproduce it, gives me hope that it's just bad configuration on my side potentially. To be clear, I'm just running the default No special reason for using Sure, here's the mountinfo |
|
I've tried the following command: is that the equivalent of what you are running? |
|
yes it is, and I can verify that running this command locally on my machine indeed works. So it must be the Kubernetes pod that makes it not work then? And indeed, when I deploy this pod into a Kubernetes cluster: and exec into it and try running So this only happens when the container is running in Kubernetes? Are there some missing privileges/capabilities needed then? |
|
how was the Kubernetes cluster created? Is it running with full privileges on the host? Could you please also show the content of |
|
Nothing special, default EKS cluster in AWS. I do not think it is the cluster configuration at fault TBH, when I change the image in the pod to Here are the contents of those files. I hope you meant from the running container, I can also get them from the node. |
|
I did a bit more testing and can confirm that this only happens for Podman versions 5.2.0, 5.2.1, 5.2.2 and 5.2.3. The last version of Podman where this is not happening is 5.1.2. Unfortunately this might not really help narrowing down where the issue lies, as 5.2.0 had quite a few changes included in comparison to 5.1.2. |
|
I've also tried running it with It seems that the |
|
I wonder if this is caused by c81f075 and the kernel is too old |
|
Could be :) Not sure which kernel version contains the new mount API, but there are newer versions of the Linux kernel, so you're probably right. Anyway, I'll very likely refrain from building a custom AMI for EKS with an updated Linux kernel. So I guess the options are either to make podman compatible with older Linux kernel versions, or I'll wait until Amazon Linux 2 updates the kernel to the newer version, restricting the podman version to 5.1.2 for that period. Is that correctly understood @giuseppe or do you see other options? |
|
it would be nice to understand find if that is really the issue :-) Seems a bit strange because the new mount API pieces should be there (but I don't know if there are other needed changes in the file system itself). Since you've access to the machine, could you please run: and share the |
|
So I ran |
|
thanks! I had expected to find a call to This is what I get locally: Another workaround could be to map the root user inside the user namespace, adding something like: |
|
Indeed it is using or more in I'm not sure why the open_tree calls are not present in the log file... |
|
interesting, they succeeded! They are not in the log file because with What fails is the static int
get_bind_mount (int dirfd, const char *src, bool recursive, bool rdonly, libcrun_error_t *err)
{
cleanup_close int open_tree_fd = -1;
struct mount_attr_s attr = {
0,
};
int recursive_flag = (recursive ? AT_RECURSIVE : 0);
int ret;
if (rdonly)
attr.attr_set = MS_RDONLY;
errno = 0;
open_tree_fd = syscall_open_tree (dirfd, src,
AT_NO_AUTOMOUNT | OPEN_TREE_CLOEXEC
| OPEN_TREE_CLONE | recursive_flag);
if (UNLIKELY (open_tree_fd < 0))
return crun_make_error (err, errno, "open_tree `%s`", src);
ret = syscall_mount_setattr (open_tree_fd, "", AT_EMPTY_PATH | recursive_flag, &attr); // <---- FAIL HERE
if (UNLIKELY (ret < 0))
return crun_make_error (err, errno, "mount_setattr `%s`", src);
return get_and_reset (&open_tree_fd);
}The |
|
Right, thanks a lot for debugging this :) So it is the Linux kernel version. What would be the appropriate action here? Do I wait until the new kernel version is supported in Amazon Linux 2 using podman 5.1.2 in the meantime? Or will podman 5.2+ support older linux kernels? |
|
I'd suggest either waiting/using a new kernel, or injecting the root user as part of the mappings (add |
|
Thanks a lot! :) the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue Description
When running a
podman runcommand with--uidmapand--network host, the container fails to start withSteps to reproduce the issue
Steps to reproduce the issue
podman run --uidmap 0:2300:1 --network host docker.io/library/ubuntuDescribe the results you received
An error appears:
Describe the results you expected
The container should start and finish successfully
podman info output
Podman in a container
Yes
Privileged Or Rootless
Privileged
Upstream Latest Release
Yes
Additional environment details
No response
Additional information
This used to work in podman 5.0.X, my hope was that the bugfix for idmap included in v5.2.3 would fix this, but that does not seem to be the case.
The text was updated successfully, but these errors were encountered: