tty device used inside toolbox gets odd permissions on host causing pinentry/gpg-agent to not work

[dustymabe]

Describe the bug

My setup includes running gpg-agent on my host and doing development (git commit, etc) inside my toolbox container. When I git-commit I sign the commits, which means the process talks to the agent that then runs a pinentry program to ask the user for the passphrase.

When running in the terminal I prefer for the passphrase to be asked to me in the terminal window and not in a popup. So I'll unset DISPLAY to force it to use the terminal window. This stopped working sometime in the past few months. I finally tracked down some more information on why.

When running inside toolbox the permissions on my tty device are as expected:

$ ls -l $(tty)
crw--w----. 1 root nobody 136, 14 Sep 27 13:37 /dev/pts/14

However outside of toolbox (from the perspective of the host) the
permissions look odd:

[dustymabe@media ~]$ id
uid=1001(dustymabe) gid=1001(dustymabe) groups=1001(dustymabe) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[dustymabe@media ~]$ ls -l /dev/pts/14
crw--w----. 1 100000 tty 136, 14 Sep 27 14:39 /dev/pts/14

I would expect the owner of the tty to be the user who launched the toolbox container (1001). If I change the ownership to the expected UID then my pinentry program appropriately asks for the passphrase on the same terminal window (inside toolbox) where I am running git commit.

I'm running Fedora 33 with:

$ rpm -q toolbox podman
toolbox-0.0.95-1.fc33.x86_64
podman-2.1.0-0.187.rc2.fc33.x86_64

Indeed if I go back to an old machine I haven't updated in a while I can see that the permissions are what I would expect them to be. This out of date machine has:

[dustymabe@hattop ~]$ rpm -q toolbox podman 
toolbox-0.0.18-1.fc31.noarch
podman-1.8.0-2.fc31.x86_64

Steps how to reproduce the behaviour

1. Launch toolbox container
2. Inspect owner inside the container: ls -l $(tty). Should show up as root since you are inside a user namespace.
3. Inspect the owner of that same device outside the container and verify the permissions looks weird.

[debarshiray]

Thanks for narrowing it down to those working and broken versions.

I wonder if this might be solved by using podman create ... --mount type=devpts,destination=/dev/pts as introduced in containers/podman#7209

I don't think Toolbox itself does anything funky with the TTY devices, but it does bind mount the entire /dev from the host into the container. So I wonder if we can reproduce the same with podman run ... ?

[dustymabe]

Thanks for narrowing it down to those working and broken versions.

No problem :) - It's worth noting those were just two data points. I don't know exactly when the issue was introduced.

I wonder if this might be solved by using podman create ... --mount type=devpts,target=/dev/pts as introduced in containers/podman#7209

Can you recreate locally? I'm assuming pretty much anyone running latest toolbox/podman should see a weird owner when running ls -l on one of the tty devices used for a toolbox session. If you see the problem maybe you can try that in a development version of toolbox?

I don't think Toolbox itself does anything funky with the TTY devices, but it does bind mount the entire /dev from the host into the container. So I wonder if we can reproduce the same with podman run ... ?

Right, I suspect it was a change in podman that caused the regression, just wasn't sure so I reported here first.

[debarshiray]

Adding --mount type=devpts,destination=/dev/pts to podman create only changes the group ownership of the TTY device inside the container from nobody to tty:

$ ls -l $(tty)
crw--w----. 1 root tty 136, 0 Oct 14 20:07 /dev/pts/0

That seems to be a step in the right direction in terms of matching the TTY device's ownership to that on the host, but I don't know if it solves your problem.

I don't know if you figured it out already, but the strange ownership that you are seeing is related to the use of --userns=keep-id. The 100000 UID on the host is the UID of root inside the container's user namespace, the nobody GID inside the container is due to the host's tty group being absent inside the container.

[dustymabe]

Right, but what we need is for the owner of the tty device on the host to be the user who started toolbox. For example, to fix this problem right now for me I sudo chown dustymabe /dev/pts/14 on the host.

Before changing the ownership whenever I git commit it will fail because I have it configured to sign commits and pinentry from the gpg-agent running on the host can't access the tty toolbox is using. After correcting the ownership I can type my password in to the pinentry program that pops up in my terminal.

[debarshiray]

By the way, I submitted #581 for the --mount type=devpts,destination=/dev/pts change. @giuseppe had been asking me to do so for a while, so better late than never.

[debarshiray]

Right, but what we need is for the owner of the tty device on the host
to be the user who started toolbox. For example, to fix this problem right
now for me I sudo chown dustymabe /dev/pts/14 on the host.

Yes, I understand.

Right now I can think of a (crude?) way to fix this. /usr/bin/toolbox is the entry point of the containers. So, instead of merely sleeping, we can install an inotify watch on /dev/pts and have it do chown <user> every time a new file is created there.

[dustymabe]

I'm not sure if this can be done inside the container or if it needs to be done as part of container setup (i.e. podman). I can't do the sudo chown dustymabe /dev/pts/14 from inside the container (inside the container you get chown: changing ownership of '/dev/pts/14': Operation not permitted). It has to be from the host.

[debarshiray]

If you have a new devpts file system mounted on /dev/pts in the container (ie., --mount type=devpts,destination=/dev/pts), then I think you can. :)

Otherwise, the /dev/pts is shared with the host, and the nobody group owner (because the host's tty group is absent inside the container) blocks the chown inside the container.

[ghost]

Just wanted to bump this up as it is unfortunately still a problem as of toolbox version 0.0.99.2 (fedora silverblue 34). The chown hack works temporarily as changing tty or exiting the toolbox resets permissions.

If it does help, the ssh credentials dialog works normally. I tried unsetting both GPG_TTY and DISPLAY, but to no avail.

[ghost]

Poked around a little bit more and found something interesting. I setup an empty git repo to test the issu. I found that this sequence of commands makes signing work inside the toolbox:

[user@fedora:repo]$ git commit -S -m 'Test' # you can cancel this
[user@fedora:repo]$ toolbox enter
[user@toolbox:repo]$ git commit -S -m 'Test' # this will sign

However this sequence will break signing both for the host and the toolbox:

[user@fedora:repo]$ toolbox enter
[user@toolbox:repo]$ git commit -S -m 'Test' # this will fail
[user@toolbox:repo]$ exit
[user@fedora:repo]$ git commit -S -m 'Test' # this will fail

Moreover, on my system the chown hack seemingly stopped working. I don't know if something magically broke or if it was working due to a rogue gpg-agent on the host...

These findings make me think that this issue also involves gpg-agent other than toolbox/podman. What exactly is going wrong I can't tell since this is out of my scope. However if any testing is needed I am happy to help

[giuseppe]

what are the permissions in /dev/pts/?

Is the container running with --mount type=devpts,destination=/dev/pts?

[ghost]

The container is running with --mount type=devpts,destination=/dev/pts and the permissions (inside the container) are the following:

$ ls -l /dev/pts
drwxr-xr-x.  2 root   root        0 Oct 27 22:03 .
drwxr-xr-x. 20 nobody nobody   4300 Oct 27 22:01 ..
crw--w----.  1 root   tty    136, 0 Oct 27 22:04 0
crw-rw-rw-.  1 root   root     5, 2 Oct 27 22:03 ptmx

[giuseppe]

that seems correct. If I understand correctly though, you would like it to be owned by your ID?

I've opened a PR for Podman to allow passing down more options like uid= and gid=: containers/podman#12124

so you'll be abe to do something like --mount type=devpts,destination=/dev/pts,uid=$UID,gid=$UID

[ghost]

Maybe starting the container directly with the correct permissions will fix the error. Right now even with chowning after toolbox enter doesn't fix it. These commands reproduce the issue:

host$ gpgconf --kill gpg-agent # Make sure the agent is not running
host$ toolbox enter
toolbox$ sudo chown user:tty /dev/pts/0
toolbox$ git commit ... # doesn't work
toolbox$ exit
host$ git commit ... # still doesn't work

This leads me to believe that this isn't simply a permission issue because, as I said in my previous comment, if the agent is started on the host then it correctly prompts for password.

OP says that this was not a problem with toolbox-0.0.18-1.fc31.noarch. From a quick glance, it didn't use --mount type=devpts,destination=/dev/pts but --volume /dev:/dev:rslave. Could this be the difference?

[giuseppe]

/dev:/dev will bind mount /dev/pts from the host as well, that could make a difference.

I don't think it is correct though, could you give a try to the patch I've proposed for Podman?

[ghost]

Did some testing using podman main at commit 0686f0bb. Using --mount type=devpts,destination=/dev/pts,uid=1000,gid=5 to mount /dev/pts with user:tty ownership still didn't fix the issue. Plus, i have noticed that by using uid and gid you lose group write permission for $(tty), which doesn't happen with --mount type=devpts,destination=/dev/pts.

I also tried passing --volume /dev:/dev:rw, omitting --mount but not --volume /dev:/dev:rw and viceversa, but nothing seemed to fix the issue. A this point I think that there was a regression or in podman or in gpg-agent, unless my setup is somehow broken, which I don't think it's the case since I am running stock fedora silverblue 35 with minimal configuration (changed prompt and added some aliases).

[debarshiray]

Revisiting this one after a long time ...

Right, but what we need is for the owner of the tty device on the host
to be the user who started toolbox. For example, to fix this problem right
now for me I sudo chown dustymabe /dev/pts/14 on the host.

Yes, I understand.

Right now I can think of a (crude?) way to fix this. /usr/bin/toolbox is the
entry point of the containers. So, instead of merely sleeping, we can install
an inotify watch on /dev/pts and have it do chown <user> every time a
new file is created there.

Oh, I am sorry. I am an idiot. :)

You wanted the owner of the (nested pseudo) terminal device on the host to be the user who started toolbox, instead of something funky; and not the owner of the device inside the container, which is what I was rambling about.

Sorry about that.

Note that entering a container (ie., podman exec --tty) creates a nested pseudo-terminal device inside the container's mount and user namespaces, and there's another one that already exists on the host for the terminal tab or window. ie., there are two terminal devices at play.

I think that one consequence of using:
podman create --mount type=devpts,destination=/dev/pts

(which is commit 494007b6cadc5fe3 or #581)

... is that it separates the devices in /dev/pts on the host and the container.

The nested pseudo-terminal device inside the container gets created as root:tty. This isn't visible from the host, and so there's no device with 100000:tty on the host.

The other consequence of that change is that the nested pseudo-terminal device inside the container gets created as root:tty, not root:nobody, which is what's documented in the commit message.

@dustymabe Does this set-up help your gpg-agent and pinentry set-up?

I'm not sure if this can be done inside the container or if it needs to be done
as part of container setup (i.e. podman). I can't do the
sudo chown dustymabe /dev/pts/14 from inside the container (inside the
container you get chown: changing ownership of '/dev/pts/14': Operation
not permitted). It has to be from the host.

If you have a new devpts file system mounted on /dev/pts in the container
(ie., --mount type=devpts,destination=/dev/pts), then I think you can. :)

Otherwise, the /dev/pts is shared with the host, and the nobody group
owner (because the host's tty group is absent inside the container) blocks
the chown inside the container.

Except, with a separate devpts file system inside the container, the container and the host can't see each other's devices. So, I don't know where this leaves you, @dustymabe :)

Note that #1016 is about changing the ownership of the nested pseudo-terminal device inside the container to $UID:tty, from the current root:tty.

[debarshiray]

You can check if your container was created with:

podman create --mount type=devpts,destination=/dev/pts

... by looking at CreateCommand in the podman inspect --type container ... output.

[debarshiray]

I spent some time debugging gpg-agent and pinentry with @dustymabe -- it's still broken and the previous workaround of fixing the ownership of the terminal device on the host doesn't work anymore.

It seems to me that the gpg-agent and pinentry processes running on the host, really want access to the secondary end of the nested pseudo-terminal device inside the container's mount and user namespaces (ie., the output of tty(1) or the /dev/pts/N device inside the container). Note how the gpg-agent(1) manual says that the GPG_AGENT environment variable should always reflect the output of tty(1).

That doesn't work at the moment because commit 494007b6cadc5fe3 or #581 mounted a separate devpts file system inside the container's mount and user and namespaces. That made the /dev/pts/N devices inside the container different from those on the host.

Back when @dustymabe originally filed this issue, the containers didn't have their own devpts file system. So, the host could see the nested pseudo-terminal device inside the container's user namespace, but it was owned by a funky user ID. Correcting the ownership meant that gpg-agent and pinentry could use the device.

Now, the host can't even see the device, so there's no ownership to be corrected.

[debarshiray]

I filed containers/crun#1158 for some clarifications and advice.

[alexpdp7]

Sorry to intrude, but this seems related to a problem I'm having running distrobox/containerbox nested inside an LXC container, see:

• How to troubleshoot pts issues in unusual environment? podman#18012
• Issues running toolbox/distrobox in Proxmox LXC alexpdp7/alexpdp7#10 (comment)

--mount type=devpts,destination=/dev/pts is causing issues in my environment. I suspect my environment is "bad" and should be "unsupported", but perhaps it's related to the problems you're seeing lately in this issue?

[heyakyra]

I don't have much to contribute here, but i wanted to offer a decent workaround for anyone who, like myself, just wants to sign git commits within a toolbox container, but always ends up hitting:

error: gpg failed to sign the data
fatal: failed to write commit object

and having to run gpgconf --kill gpg-agent and exiting the toolbox container to try again.

Since signing works as long as the gpg-agent was started on the host machine outside of the toolbox, you can just run gpg-agent --daemon before toolbox enter. If you do this often enough, create an alias like: alias tbx='gpg-agent --daemon; toolbox enter [container name]' so you can just run tbx

[dustymabe]

@heyakyra, I'm not sure that's a sufficient workaround for the problem I experience. My gpg agent is always run from outside of toolbox. The problem I have is that when I'm not on my local system (i.e. I'm SSHing into it) I can't sign commits.

When I am on my local system (i.e. DISPLAY=:0) things are fine because a window popup asks me for my password (doesn't need the terminal tty). When I'm remote (SSH) I need the terminal tty and hit this issue.

[losuler]

I have no idea why this workaround works, but if you run this a few times in quick succession from within the toolbox (never works if just ran once):

systemctl --user restart gpg-agent.socket

Then for some reason the popup window in GNOME will start working, and I can put in my password for my GPG key I use to sign my git commits.

$ rpm -q toolbox podman
toolbox-0.0.99.4-1.fc38.x86_64
podman-4.5.1-1.fc38.x86_64