Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Users may need to ignore 127.0.0.1 in SSH config to use Podman 5.3 #24567

Closed
calebevans opened this issue Nov 14, 2024 · 7 comments · Fixed by #24568
Closed

Users may need to ignore 127.0.0.1 in SSH config to use Podman 5.3 #24567

calebevans opened this issue Nov 14, 2024 · 7 comments · Fixed by #24568
Assignees
Labels
5.3 kind/bug Categorizes issue or PR as related to a bug. macos MacOS (OSX) related regression remote Problem is in podman-remote

Comments

@calebevans
Copy link

Issue Description

After updating to 5.3, I started receiving the following error:

caevans@caevans-mac reportportal-connector % podman --log-level=debug ps                            
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps) 
DEBU[0000] ssh_config alias found: 127.0.0.1            
DEBU[0000]   User: core                                 
DEBU[0000]   Hostname: ssh://[email protected]:49744/run/user/501/podman/podman.sock 
DEBU[0000]   Port: 49744                                
DEBU[0000]   IdentityFile: "~/.ssh/id_ed25519"          
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_ed25519": open ~/.ssh/id_ed25519: no such file or directory
DEBU[0000] Shutting down engines

I found that adding !127.0.0.1 and !localhost to the wildcard rule resolved my issue:

Host * !127.0.0.1 !localhost
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519

Steps to reproduce the issue

Steps to reproduce the issue

  1. Install Podman 5.3
  2. Don't exclude 127.0.0.1 or localhost in your SSH config rules
  3. Execute a Podman command

Describe the results you received

OUTPUT

caevans@caevans-mac reportportal-connector % podman --log-level=debug ps                            
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps) 
DEBU[0000] ssh_config alias found: 127.0.0.1            
DEBU[0000]   User: core                                 
DEBU[0000]   Hostname: ssh://[email protected]:49744/run/user/501/podman/podman.sock 
DEBU[0000]   Port: 49744                                
DEBU[0000]   IdentityFile: "~/.ssh/id_ed25519"          
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_ed25519": open ~/.ssh/id_ed25519: no such file or directory
DEBU[0000] Shutting down engines

Describe the results you expected

Expected to execute Podman commands without modifying ssh config.

podman info output

host:
  arch: arm64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.76
    systemPercent: 0.12
    userPercent: 0.12
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2044
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.11.3-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 7222734848
  memTotal: 7716302848
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.aarch64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: unix:///run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 8h 28m 41.00s (Approximately 0.33 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 7329873920
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 49
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.0
  Built: 1731456000
  BuiltTime: Tue Nov 12 17:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/arm64
  Version: 5.3.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

SYSTEM INFO

  • macOS 15.1
  • Apple Silicon CPU
  • Podman:
caevans@caevans-mac ~ % podman version
Client:       Podman Engine
Version:      5.3.0
API Version:  5.3.0
Go Version:   go1.23.3
Git Commit:   874bf2c301ecf0ba645f1bb45f81966cc755b7da
Built:        Tue Nov 12 09:10:17 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.3.0
API Version:  5.3.0
Go Version:   go1.22.7
Built:        Tue Nov 12 17:00:00 2024
OS/Arch:      linux/arm64

Additional information

I found that adding !127.0.0.1 and !localhost to the wildcard rule resolved my issue:

Host * !127.0.0.1 !localhost
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519
@calebevans calebevans added the kind/bug Categorizes issue or PR as related to a bug. label Nov 14, 2024
@github-actions github-actions bot added macos MacOS (OSX) related remote Problem is in podman-remote labels Nov 14, 2024
@calebevans calebevans changed the title Podman is using default wildcard in the ~/.ssh/config file Users may need to ignore 127.0.0.1 in SSH config to use Podman 5.3 Nov 14, 2024
@Luap99 Luap99 self-assigned this Nov 14, 2024
Luap99 added a commit to Luap99/libpod that referenced this issue Nov 14, 2024
When we alreayd get a full URL with user, port and identity then we
should not read the config file just to overwrite them with wrong
values. This is a bad regression for user using * wildcard in their
ssh_config as it makes podman machine unusable.

Fixes: containers#24567

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/libpod that referenced this issue Nov 14, 2024
When we alreayd get a full URL with user, port and identity then we
should not read the config file just to overwrite them with wrong
values. This is a bad regression for user using * wildcard in their
ssh_config as it makes podman machine unusable.

Fixes: containers#24567
Fixes: e523734 ("Add support for ssh_config for connection")

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/libpod that referenced this issue Nov 14, 2024
When we alreadty get a full URL with user, port and identity then we
should not read the config file just to overwrite them with wrong
values. This is a bad regression for user using * wildcard in their
ssh_config as it makes podman machine unusable.

Fixes: containers#24567
Fixes: e523734 ("Add support for ssh_config for connection")

Signed-off-by: Paul Holzinger <[email protected]>
@Luap99
Copy link
Member

Luap99 commented Nov 14, 2024

Fix in #24568, we are planning for a 5.3.1 release early next week with some other fixes

@afbjorklund
Copy link
Contributor

afbjorklund commented Nov 17, 2024

Was the root cause here that we didn't tilde-expand the key path (for Host *)?

Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_ed25519": open ~/.ssh/id_ed25519: no such file or directory

The workaround should be fine to fix the regression, but it should cope with:

Host *
    IdentityFile ~/.ssh/id_rsa

But it doesn't:

Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_rsa": open ~/.ssh/id_rsa: no such file or directory

Even if it is the default (and thus was never tested, when it came to the parser)


Like so:

--- a/pkg/bindings/connection.go
+++ b/pkg/bindings/connection.go
@@ -189,8 +189,13 @@ func sshClient(_url *url.URL, uri string, identity string, machine bool) (Connec
                }
        }
        if val := cfg.Get(alias, "IdentityFile"); val != "" {
+               h, err := os.UserHomeDir()
+               if err != nil {
+                       return connection, fmt.Errorf("user homedir could not be determined: %w", err)
+               }
                if val != ssh_config.Default("IdentityFile") {
                        identity = strings.Trim(val, "\"")
+                       identity = strings.Replace(identity, "~", h, 1)
                        found = true
                }
        }

Then it goes back to failing "properly", i.e. due to overriding the ssh key for all hosts.

Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

@afbjorklund
Copy link
Contributor

afbjorklund commented Nov 17, 2024

The real issue was that we only had a single key, and didn't support multiple.

It was supposed to start with the Host *, and then continue with the Host...

That is why the podman connection failed, even though ssh is still successful:

debug1: Next authentication method: publickey
debug1: Offering public key: /home/anders/.ssh/id_rsa RSA SHA256:... explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /home/anders/.lima/_config/user ED25519 SHA256:... explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60

Changing the order of the ssh config lines (so that Host * comes after) makes it work.

The current ssh_config parser returns the first key that it can find, but only a single one.


It still needs to expand tilde on the path, though:

And apparently there are bugs in support for the !?

The string * !127.0.0.1 !localhost matches all hosts...

@Luap99
Copy link
Member

Luap99 commented Nov 17, 2024

@afbjorklund There are multiple issues, see the commits in the PR. But the biggest issue from your PR was that you overwrote settings that were already set in the ssh URL. For the machine connection we already have the proper user, port and identity file set so it should not pick the ones from the shh config as these are defaults not overwrites.

openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/podman that referenced this issue Nov 18, 2024
When we alreadty get a full URL with user, port and identity then we
should not read the config file just to overwrite them with wrong
values. This is a bad regression for user using * wildcard in their
ssh_config as it makes podman machine unusable.

Fixes: containers#24567
Fixes: e523734 ("Add support for ssh_config for connection")

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/libpod that referenced this issue Nov 19, 2024
When we alreadty get a full URL with user, port and identity then we
should not read the config file just to overwrite them with wrong
values. This is a bad regression for user using * wildcard in their
ssh_config as it makes podman machine unusable.

Fixes: containers#24567
Fixes: e523734 ("Add support for ssh_config for connection")

Signed-off-by: Paul Holzinger <[email protected]>
(cherry picked from commit a7120b5)
@unexist
Copy link

unexist commented Nov 20, 2024

A similar problem occurs when your ssh/config contains something like this:

Host *
    User unexist

This leads to this problem:

podman --log-level=debug ps
INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps)
DEBU[0000] ssh_config alias found: 127.0.0.1
DEBU[0000]   User: unexist
DEBU[0000]   Hostname: ssh://[email protected]:56486/run/user/501/podman/podman.sock
DEBU[0000]   Port: 56486
DEBU[0000]   IdentityFile: "/Users/unexist/.local/share/containers/podman/machine/machine"
DEBU[0000] SSH Ident Key "/Users/unexist/.local/share/containers/podman/machine/machine" SHA256:gwXfNN5TwvnFTWi7veLCoRMLjb1IBZJ7avDlsrEJO+M ssh-ed25519
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
DEBU[0000] Shutting down engines

This is probably completely my fault, but it took me quite a few hours to narrow this down and it never caused issues before. So this is just a heads-up for people who stumble upon the same issue.

@fpigeonjr
Copy link

can confirm that adding this to my ~/ssh/config worked for me as well:

Host * !127.0.0.1 !localhost
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519

@basenc
Copy link

basenc commented Dec 9, 2024

Image

The error is still present. It would switch to using file identities instead of ssh-agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5.3 kind/bug Categorizes issue or PR as related to a bug. macos MacOS (OSX) related regression remote Problem is in podman-remote
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants