Problem running podman commands in windows but podman ssh is working

[mwoodpatrick]

Issue Description

On Windows, I'm sometimes getting into a state where I can't run some commands from the terminal including "podman version" but I can run "podman machine ssh" without issue and then run these commands inside the machine without issue. What's the best way to debug or fix?

Steps to reproduce the issue

Create podman machine run, later run podman commands

Describe the results you received

Run the command
podman version

returns

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v5.0.1/libpod/_ping": ssh: rejected: connect failed (open failed)

But the command:

podman machine ssh

works as expected:

Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Last login: Sat Apr 13 06:22:50 2024 from 127.0.0.1

Describe the results you expected

podman version runs the current version

podman info output

host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.10-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 2dcd736e46ded79a53339462bc251694b150f870'
  cpuUtilization:
    idlePercent: 99.94
    systemPercent: 0.03
    userPercent: 0.02
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2043
  hostname: mlwphpenvy360
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.150.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 32337616896
  memTotal: 33509527552
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.10.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: /usr/lib/podman/netavark is owned by netavark 1.10.3-1
    path: /usr/lib/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.14.4-1
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/sbin/pasta
    package: /usr/bin/pasta is owned by passt 2024_04_05.954589b-1
    version: |
      pasta 2024_04_05.954589b
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/sbin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.3-1
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 7757352960
  swapTotal: 8589934592
  uptime: 13h 12m 32.00s (Approximately 0.54 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/mwoodpatrick/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 1
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/mwoodpatrick/.local/share/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 16177856512
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/mwoodpatrick/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.1
  Built: 1712088128
  BuiltTime: Tue Apr  2 13:02:08 2024
  GitCommit: 946d055df324e4ed6c1e806b561af4740db4fea9-dirty
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

wsl --version

WSL version: 2.2.2.0
Kernel version: 5.15.150.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22635.3430

Additional information

Any pointers on the best way to debug would be appreciated,

[mwoodpatrick]

Why is podman using the URL:

"http://d/v5.0.1/libpod/_ping"

When trying to connect?

Running:

curl http://d/v5.0.1/libpod/_ping

Returns:

curl: (6) Could not resolve host: d

and

podman system connection list

gives:

Name                         URI                                                                  Identity
                                        Default     ReadWrite
podman-machine-default       ssh://[email protected]:53726/run/user/1000/podman/podman.sock  C:\Users\mlwp\.local\share\containers\podman\machine\machine  true        true
podman-machine-default-root  ssh://[email protected]:53726/run/podman/podman.sock                    C:\Users\mlwp\.local\share\containers\podman\machine\machine  false       true

[baude]

this could be any number of problems. it sounds like it works and then something is going on that causes it to fail. right ?

[mwoodpatrick]

Agreed, I'm trying to figure best way to debug why the connection is failing. It seems like an issue with the REST API since the commands in the WSL-2 distro created by Podman machine are working

[baude]

i dont think the REST API as much as the socket

[mwoodpatrick]

Could easily be a socket issue, Any suggestions on how to debug?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[er-eg]

Same problem here on Windows 11 and Podman 5.1.0 (curl: (6) Could not resolve host: d), but system connection list is empty.

Podman machine is running and container can be started from "Podman Desktop" but not from terminal.

podman start {container}

Error: unable to connect to Podman socket: Get "http://d/v5.1.0/libpod/_ping": dial unix /run/podman/podman.sock: connect: A socket operation encounte

[halberom]

I've also hit this. I can replicate the error by performing podman machine init --username foo. I can then podman machine ssh, but not podman ps or anything else. If I init without setting the username, it works as expected.

[keshavsharma-github]

I am also facing the same issue on Windows 11.
Do we have any potential fix for this?

[carrulop]

Have you tried to create the podman machine with root privileges? I had the same issue than you, and I realized that I had created the machine without root privileges, but with podman desktop I created the machine with root privilege, and now, I can do things like podman ps, or podman version.

[amiyadas77]

on Windows 11 execute below command and it worked
podman machine stop
podman machine set --rootful
podman machine start

[salvo80]

Same error here! It works with podman-compose downgraded to 1.0.6

windows 11
podman 5.2.0
podman-compose installed by "pip3 install podman-compose"