Unable to pull any images (even from dockerhub.io)

[Mo0rBy]

Issue Description

I have used Podman in the past before and didn't encounter this issue.
I switched back to using Docker for some reason because I needed to use docker-compose and it wasn't working out of the box with Podman. Anyway, I have tried to use Podman again, but I am unable to pull even official images from Dockerhub due to an X509 certificate issue.

So all I have done is update Podman to latest version and tried to pull node:latest

Steps to reproduce the issue

Steps to reproduce the issue

1. Install/update Podman to latest (at time of writing, my podman -v output is podman version 5.3.1)
2. Run podman pull node:latest (or attempt to pull some other iamge from dockerhub). I have also the the full name podman pull docker.io/library/node:latest.
3. See error related to X509 certificate

Describe the results you received

Here is the output of when I run podman pull node:latest --log-level=debug:

INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull node:latest --log-level=debug)
DEBU[0000] SSH Ident Key "/Users/wmoorby/.local/share/containers/podman/machine/machine" SHA256:u3+Ua/vygSwGCrqDvAPSO8D9D7qBBGNGBLfOcTWyWKc ssh-ed25519
DEBU[0000] DoRequest Method: GET URI: http://d/v5.3.1/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] No credentials matching docker.io found in /Users/wmoorby/.config/containers/auth.json
DEBU[0000] No credentials matching docker.io found in /Users/wmoorby/.config/containers/auth.json
DEBU[0000] Found an empty credential entry "https://index.docker.io/v1/" in "/Users/wmoorby/.docker/config.json" (an unhandled credential helper marker?), moving on
DEBU[0000] No credentials matching docker.io found in /Users/wmoorby/.dockercfg
DEBU[0000] No credentials for docker.io found
DEBU[0000] No credentials matching ***.amazonaws.com found in /Users/wmoorby/.config/containers/auth.json
DEBU[0000] No credentials matching ***.amazonaws.com found in /Users/wmoorby/.config/containers/auth.json
DEBU[0000] Found an empty credential entry "***.amazonaws.com" in "/Users/wmoorby/.docker/config.json" (an unhandled credential helper marker?), moving on
DEBU[0000] No credentials matching ***.amazonaws.com found in /Users/wmoorby/.dockercfg
DEBU[0000] No credentials for ***.amazonaws.com found
DEBU[0000] No credentials matching ***.amazonaws.com found in /Users/wmoorby/.config/containers/auth.json
DEBU[0000] No credentials matching ***.amazonaws.com found in /Users/wmoorby/.config/containers/auth.json
DEBU[0000] Found an empty credential entry "***.amazonaws.com" in "/Users/wmoorby/.docker/config.json" (an unhandled credential helper marker?), moving on
DEBU[0000] No credentials matching ***.amazonaws.com found in /Users/wmoorby/.dockercfg
DEBU[0000] No credentials for ***.amazonaws.com found
DEBU[0000] DoRequest Method: POST URI: http://d/v5.3.1/libpod/images/pull
Resolved "node" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/node:latest...
Error: initializing source docker://node:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
DEBU[0000] Shutting down engines

Judging from this output, I am missing some Dockerhub auth somewhere, but I'm not sure where exactly. You can see that podman is picking up some creds for AWS accounts, but that is not related to this as it is just going to Dockerhub.

Also, I would expect the error message to indicate my authentication is bad, not that the certificate is signed by an unknown author.

I am able to pull this image using docker pull node:latest, so this is just a problem with my podman setup.

Describe the results you expected

For podman to pull the latest node image successfully and be able to connect to Dockerhub

podman info output

host:
  arch: arm64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.78
    systemPercent: 0.13
    userPercent: 0.08
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.11.3-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1539432448
  memTotal: 2042908672
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.aarch64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: unix:///run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 12m 52.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 4700618752
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.1
  Built: 1732147200
  BuiltTime: Thu Nov 21 00:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/arm64
  Version: 5.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Apple M1 Pro, Sequoia 15.1.1

Additional information

No response

[Luap99]

Are you behind a HTTPS proxy that intercepts all requests and needs its own cert?
https://github.com/containers/podman/blob/main/docs/tutorials/podman-install-certificate-authority.md

[Mo0rBy]

This is on my work laptop, so that is a possibility, however I am able to pull the same images when using docker CLI command.
So if it was a proxy, would it not affect all traffic? Not just what Podman is trying to do.

I will ask around tomorrow to see if other people have had the same problem or not.

[Mo0rBy]

ok so I ran curl -vvvv https://registry-1.docker.io to see the output, and I can see that my local machine is connecting using a custom certificate from my companies new network blocker application.

I'm guessing that the Podman VM does not contain this cert, but still goes through my local machine to go to the internet, and therefore Podman is unhappy with this custom cert.

I'm tried using Buildkit and I get the same results. I see in Buildkit that you can specify some certFilePath's, I don't know if there is something similar in Podman or not, so I'm going to try with Buildkit and see what I can do.

Can someone please advise or point me to some docs that explain how to exec into the Podman VM or the command flags needed to spin up the Podman VM with custom certs on the localhost system?

[Luap99]

The link I gave you should shows the steps the first command is podman machine ssh ...

[Mo0rBy]

I thought I added a comment + closed the issue this was originally from, but I guess not. Whoops!

Just wanted to finalise this and say, yes it was a company networking cert issue.
Followed the docs linked earlier in thread to add certs into the Podman VM, and it worked!

I've raised it to relevant people within my company, and learnt how to add certs to Podman, but if you have these sort of issues, it's probably a company networking thing.

Thanks for your help @Luap99 ! I will close this now (as I thought I closed the issue before, but clearly I didn't)