Podman-docker not working with pterodactyl wings

[leonpano2006]

Issue Description

It should run without any strange issues
And please check here
pterodactyl/panel#4928
and issue won’t occur under normal docker-ce

Steps to reproduce the issue

Steps are described in
pterodactyl/panel#4928

Describe the results you received

Things will fail to start

Describe the results you expected

Runs fine just like on docker

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: aadb7c890ac6283eb4666d92690238e5fbdec5c7'
  cpuUtilization:
    idlePercent: 99.97
    systemPercent: 0.02
    userPercent: 0.02
  cpus: 64
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "9.3"
  eventLogger: journald
  freeLocks: 2043
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
  kernel: 6.6.15
  linkmode: dynamic
  logDriver: journald
  memFree: 402122006528
  memTotal: 405075681280
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.el9.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-2.el9_3.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.7-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.7
      commit: 53a9996ce82d1ee818349bdcc64797a1fa0433c4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230818.g0af928e-4.el9.x86_64
    version: |
      pasta 0^20230818.g0af928e-4.el9.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 17179865088
  swapTotal: 17179865088
  uptime: 5h 19m 46.00s (Approximately 0.21 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/himserver/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/himserver/.local/share/containers/storage
  graphRootAllocated: 581926912000
  graphRootUsed: 40216023040
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/himserver/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1701529524
  BuiltTime: Sat Dec  2 23:05:24 2023
  GitCommit: ""
  GoVersion: go1.20.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

RHEL9 on Linux 6.6(Ubuntu like kernel)

Additional information

No response

[rhatdan]

I would figure that the link for docker.sock does not exists yet?

ls -l /var/run/docker.sock

Should point at the podman.sock

You can use DOCKER_HOST environment variable to point at the podman.socket.

[xx445469]

It is linked

[leonpano2006]

I would figure that the link for docker.sock does not exists yet?

ls -l /var/run/docker.sock

Should point at the podman.sock

You can use DOCKER_HOST environment variable to point at the podman.socket.

@rhatdan
It is linked as @xx445469 says
I am partner form @xx445469

[parkervcp]

It is linked

The fact that this is highlighted red means it's wrong.

[leonpano2006]

It is linked

The fact that this is highlighted red means it's wrong.

I find out it works with sudo
Or i will get other error in first place

For some reason that path is not possible to view without root privileges

[parkervcp]

You link the other way around. It's from -> to in the ls command

ln -s /run/podman/podman.sock /var/run/docker.sock

[leonpano2006]

You link the other way around. It's from -> to in the ls command

ln -s /run/podman/podman.sock /var/run/docker.sock

Is this my mistake or upstream?

[leonpano2006]

You link the other way around. It's from -> to in the ls command

ln -s /run/podman/podman.sock /var/run/docker.sock

I didn't manually link it
But I think pull request won't fix pterodactyl wings errors when used with podman-docker since I run wings with sudo

But I still that pull request is also important

There might be more fixes required

[leonpano2006]

this time with fresh install

[leonpano2006]

@rhatdan can you help me please?

[rhatdan]

Sure what is the problem? The rootful socket looks good.

[leonpano2006]

Sure what is the problem? The rootful socket looks good.

i find out few things

if OOM killer is disabled then i can get this error with workaround mentioned here pterodactyl/panel#4928

ERROR: [Feb 15 23:55:45.709] failed to run install process for server error=Error response from daemon: crun: cannot disable OOM killer with cgroupv2: OCI runtime error server=75950f46-49c0-4911-ade7-7007ed01eba7

Stacktrace:
Error response from daemon: crun: cannot disable OOM killer with cgroupv2: OCI runtime error

and if OOM killer is enabled and server may run with workaround mentioned here pterodactyl/panel#4928
but i can still get errors like bellow

DEBUG: [Feb 15 23:57:34.226] syncing server settings with environment server=2cdfe3a2-9098-4cd8-b31a-d381eac13b58
DEBUG: [Feb 15 23:57:34.226] syncing stop configuration with configured docker environment server=2cdfe3a2-9098-4cd8-b31a-d381eac13b58
 INFO: [Feb 15 23:57:34.226] performing server limit modification on-the-fly server=2cdfe3a2-9098-4cd8-b31a-d381eac13b58
 WARN: [Feb 15 23:57:34.230] failed to perform on-the-fly update of the server environment error=environment/docker: could not update container: Error response from daemon: Not Found server=2cdfe3a2-9098-4cd8-b31a-d381eac13b58

Stacktrace:
Error response from daemon: Not Found
environment/docker: could not update container
github.com/pterodactyl/wings/environment/docker.(*Environment).InSituUpdate
        github.com/pterodactyl/wings/environment/docker/container.go:129
github.com/pterodactyl/wings/server.(*Server).SyncWithEnvironment
        github.com/pterodactyl/wings/server/update.go:51
github.com/pterodactyl/wings/server.(*Server).Sync
        github.com/pterodactyl/wings/server/server.go:200
github.com/pterodactyl/wings/server.(*Server).onBeforeStart
        github.com/pterodactyl/wings/server/power.go:174
github.com/pterodactyl/wings/server.(*Server).HandlePowerAction
        github.com/pterodactyl/wings/server/power.go:131
github.com/pterodactyl/wings/router/websocket.(*Handler).HandleInbound
        github.com/pterodactyl/wings/router/websocket/websocket.go:363
github.com/pterodactyl/wings/router.getServerWebsocket.func3
        github.com/pterodactyl/wings/router/router_server_ws.go:85
runtime.goexit
        runtime/asm_amd64.s:1598

and log driver would be an issue too as here pterodactyl/panel#4928 (you didn't make pull request to fix log driver)

but these error wont happen on docker-ce but on podman-docker will occur

and there is more logs is here
https://gist.github.com/leonpano2006/7958c8751b000f53ecb15d8d459f0e70

[rhatdan]

Please open a new issue on this since it has nothing to do with podman-docker. Errors should be opened with crun.

If you try with runc, do you see similar issues.

[leonpano2006]

Please open a new issue on this since it has nothing to do with podman-docker. Errors should be opened with crun.

If you try with runc, do you see similar issues.

is this part issue with crun?

Error response from daemon: Not Found
environment/docker: could not update container
github.com/pterodactyl/wings/environment/docker.(*Environment).InSituUpdate
        github.com/pterodactyl/wings/environment/docker/container.go:129
github.com/pterodactyl/wings/server.(*Server).SyncWithEnvironment
        github.com/pterodactyl/wings/server/update.go:51
github.com/pterodactyl/wings/server.(*Server).Sync
        github.com/pterodactyl/wings/server/server.go:200
github.com/pterodactyl/wings/server.(*Server).onBeforeStart
        github.com/pterodactyl/wings/server/power.go:174
github.com/pterodactyl/wings/server.(*Server).HandlePowerAction
        github.com/pterodactyl/wings/server/power.go:131
github.com/pterodactyl/wings/router/websocket.(*Handler).HandleInbound
        github.com/pterodactyl/wings/router/websocket/websocket.go:363
github.com/pterodactyl/wings/router.getServerWebsocket.func3
        github.com/pterodactyl/wings/router/router_server_ws.go:85
runtime.goexit
        runtime/asm_amd64.s:1598

and is log driver something to do with crun?
and why this thing can run fine with docker-ce, or crun is not used in docker-ce?

[leonpano2006]

runc

and what is runc?

[rhatdan]

--runtime=runc

podman defaults to crun, and docker defaults to runc. They are OCI Runtimes. Anyways open an issue with crun.

[rhatdan]

@giuseppe FYI ^^

[leonpano2006]

--runtime=runc

podman defaults to crun, and docker defaults to runc. They are OCI Runtimes. Anyways open an issue with crun.

containers/crun#1416

[leonpano2006]

--runtime=runc

podman defaults to crun, and docker defaults to runc. They are OCI Runtimes. Anyways open an issue with crun.

and there is issue also for service of podman-docker (i dont think this have to do with crun)
it dosen't emulate docker.service
so this can happen

him@localhost:~$ sudo systemctl enable --now wings
Failed to start wings.service: Unit docker.service not found.

and i need to manually change docker.service to podman.service in order to work

maybe there is something can be done to fix this

[rhatdan]

Please open a separate issue for this. So this is a packaged service which requires the docker.service to be running?