Podman does not start sql server container

[huestack]

Issue Description

I have been running sql-server database in podman container from a custom image from tar file. Today after update to system, the container fails to start without any error message. Initially, I thought something happened in my database, but for a test I simply pulled fresh sql server image, and ran a container from it. And still, the container failed to run.

In my other system having same configuration, I ran same image in docker, and it worked.
Here is log file generated with sudo podman logs sql-test: sql-test.log.

Steps to reproduce the issue

Steps to reproduce the issue

1. Pull sql server image

sudo docker pull mcr.microsoft.com/mssql/server:2022-latest

2. Run the container

sudo podman run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=Xserver@101" -p 1433:1433 --name sql-test --hostname sql-test -d mcr.microsoft.com/mssql/server

3. Check the list of running containers

sudo podman ps

Describe the results you received

sudo podman ps list is empty.

Describe the results you expected

The running container should be listed, and sql server should work.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.10-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 2dcd736e46ded79a53339462bc251694b150f870'
  cpuUtilization:
    idlePercent: 96.56
    systemPercent: 2.61
    userPercent: 0.83
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: endeavouros
    version: unknown
  eventLogger: journald
  freeLocks: 2045
  hostname: the-desktop
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.7.0-arch3-1
  linkmode: dynamic
  logDriver: journald
  memFree: 10829881344
  memTotal: 33341288448
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: /usr/lib/podman/netavark is owned by netavark 1.9.0-1
    path: /usr/lib/podman/netavark
    version: netavark 1.9.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.12-1
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 6h 4m 54.00s (Approximately 0.25 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /media/others/Programs/Containers
  graphRootAllocated: 365001289728
  graphRootUsed: 29780652032
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /media/others/Programs/Containers/volumes
version:
  APIVersion: 4.8.3
  Built: 1704384851
  BuiltTime: Thu Jan  4 21:44:11 2024
  GitCommit: 85dc30df56566a654700722a4dd190e1b9680ee7-dirty
  GoVersion: go1.21.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.3

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Client:       Podman Engine
Version:      4.8.3
API Version:  4.8.3
Go Version:   go1.21.5
Git Commit:   85dc30df56566a654700722a4dd190e1b9680ee7-dirty
Built:        Thu Jan  4 21:44:11 2024
OS/Arch:      linux/amd64

[giuseppe]

could it be selinux or seccomp?

Try adding --security-opt label=disable or --security-opt seccomp=unconfined to your podman run command

[huestack]

@giuseppe No. Both don't work.

[giuseppe]

could you run sudo strace -f -o strace.log -v -s 1000 podman run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=Xserver@101" -p 1433:1433 --name sql-test --hostname sql-test mcr.microsoft.com/mssql/server and attach here the strace.log file?

[huestack]

Here it is.
strace.tar.gz

[giuseppe]

thanks.

I see a bunch of invalid arch_prctl syscalls:

arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd61e52b70) = -1 EINVAL (Invalid argument)

that might be the reason for the failure.

That doesn't depend from Podman, it is sql server doing the wrong thing. Please report the error to sql server.

[kingOf0]

For anyone come from google, looks like kernel 6.7.0 breaks mssql-server. (microsoft/mssql-docker#868) Downgrading the kernel fixes. (See https://wiki.archlinux.org/title/Downgrading_packages for arch)