Podman run/build is painfully slow compared to docker

[stefanschober]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. run a container using podman with time (time podman run

2. use the exactly same image to run the container using 'time docker ...'

Describe the results you received:
I use a crosscompilation container to run a python script. With the option '-h' this script shows the argparse help message and exits.
I see massive performance differences. Podman on my system is slow as the continental drift, compared to docker.

time docker run --rm -t -v ${PWD}:/work -w /work crosscompile ./buildprj.py -h:
real 0m0.636s
user 0m0.032s
sys 0m0.039s

time podman run --rm -t -v ${PWD}:/work -w /work crosscompile ./buildprj.py -h
real 2m15.022s
user 0m1.862s
sys 0m17.472s

Describe the results you expected:
comparable execution times

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ad
Built: Thu Dec 9 19:30:40 2021
OS/Arch: linux/amd64

Output of podman info --debug:
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:

• memory
• pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
  package: /usr/bin/conmon is owned by conmon 1:2.0.32-2
  path: /usr/bin/conmon
  version: 'conmon version 2.0.32, commit: 436b460d1586c2e4ab4e845448449ddd9136767a'
  cpus: 8
  distribution:
  distribution: manjaro
  version: unknown
  eventLogger: journald
  hostname: troja
  idMappings:
  gidmap:
  • container_id: 0
    host_id: 1000
    size: 1
  • container_id: 1
    host_id: 200000
    size: 65536
    uidmap:
  • container_id: 0
    host_id: 1000
    size: 1
  • container_id: 1
    host_id: 200000
    size: 65536
    kernel: 5.16.2-1-MANJARO
    linkmode: dynamic
    logDriver: journald
    memFree: 5134245888
    memTotal: 33525063680
    ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.4-1
    path: /usr/bin/crun
    version: |-
    crun version 1.4
    commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
    spec: 1.0.0
    +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
    os: linux
    remoteSocket:
    path: /run/user/1000/podman/podman.sock
    security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
    serviceIsRemote: false
    slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
    slirp4netns version 1.1.12
    commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
    libslirp: 4.6.1
    SLIRP_CONFIG_VERSION_MAX: 3
    libseccomp: 2.5.3
    swapFree: 66602639360
    swapTotal: 68719472640
    uptime: 223h 10m 19.68s (Approximately 9.29 days)
    plugins:
    log:
• k8s-file
• none
• journald
  network:
• bridge
• macvlan
  volume:
• local
  registries:
  localhost:
  Blocked: false
  Insecure: true
  Location: localhost
  MirrorByDigestOnly: false
  Mirrors: null
  Prefix: localhost
  search:
• localhost
• docker.io
  store:
  configFile: /home/stefan/.config/containers/storage.conf
  containerStore:
  number: 0
  paused: 0
  running: 0
  stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/stefan/.local/share/containers/storage
  graphStatus: {}
  imageStore:
  number: 14
  runRoot: /run/user/1000/containers
  volumePath: /home/stefan/.local/share/containers/storage/volumes
  version:
  APIVersion: 3.4.4
  Built: 1639074640
  BuiltTime: Thu Dec 9 19:30:40 2021
  GitCommit: f6526ad
  GoVersion: go1.17.4
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):
[stefan]$ pacman -Qi podman
Name : podman
Version : 3.4.4-1
Description : Tool and library for running OCI-based containers in pods
Architecture : x86_64
URL : https://github.com/containers/podman
Licenses : Apache
Groups : None
Provides : None
Depends On : cni-plugins conmon containers-common crun fuse-overlayfs iptables libdevmapper.so=1.02-64
libgpgme.so=11-64 libseccomp.so=2-64 slirp4netns
Optional Deps : apparmor: for AppArmor support
btrfs-progs: support btrfs backend devices [installed]
catatonit: --init flag support
podman-docker: for Docker-compatible CLI
Required By : cockpit-podman podman-compose
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 72.79 MiB
Packager : David Runge dvzrv@archlinux.org
Build Date : Thu 09 Dec 2021 07:30:40 PM CET
Install Date : Fri 17 Dec 2021 10:47:22 AM CET
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

graphDriverName: vfs

There's your problem. You're using the VFS storage driver, which is not a proper laying filesystem, and as such requires a lot of expensive copy operations just to launch containers.

Do you have fuse-overlayfs installed? Alternatively, a sufficiently-new kernel should have support for native kernel Overlayfs with rootless.

Finally, I'll note that you seem to be comparing root Docker to rootless Podman, which we don't really expect to perform comparably - rootless will be slower, due to the compromises it has to make in the name of improved security. However, I think we'd expect it to be ~15% slower, not the figures you are seeing.

[rhatdan]

After installing fuse-overlayfs
do a
$ podman system reset

And then podman info to make sure podman is using overlay. Then compare again, we feel they will be much closer.

[stefanschober]

Thanks for the extremely quick and helpful support :)
after the update:
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/stefan/.local/share/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"

the same python script as above with podman and docker:
`docker:
real 0m0.705s
user 0m0.025s
sys 0m0.044s

podman:
real 0m1.289s
user 0m0.184s
sys 0m0.108s
`

[nmirzayev]

Thanks for the extremely quick and helpful support :) after the update: graphDriverName: overlay graphOptions: {} graphRoot: /home/stefan/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false"

the same python script as above with podman and docker: `docker: real 0m0.705s user 0m0.025s sys 0m0.044s

podman: real 0m1.289s user 0m0.184s sys 0m0.108s `

hey. how were you able to set Native Overlay Diff: "true". I am experiencing the same, the performance issue with podman. I am using overlay but couldnt set Native Overlay Diff to true.

here is my podman info:
graphDriverName: overlay
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphRootAllocated: 133003395072
graphRootUsed: 36006727680
graphStatus:
Backing Filesystem: overlayfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp

[zhucan]

still slowly than docker:

docker build needs 17s:

❯ docker build --no-cache -t reg.deeproute.ai/deeproute-public/zhucan_test8 .
Sending build context to Docker daemon  1.074GB
Step 1/2 : FROM nginx
 ---> a99a39d070bf
Step 2/2 : COPY ./tempfile /app/
 ---> 1fc6c13383a2
Successfully built 1fc6c13383a2
Successfully tagged reg.deeproute.ai/deeproute-public/zhucan_test8:latest

deeproute in rook-node01 in ~/podman took **17s** 

podman build needs 32s:

❯ podman build --no-cache -t reg.deeproute.ai/deeproute-public/zhucan_test7 .
STEP 1/2: FROM nginx
STEP 2/2: COPY ./tempfile /app/
COMMIT reg.deeproute.ai/deeproute-public/zhucan_test7
--> bf8dbf367d4
Successfully tagged reg.deeproute.ai/deeproute-public/zhucan_test7:latest
bf8dbf367d46d0de8a87edc13924f2558edb237e2fa454012284fc8f13dbf3b1

deeproute in rook-node01 in ~/podman took **32s** 

podman info:

store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1675248852
  BuiltTime: Wed Feb  1 18:54:12 2023
  GitCommit: ""
  GoVersion: go1.18.4
  OsArch: linux/amd64
  Version: 3.4.4

[zhucan]

@rhatdan

[rhatdan]

Please open a new issue with all of the required information and a reproducer.

[hydrargyrum]

After installing fuse-overlayfs do a $ podman system reset

And then podman info to make sure podman is using overlay. Then compare again, we feel they will be much closer.

For anyone coming across this ticket, this solution worked for me, but it can't be stressed enough that:

• not only you need to stop all your containers for this to work
• but also that it will destroy every volume, every container, everything you have in podman
• and depending on how many containers/images you had, podman system reset will be extremely slow (even with only a couple of images/containers it's painfully slow, it takes hours)

So:

• backup the volumes/containers and everything you don't want to lose, you will need to use podman volume export for example
• podman system reset, don't forget to take into account the massive downtime incurred and plan ahead
• then you might need to force [storage] driver = "overlay" in storage.conf (e.g. ~/.config/containers/storage.conf)
• check with podman info --debug that it really worked
• podman volume import and restart your containers etc.