[Containerfile] HEREDOC with RUN instruction does not invalidate cache

[GrabbenD]

Issue Description

HEREDOC is useful for providing a clean syntax in Contailerfile. It allows you to create multi-line commands which supports comments without the need of using any special operators like &&.

However, the cache does not get invalidated if you update a cached HEREDOC RUN instruction in Containerfile:

Steps to reproduce the issue

1. Create a RUN instruction using HEREDOC:
   $ podman build -t localhost/test/heredoc:latest -f ./Containerfile

   # Containerfile
   FROM docker.io/archlinux/archlinux:latest
   
   RUN <<EXE
       echo 1
       echo 2
   EXE
   
   RUN echo finish

   STEP 2/3: RUN <<EXE
   1
   2
   --> 051e073b13d1

2. Add a new command to an existing HEREDOC RUN instruction:
   $ podman build -t localhost/test/heredoc:latest -f ./Containerfile

   # Containerfile
   FROM docker.io/archlinux/archlinux:latest
   
   RUN <<EXE
       echo 1
       echo 2
       echo 3
   EXE
   
   RUN echo finish

   STEP 2/3: RUN <<EXE
   --> Using cache 051e073b13d16863c61c6d43f5e9f518af6d5c5870d718e5c3a6eed2097d50ee
   --> 051e073b13d1

Describe the results you received

Using the above example, echo 3 is never executed.

Describe the results you expected

Changes to any RUN instruction should invalidate the cache even if it uses HEREDOC.

This issue can lead to a lot of confusion and frustration when pushing code into production which uses HEREDOC.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.10-1.1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 2dcd736e46ded79a53339462bc251694b150f870'
  cpuUtilization:
    idlePercent: 99.72
    systemPercent: 0.13
    userPercent: 0.15
  cpus: 32
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2042
  hostname: cachyos
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.7.3-1-cachyos-eevdf-lto
  linkmode: dynamic
  logDriver: journald
  memFree: 22890782720
  memTotal: 33635684352
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: /usr/lib/podman/netavark is owned by netavark 1.10.2-1.1
    path: /usr/lib/podman/netavark
    version: netavark 1.10.2
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.14-1.1
    path: /usr/bin/crun
    version: |-
      crun version 1.14
      commit: 667e6ebd4e2442d39512e63215e79d693d0780aa
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1.1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 3h 5m 28.00s (Approximately 0.12 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 1
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 214374158336
  graphRootUsed: 113286766592
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 268
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.9.1
  Built: 1706799366
  BuiltTime: Thu Feb  1 15:56:06 2024
  GitCommit: 118829d7fc68c34d5a317cda90b69884f3446f5c-dirty
  GoVersion: go1.21.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Related: containers/buildah#3474

Additional information

To workaround this issue, simply change the name of the HEREDOC (EXE -> V2):

RUN <<V2
    echo 1
    echo 2
    # This will now be executed:
    echo 3
V2

[rhatdan]

@flouthoc PTAL

[GrabbenD]

Alternative 2

I found the workaround from above to be tedious, instead I'll be using this temporarily (for multi-line commands with comments):

RUN echo 1 \
 && echo 2 \
    still 2 \
 # Comments work!
 && echo 3

STEP 2/3: RUN echo 1 && echo 2 still 2 && echo 3
1
2 still 2
3

[flouthoc]

@GrabbenD I think I fixed it here: containers/buildah#5261

[flouthoc]

Closing this since this looks already resolved, please feel free to re-open if you think this is still a bug.