Source file mtimes are always set to Jan 1, 1980

[matthewrobertson]

Summary

All buildpack builders reset the mtime of all application source files to Jan 1, 1980, even if SOURCE_DATE_EPOCH or the pack --creation-time flag is set.

---

Reproduction

See the sample app here: https://github.com/matthewrobertson/bp-mtime-issue

Steps

(more comprehensive instructions in the linked repo)

1. clone the sample app: https://github.com/matthewrobertson/bp-mtime-issue
2. Build the app:

pack build bp-mtime-issue \
  --builder docker.io/paketobuildpacks/builder-jammy-full:latest \
  --creation-time now

3. Run the app:

docker run -it -ePORT=8080 -p8080:8080 bp-mtime-issue

4. Load the app in your browser: http://localhost:8080/

5. Repeat steps 1 - 3.

Current behavior

The content on the page does not change unless you do hard refresh in the browser because of HTTP caching. This is because the public/index.js file in the container has an mtime of Jan 1, 1980...:

docker run --entrypoint launcher -it bp-mtime-issue "ls -al ./public"
total 12
drwxr-x--- 2 1001 cnb 4096 Jan  1  1980 .
drwxrwxrwx 4 1001 cnb 4096 Jan  1  1980 ..
-rw-r----- 1 1001 cnb   63 Jan  1  1980 index.html

Expected behavior

I would expect the cache to get busted every time I rebuild because I passed the --creation-time now flag.

---

Context

lifecycle version

0.19.6

platform version(s)

❯ pack report
Pack:
  Version:  0.31.0+git-3a994bd.build-5086
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.17.1

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config:
  [[trusted-builders]]
    name = "[REDACTED]"

[natalieparellano]

@matthewrobertson I think this is actually by design. The RFC has a bit more detail, but basically we were worried that setting layer timestamps (as opposed to just image create time) would break caching. However, we've had a few requests for layer timestamps since this feature was introduced, so maybe we should add it as a configurable option. In the meantime, we can probably update the help text in pack to make it clearer that layer timestamps aren't included for now.

[matthewrobertson]

Thanks for the context @natalieparellano. I am actually not sure that SOURCE_DATE_EPOCH is even relevant here (although the behaviour was surprising so I wanted to highlight it). My opinion is that we shouldn't be changing the mtime timestamps of workspace files at all. Reproducible builds require identical inputs to produce identical outputs, but if an mtime has changed, then the input has changed and the digest of the image should change.

We have gotten a handful of customer bug reports that zeroing out mtimes causes caching issues in various HTTP frameworks. If we are worried about side effects of changing this, how would we feel about a new setting to disable it CNB_RESPECT_APP_DIR_MTIME?

[natalieparellano]

I'll bring this up for discussion at our next working group (Thurs 6/6).

[natalieparellano]

Hey @matthewrobertson we talked about this today - generally folks are in favor of supporting something like this. We would want a light-weight RFC in the style of https://github.com/buildpacks/rfcs/blob/main/text/0103-source-date-epoch.md. There was some debate about whether this should be the default behavior that folks opt out of, to be discussed further. In either case, it would require a Platform API version bump.

Implementation would require some care, as buildpacks can (and do) make changes to the app directory, and we would want to ensure that mtimes are zeroed out when buildpacks make changes as otherwise build reproducibility is broken. One possible path forward would be to zero out mtimes if the existing time is later than the build start time.

Also noted was the fact that buildpacks for some language families delete the source code entirely, so this change would be relevant in only a subset of cases.

Feel free to reach out if you'd like any help with writing the RFC. @AidanDelaney has kindly agreed to provide guidance here.

[matthewrobertson]

Sounds good. I will start working on something and try to attend the next working group meeting to discuss if I have any questions.

[joanna-polec]

Hi, do you have any update about this issue? Can we expect implementation of this feature any time soon?

[natalieparellano]

@joanna-polec this still requires an RFC, and someone to drive the implementation. If you are interested in taking this on, please let us know and a maintainer would be happy to guide you.