-
Notifications
You must be signed in to change notification settings - Fork 293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"pack builder create" runs out of memory when using pack 0.30 #1886
Comments
I think this bug was generated by the changes made to the builder creation to support the note We are planning to simplify the |
I was able to reproduce the error with the steps provided by @bruthphorce (but using a 4GB file). I ran a container with a memory limit to simulate the environment. docker run -v /var/run/docker.sock:/var/run/docker.sock -v ./:/home/root -it --memory="2g" --memory-reservation="1g" ubuntu /bin/bash Once inside the container, I ran the Creating builder with the following buildpacks:
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> [email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
gc 7 @13.557s 0%: 0.055+4.7+0.076 ms clock, 0.44+0/4.6/14+0.60 ms cpu, 5->6->3 MB, 5 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 8 @13.570s 0%: 0.070+8.2+0.003 ms clock, 0.56+0/8.3/1.1+0.025 ms cpu, 7->7->3 MB, 7 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 9 @13.590s 0%: 0.066+9.5+0.092 ms clock, 0.52+0/10/0.85+0.74 ms cpu, 9->9->4 MB, 9 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 10 @13.615s 0%: 0.059+5.2+0.33 ms clock, 0.47+0.22/2.6/0.59+2.7 ms cpu, 9->9->3 MB, 9 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 11 @13.631s 0%: 0.046+12+0.074 ms clock, 0.37+0.066/1.7/0.75+0.59 ms cpu, 6->9->6 MB, 7 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 12 @13.667s 0%: 0.099+13+0.12 ms clock, 0.79+0/13/0.77+1.0 ms cpu, 15->15->5 MB, 15 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 13 @13.695s 0%: 0.041+12+0.14 ms clock, 0.33+0/12/0.76+1.1 ms cpu, 15->15->7 MB, 15 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 14 @13.716s 0%: 0.038+36+0.096 ms clock, 0.30+0/37/0.95+0.77 ms cpu, 14->14->9 MB, 16 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 15 @13.775s 0%: 0.048+6.8+0.10 ms clock, 0.38+0/7.1/0.34+0.87 ms cpu, 27->27->12 MB, 27 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 16 @13.797s 0%: 0.042+12+0.096 ms clock, 0.33+0/12/0.94+0.77 ms cpu, 25->25->15 MB, 25 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 17 @13.826s 0%: 0.070+25+0.082 ms clock, 0.56+0/25/0.79+0.65 ms cpu, 31->31->18 MB, 31 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 18 @13.872s 0%: 0.045+24+0.069 ms clock, 0.36+0/25/1.3+0.55 ms cpu, 38->38->22 MB, 38 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 19 @13.912s 0%: 0.041+3.7+0.079 ms clock, 0.33+0/3.9/0.73+0.63 ms cpu, 47->47->27 MB, 47 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 20 @13.959s 0%: 0.039+29+0.095 ms clock, 0.31+0/29/0.98+0.76 ms cpu, 59->59->33 MB, 59 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 21 @14.120s 0%: 0.17+25+0.008 ms clock, 1.4+0/1.4/25+0.070 ms cpu, 67->67->7 MB, 67 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 22 @14.151s 0%: 0.33+2.0+0.003 ms clock, 2.6+0.24/2.9/0.50+0.028 ms cpu, 22->23->10 MB, 23 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 23 @14.160s 0%: 0.15+2.5+0.058 ms clock, 1.2+0/2.8/0.98+0.46 ms cpu, 33->33->15 MB, 33 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 24 @14.166s 0%: 0.18+3.4+0.12 ms clock, 1.4+0/3.7/0.78+1.0 ms cpu, 31->31->18 MB, 31 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 25 @14.174s 0%: 0.12+5.7+0.15 ms clock, 1.0+0/6.1/0.61+1.2 ms cpu, 38->38->22 MB, 38 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 26 @14.185s 0%: 0.13+40+0.067 ms clock, 1.0+0/40/0.98+0.53 ms cpu, 47->47->27 MB, 47 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 27 @14.234s 0%: 0.15+5.4+0.14 ms clock, 1.2+0/5.6/0.82+1.1 ms cpu, 58->58->33 MB, 58 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 28 @14.250s 0%: 0.038+43+0.003 ms clock, 0.30+0/43/0.85+0.024 ms cpu, 72->72->41 MB, 72 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 29 @14.306s 0%: 0.067+33+0.11 ms clock, 0.53+0/0.82/33+0.88 ms cpu, 89->89->50 MB, 89 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 30 @14.355s 0%: 0.045+11+0.11 ms clock, 0.36+0/1.2/12+0.94 ms cpu, 110->110->62 MB, 111 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 31 @14.384s 0%: 0.044+50+0.15 ms clock, 0.35+0/50/0.77+1.2 ms cpu, 138->138->77 MB, 138 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 32 @14.459s 0%: 0.040+53+0.13 ms clock, 0.32+0/53/0.83+1.0 ms cpu, 171->171->96 MB, 171 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 33 @14.541s 0%: 0.042+18+0.11 ms clock, 0.34+0/18/1.1+0.91 ms cpu, 214->214->120 MB, 214 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 34 @14.591s 0%: 0.045+29+0.12 ms clock, 0.36+0/0.87/29+0.97 ms cpu, 267->267->149 MB, 267 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 35 @14.667s 0%: 0.079+83+0.036 ms clock, 0.63+0/83/0.79+0.29 ms cpu, 333->333->186 MB, 333 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 36 @14.803s 0%: 0.086+33+0.16 ms clock, 0.69+0/33/1.3+1.3 ms cpu, 415->415->232 MB, 415 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 37 @14.911s 0%: 0.044+82+0.085 ms clock, 0.35+0/0.94/82+0.68 ms cpu, 519->519->289 MB, 519 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 38 @15.076s 0%: 0.038+133+0.30 ms clock, 0.30+0/133/0.95+2.4 ms cpu, 648->648->361 MB, 648 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 39 @15.314s 0%: 0.044+69+0.003 ms clock, 0.35+0/69/2.2+0.031 ms cpu, 809->809->450 MB, 809 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 40 @15.500s 0%: 0.045+154+0.32 ms clock, 0.36+0/154/1.1+2.5 ms cpu, 1011->1011->563 MB, 1011 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 41 @15.831s 0%: 0.049+1057+0.007 ms clock, 0.39+0/1.9/1057+0.058 ms cpu, 1263->1263->703 MB, 1263 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 42 @17.113s 2%: 0.039+2120+0.064 ms clock, 0.31+0/2120/1.6+0.51 ms cpu, 1578->1578->878 MB, 1578 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 43 @19.651s 4%: 0.057+6700+0.006 ms clock, 0.46+0/6697/1.3+0.048 ms cpu, 1972->1972->1097 MB, 1972 MB goal, 0 MB stacks, 0 MB globals, 8 P
Killed We can see how the heap memory is increasing #->#-># MB But when using pack 0.29 with the same configuration we get the following output Creating builder with the following buildpacks:
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> [email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
gc 7 @6.168s 0%: 0.042+1.5+0.16 ms clock, 0.34+0/1.7/0.89+1.3 ms cpu, 5->5->3 MB, 5 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 8 @6.178s 0%: 0.041+4.6+0.004 ms clock, 0.33+0/4.2/1.4+0.033 ms cpu, 7->7->3 MB, 6 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 9 @6.190s 0%: 0.043+1.5+0.006 ms clock, 0.34+0.53/1.4/1.0+0.048 ms cpu, 6->6->4 MB, 7 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 10 @6.198s 0%: 0.043+18+0.080 ms clock, 0.34+0/18/0.83+0.64 ms cpu, 9->9->5 MB, 9 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 11 @6.231s 0%: 0.048+3.4+0.24 ms clock, 0.38+0.16/1.7/0+1.9 ms cpu, 9->10->4 MB, 10 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 12 @6.245s 0%: 0.041+4.7+0.31 ms clock, 0.32+0/4.4/0.98+2.5 ms cpu, 9->9->4 MB, 9 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 13 @6.264s 0%: 0.073+3.5+0.005 ms clock, 0.58+0/3.2/2.1+0.043 ms cpu, 10->10->5 MB, 9 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 14 @6.296s 0%: 0.068+15+0.10 ms clock, 0.54+0/15/0.97+0.82 ms cpu, 15->15->7 MB, 12 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 15 @6.319s 0%: 0.036+9.3+0.005 ms clock, 0.29+0/9.6/1.3+0.040 ms cpu, 14->14->9 MB, 15 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 16 @6.338s 0%: 0.036+1.8+0.075 ms clock, 0.29+0/1.9/0.88+0.60 ms cpu, 17->17->10 MB, 18 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 17 @6.349s 0%: 0.042+12+0.092 ms clock, 0.33+0/12/1.0+0.73 ms cpu, 20->20->12 MB, 21 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 18 @6.377s 0%: 0.035+2.3+0.090 ms clock, 0.28+0/2.6/0.87+0.72 ms cpu, 25->25->15 MB, 25 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 19 @6.395s 0%: 0.035+32+0.099 ms clock, 0.28+0/32/0.67+0.79 ms cpu, 31->31->18 MB, 30 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 20 @6.448s 0%: 0.037+21+0.10 ms clock, 0.29+0/21/0.88+0.82 ms cpu, 38->38->22 MB, 37 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 21 @6.499s 0%: 0.038+5.3+0.18 ms clock, 0.30+0/5.6/1.0+1.4 ms cpu, 47->47->46 MB, 45 MB goal, 0 MB stacks, 0 MB globals, 8 P
Adding buildpack [email protected] (diffID=sha256:89ae2cc6518d1b76c06062be7fb5cd02405f155f9071f1d977cf821cbf73c61c)
Adding buildpack samples/[email protected] (diffID=sha256:a17feab82bbb1ec651086f769d4178125792d5dbf8913672222c958eab4ee5c0)
Adding buildpack samples/[email protected] (diffID=sha256:7c7dd7e20b20ec6da8b4c1a7d879d5e142e8380e6ae1486a99b72f203eadcdd3)
Adding buildpack samples/[email protected] (diffID=sha256:b4f741f100ff0cda0834efc43b8e61dccf08d538807c3359555f4db126dda1ad)
Adding buildpack samples/[email protected] (diffID=sha256:42b8a74a77084e1e9a7cbbd75a1418271bcf907359421b4a29a1e391bf35b34c)
Adding buildpack samples/[email protected] (diffID=sha256:851c15ba18fddb4d49b4759ba18c5e09c5de0e8b932f9350337bd2b9d6d695e2)
Adding buildpack samples/[email protected] (diffID=sha256:9d6b31f4fb43fd948fe010dc0e3f8dac6d2b2bdd8a2a1d9097ae9c4e3d2b7f0e)
Successfully created builder image my-builder:jammy
Tip: Run pack build <image-name> --builder my-builder:jammy to use this builder Digging into a little bit more, I noticed the problem must be happening reading the tar created during processing the buildpacks. The problem was here, using Creating builder with the following buildpacks:
-> [email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
-> samples/[email protected]
gc 8 @106.374s 0%: 0.048+0.92+0.065 ms clock, 0.39+0.089/0.97/0.96+0.52 ms cpu, 5->5->2 MB, 5 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 9 @106.382s 0%: 0.039+6.2+0.20 ms clock, 0.31+0/5.9/0.65+1.6 ms cpu, 5->5->3 MB, 5 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 10 @106.397s 0%: 0.043+2.2+0.17 ms clock, 0.35+0/2.4/1.2+1.4 ms cpu, 6->6->3 MB, 6 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 11 @106.406s 0%: 0.046+2.2+0.14 ms clock, 0.37+0/2.2/0.97+1.1 ms cpu, 7->7->4 MB, 8 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 12 @106.423s 0%: 0.037+1.4+0.097 ms clock, 0.29+0/1.3/0.88+0.77 ms cpu, 9->9->3 MB, 9 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 13 @106.432s 0%: 0.039+9.2+0.17 ms clock, 0.31+0/8.6/0.78+1.4 ms cpu, 6->6->3 MB, 6 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 14 @106.453s 0%: 0.048+7.9+0.17 ms clock, 0.38+0/2.4/0.22+1.4 ms cpu, 8->9->4 MB, 9 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 15 @106.468s 0%: 0.040+2.4+0.16 ms clock, 0.32+0/2.5/0.78+1.3 ms cpu, 10->10->5 MB, 10 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 16 @106.503s 0%: 0.043+6.3+0.21 ms clock, 0.34+0/7.2/0+1.7 ms cpu, 15->15->7 MB, 15 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 17 @106.546s 0%: 0.051+8.1+0.34 ms clock, 0.41+0/2.0/0.74+2.7 ms cpu, 22->22->17 MB, 22 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 18 @106.591s 0%: 0.038+6.9+0.003 ms clock, 0.31+0/7.1/1.4+0.031 ms cpu, 40->40->15 MB, 40 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 19 @106.619s 0%: 0.065+72+0.097 ms clock, 0.52+0/73/1.8+0.77 ms cpu, 31->31->18 MB, 31 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 20 @106.722s 0%: 0.046+12+0.13 ms clock, 0.36+0/12/0.64+1.0 ms cpu, 38->38->22 MB, 38 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 21 @106.755s 0%: 0.044+10+0.13 ms clock, 0.35+0/3.7/15+1.0 ms cpu, 47->47->27 MB, 47 MB goal, 0 MB stacks, 0 MB globals, 8 P
gc 22 @106.827s 0%: 0.043+25+0.091 ms clock, 0.34+0/1.3/24+0.73 ms cpu, 59->59->33 MB, 59 MB goal, 0 MB stacks, 0 MB globals, 8 P
Adding buildpack [email protected] (diffID=sha256:89ae2cc6518d1b76c06062be7fb5cd02405f155f9071f1d977cf821cbf73c61c)
Adding buildpack samples/[email protected] (diffID=sha256:a17feab82bbb1ec651086f769d4178125792d5dbf8913672222c958eab4ee5c0)
Adding buildpack samples/[email protected] (diffID=sha256:7c7dd7e20b20ec6da8b4c1a7d879d5e142e8380e6ae1486a99b72f203eadcdd3)
Adding buildpack samples/[email protected] (diffID=sha256:b4f741f100ff0cda0834efc43b8e61dccf08d538807c3359555f4db126dda1ad)
Adding buildpack samples/[email protected] (diffID=sha256:42b8a74a77084e1e9a7cbbd75a1418271bcf907359421b4a29a1e391bf35b34c)
Adding buildpack samples/[email protected] (diffID=sha256:851c15ba18fddb4d49b4759ba18c5e09c5de0e8b932f9350337bd2b9d6d695e2)
Adding buildpack samples/[email protected] (diffID=sha256:9d6b31f4fb43fd948fe010dc0e3f8dac6d2b2bdd8a2a1d9097ae9c4e3d2b7f0e)
Successfully created builder image my-builder:jammy-0.30
Tip: Run pack build <image-name> --builder my-builder:jammy-0.30 to use this builder |
Awesome, thank you so much! |
Summary
We are running a
pack builder create
in our CICD pipeline on runners that have 7GB of memory available. One of the buildpacks included in the builder is 4GB in size (because of GPU drivers necessary to be in there). Up until pack 0.29, this was working fine.However, when testing the upgrade to pack 0.30, the
pack builder create
command fails with exit code 137. The last logs that are written:The process crashes approximately 30 seconds after this.
When we run the same process locally we initially didn't encounter any problems with pack 0.30, it seems to work fine given that enough memory is available. As a rough guess, I would say the process fails once the size of the buildpacks exceeds the available memory divided by 2.
Reproduction
Steps
I have reproduced the issue as follows:
Create directory:
mkdir buildpacks/memory-test
Define include-files:
vi buildpacks/memory-test/buildpack.toml
Create sample large file:
dd if=/dev/zero of=buildpacks/memory-test/testfile bs=1G count=1
Then running the following command works fine using pack 0.29:
pack builder create my-builder:jammy --config ./builder.toml
But using pack 0.30 it fails:
echo $?
returns137
Current behavior
Pack exits with an error while running
pack builder create
Expected behavior
Builder creation succeeds as before
Environment
Microsoft-hosted agent for Azure, Colima
pack info
From Colima:
docker info
From Colima:
The text was updated successfully, but these errors were encountered: