Introduce artifact max size limit of 50MiB #929
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stefanprodan
force-pushed
the
artifact-max-size
branch
from
0979fcc to
13b0783
Compare
stefanprodan
force-pushed
the
artifact-max-size
branch
from
13b0783 to
c68d2dd
Compare
pjbgf
reviewed
Oct 10, 2022
controllers/storage.go
Outdated
Comment on lines
439
to
444
| if s.ArtifactMaxSize > 0 && sz.written > s.ArtifactMaxSize { | ||
| return fmt.Errorf("artifact size %d exceeds the max limit of %d bytes", sz.written, s.ArtifactMaxSize) | ||
| } | ||
|
|
There was a problem hiding this comment.
Is the writerCounter incremented mid-flight as a stream? If so, within Walk we could short-circuit this process and stop other operations once the pre-set limit is reached.
There was a problem hiding this comment.
Yes but my intention here was to tell users which is the final size of the artifact.
There was a problem hiding this comment.
I would short circuit the process instead, as otherwise half of the usefulness of the bound is lost.
There was a problem hiding this comment.
If I move the check inside Walk then sz.written is always zero.
stefanprodan
force-pushed
the
artifact-max-size
branch
from
c68d2dd to
0dc411f
Compare
darkowlzz
reviewed
Oct 11, 2022
darkowlzz
reviewed
Oct 11, 2022
darkowlzz
reviewed
Oct 11, 2022
darkowlzz
reviewed
Oct 11, 2022
stefanprodan
force-pushed
the
artifact-max-size
branch
2 times, most recently
from
dd1e55b to
af969cf
Compare
Add a controller flag named "--artifact-max-size=<bytes>" with the default value of 50MiB. To disable the limit, the value can be set to "--artifact-max-size=-1". The flag enforces a max size limit for the artifact contents produced by source-controller, to avoid out-of-memory crashes of consumers such as kustomize-controller. Signed-off-by: Stefan Prodan <[email protected]>
stefanprodan
force-pushed
the
artifact-max-size
branch
from
af969cf to
3168bb6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a controller flag named
--artifact-max-size=<bytes>with the default value of 50MiB. To disable the limit, the value can be set to--artifact-max-size=-1.The flag enforces a max size limit for the artifact contents produced by source-controller, to avoid out-of-memory crashes of consumers such as kustomize-controller.
Closes: #901