[DO NOT MERGE] Use distroless to build the python container #29001

liferoad · 2023-10-16T00:36:08Z

distroless does not provide the images with different python versions. And the Beam python containers also contains extras system packages.

Ideas:

Use FROM python:"${py_version}"-bookworm as python-base as the python base image to install everything Beam needs including both system packages and python packages under /venv
copy all the packages over to the distroless image

Using the python bullseye as the base,

Using the distroless as the base,

Fix #28991

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

codecov · 2023-10-16T00:44:38Z

Codecov Report

Merging #29001 (a8dbd7b) into master (87ca614) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #29001   +/-   ##
=======================================
  Coverage   38.29%   38.30%           
=======================================
  Files         690      690           
  Lines      102048   102048           
=======================================
+ Hits        39082    39085    +3     
+ Misses      61382    61380    -2     
+ Partials     1584     1583    -1

Flag	Coverage Δ
go	`53.45% <ø> (+<0.01%)`	⬆️
python	`29.88% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

sdks/python/container/Dockerfile

bvolpato

Thanks! It makes sense that we have to use multi-stage, and this looks a bit more complex.

Can we write some header comments just sharing in general what is the Dockerfile doing? This will help others get some context instead of figuring out why we have some duplicated steps.

sdks/python/container/Dockerfile

robertwb · 2023-10-18T22:27:32Z

What is the underlying motivation here?

…

On Wed, Oct 18, 2023, 3:24 PM tvalentyn ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sdks/python/container/Dockerfile <#29001 (comment)>: > - pip install --upgrade pip setuptools wheel && \ +RUN python -m venv venv-beam Beam SDK workers create a venv as well, which is superimposed upon the default environment. I think having Beam dependencies packages in a separate venv would break this setup. — Reply to this email directly, view it on GitHub <#29001 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADWVAOZSTTFQKPC3E7CVYLYABJKDAVCNFSM6AAAAAA6BMBU6WVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTMOBWGMYDANRTGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

sdks/python/container/Dockerfile

tvalentyn · 2023-10-18T22:39:55Z

I also wonder if the maintainers of python images are aware of the potential vulnerability concerns and perhaps those issues could be resolved upstream of Beam without us not getting into building Python images from scratch.

liferoad · 2023-10-19T13:32:36Z

What is the underlying motivation here?
…
On Wed, Oct 18, 2023, 3:24 PM tvalentyn @.> wrote: @.* commented on this pull request. ------------------------------ In sdks/python/container/Dockerfile <#29001 (comment)>: > - pip install --upgrade pip setuptools wheel && \ +RUN python -m venv venv-beam Beam SDK workers create a venv as well, which is superimposed upon the default environment. I think having Beam dependencies packages in a separate venv would break this setup. — Reply to this email directly, view it on GitHub <#29001 (review)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADWVAOZSTTFQKPC3E7CVYLYABJKDAVCNFSM6AAAAAA6BMBU6WVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTMOBWGMYDANRTGE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

We just create the temp venv in the python-base image to install all the packages and later copy them to the distroless image.

liferoad · 2023-10-28T04:46:09Z

Tested the wordcount example using Dataflow. It works.

github-actions · 2023-10-28T06:06:37Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @AnandInguva for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

liferoad · 2023-10-28T12:17:58Z

Run Python_Transforms PreCommit

sdks/python/container/Dockerfile

tvalentyn · 2023-11-02T17:26:32Z

sdks/python/container/Dockerfile

+ARG TARGETARCH
+
+# copy commands & libs to distroless
+COPY --from=python-base /bin /bin


for my information:

how did we choose what content to copy from base?

do distroless images have a package manager (apt) ? Will users be able to install additional software into these images if they want?

/bin contains some system commands like ls. We do not need them in theory.

no apt on distroless. If you are talking about the customer container, it will be better for users to use other images as base or use the way in this PR to copy the packages over.

I think we have to announce this in breaking changes in CHANGES.MD and in make adjustments to custom container documentation in Beam / Dataflow docs. I do see that this could complicate the UX for some custom container users.

Sounds good. I will update CHANGES.md after we all agree with using distroless.

Not a blocker here, just a general comment: I'd rather pick specific binaries rather than copying every /bin binary. I understand it probably means more breakages, but it's also a smaller vulnerability surface.

I think lack of apt is the main inconvenience for me. I think this is a real friction point, while the vulnerabilities reported by docker image checkers more often than not, are not applicable to execution of Beam pipelines.

I do not think we need /bin. I added it here only for the convenience. @tvalentyn I doubt users care about apt since if they need to touch this file, they usually build their own images. The purpose for us to adopt distorless is only to reduce the vulnerabilities when users use our containers as the default.

Before custom containers we also mentioned running custom commands in:

https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#nonpython

Not sure how much usage this has though.

sdks/python/container/Dockerfile

tvalentyn · 2023-11-03T16:57:12Z

Run Python ValidatesContainer Dataflow ARM 3.8

tvalentyn · 2023-11-03T16:57:26Z

Run Python ValidatesContainer Dataflow ARM 3.11

tvalentyn · 2023-11-03T16:58:27Z

Run Python Dataflow ValidatesContainer

tvalentyn · 2023-11-03T16:58:54Z

Run Python ValidatesContainer Dataflow ARM 3.11

liferoad · 2023-11-04T14:38:05Z

Run Python_Integration PreCommit 3.8

liferoad · 2023-11-07T04:05:08Z

Run Python Dataflow ValidatesContainer

liferoad · 2023-11-07T04:05:24Z

Run Python ValidatesContainer Dataflow ARM 3.11

tvalentyn · 2023-11-08T03:18:08Z

I can find test results - could you paste a link for container test suites please after we iron out any remaining issues with test infra?

tvalentyn · 2023-11-08T03:18:31Z

I am not sure why ARM suite is not showing up, it's on github actions, I thought only Jenkins had issues

tvalentyn · 2023-11-08T03:18:52Z

Run Python ValidatesContainer Dataflow ARM 3.8

tvalentyn · 2023-11-08T03:18:56Z

Run Python ValidatesContainer Dataflow ARM

tvalentyn · 2023-11-08T03:19:27Z

Run Python 3.8 Postcommit

tvalentyn · 2023-11-08T03:21:39Z

Regular postcommits wouldn't pick up these changes. ARM postcommits should

tvalentyn · 2023-11-08T03:21:43Z

Run Python PostCommit Arm

github-actions · 2023-11-17T12:13:55Z

Reminder, please take a look at this pr: @AnandInguva

github-actions · 2023-11-21T12:14:03Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @riteshghorse for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

liferoad · 2023-11-21T14:59:19Z

Close this for now since we plan to support more container images with different purposes.

liferoad added 2 commits October 13, 2023 17:15

use distroless to build python sdk docker images

9d7dbf8

[TEST] Build python container using distroless

6e355ae

github-actions bot added python docker labels Oct 16, 2023

add more libs

ad1720b

bvolpato reviewed Oct 16, 2023

View reviewed changes

sdks/python/container/Dockerfile Show resolved Hide resolved

bvolpato reviewed Oct 16, 2023

View reviewed changes

sdks/python/container/Dockerfile Outdated Show resolved Hide resolved

bvolpato reviewed Oct 16, 2023

View reviewed changes

sdks/python/container/Dockerfile Outdated Show resolved Hide resolved

bvolpato reviewed Oct 16, 2023

View reviewed changes

tvalentyn reviewed Oct 18, 2023

View reviewed changes

sdks/python/container/Dockerfile Outdated Show resolved Hide resolved

tvalentyn reviewed Oct 18, 2023

View reviewed changes

sdks/python/container/Dockerfile Outdated Show resolved Hide resolved

sdks/python/container/Dockerfile Outdated Show resolved Hide resolved

liferoad added 4 commits October 27, 2023 10:28

Merge branch 'master' into python-distroless-1

f7b6009

polished the Dockerfile

ca64035

polish the Dockerfile

c2383b5

polished the copy

53bdcb4

liferoad marked this pull request as ready for review October 28, 2023 04:45

liferoad requested review from tvalentyn and bvolpato October 28, 2023 04:45

github-actions bot added the Next Action: Reviewers label Oct 28, 2023

tvalentyn reviewed Nov 2, 2023

View reviewed changes

liferoad added 2 commits November 4, 2023 07:49

Merge branch 'master' into python-distroless-1

95026a5

addressed some comments

a8dbd7b

bvolpato approved these changes Nov 7, 2023

View reviewed changes

liferoad changed the title ~~Use distroless to build the python container~~ [DO NOT MERGE] Use distroless to build the python container Nov 9, 2023

github-actions bot added the slow-review label Nov 17, 2023

github-actions bot removed the slow-review label Nov 21, 2023

liferoad closed this Nov 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE] Use distroless to build the python container #29001

[DO NOT MERGE] Use distroless to build the python container #29001

liferoad commented Oct 16, 2023 •

edited

Loading

codecov bot commented Oct 16, 2023 •

edited

Loading

bvolpato left a comment

robertwb commented Oct 18, 2023 via email

tvalentyn commented Oct 18, 2023

liferoad commented Oct 19, 2023

liferoad commented Oct 28, 2023

github-actions bot commented Oct 28, 2023

liferoad commented Oct 28, 2023

tvalentyn Nov 2, 2023

liferoad Nov 2, 2023

tvalentyn Nov 3, 2023

liferoad Nov 4, 2023

bvolpato Nov 7, 2023

tvalentyn Nov 7, 2023 •

edited

Loading

liferoad Nov 7, 2023

tvalentyn Nov 8, 2023

tvalentyn commented Nov 3, 2023

tvalentyn commented Nov 3, 2023

tvalentyn commented Nov 3, 2023

tvalentyn commented Nov 3, 2023

liferoad commented Nov 4, 2023

liferoad commented Nov 7, 2023

liferoad commented Nov 7, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Nov 21, 2023

liferoad commented Nov 21, 2023

[DO NOT MERGE] Use distroless to build the python container #29001

[DO NOT MERGE] Use distroless to build the python container #29001

Conversation

liferoad commented Oct 16, 2023 • edited Loading

GitHub Actions Tests Status (on master branch)

codecov bot commented Oct 16, 2023 • edited Loading

Codecov Report

bvolpato left a comment

Choose a reason for hiding this comment

robertwb commented Oct 18, 2023 via email

tvalentyn commented Oct 18, 2023

liferoad commented Oct 19, 2023

liferoad commented Oct 28, 2023

github-actions bot commented Oct 28, 2023

liferoad commented Oct 28, 2023

tvalentyn Nov 2, 2023

Choose a reason for hiding this comment

liferoad Nov 2, 2023

Choose a reason for hiding this comment

tvalentyn Nov 3, 2023

Choose a reason for hiding this comment

liferoad Nov 4, 2023

Choose a reason for hiding this comment

bvolpato Nov 7, 2023

Choose a reason for hiding this comment

tvalentyn Nov 7, 2023 • edited Loading

Choose a reason for hiding this comment

liferoad Nov 7, 2023

Choose a reason for hiding this comment

tvalentyn Nov 8, 2023

Choose a reason for hiding this comment

tvalentyn commented Nov 3, 2023

tvalentyn commented Nov 3, 2023

tvalentyn commented Nov 3, 2023

tvalentyn commented Nov 3, 2023

liferoad commented Nov 4, 2023

liferoad commented Nov 7, 2023

liferoad commented Nov 7, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

tvalentyn commented Nov 8, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Nov 21, 2023

liferoad commented Nov 21, 2023

liferoad commented Oct 16, 2023 •

edited

Loading

codecov bot commented Oct 16, 2023 •

edited

Loading

tvalentyn Nov 7, 2023 •

edited

Loading