[Feature Request]: Beam SDK as a pure python package

[prassanna-ravishankar]

What would you like to happen?

While writing a dockerfile for beam python runners, we have to do the following

COPY --from=apache/beam_python3.9_sdk:2.55.1 /opt/apache/beam /opt/apache/beam

Is there any way we can introduce this as a python package instead (with necessary shared libraries)? It'll help us use poetry like resolvers to better manage dependency issues. CUDA, for example is able to do this.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[liferoad]

That is just the entrypoint for Beam Runner. No need to put that into poetry. All you need is to put the Beam version as the variable when you build the image, for example, https://github.com/google/dataflow-ml-starter/blob/main/tensorflow_gpu.Dockerfile

[prassanna-ravishankar]

I might be misunderstanding something. If one uses things like pyenv to maintain python version, this doesn't really work well with an alternate entrypoint.

[liferoad]

/opt/apache/beam is written in Go. As long as you copy it in your Dockerfile, it does not matter.