[Bug]: ProtoCoder fails on super large protobufs

[snallapa]

What happened?

ProtoCoder can't encode messages that are above the Integer.MAX_VALUE bytes (2.147483647gb). This is due to the following https://github.com/protocolbuffers/protobuf/blob/d41b9e7a4b2834572cc396c41809fd46ea5eb5d4/java/core/src/main/java/com/google/protobuf/MessageLite.java#L55 can return negative. It seems like there are a lot of undefined behaviors when there are protobuf this large.

However, it at least seems that ProtoEncoder can check the size and writeDelimited to the output stream to solve this in this method

beam/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java

Line 190 in b876299

if (context.isWholeStream) {

Naturally, no one should probably have protobuf's this large anyways, but here I am!

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[liferoad]

cc @robertwb @kennknowles

[kennknowles]

@snallapa want to contribute the fix? 😄 😄 😄

[snallapa]

@kennknowles if I find the time in the next week or so I will try to!