[Bug]: SizeLimitExceededException thrown while sending a small message to GCP PubSub

[jkosternl]

What happened?

Since Beam v2.48 (Java), there is a bug introduced in the logic that verifies the total size of the message send to PubSub. The message size is checked against the amount of messages inside a batch (100). In v2.47 and before this was working fine.

Effect of the bug
Currently, in v2.48, it is not possible to use PubsubIO class to send messages to GCP PubSub from a Pipeline. It will cause a SizeLimitExceededException thrown like this:
"Pubsub message of length XXX exceeds maximum of 100 bytes, when considering the payload and attributes. See https://cloud.google.com/pubsub/quotas#resource_limits "

Cause
The bug is caused by commit specifically file sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java at line 100, committed by @reuvenlax

Solution
In Beam v2.47, a similar check was performed against maxPublishBatchByteSize in file PubsubIO which is totally fine. So, please use the idea of comparing bytes with bytes, as was done there.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[chamikaramj]

This seems like a regression for 2.48.0 caused by #26063. @reuvenlax can you check ?

[chamikaramj]

Also making this a blocker for the 2.49.0 release.

[chamikaramj]

Ah, seems like it was already fixed: #27000

Closing this issue. Please re-open if the issue was not addressed by the above.

[jkosternl]

Ah yes, you're right. I didn't find that bug while searching, unfortunately. Thanks @chamikaramj