[Bug]: Prism runner "received message larger than max" error

[miracvbasaran]

What happened?

While updating Privacy on Beam's dependency on Apache Beam Go to version v2.51.0, I am receiving the following error:

2023/10/26 15:12:58 stopping worker job-001[go-job-1-1698325959973818582]_go
    distinct_id_test.go:318: DistinctPrivacyID didn't add any Laplace noise: job failed to prepare
        	caused by:
        rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4236507 vs. 4194304)
        error may be due to Apache Beam Go's migration from the direct runner to the prism runner. While the failure(s) should be fixed, you can continue to use the direct runner with this TestMain override: `func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }`
--- FAIL: TestDistinctPrivacyIDAddsNoise (19.35s)

on this test. Curiously, it works fine for all the other tests.

Is this a known issue? Are there any actions I can take to resolve this?

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[lostluck]

Ah good catch. I thought I increased message size to maximum everywhere.

Do note that prism isn't yet optimized for performance. It will presently be slower than the direct runner. In the short term though, as the error says, override to use the direct runner.

[lostluck]

OH! Job failed to prepare means it's failing on pipeline submission.

Your pipeline is > 4MB serialized.

(Still can be fixed however.)

[miracvbasaran]

Thanks for the fix!