Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[prism] Java timer families cannot be looked up SDK side (prism re-write issue) #32221

Closed
Tracked by #29650
lostluck opened this issue Aug 16, 2024 · 3 comments · Fixed by #32223
Closed
Tracked by #29650

[prism] Java timer families cannot be looked up SDK side (prism re-write issue) #32221

lostluck opened this issue Aug 16, 2024 · 3 comments · Fixed by #32223
Assignees

Comments

@lostluck
Copy link
Contributor

lostluck commented Aug 16, 2024

Java SDK timer tests fail, being unable to look up timers SDK side, even though the Go and Python SDKs don't seem to have this issue.

The error occurs in BeamFnDataInboundObserver.

error="bundle inst003 stage-004 failed:java.lang.IllegalStateException: Unable to find inbound timer receiver for instruction \"inst003\", transform \"first-ParMultiDo-Anonymous-\", and timer family \"\" No endpoint. Available keys: [ts-foo]\n\tat org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements(BeamFnDataInboundObserver.java:193 ...

So we do see that there is a timer there, so question is why does it look like Prism isn't actually populating the timer family being sent to the SDK.

Full Suite iteration (two suites)

TEST=org.apache.beam.sdk.transforms.ParDoTest\$TimerTests
./gradlew :runners:portability:java:ulrLoopbackValidatesRunnerTests -PjobEndpoint=localhost:8073 --tests="$TEST"

31 tests completed, 19 failed

TEST=org.apache.beam.sdk.transforms.ParDoTest\$TimerFamilyTests
./gradlew :runners:portability:java:ulrLoopbackValidatesRunnerTests -PjobEndpoint=localhost:8073 --tests="$TEST"

7 tests completed, 6 failed

Going to iterate with the following to investigate.

TEST=org.apache.beam.sdk.transforms.ParDoTest\$TimerTests.testEventTimeTimerLoop

@lostluck
Copy link
Contributor Author

The answer is... prism isn't closing the timer stream properly. It's apparently per timer family not just per transform.

@lostluck
Copy link
Contributor Author

I now have a quick fix for the root of the timer issue for Java. Need to validate that Python and Go continue to function as well.

But now the remaining test appears to have found a real bug in Prism! AFAICT this test will never terminate.
org.apache.beam.sdk.transforms.ParDoTest\$TimerTests.testEventTimeTimerOrderingWithCreate

I'll need to statically read what the test is doing to see how it should terminate, so I can figure out what prism is doing wrong.

@lostluck
Copy link
Contributor Author

Filed #32222 for the non-terminating test.

lostluck added a commit to lostluck/beam that referenced this issue Aug 19, 2024
lostluck added a commit that referenced this issue Aug 20, 2024
…r. (#32223)

* [#32221] Mark is-last for each timer stream correctly.

* Remove test override for clear, since it now passes unmodified.

* delint

---------

Co-authored-by: lostluck <[email protected]>
@github-actions github-actions bot added this to the 2.59.0 Release milestone Aug 20, 2024
reeba212 pushed a commit to reeba212/beam that referenced this issue Dec 4, 2024
…rm pair. (apache#32223)

* [apache#32221] Mark is-last for each timer stream correctly.

* Remove test override for clear, since it now passes unmodified.

* delint

---------

Co-authored-by: lostluck <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant