numpy.int64 type is not serialized correctly in Python 3.11 and Python 3.12 #33137
Conversation
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Assigning reviewers. If you would like to opt out of this review, comment R: @jrmccluskey for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
Failed workflows are not related to this PR. |
| @@ -510,6 +510,9 @@ def encode_special_deterministic(self, value, stream): | |||
| stream.write_byte(NESTED_STATE_TYPE) | |||
| self.encode_type(type(value), stream) | |||
| state_value = value.__getstate__() | |||
| if value is not None and state_value is None: | |||
There was a problem hiding this comment.
Why are we checking that value is not None here? I don't think we want to use this protocol for None.
There was a problem hiding this comment.
This just indicates __getstate__ might not be implemented. And we know this is true for this if checks.
| @@ -510,6 +510,9 @@ def encode_special_deterministic(self, value, stream): | |||
| stream.write_byte(NESTED_STATE_TYPE) | |||
| self.encode_type(type(value), stream) | |||
| state_value = value.__getstate__() | |||
There was a problem hiding this comment.
I am concerned that just checking that this value is not None is insufficient, as objects with custom pickling protocols can still use this method and the root object now defines a default for anything with dict or slots.
There was a problem hiding this comment.
I agree it is insufficient.
Fixes #33020 and #33021. Basically, Beam should return TypeError like
TypeError: Unable to deterministically encode '2000-01-01' of type '<class 'numpy.datetime64'>', please provide a type hint for the input of 'GroupByKey' [while running 'CreatePInput0/Map(decode)']. The user needs to define their own deterministic coder to use this type as the groupby key.This is a hacky fix by checking whether
__getstate__returns reasonable results.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.