BUG: rolling with datetime ArrowDtype #56370
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mroeschke
added
Datetime
| @@ -2514,4 +2515,8 @@ def dtype_to_unit(dtype: DatetimeTZDtype | np.dtype) -> str: | |||
| """ | |||
| if isinstance(dtype, DatetimeTZDtype): | |||
| return dtype.unit | |||
| elif isinstance(dtype, ArrowDtype): | |||
| if dtype.kind not in "mM": | |||
| raise ValueError(f"{dtype=} does not have a resolution.") | |||
There was a problem hiding this comment.
I don't think so? (If you're referring to dtype=, it was intentional to render e.g. "dtype=int64[pyarrow] does not have a resolution")
There was a problem hiding this comment.
maybe im misunderstanding how f-strings work. id expect this to render "int64[pyarrow]= does not have a resolution"
There was a problem hiding this comment.
In [2]: import pandas as pd
In [3]: import pyarrow as pa
In [4]: dtype = pd.ArrowDtype(pa.int64())
In [5]: f"{dtype=} does not have a resolution"
Out[5]: 'dtype=int64[pyarrow] does not have a resolution'| if isinstance(self._on, (PeriodIndex, DatetimeIndex, TimedeltaIndex)): | ||
| return self._on.asi8 | ||
| elif isinstance(self._on.dtype, ArrowDtype): | ||
| return self._on.to_numpy(dtype=np.int64) |
There was a problem hiding this comment.
comment/assertion about self._on.dtype.kind?
There was a problem hiding this comment.
Yup good point. Added a kind check
pandas/core/window/rolling.py
Outdated
| @@ -1871,7 +1875,7 @@ def _validate(self): | |||
| # we allow rolling on a datetimelike index | |||
| if ( | |||
| self.obj.empty | |||
| or isinstance(self._on, (DatetimeIndex, TimedeltaIndex, PeriodIndex)) | |||
| or (isinstance(self._on, PeriodIndex) or self._on.dtype.kind in "Mm") | |||
There was a problem hiding this comment.
id err on the side of being more explicit about the supported dtypes being ArrowDtype
|
Since there's no more comments here going to merge. Can follow up if needed |
meeseeksmachine
pushed a commit
to meeseeksmachine/pandas
that referenced
this pull request
Dec 28, 2023
phofl
pushed a commit
that referenced
this pull request
Dec 28, 2023
…type) (#56665) Backport PR #56370: BUG: rolling with datetime ArrowDtype Co-authored-by: Matthew Roeschke <[email protected]>
cbpygit
pushed a commit
to cbpygit/pandas
that referenced
this pull request
Jan 2, 2024
* BUG: rolling with datetime ArrowDtype * Dont modify needs_i8_conversion * More explicit tests * Fix arrow to_numpy
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.Additionally renamed _apply_blockwise to _apply_columnwise to fit the actual methodology