Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Index.getitem returning wrong result with negative step for arrow #55832

Merged
merged 13 commits into from
Nov 22, 2023

Conversation

phofl
Copy link
Member

@phofl phofl commented Nov 5, 2023

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Tests added and passed if fixing a bug or adding a new feature
  • All code checks passed.
  • Added type annotations to new arguments/methods/functions.
  • Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
idx = Index(list("bcdxy"), dtype="string[pyarrow_numpy]")
idx[slice(4, -6, -1)]

This omits b for arrow dtypes and looks like a bug on the arrow side, cc @jorisvandenbossche

@phofl phofl added this to the 2.1.3 milestone Nov 5, 2023
if isinstance(item, slice):
if item.start == item.stop:
pass
elif item.start == -len(self) - 1:
Copy link
Member

@lukemanley lukemanley Nov 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be item.start < -len(self)?

Not sure if you want to mimic numpy behavior here but numpy allows slicing beyond the len of the array:

In [1]: import numpy as np

In [2]: np.array(["a", "b", "c"])[slice(-10, 10)]
Out[2]: array(['a', 'b', 'c'], dtype='<U1')

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that makes sense, updated

@mroeschke mroeschke added Indexing Related to indexing on series/frames, not to indexes themselves Arrow pyarrow functionality labels Nov 6, 2023
@jorisvandenbossche jorisvandenbossche modified the milestones: 2.1.3, 2.1.4 Nov 13, 2023
Comment on lines +560 to +564
item.stop is not None
and item.stop < -len(self)
and item.step is not None
and item.step < 0
):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment about why we do this (or refer to a pyarrow bug report or this PR)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added the arrow ref

@jorisvandenbossche
Copy link
Member

Opened an issue on the Arrow side -> apache/arrow#38768

@phofl phofl requested a review from mroeschke November 21, 2023 22:30
@mroeschke mroeschke merged commit 196e907 into pandas-dev:main Nov 22, 2023
40 checks passed
@mroeschke
Copy link
Member

Thanks @phofl

Copy link

lumberbot-app bot commented Nov 22, 2023

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

  1. Checkout backport branch and update it.
git checkout 2.1.x
git pull
  1. Cherry pick the first parent branch of the this PR on top of the older branch:
git cherry-pick -x -m1 196e907a4d88f6cd11062f3fb02b3c76faf51b6b
  1. You will likely have some merge/cherry-pick conflict here, fix them and commit:
git commit -am 'Backport PR #55832: BUG: Index.getitem returning wrong result with negative step for arrow'
  1. Push to a named branch:
git push YOURFORK 2.1.x:auto-backport-of-pr-55832-on-2.1.x
  1. Create a PR against branch 2.1.x, I would have named this PR:

"Backport PR #55832 on branch 2.1.x (BUG: Index.getitem returning wrong result with negative step for arrow)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

@phofl phofl deleted the indexing_negative_steps branch November 22, 2023 22:01
phofl added a commit to phofl/pandas that referenced this pull request Nov 22, 2023
pandas-dev#55832)

* BUG: Index.getitem returning wrong result with negative step for arrow

* Update

* Update

* Fix

* Update array.py

* Fix

* Add gh ref

* Update

(cherry picked from commit 196e907)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Arrow pyarrow functionality Indexing Related to indexing on series/frames, not to indexes themselves
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants