BUG: String slicing produces different results with pyarrow string datatype compared to python string type

[rohanjain101]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> a = pd.Series(["abcdefghijklmnabcdefghijkln"], dtype=pd.ArrowDtype(pa.string()))
>>> a.str.slice(None, 8, -9)
0
dtype: string[pyarrow]

Issue Description

The string above contains pure ascii characters, yet the result does not match the python string result.

>>> "abcdefghijklmnabcdefghijkln"[None:8:-9]
'nd'
>>>

Expected Behavior

Behavior with python strings.

>>> a = pd.Series(["abcdefghijklmnabcdefghijkln"], dtype=pd.StringDtype())
>>> a.str.slice(None, 8, -9)
0    nd
dtype: string
>>>

Installed Versions

pd.show_versions()

INSTALLED VERSIONS

commit : 478d340
python : 3.8.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22621
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.0.0
numpy : 1.24.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.5.1
pip : 23.0.1
Cython : None
pytest : 7.0.1
hypothesis : None
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.12.0
pandas_datareader: None
bs4 : 4.12.0
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[rohanjain101]

Since pandas implementation is just delegating to pyarrow utf8_slice_codeunits, seems difference is caused by upstream arrow code. Is arrow slice intended to be different than python slice?

>>> pa.compute.utf8_slice_codeunits("abcdefghijklmnabcdefghijkln", start=0, stop=8, step=-9)
<pyarrow.StringScalar: ''>
>>>

EDIT: This example is wrong, if step is negative, then start should be len - 1.

>>> pa.compute.utf8_slice_codeunits("abcdefghijklmnabcdefghijkln", start=26, stop=8, step=-9)
<pyarrow.StringScalar: 'nd'>
>>>

Seems pandas in _str_slice does not take into account negative step value when calling arrow compute, its always passed as 0.

[mroeschke]

Yeah looks like an issue with the upstream utf8_slice_codeunits, mind opening an issue with arrow and linking it here?

[rohanjain101]

@mroeschke I apologize for the confusion, my example was wrong, it seems arrow is producing the correct behavior, for the example:

>>> "abcdefghijklmnabcdefghijkln"[None:8:-9]
'nd'

>>> pa.compute.utf8_slice_codeunits("abcdefghijklmnabcdefghijkln", start=26, stop=8, step=-9)
<pyarrow.StringScalar: 'nd'>
>>>

It looks like pandas is not taking into account that if step is negative, start needs to be len - 1, seems like it's just passing 0. I don't believe this is an upstream issue.

[mroeschke]

The difficulty is that you cannot specify len - 1 for an array of strings with different lengths.

xref apache/arrow#28940

[rohanjain101]

Ah, I see. Is this still something that should be reported upstream? As the linked issue has been closed.

[mroeschke]

Yeah would be worth an issue in pyarrow. apache/arrow#14991 is somewhat related in the sense that utf8_slice_codeunits doesn't seem to handle None start or end bounds of different lengths

[rohanjain101]

Opened apache/arrow#34917

[mroeschke]

Closing as awaiting resolution in apache/arrow#34917