BUG: Interchange protocol fails with pyarrow backed types

[char101]

Pandas version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd, pyarrow as pa
from datetime import datetime

pa.interchange.from_dataframe(pd.DataFrame({'a': [datetime.now()]}, dtype='date32[pyarrow]'))

NotImplementedError                       Traceback (most recent call last)
Cell In [31], line 1
----> 1 pl.from_dataframe(pd.DataFrame({'a': [datetime.now()]}, dtype='date32[pyarrow]'))

File C:\python\3.11\Lib\site-packages\pyarrow\interchange\from_dataframe.py:85, in from_dataframe(df, allow_copy)
     82 if not hasattr(df, "__dataframe__"):
     83     raise ValueError("`df` does not support __dataframe__")
---> 85 return _from_dataframe(df.__dataframe__(allow_copy=allow_copy),
     86                        allow_copy=allow_copy)

File C:\python\3.11\Lib\site-packages\pyarrow\interchange\from_dataframe.py:108, in _from_dataframe(df, allow_copy)
    106 batches = []
    107 for chunk in df.get_chunks():
--> 108     batch = protocol_df_chunk_to_pyarrow(chunk, allow_copy)
    109     batches.append(batch)
    111 table = pa.Table.from_batches(batches)

File C:\python\3.11\Lib\site-packages\pyarrow\interchange\from_dataframe.py:143, in protocol_df_chunk_to_pyarrow(df, allow_copy)
    141     raise ValueError(f"Column {name} is not unique")
    142 col = df.get_column_by_name(name)
--> 143 dtype = col.dtype[0]
    144 if dtype in (
    145     DtypeKind.INT,
    146     DtypeKind.UINT,
   (...)
    149     DtypeKind.DATETIME,
    150 ):
    151     columns[name] = column_to_array(col, allow_copy)

File C:\python\3.11\Lib\site-packages\pandas\_libs\properties.pyx:36, in pandas._libs.properties.CachedProperty.__get__()

File C:\python\3.11\Lib\site-packages\pandas\core\interchange\column.py:126, in PandasColumn.dtype(self)
    124     raise NotImplementedError("Non-string object dtypes are not supported yet")
    125 else:
--> 126     return self._dtype_from_pandasdtype(dtype)

File C:\python\3.11\Lib\site-packages\pandas\core\interchange\column.py:141, in PandasColumn._dtype_from_pandasdtype(self, dtype)
    137 if kind is None:
    138     # Not a NumPy dtype. Check if it's a categorical maybe
    139     raise ValueError(f"Data type {dtype} not supported by interchange protocol")
--> 141 return kind, dtype.itemsize * 8, dtype_to_arrow_c_fmt(dtype), dtype.byteorder

File C:\python\3.11\Lib\site-packages\pandas\core\interchange\utils.py:91, in dtype_to_arrow_c_fmt(dtype)
     88     resolution = re.findall(r"\[(.*)\]", typing.cast(np.dtype, dtype).str)[0][:1]
     89     return ArrowCTypes.TIMESTAMP.format(resolution=resolution, tz="")
---> 91 raise NotImplementedError(
     92     f"Conversion of {dtype} to Arrow C format string is not implemented."
     93 )

NotImplementedError: Conversion of date32[day][pyarrow] to Arrow C format string is not implemented.

Issue Description

Conversion of a dataframe containing date32 to pyarrow table throws an exception.

Expected Behavior

date32[day][pyarrow] should be able to be converted back to a pyarrow table.

Installed Versions

INSTALLED VERSIONS

commit : ceef0da
python : 3.11.1.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19043
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_Indonesia.1252

pandas : 2.1.0.dev0+368.gceef0da443
numpy : 1.24.2
pytz : 2022.6
dateutil : 2.8.2
setuptools : 65.6.3
pip : 23.0.1
Cython : 0.29.32
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : 2.9.5
jinja2 : 3.1.2
IPython : 8.6.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : 1.3.7
brotli : 1.0.9
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.0
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : 1.4.45
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2022.6
qtpy : 2.3.0
pyqt5 : None

[char101]

pandas/pandas/core/interchange/utils.py

Line 81 in 44eb283

format_str = getattr(ArrowCTypes, dtype.name.upper(), None)

pd.Series([datetime.now()], dtype='date32[pyarrow]').dtype.name.upper()
Out[36]: 'DATE32[DAY][PYARROW]'

Since the dtype name contains [DAY][PYARROW] would the code above requires .split('[')[0]?

[char101]

I added the .split and now I get this error:

Traceback (most recent call last):
  File "C:\python\3.11\Lib\site-packages\pyarrow\interchange\from_dataframe.py", line 85, in from_dataframe
    return _from_dataframe(df.__dataframe__(allow_copy=allow_copy),
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\python\3.11\Lib\site-packages\pyarrow\interchange\from_dataframe.py", line 108, in _from_dataframe
    batch = protocol_df_chunk_to_pyarrow(chunk, allow_copy)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\python\3.11\Lib\site-packages\pyarrow\interchange\from_dataframe.py", line 143, in protocol_df_chunk_to_pyarrow
    dtype = col.dtype[0]
            ^^^^^^^^^
  File "pandas\_libs\properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
  File "C:\python\3.11\Lib\site-packages\pandas\core\interchange\column.py", line 126, in dtype
    return self._dtype_from_pandasdtype(dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\python\3.11\Lib\site-packages\pandas\core\interchange\column.py", line 141, in _dtype_from_pandasdtype
    return kind, dtype.itemsize * 8, dtype_to_arrow_c_fmt(dtype), dtype.byteorder
                                                                  ^^^^^^^^^^^^^^^
AttributeError: 'ArrowDtype' object has no attribute 'byteorder'

[mroeschke]

Thanks for the report. The interchange protocol was first implemented with numpy dtypes in mind. I suspect all arrow backed types will have a similar error

[schradert]

I encountered a similar error NotImplementedError: dt.tz is not supported for date32[day][pyarrow] when calling .to_sql. Converting and casting to timestamp[us][pyarrow] funnily enough worked.

[mroeschke]

Now it appears this is failing on the pyarrow side so going to close as an upstream issue

File /opt/miniconda3/envs/pandas-dev/lib/python3.10/site-packages/pyarrow/interchange/from_dataframe.py:324, in parse_datetime_format_str(format_str)
    320         unit += "s"
    322     return unit, tz
--> 324 raise NotImplementedError(f"DateTime kind is not supported: {format_str}")

NotImplementedError: DateTime kind is not supported: tdD