ENH: add Series.struct accessor for ArrowDtype[struct] #54977

tswast · 2023-09-03T02:39:36Z

Features:

Series.struct.dtypes -- see dtypes and field names
Series.struct.field(name_or_index) -- extract a field as a Series
Series.struct.to_frame() -- convert all fields into a DataFrame

closes ENH: Series.struct accessor with Series.struct.field("sub-column name") for ArrowDtype #54938 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

pandas/core/arrays/arrow/accessors.py

mroeschke · 2023-09-07T18:16:48Z

pandas/core/arrays/arrow/accessors.py

+    def _validate(self, data):
+        dtype = data.dtype
+        if not isinstance(dtype, ArrowDtype):
+            raise AttributeError(self._validation_msg)


IMO I think TypeError would be more appropriate here.

Also could we include the incoming type in the error message like "f..., not {dtype}"

I think AttributeError was likely preventing this test failure.

pandas/tests/series/test_api.py:175: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../../../micromamba/envs/test/lib/python3.9/inspect.py:351: in getmembers value = getattr(object, key) pandas/core/accessor.py:224: in __get__ accessor_obj = self._accessor(obj) pandas/core/arrays/arrow/accessors.py:38: in __init__ self._validate(data) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <pandas.core.arrays.arrow.accessors.StructAccessor object at 0x7ff513c9cbe0> data = Series([], dtype: object) def _validate(self, data): dtype = data.dtype if not isinstance(dtype, ArrowDtype): > raise TypeError(self._validation_msg.format(dtype=dtype)) E TypeError: Can only use the '.struct' accessor with 'struct[pyarrow]' dtype, not object. pandas/core/arrays/arrow/accessors.py:43: TypeError

pandas/core/arrays/arrow/accessors.py

pandas/tests/series/accessors/test_struct_accessor.py

pandas/core/arrays/arrow/accessors.py

tswast · 2023-09-08T23:27:20Z

@mroeschke Thanks for the feedback! I believe I've now addressed all of your comments.

pandas/core/arrays/arrow/accessors.py

pandas/tests/series/accessors/test_struct_accessor.py

pandas/core/arrays/arrow/accessors.py

mroeschke · 2023-09-11T16:23:28Z

doc/source/whatsnew/v2.2.0.rst

+:meth:`Series.struct.explode` converts PyArrow structured data to a pandas
+DataFrame. (:issue:`54938`)
+
+.. code-block:: ipython


Could you use a .. ipython:: python directive here instead?

Good catch. Done. While I was at it, I updated the sample to be more meaningful, per https://pandas.pydata.org/docs/development/contributing_docstring.html#conventions-for-the-examples

This example is taken from the "file" struct schema in https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/#data-schema

mroeschke

Some small comments otherwise looks good!

Features: * Series.struct.dtypes -- see dtypes and field names * Series.struct.field(name_or_index) -- extract a field as a Series * Series.struct.explode() -- convert all fields into a DataFrame

TomAugspurger · 2023-09-12T01:37:27Z

pandas/core/arrays/arrow/accessors.py

+        names = [struct.name for struct in pa_type]
+        return Series(types, index=Index(names))
+
+    def field(self, name_or_index: str | int) -> Series:


I see that https://arrow.apache.org/docs/python/generated/pyarrow.compute.struct_field.html says you can do things like struct_field(array, indices=['a', 'b']) to do nested lookups like a.b. Any reason to disallow that?

That's a good idea. Extracting the name to use for the Series with such inputs is turning out to be non-trivial though. Perhaps best left as a follow-up enhancement?

A follow-up makes sense to me.

mroeschke

LGTM. Will keep open for a few day just in case there's other feedback

TomAugspurger · 2023-09-17T18:00:46Z

@mroeschke OK to merge now?

mroeschke · 2023-09-18T16:15:16Z

Nice thanks @tswast

) Features: * Series.struct.dtypes -- see dtypes and field names * Series.struct.field(name_or_index) -- extract a field as a Series * Series.struct.explode() -- convert all fields into a DataFrame

tswast mentioned this pull request Sep 3, 2023

ENH: Series.struct accessor with Series.struct.field("sub-column name") for ArrowDtype #54938

Closed

3 tasks

tswast force-pushed the issue54938-struct-accessor branch from 66ff669 to 3996cb3 Compare September 4, 2023 03:10

tswast marked this pull request as ready for review September 4, 2023 03:10

tswast force-pushed the issue54938-struct-accessor branch 2 times, most recently from 1fd8d43 to ae4088a Compare September 4, 2023 12:16

tswast commented Sep 4, 2023

View reviewed changes

pandas/core/arrays/arrow/accessors.py Outdated Show resolved Hide resolved

tswast force-pushed the issue54938-struct-accessor branch 2 times, most recently from 00dff2c to adc9e26 Compare September 5, 2023 11:18

tswast added Enhancement Arrow pyarrow functionality Accessors accessor registration mechanism (not .str, .dt, .cat) labels Sep 6, 2023

tswast added this to the 2.2 milestone Sep 6, 2023

mroeschke reviewed Sep 7, 2023

View reviewed changes

pandas/core/arrays/arrow/accessors.py Outdated Show resolved Hide resolved

mroeschke reviewed Sep 7, 2023

View reviewed changes

pandas/core/arrays/arrow/accessors.py Show resolved Hide resolved

mroeschke reviewed Sep 7, 2023

View reviewed changes

pandas/tests/series/accessors/test_struct_accessor.py Outdated Show resolved Hide resolved

mroeschke reviewed Sep 7, 2023

View reviewed changes

pandas/tests/series/accessors/test_struct_accessor.py Outdated Show resolved Hide resolved

mroeschke reviewed Sep 8, 2023

View reviewed changes

pandas/core/arrays/arrow/accessors.py Outdated Show resolved Hide resolved

tswast force-pushed the issue54938-struct-accessor branch from b19ab49 to 2f64ded Compare September 8, 2023 23:26

tswast requested a review from mroeschke September 8, 2023 23:26

tswast commented Sep 9, 2023

View reviewed changes

pandas/core/arrays/arrow/accessors.py Outdated Show resolved Hide resolved

pandas/core/arrays/arrow/accessors.py Outdated Show resolved Hide resolved

pandas/tests/series/accessors/test_struct_accessor.py Outdated Show resolved Hide resolved

tswast force-pushed the issue54938-struct-accessor branch from e3dd736 to bbf9c72 Compare September 9, 2023 02:41

mroeschke reviewed Sep 11, 2023

View reviewed changes

pandas/core/arrays/arrow/accessors.py Outdated Show resolved Hide resolved

mroeschke reviewed Sep 11, 2023

View reviewed changes

ENH: add Series.struct accessor for ArrowDtype[struct]

d47b6f9

Features: * Series.struct.dtypes -- see dtypes and field names * Series.struct.field(name_or_index) -- extract a field as a Series * Series.struct.explode() -- convert all fields into a DataFrame

tswast force-pushed the issue54938-struct-accessor branch from 1524dfb to d47b6f9 Compare September 11, 2023 19:47

tswast requested a review from mroeschke September 11, 2023 19:48

Merge branch 'main' into issue54938-struct-accessor

f3d57c8

TomAugspurger approved these changes Sep 12, 2023

View reviewed changes

mroeschke approved these changes Sep 12, 2023

View reviewed changes

tswast added 3 commits September 13, 2023 13:27

Merge branch 'main' into issue54938-struct-accessor

d3a7eb4

Merge branch 'main' into issue54938-struct-accessor

40103d5

Merge branch 'main' into issue54938-struct-accessor

6006e77

Merge branch 'main' into issue54938-struct-accessor

3fe9291

mroeschke merged commit 36aa531 into pandas-dev:main Sep 18, 2023
32 of 33 checks passed

tswast deleted the issue54938-struct-accessor branch September 18, 2023 16:50

TomAugspurger mentioned this pull request Nov 19, 2023

ENH: Expand index types in Series.struct.field method #56065

Closed

3 tasks

MarcoGorelli mentioned this pull request Dec 20, 2023

Move year to .dt.year, and other namespace-specific function data-apis/dataframe-api#341

Closed

tswast mentioned this pull request Aug 23, 2024

ENH: DataFrame.struct.explode(column, *, separator=".") method to pull struct subfields into the parent DataFrame #59585

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add Series.struct accessor for ArrowDtype[struct] #54977

ENH: add Series.struct accessor for ArrowDtype[struct] #54977

tswast commented Sep 3, 2023 •

edited

Loading

mroeschke Sep 7, 2023

tswast Sep 9, 2023

tswast commented Sep 8, 2023

mroeschke Sep 11, 2023

tswast Sep 11, 2023

mroeschke left a comment

TomAugspurger Sep 12, 2023

tswast Sep 12, 2023

TomAugspurger Sep 13, 2023

mroeschke left a comment

TomAugspurger commented Sep 17, 2023

mroeschke commented Sep 18, 2023

ENH: add Series.struct accessor for ArrowDtype[struct] #54977

ENH: add Series.struct accessor for ArrowDtype[struct] #54977

Conversation

tswast commented Sep 3, 2023 • edited Loading

mroeschke Sep 7, 2023

Choose a reason for hiding this comment

tswast Sep 9, 2023

Choose a reason for hiding this comment

tswast commented Sep 8, 2023

mroeschke Sep 11, 2023

Choose a reason for hiding this comment

tswast Sep 11, 2023

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

TomAugspurger Sep 12, 2023

Choose a reason for hiding this comment

tswast Sep 12, 2023

Choose a reason for hiding this comment

TomAugspurger Sep 13, 2023

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

TomAugspurger commented Sep 17, 2023

mroeschke commented Sep 18, 2023

tswast commented Sep 3, 2023 •

edited

Loading