BUG: DataFrame.groupby returns invalid value when dropna=False

[michaelpradel]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame({'A': ['foo', 'bar', None], 'B': [1, 2, 3]})
grouped = df.groupby('A', dropna=False)
len(grouped)  # raises exception with pandas 2.2.2; returns 3 with current main branch

Issue Description

The above example raises the following exception with pandas 2.2.2:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 787, in __len__
    return len(self.groups)
               ^^^^^^^^^^^
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 851, in groups
    return self._grouper.groups
           ^^^^^^^^^^^^^^^^^^^^
  File "properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 717, in groups
    return self.groupings[0].groups
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/groupby/grouper.py", line 842, in groups
    cats = Categorical.from_codes(self.codes, self._group_index, validate=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/arrays/categorical.py", line 745, in from_codes
    dtype = CategoricalDtype._from_values_or_dtype(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/dtypes/dtypes.py", line 338, in _from_values_or_dtype
    dtype = CategoricalDtype(categories, ordered)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/dtypes/dtypes.py", line 221, in __init__
    self._finalize(categories, ordered, fastpath=False)
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/dtypes/dtypes.py", line 378, in _finalize
    categories = self.validate_categories(categories, fastpath=fastpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/myen/lib/python3.11/site-packages/pandas/core/dtypes/dtypes.py", line 576, in validate_categories
    raise ValueError("Categorical categories cannot be null")
ValueError: Categorical categories cannot be null

My understanding of the API documentation is that len(grouped) should be 3. Indeed, this is the behavior of the current main branch. The change in behavior is due to #57595, which was supposed to only improve performance, but coincidentally fixed this bug.

As the bug is still present in the latest version (2.2.2), it may be worth backporting the (unintended) fix of #57595.

Expected Behavior

len(grouped) should be 3, as there are three groups: 'foo', 'bar', and None.

Installed Versions

INSTALLED VERSIONS ------------------ commit : d9cdd2e python : 3.11.6.final.0 python-bits : 64 OS : Linux OS-release : 6.5.0-28-generic Version : #29-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 28 23:46:48 UTC 2024 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 68.1.2
pip : 23.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report. This cannot be backported because #57595 depends on #55738 which is a large change.

In general, we only backport regression fixes to keep them minimal. Bugfixes can introduce new bugs, and so we do not backport unless it is a regression.

[michaelpradel]

Thanks for explaining. Makes sense.

Nit: You may want to update the "whatsnew" of #57595 because it's actually not only a performance improvement, but also a fix to this bug.