BUG: Creating a string column on a mask results in NaN being stringyfied and potentially truncated based on the 'maxchar' value of the column

[CarlinCyril]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

df = pd.DataFrame({"a": [1,2], "b":[3,4]})
df.loc[lambda x: x.a == 1, "c"] = "1"

// Result //
   a  b  c
0  1  3  1
1  2  4  n

Issue Description

When partially defining a column on a mask, rows not included in the mask should be set to NaN. With the new way of handling string in pandas, it seems that NaN values are cast to string ("nan") and then the space optimisation applying a maxchar on the column is potentially truncating the "nan" to "na" or even "n".

Expected Behavior

// Result //
a b c
0 1 3 1
1 2 4 NaN

Installed Versions

INSTALLED VERSIONS

commit : 2a953cf
python : 3.10.11.final.0
python-bits : 64
OS : Linux
OS-release : 4.18.0-477.27.1.el8_8.x86_64
Version : #1 SMP Thu Aug 31 10:29:22 EDT 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.3
numpy : 1.26.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.2.0
pip : 23.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[asishm]

Seems to be a regression (haven't run a bisect yet). Was working as expected in 2.0.3

[richard-younergy]

I've also come across this, running on Mac OS X. The change seems to occur from release 2.1.2 onwards (2.1.1 is ok).

Expanded Example

import pandas as pd
data = range(0,10)
d = {'col1': data}
df = pd.DataFrame(data=d)
even_mask = [val % 2 != 0 for val in data]
df.loc[even_mask,'is_even_string'] = 'yes'
df.loc[even_mask,'is_even_char'] = 'y'
df.loc[even_mask,'is_even_int'] = 1
df.loc[even_mask,'is_even_float'] = 1.0
print(df)

Under pandas <= 2.1.1 get expected result where all four columns set from a partial mask have NaN for the undefined odd rows:

   col1 is_even_string is_even_char  is_even_int  is_even_float
0     0            yes            y          1.0            1.0
1     1            NaN          NaN          NaN            NaN
2     2            yes            y          1.0            1.0
3     3            NaN          NaN          NaN            NaN
4     4            yes            y          1.0            1.0
5     5            NaN          NaN          NaN            NaN
6     6            yes            y          1.0            1.0
7     7            NaN          NaN          NaN            NaN
8     8            yes            y          1.0            1.0
9     9            NaN          NaN          NaN            NaN

Swapping to pandas >= 2.1.2 the string or "char" are both turned into (truncated strings):

   col1 is_even_string is_even_char  is_even_int  is_even_float
0     0            yes            y          1.0            1.0
1     1            nan            n          NaN            NaN
2     2            yes            y          1.0            1.0
3     3            nan            n          NaN            NaN
4     4            yes            y          1.0            1.0
5     5            nan            n          NaN            NaN
6     6            yes            y          1.0            1.0
7     7            nan            n          NaN            NaN
8     8            yes            y          1.0            1.0
9     9            nan            n          NaN            NaN

[asishm]

Thanks @richard-younergy for narrowing it down further.

Bisect seems to give (not sure if I bisected correctly using meson or not) #55249 cc @MarcoGorelli

fd51bb6 is the first bad commit
commit fd51bb6
Author: Lumberbot (aka Jack) [email protected]
Date: Fri Sep 22 16:47:52 2023 -0700

Backport PR #55249 on branch 2.1.x (BUG: incompatible dtype when creating string column with loc) (#55251)

Backport PR #55249: BUG: incompatible dtype when creating string column with loc

Co-authored-by: Marco Edward Gorelli <[email protected]>

doc/source/whatsnew/v2.1.2.rst | 2 +-
pandas/core/dtypes/missing.py | 3 +++
pandas/tests/frame/indexing/test_indexing.py | 15 +++++++++++++++
pandas/tests/frame/indexing/test_set_value.py | 5 +----
4 files changed, 20 insertions(+), 5 deletions(-)
bisect found first bad commit

[MarcoGorelli]

thanks for the report, this looks pretty bad, will take a loook

[CarlinCyril]

Thanks for the help @richard-younergy. I had my suspicions as well on 2.1.2 as I upgraded from 2.1.1 to 2.1.3 and got the bug.
If I can help in any way @MarcoGorelli, tell me. Never helped on pandas but would be a pleasure to help on a package I use everyday :)