Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: enlarging a DataFrame by adding an extra column with a tz-aware datetime results in object dtype #55423

Open
3 tasks done
MarcoGorelli opened this issue Oct 6, 2023 · 2 comments
Labels
Bug Datetime Datetime data dtype Dtype Conversions Unexpected or buggy dtype conversions

Comments

@MarcoGorelli
Copy link
Member

MarcoGorelli commented Oct 6, 2023

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

import datetime as dt

df = pd.DataFrame([{'id': 1},{'id': 2},{'id': 3}])
_time = dt.datetime.utcfromtimestamp(1695887042)
_time = _time.replace(tzinfo=dt.timezone.utc)
df.loc[df.id>=2, 'time'] = _time
print(df)
print(df.dtypes)

Issue Description

the above outputs

   id                       time
0   1                        NaT
1   2  2023-09-28 07:44:02+00:00
2   3  2023-09-28 07:44:02+00:00
id       int64
time    object
dtype: object

Expected Behavior

it would be nice to get to

   id                      time
0   1                       NaT
1   2 2023-09-28 07:44:02+00:00
2   3 2023-09-28 07:44:02+00:00
id                    int64
time    datetime64[ns, UTC]
dtype: object

right away

Installed Versions

[1/1] Generating write_version_file with a custom command

  • /home/marcogorelli/.virtualenvs/pandas/bin/ninja

INSTALLED VERSIONS

commit : 99efe62
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.102.1-microsoft-standard-WSL2
Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 2.1.0rc0+99.g99efe62afd.dirty
numpy : 1.24.4
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 59.6.0
pip : 23.2.1
Cython : 0.29.33
pytest : 7.4.2
hypothesis : 6.86.2
sphinx : 6.2.1
blosc : 1.11.1
feather : None
xlsxwriter : 3.1.4
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.7
jinja2 : 3.1.2
IPython : 8.15.0
pandas_datareader : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat: None
fastparquet : 2023.8.0
fsspec : 2023.9.1
gcsfs : 2023.9.1
matplotlib : 3.7.3
numba : 0.57.1
numexpr : 2.8.6
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : 1.2.3
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2023.9.1
scipy : 1.11.2
sqlalchemy : 2.0.21
tables : 3.8.0
tabulate : 0.9.0
xarray : 2023.8.0
xlrd : 2.0.1
zstandard : 0.21.0
tzdata : 2023.3
qtpy : None
pyqt5 : None
None

@jbrockmendel
Copy link
Member

the title suggests the issue is about the Series constructor, but the example seems to be about setitem-with-expansion. Is the example minimal?

@MarcoGorelli MarcoGorelli changed the title BUG: initialising a Series with a tz-aware datetime results in object dtype BUG: enlarging a DataFrame by adding an extra column with a tz-aware datetime results in object dtype Oct 26, 2023
@MarcoGorelli MarcoGorelli added the Dtype Conversions Unexpected or buggy dtype conversions label Oct 26, 2023
@MarcoGorelli
Copy link
Member Author

thanks, have updated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Datetime Datetime data dtype Dtype Conversions Unexpected or buggy dtype conversions
Projects
None yet
Development

No branches or pull requests

2 participants