Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: pandas.read_excel() with dtype=object is broken in pandas 2.1.1 #55489

Closed
3 tasks done
VisualDev-FR opened this issue Oct 11, 2023 · 8 comments · Fixed by #55757
Closed
3 tasks done

BUG: pandas.read_excel() with dtype=object is broken in pandas 2.1.1 #55489

VisualDev-FR opened this issue Oct 11, 2023 · 8 comments · Fixed by #55757
Labels

Comments

@VisualDev-FR
Copy link

VisualDev-FR commented Oct 11, 2023

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Create an excel file nammed xl_file.xlsx, with one worksheet named Sheet1 as following:

A B C
2 'dummy_str

Then save it and run this code in the directory where you created the excel file :

import pandas as pd
from pathlib import Path

df = pd.read_excel(
    io=Path("xl_file.xlsx"),
    sheet_name="Sheet1",
    dtype=object,
)

dtypes_1 = df.dtypes
dtypes_2 = df.fillna("").dtypes
>>> print(dtypes_1)
0   A    object # <- expected int, or int64
1   B    object
2   C    object
>>> print(dtypes_2)
0   A     int64 # <- when we use df.fillna(""), we have the expected behavior
1   B    object
2   C    object

Issue Description

The method pandas.read_excel() with argument dtypes=object does not prevent pandas from converting data types, as described on the offical documentation page

Expected Behavior

when we use the method pandas.read_excel() with argument dtypes=object, on an Excel workbook which contains int values, the dtypes of this columns should be int64

Installed Versions

INSTALLED VERSIONS ------------------ commit : e86ed37 python : 3.11.5.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 141 Stepping 1, GenuineIntel byteorder : little LC_ALL : None LANG : fr_FR.UTF-8 LOCALE : fr_FR.cp1252

pandas : 2.1.1
numpy : 1.24.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 65.5.0
pip : 23.2.1
Cython : None
pytest : 7.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.7
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

@VisualDev-FR VisualDev-FR added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Oct 11, 2023
@VisualDev-FR VisualDev-FR changed the title BUG: pandas.read_excel() with dtypes=object is broken in pandas 2.1.1 BUG: pandas.read_excel() with dtype=object is broken in pandas 2.1.1 Oct 11, 2023
@asishm-wk
Copy link

Can you share the excel file directly or a way to generate the excel file, e.g. using pd.DataFrame(...).to_excel(...) , that reproduces the issue?

@rhshadrach
Copy link
Member

rhshadrach commented Oct 12, 2023

Thanks for the report, is specifying dtype=None expected to give the same result as dtype=object? I wonder if we should just change the docs to say None instead of object.

@rhshadrach rhshadrach added the IO Excel read_excel, to_excel label Oct 12, 2023
@rhshadrach
Copy link
Member

cc @WillAyd

@rhshadrach
Copy link
Member

This was added in 588e29d. I think this is behaving as expected.

df = pd.DataFrame({'a': [1, 1, 2], 'b': ['x', 'y', 'z']})
df.to_excel('test.xlsx')
print(type(pd.read_excel('test.xlsx', dtype=object).iloc[0, 1]))
# <class 'int'>

There is no inference done - including inferring that the value is an integer! So we must store them as Python objects, but they are indeed Python ints.

I think the doc could be clarified better - explicitly mentioning that the result will be object dtype.

@rhshadrach rhshadrach added Docs good first issue and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Oct 15, 2023
@aryanVanaik
Copy link

Hi, could I work on this, would you prefer to change the docs maybe perhaps pointing me in the right direction

@rhshadrach
Copy link
Member

Certainly! See the contributor guide.

For this particular issue, we're looking to better explain that specifying dtype=object will result in a DataFrame with object dtype. Perhaps just adding This will necessarily result in ``object`` dtype in the docstring?

@aryanVanaik
Copy link

perfect ill get that done :)

ziadk pushed a commit to ziadk/pandas that referenced this issue Oct 22, 2023
@ziadk ziadk mentioned this issue Oct 22, 2023
gadeatric added a commit to gadeatric/october-2023-pydata that referenced this issue Oct 30, 2023
@gadeatric
Copy link
Contributor

Hi, @rhshadrach ! As I saw that the issue was still open, I decided to fix it considering the previous attempts , I hope that's OK - let me know if you have any questions or clarifications. Thanks! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants