PERF: pd.read_parquet() consistently slower than pq.read_table().to_pandas()

[angusl-gr]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this issue exists on the latest version of pandas.

• I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

Loading a Parquet file via PyArrow and then converting to Pandas is consistently faster than loading directly via Pandas on my machine (I have verified on both Windows and Linux machines). I'm really struggling to see why, as the source for pd.read_parquet() effectively just wraps the PyArrow version here as far as I can tell. Profiling in VizTracer shows both versions ultimately spending most of their time in pyarrow._dataset.FileSystemDataset.to_table(), but the PyArrow version always takes less time in this call for some reason.

If anybody has any idea what is going on I'd be very interested to hear.

Thanks!

import numpy as np
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
pd.show_versions()

data = np.random.rand(10 ** 7, 100).astype(np.float32)
df = pd.DataFrame(data)
df.columns = [str(col) for col in df.columns]
df.to_parquet('data.parquet')

%%timeit
df1 = pd.read_parquet('data.parquet')
# This takes ~2.8s on my machine

%%timeit
df2 = pq.read_table('data.parquet').to_pandas()
# This takes ~2.2s on my machine

Installed Versions

INSTALLED VERSIONS

commit : e8093ba
python : 3.8.7.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United Kingdom.1252

pandas : 1.4.3
numpy : 1.22.2
pytz : 2021.1
dateutil : 2.8.2
setuptools : 60.5.0
pip : 21.2.2
Cython : 0.29.24
pytest : 6.2.5
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 8.0.0
pandas_datareader: None
bs4 : None
bottleneck : 1.3.2
brotli : None
fastparquet : 0.7.1
fsspec : 2021.07.0
gcsfs : None
markupsafe : 2.0.1
matplotlib : 3.4.2
numba : 0.53.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 8.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : 0.19.0
xlrd : None
xlwt : None
zstandard : None

Prior Performance

No response

[angusl-gr]

Bumping in case anybody has any insights.

[philip-sterne]

I know this is a bit old, but does the effect persist if you swap the order? (ie. do the pyarrow first, and then the pandas version second)

It could be a cache effect, so whichever method you call first is going to be the slower method as the file has to be loaded into cache.

[ei-grad]

This issue is not related to caching. The performance discrepancy is, at first glance, so apparent that many of us, including myself, don't immediately question why it's the case or consider if it can be improved upon. It's likely that a significant part of the problem is due to the more efficient use of multithreading in Arrow, which might not be as effectively leveraged in Pandas.