Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOC: Provide examples of using read_parquet #49739

Open
1 task done
wjones127 opened this issue Nov 17, 2022 · 3 comments
Open
1 task done

DOC: Provide examples of using read_parquet #49739

wjones127 opened this issue Nov 17, 2022 · 3 comments
Labels
Docs IO Parquet parquet, feather

Comments

@wjones127
Copy link

Pandas version checks

  • I have checked that the issue still exists on the latest versions of the docs on main here

Location of the documentation

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html

Documentation problem

For the pyarrow engine, there are some important features behind the kwargs that aren't aren't described here, and it might not be obvious to users where to look in PyArrow. For example:

  • Using filters, users can prune which files and/or row groups are read.
  • Using filesystem, users can configure a filesystem such as S3

Suggested fix for documentation

At the very least, we should document for each engine where those kwargs are passed. But it might even be worthwhile to provide examples of filters, reading partitioned datasets, and configuring remote filesystems. Does that seem reasonable?

@wjones127 wjones127 added Docs Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 17, 2022
@rhshadrach rhshadrach added IO Parquet parquet, feather and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 17, 2022
@rhshadrach
Copy link
Member

Thanks for the report! +1 on saying the function that's called from pandas and linking to it's documentation. However, if we were to document which kwargs there's a bit more maintenance burden keeping it in sync (e.g. what can be passed for filters just changed in 10.0.0) and I don't think it provides significant benefit to the user.

@phofl
Copy link
Member

phofl commented Nov 17, 2022

Yep agreed, would rather link to the functions itself too

vvaidy pushed a commit to vvaidy/pandas that referenced this issue Jul 15, 2023
vvaidy pushed a commit to vvaidy/pandas that referenced this issue Jul 15, 2023
vvaidy pushed a commit to vvaidy/pandas that referenced this issue Jul 15, 2023
vvaidy pushed a commit to vvaidy/pandas that referenced this issue Jul 16, 2023
Fixed typos that were causing tests to fail. Oops.
vvaidy pushed a commit to vvaidy/pandas that referenced this issue Jul 18, 2023
vvaidy pushed a commit to vvaidy/pandas that referenced this issue Jul 18, 2023
…d read_parquet from code_checks.sh as requested by @mroeschke
mroeschke pushed a commit that referenced this issue Jul 18, 2023
* DOC: Provide examples of using read_parquet #49739

* DOC: Provide examples of using read_parquet #49739

* DOC: Provide examples of using read_parquet #49739 (with minor fixes)

* DOC: Provide examples of using read_parquet #49739
Fixed typos that were causing tests to fail. Oops.

* DOC: Provide examples of using read_parquet #49739 - fix formatting failed checks

* DOC: Provide examples of using read_parquet #49739 - removed read_parquet from code_checks.sh as requested by @mroeschke

---------

Co-authored-by: Vijay Vaidyanathan <[email protected]>
@LuchiLucs
Copy link

I'm interested in examples of:

  1. leveraging the filter argument to filter rows based on their index, for instance if the index is a datetime, to return rows in a given datetime interval
  2. leveraging the columns argument considering the case of unknown existence. For instance, if given columns do exist, filter based on them, if a sub-set do exist and a sub-set do not, filter based only on those existing without raising.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Docs IO Parquet parquet, feather
Projects
None yet
Development

No branches or pull requests

4 participants