Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for postgres COPY TO/FROM command #160

Open
DeeeeLAN opened this issue Dec 5, 2024 · 0 comments
Open

Add support for postgres COPY TO/FROM command #160

DeeeeLAN opened this issue Dec 5, 2024 · 0 comments

Comments

@DeeeeLAN
Copy link

DeeeeLAN commented Dec 5, 2024

It would be nice for bigger datasets to be able to utilize postgres COPY TO/FROM to speed up data IO dramatically. Here is another object manager that is set up to support it: https://github.com/palewire/django-postgres-copy

Since COPY TO/FROM reads and dumps in CSV format, all django-pandas would need to do is convert between CSV and a DataFrame.

For context, I just ran a comparison between the two operations.

Database row retrieval count: 236,170,980
to_dataframe() time: 774 seconds
COPY TO to_csv() time: 223 seconds
converting csv to dataframe time: 24 seconds

@DeeeeLAN DeeeeLAN changed the title Add support for postgres COPY FROM command Add support for postgres COPY TO/FROM command Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant