You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It would be nice for bigger datasets to be able to utilize postgres COPY TO/FROM to speed up data IO dramatically. Here is another object manager that is set up to support it: https://github.com/palewire/django-postgres-copy
Since COPY TO/FROM reads and dumps in CSV format, all django-pandas would need to do is convert between CSV and a DataFrame.
For context, I just ran a comparison between the two operations.
It would be nice for bigger datasets to be able to utilize postgres COPY TO/FROM to speed up data IO dramatically. Here is another object manager that is set up to support it: https://github.com/palewire/django-postgres-copy
Since COPY TO/FROM reads and dumps in CSV format, all django-pandas would need to do is convert between CSV and a DataFrame.
For context, I just ran a comparison between the two operations.
Database row retrieval count: 236,170,980
to_dataframe()
time: 774 secondsCOPY TO
to_csv()
time: 223 secondsconverting csv to dataframe time: 24 seconds
The text was updated successfully, but these errors were encountered: