-
Notifications
You must be signed in to change notification settings - Fork 413
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Javascript (server side) bindings via n-api #2342
Comments
@rtyler I'd be happy to contribute to this one. Feel free to assign it to me. |
@universalmind303 I tried creating a reader with polars rust directly but the Scan is not flexible enough to read schema evolved table unfortunately : ( So curious to see how you are planning to do this with polars 😄 |
for polars, there is a way to push down some operations to the datasource during planning. |
@universalmind303 I tried using a LogicalPlan::FileScanfor parquet but polars will error out during collection, since it concats the dfs instead of doing a diagonal concat. Also if you provide a schema to the reader and a column is not in the parquet, it will error out instead of creating a null array Here you can see old branch where I was playing around with the scan: https://github.com/ion-elgreco/polars-deltalake/blob/feat/cloud_reads/python/src/lib.rs |
What is the update on this? Would also love this to use read/write delta with nodejs , huge unlock with the amount of js devs imo. |
I had an initial POC for it here, but since then, the python API has changed quite a bit. I made a bit of progress getting it to feature parity with the python one, but not quite there yet. I'm hoping to find some time to work on this again soon. If anyone else wants to work on it in the meantime, by all means! |
Description
I'd like a similar sdk as the python sdk but available in server side js (node, deno, bun, ...). The sdk should closely resemble the python one, and only deviate when either necessary due to napi limitations, or when it is unidiomatic in JS.
napi.rs is very similar in nature to pyo3 and is relatively easy to use.
Use Case
I want to read delta files via a js application.
Related Issue(s)
pola-rs/nodejs-polars#176
The text was updated successfully, but these errors were encountered: