Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include Dask Support #105

Open
jakirkham opened this issue Aug 8, 2017 · 2 comments
Open

Include Dask Support #105

jakirkham opened this issue Aug 8, 2017 · 2 comments

Comments

@jakirkham
Copy link
Member

Would be great if Dask Array's could be an optionally supported input/output type. This way one could construct a graph of the computation using rank_filter with proper haloing and compute in parallel when ready.

@jakirkham
Copy link
Member Author

jakirkham commented Aug 16, 2018

Alternatively it might be reasonable to just use multiprocessing.dummy.Pool, which would give us a way to run the filter in multiple threads and avoid any additional dependencies.

Edit: Can probably do something like this.

multiprocessing.dummy.Pool().map_async(<func>, <iterable>).wait()

Edit 2: This wound up being slower than not using a thread pool. The overhead of scheduling and lock contention must be making this not viable.

@jakirkham
Copy link
Member Author

As a step in this direction, the filter now releases the GIL for the entirety of the for-loop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant