Add rust extension lib alluxiocommon to the repo #39
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…hread 5f80059 small fixes for REST and metrics 1c426c2 add access debug logs for all fsspec ops & add quick sanity unit tests for read 7dcb833 checkin multi process http benchmark for worker http server 4b4ebce use alluxiocommon pyo3 package to try to do multithread http req 523f748 add rust pyo3 extension 8979d2b remove first 0367659 add related files 4d0eebe small fixes a0519b3 if any error on any thread running result, raise exception back to python directly
…llowing exception
…_alluxio_fsspec.py 2. make tests/client/test_read_range_docker.py (now named as test_read_docker.py to handle full read as well add alluxiocommon fixture in it too
lucyge2022
force-pushed
the
rust-extension
branch
2 times, most recently
from
a448110 to
647e3ea
Compare
jja725
reviewed
Apr 26, 2024
|
|
||
| [dependencies] | ||
| reqwest = { version = "0.11", features = ["blocking"] } | ||
| tokio = { version = "1", features = ["full"] } |
There was a problem hiding this comment.
full tokio is a big stuff, probably we only need a specific part of it?
There was a problem hiding this comment.
understood, will check the related pieces to check in
| } | ||
|
|
||
| for sender in senders { | ||
| let result = sender.unwrap().blocking_recv(); |
There was a problem hiding this comment.
I feel like this could be simplified with rayon::iter::ParallelIterator.try_for_each. We just generate a iterate of pageid and use ParallelIterator, the output is ordered so in that case we can get rid of the oneshot channel as well.
There was a problem hiding this comment.
yeah u r right i will try that, i plan to clean up this lib after i check this extension it, will address this in later PR
lucyge2022
force-pushed
the
rust-extension
branch
from
ae3710b to
038ad4e
Compare
lucyge2022
force-pushed
the
rust-extension
branch
from
b0ec3b5 to
3eccac5
Compare
… rust-native-tls requires dynamic link of tls in runtime env
lucyge2022
force-pushed
the
rust-extension
branch
from
62630a0 to
e509554
Compare
lucyge2022
force-pushed
the
rust-extension
branch
from
a8f0806 to
2017c4a
Compare
lucyge2022
force-pushed
the
rust-extension
branch
from
2017c4a to
b2deaff
Compare
|
alluxio-bot, merge this please. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add rust extension lib alluxiocommon to the repo, it currently only handles:
spawn on demand or preallocated threadpool to make multiple http getpage request concurrently to fetch pages
this extension can be enabled or disabled via options in AlluxioFileSystem as well as in AlluxioClient