Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support bypassMergeSort Suffle #15

Open
pgaref opened this issue Jul 30, 2019 · 0 comments
Open

Support bypassMergeSort Suffle #15

pgaref opened this issue Jul 30, 2019 · 0 comments
Assignees
@pgaref
Labels
enhancement New feature or request
Milestone
Shuffle support

Comments

@pgaref
Copy link
Collaborator

pgaref commented Jul 30, 2019 •
edited
Loading

This is the default hash-ish shuffle (supported by SparkSQL) using serialized objects when there is no need for map-side aggregation.

Part of #9
@pgaref pgaref added the enhancement New feature or request label Jul 30, 2019
@pgaref pgaref self-assigned this Jul 30, 2019
pgaref added a commit that referenced this issue Jul 30, 2019
@pgaref
ByPassSort shuffle test case going through SGXWorker now working #15
71d2484 
Multiple partitions supported
@pgaref pgaref mentioned this issue Jul 30, 2019
Merged
@pgaref pgaref added this to the Shuffle support milestone Jul 30, 2019
pgaref added a commit that referenced this issue Aug 1, 2019
@pgaref
Supporting byPassMergeSort shuffle for SGX (#17)
c250aa7 
* Introducing new SHUFFLE_MAP type for bypassMerge sort shuffle

* Adding numOfPartitions as part of SGX RDD (needed by the shuffle partitioner)

* Introducing SGX partitioner - hash-based for now (to be changed by any pseudorandom function we want)

* Introducing sgxWrite interface as part of Shuffle writter
When SGX is enable we use this path instead
The method takes as arguments both the encrypted records AND the record-partition mapping (recordMapping) as return by the enclave worker

* Handling SHUFFLE_MAP_BYPASS task type in SGX worker - return a record-partition mapping iterator
* ShuffleMapTask checks if SGX is enabled and calls the appropriate shuffle interface

* ByPassSort shuffle test case going through SGXWorker now working #15
Multiple partitions supported
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pgaref pgaref

Labels
enhancement New feature or request
Projects
None yet
Milestone
Shuffle support
Development

No branches or pull requests
1 participant
@pgaref