Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supporting byPassMergeSort shuffle for SGX #17

Merged
merged 5 commits into from
Aug 1, 2019
Merged

Supporting byPassMergeSort shuffle for SGX #17

merged 5 commits into from
Aug 1, 2019

Conversation

pgaref
Copy link
Collaborator

@pgaref pgaref commented Jul 30, 2019

Addressing #15

  • Introducing new SHUFFLE_MAP type for bypassMerge sort shuffle
  • Adding numOfPartitions as part of SGX RDD (needed by the shuffle partitioner)
  • Introducing SGX partitioner - hash-based for now (to be changed by any pseudorandom function we want)
  • Introducing sgxWrite interface as part of Shuffle writer, when SGX is enabled we use this path instead. The method takes as arguments both the encrypted records AND the record-partition mapping (recordMapping) as returned by the enclave worker
  • Handling SHUFFLE_MAP_BYPASS task type in SGX worker - return a record-partition mapping iterator
  • ShuffleMapTask checks if SGX is enabled and calls the appropriate shuffle interface
  • ByPassSort shuffle test case with multiple partitions

pgaref added 5 commits July 30, 2019 19:10
* Adding numOfPartitions as part of SGX RDD (needed by the shuffle partitioner)
When SGX is enable we use this path instead
The method takes as arguments both the encrypted records AND the record-partition mapping (recordMapping) as return by the enclave worker
…rd-partition mapping iterator

* ShuffleMapTask checks if SGX is enabled and calls the appropriate shuffle interface
@pgaref pgaref merged commit c250aa7 into master Aug 1, 2019
@pgaref pgaref deleted the shuffle branch August 11, 2019 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant