-
Notifications
You must be signed in to change notification settings - Fork 103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
In memory shuffle #135
Open
jegonzal
wants to merge
2
commits into
amplab:vldb
Choose a base branch
from
jegonzal:in-memory-shuffle
base: vldb
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
In memory shuffle #135
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ankurdave
added a commit
that referenced
this pull request
Mar 28, 2014
ankurdave
added a commit
to ankurdave/graphx
that referenced
this pull request
Apr 20, 2014
(cherry picked from commit 5ec645d) Conflicts: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala core/src/main/scala/org/apache/spark/storage/BlockManager.scala graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
ankurdave
added a commit
to ankurdave/graphx
that referenced
this pull request
Apr 20, 2014
This reverts commit 5b65b1f. Conflicts: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
dcrankshaw
pushed a commit
to dcrankshaw/graphx
that referenced
this pull request
Apr 28, 2014
ankurdave
added a commit
to amplab/graphx2
that referenced
this pull request
Apr 30, 2014
This reverts commit 5ec645d. Conflicts: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
ankurdave
added a commit
to ankurdave/spark
that referenced
this pull request
Aug 19, 2014
1. In-memory shuffle 2. Checking Iterator.hasNext 3. In-place/destructive updates in vertex replication 4. WikiPipelineBenchmark 5. K-Core 6. Dataflow (naive Spark) PageRank Squashed commit of the following: commit ad6590c Author: Ankur Dave <[email protected]> Date: Mon Apr 21 17:56:24 2014 -0700 Add unpersist option to Pregel commit 3bcaa2f Author: Dan Crankshaw <[email protected]> Date: Wed Apr 30 05:58:11 2014 +0000 Added faster version of kcore and fixed compile issue with dataflow pagerank commit 03f5d76 Author: Dan Crankshaw <[email protected]> Date: Wed Apr 30 02:22:13 2014 +0000 Added more connected components. commit cfd209b Author: Dan Crankshaw <[email protected]> Date: Wed Apr 30 00:01:24 2014 +0000 Initial CC dataflow implementation. commit 255db45 Author: Dan Crankshaw <[email protected]> Date: Tue Apr 29 19:38:26 2014 +0000 Updated pregel logging. commit 5032924 Author: Dan Crankshaw <[email protected]> Date: Tue Apr 29 08:52:07 2014 +0000 Added dataflow pagerank. Having issues with it and in-memory shuffle. commit f483ca4 Merge: 8d22500 9e59642 Author: Ankur Dave <[email protected]> Date: Tue Apr 29 00:05:49 2014 -0700 Merge pull request apache#3 from dcrankshaw/osdi_with_kcore_for_merge Osdi with kcore for merge commit 9e59642 Author: Dan Crankshaw <[email protected]> Date: Mon Apr 28 20:51:24 2014 +0000 Added more logging. Conflicts: graphx/src/main/scala/org/apache/spark/graphx/lib/Analytics.scala commit 76a6a54 Author: Dan Crankshaw <[email protected]> Date: Sun Apr 13 03:00:43 2014 +0000 Fixed kcore. now works. commit 1e924df Author: Dan Crankshaw <[email protected]> Date: Sun Apr 13 00:06:40 2014 +0000 Changed kcore filenames. commit a641dc1 Author: Dan Crankshaw <[email protected]> Date: Sat Apr 12 01:59:17 2014 +0000 Kcore debugging. commit dbe5180 Author: Dan Crankshaw <[email protected]> Date: Fri Apr 11 22:49:38 2014 +0000 Fixed results of cherry-pick commit 4ddc552 Author: Dan Crankshaw <[email protected]> Date: Tue Jan 7 13:44:46 2014 -0800 Added kcore implementation. Conflicts: graph/src/main/scala/org/apache/spark/graph/Pregel.scala commit 250199a Author: Dan Crankshaw <[email protected]> Date: Mon Apr 28 22:23:49 2014 +0000 Minor logging changes. commit 8d22500 Author: Ankur Dave <[email protected]> Date: Fri Apr 25 16:34:53 2014 -0700 EdgePartition.size should be val commit bb36cc8 Author: Ankur Dave <[email protected]> Date: Fri Apr 25 03:08:27 2014 -0700 Set locality wait commit ad4c874 Author: Ankur Dave <[email protected]> Date: Sat Apr 19 20:08:57 2014 -0700 In GraphLoader, coalesce to minEdgePartitions commit 2898126 Author: Ankur Dave <[email protected]> Date: Sat Apr 19 20:10:36 2014 -0700 In Analytics, take PageRank numIter commit 719b04a Author: Ankur Dave <[email protected]> Date: Sat Apr 19 20:09:24 2014 -0700 Log current Pregel iteration commit 8609184 Author: Ankur Dave <[email protected]> Date: Fri Mar 28 15:15:44 2014 -0700 Clean up WikiPipelineBenchmark commit e8be08e Author: Ankur Dave <[email protected]> Date: Fri Mar 28 15:14:50 2014 -0700 for-loop to while-loop in WikiArticle commit 540c267 Author: Ankur Dave <[email protected]> Date: Fri Mar 28 15:13:56 2014 -0700 Make mutable RDDs optional commit e27da44 Author: Ankur Dave <[email protected]> Date: Fri Mar 28 15:12:58 2014 -0700 Check for empty iterators commit 5ec645d Author: Ankur Dave <[email protected]> Date: Fri Mar 28 15:10:41 2014 -0700 In memory shuffle (cherry-picked from amplab/graphx#135) commit f36e576 Author: Ankur Dave <[email protected]> Date: Fri Mar 28 15:09:23 2014 -0700 Log warning on partition recompute commit c289e91 Author: Ankur Dave <[email protected]> Date: Fri Mar 28 15:08:06 2014 -0700 Untrack conf files commit a61be7a Author: Dan Crankshaw <[email protected]> Date: Tue Mar 25 07:03:41 2014 +0000 More timing logging and fixing the iterator issue. commit 0645183 Author: Dan Crankshaw <[email protected]> Date: Mon Mar 24 21:34:00 2014 +0000 Fixed IO bugs. commit a13f3b9 Author: Dan Crankshaw <[email protected]> Date: Mon Mar 24 19:52:46 2014 +0000 Fixed compile errors commit e6fc93b Author: Dan Crankshaw <[email protected]> Date: Mon Mar 24 03:29:19 2014 +0000 Updated pipeline benchmark to handle other systems. commit 0a04cb9 Author: Dan Crankshaw <[email protected]> Date: Sun Mar 23 20:57:08 2014 +0000 Renamed prepostprocess commit a262b07 Merge: 470a950 3342751 Author: Ankur Dave <[email protected]> Date: Fri Mar 21 11:39:02 2014 -0700 Merge remote-tracking branch 'dcrankshaw-incubator-spark/gx-vldb-bench' into vldb Conflicts: project/SparkBuild.scala commit 470a950 Merge: e09139d 3407a8f Author: Ankur Dave <[email protected]> Date: Fri Mar 21 11:30:12 2014 -0700 Merge branch 'mutable-rdd' into vldb commit 3342751 Author: Dan Crankshaw <[email protected]> Date: Sun Mar 2 02:14:46 2014 +0000 Fixed partitioning issues. commit 3407a8f Author: Ankur Dave <[email protected]> Date: Sat Mar 1 16:39:09 2014 -0800 Try mutating old RDDs for delta updates commit fad630f Author: Dan Crankshaw <[email protected]> Date: Wed Feb 26 02:33:01 2014 +0000 more debugging. Seems like we have empty partitions. commit b788bfb Author: Dan Crankshaw <[email protected]> Date: Tue Feb 25 11:10:16 2014 +0000 added more debug logging. commit 75b2da7 Author: Dan Crankshaw <[email protected]> Date: Tue Feb 25 09:47:32 2014 +0000 Fixed pipeline. Now seems to work well. commit 7799b3e Author: Dan Crankshaw <[email protected]> Date: Mon Feb 24 22:31:47 2014 +0000 updated slaves and conf. commit 95e19ea Author: Dan Crankshaw <[email protected]> Date: Sun Feb 23 06:19:21 2014 +0000 Readadded prepost commit da93886 Author: Dan Crankshaw <[email protected]> Date: Sun Feb 23 06:13:24 2014 +0000 Stringify still not working. FML. commit 6a4e492 Author: Dan Crankshaw <[email protected]> Date: Sun Feb 23 03:34:00 2014 +0000 Added debugging. Issue with stringify. commit 99d2713 Author: Dan Crankshaw <[email protected]> Date: Sat Feb 22 04:40:45 2014 +0000 Added new conf commit 7a036cb Author: Dan Crankshaw <[email protected]> Date: Fri Feb 21 20:35:37 2014 -0800 Fixed compilation issues. commit d3bbfd0 Author: Dan Crankshaw <[email protected]> Date: Wed Feb 19 06:15:14 2014 +0000 Beginning of new pipeline, not compiled yet. commit 2983132 Author: root <[email protected]> Date: Wed Feb 19 01:23:15 2014 +0000 Initial vldb benchmark commmit. Added slaves file.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.