Skip to content
This repository has been archived by the owner on May 1, 2024. It is now read-only.

WIP-Spark Tasks #476

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

WIP-Spark Tasks #476

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
097b826
convert TotalEventsDailyTask task to spark #486
rao-abdul-mannan Jan 24, 2018
d615749
convert user activity task to spark
rao-abdul-mannan Jan 24, 2018
d06f87f
move spark runtime options to luigi config
rao-abdul-mannan Feb 21, 2018
78a842f
replace username with user_id
rao-abdul-mannan May 5, 2018
0b8b937
rebased with master before location history job conversion
rao-abdul-mannan May 22, 2018
01f6b2b
user location history spark task
rao-abdul-mannan May 28, 2018
b863f58
using manifest file as input source to spark
rao-abdul-mannan May 31, 2018
744684e
Enable logging in spark tasks
rao-abdul-mannan Jun 7, 2018
e204e49
Process event logs directly
rao-abdul-mannan Jun 8, 2018
2b7b94b
using wildcards with dataframe source
rao-abdul-mannan Jun 10, 2018
64308be
with rdd
rao-abdul-mannan Jun 12, 2018
1a42858
enable verbose logging
rao-abdul-mannan Jun 13, 2018
5f6726d
address PR review changes
rao-abdul-mannan Jun 14, 2018
e5d3e87
Parallelize file loading to reduce in-activity delays
rao-abdul-mannan Jun 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions config/devstack.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,3 +131,8 @@ api_root_url = http://localhost:8000/api/courses/v1/courses/

[course-blocks]
api_root_url = http://localhost:8000/api/courses/v1/blocks/

[spark]
driver-memory=3g
executor-memory=3g
executor-cores=1
Loading