-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added run_jobs_using_params() #36
Changes from all commits
ad23d4a
a3a024c
a30396b
7c7126e
f620e39
4b772fe
96a2d1a
fc78481
a3a8aa7
1f4200b
f630a43
9d522ab
ad6cee5
cf3620d
ac0022e
9acc3e0
9741a0b
8ecfb75
b30d6b3
7950ec4
cbcb699
d6ee3f1
0027ec8
ed92d30
d0fbd7b
0146fad
d07d877
6b01d2c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,6 +18,12 @@ | |
from IPython.display import display, Markdown, Javascript | ||
|
||
|
||
class ParamAccumulator: | ||
def __init__(self, | ||
params={}): | ||
self.params = params | ||
|
||
|
||
class CyberGISCompute: | ||
"""CyberGISCompute class | ||
An inteface that handles all interactions with the HPC backend | ||
|
@@ -210,6 +216,25 @@ def create_job(self, maintainer='community_contribution', hpc=None, hpcUsername= | |
self.login() | ||
return Job(maintainer=maintainer, hpc=hpc, id=None, hpcUsername=hpcUsername, hpcPassword=hpcPassword, client=self.client, isJupyter=self.isJupyter, jupyterhubApiToken=self.jupyterhubApiToken, printJob=verbose) | ||
|
||
def run_job_using_params(self, | ||
input_params=[], | ||
maintainer='community_contribution', | ||
hpc="keeling_community", | ||
hpcUsername=None, | ||
hpcPassword=None, | ||
localExecutableFolder={"type": "git", | ||
"gitId": "hello_world"}, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems this is how the user would pass in the "job"/"model" which will probably not be intuitive to them. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yup I will handle that in ParamAccumulator (Will maybe change the name to something else since its handling "job"/"mode" preprocessing as well). |
||
localDataFolder=None, | ||
localResultFolder=None, | ||
env=None, | ||
slurm=None, | ||
verbose=True): | ||
for params in input_params: | ||
param_acc = ParamAccumulator(params) | ||
job = self.create_job(maintainer, hpc, hpcUsername, hpcPassword) | ||
job.set(localExecutableFolder, localDataFolder, localResultFolder, param_acc.params, env, slurm) | ||
job.submit() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This makes sense and works (tested it with a couple of jobs), but we may also need to report the status of jobs. I imagine the way this is going to be used, end-users will want to know that all of their jobs ran and didn't throw errors. Maybe next week we can sit down with Drew and get feedback on that. |
||
|
||
def get_job_by_id(self, id=None, verbose=True): | ||
""" | ||
Returns Job object with the specified id | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not quite understanding what the role of this helper class is at this point. It seems to be a wrapper around a dictionary that doesn't provide any additional functionality.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah the role for that class is to handle all of the input preprocessing that might be needed. For e.g. json paths ( working on it rn ) or some other input format that we might wanna support.