Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added run_jobs_using_params() #36

Merged
merged 28 commits into from
Apr 29, 2024
Merged
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
ad23d4a
added json functionality to slurm params
mitkotak Jul 13, 2022
a3a024c
Added alternative path if parameter not found
mitkotak Jul 13, 2022
a30396b
added try catch
mitkotak Jul 13, 2022
7c7126e
closing json file
mitkotak Jul 13, 2022
f620e39
updating existing json file
mitkotak Jul 13, 2022
4b772fe
modified updateconfig to read and write in the same call
mitkotak Jul 13, 2022
96a2d1a
added print statements for debugging
mitkotak Jul 14, 2022
fc78481
improved print statements
mitkotak Jul 14, 2022
a3a8aa7
added json print statement
mitkotak Jul 14, 2022
1f4200b
created save param button
mitkotak Jul 14, 2022
f630a43
globalized savedParam variable
mitkotak Jul 14, 2022
9d522ab
hopefully button is showing up now
mitkotak Jul 14, 2022
ad6cee5
added button display
mitkotak Jul 14, 2022
cf3620d
fixed rendering
mitkotak Jul 14, 2022
ac0022e
resolved flake8
mitkotak Jul 14, 2022
9acc3e0
enable overwriting file
mitkotak Jul 14, 2022
9741a0b
close file after loading
mitkotak Jul 14, 2022
8ecfb75
fixed flake8 again
mitkotak Jul 14, 2022
b30d6b3
Merge branch 'v2' into ui_params
mitkotak Aug 31, 2022
7950ec4
Merge branch 'v2' into ui_params
mitkotak Sep 8, 2022
cbcb699
added run job using params
mitkotak Sep 8, 2022
d6ee3f1
removed json ( for testing )
mitkotak Sep 8, 2022
0027ec8
added default executable folder
mitkotak Sep 8, 2022
ed92d30
removing print statements
mitkotak Sep 8, 2022
d0fbd7b
removed double login
mitkotak Sep 8, 2022
0146fad
added hpc
mitkotak Sep 8, 2022
d07d877
Merge branch 'v2' of https://github.com/cybergis/cybergis-compute-pyt…
mitkotak Sep 15, 2022
6b01d2c
Merge branch 'v2' into ui_params
mitkotak Apr 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions cybergis_compute_client/CyberGISCompute.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@
from IPython.display import display, Markdown, Javascript


class ParamAccumulator:
Copy link
Member

@alexandermichels alexandermichels Sep 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite understanding what the role of this helper class is at this point. It seems to be a wrapper around a dictionary that doesn't provide any additional functionality.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the role for that class is to handle all of the input preprocessing that might be needed. For e.g. json paths ( working on it rn ) or some other input format that we might wanna support.

def __init__(self,
params={}):
self.params = params


class CyberGISCompute:
"""CyberGISCompute class
An inteface that handles all interactions with the HPC backend
Expand Down Expand Up @@ -210,6 +216,25 @@ def create_job(self, maintainer='community_contribution', hpc=None, hpcUsername=
self.login()
return Job(maintainer=maintainer, hpc=hpc, id=None, hpcUsername=hpcUsername, hpcPassword=hpcPassword, client=self.client, isJupyter=self.isJupyter, jupyterhubApiToken=self.jupyterhubApiToken, printJob=verbose)

def run_job_using_params(self,
input_params=[],
maintainer='community_contribution',
hpc="keeling_community",
hpcUsername=None,
hpcPassword=None,
localExecutableFolder={"type": "git",
"gitId": "hello_world"},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this is how the user would pass in the "job"/"model" which will probably not be intuitive to them.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup I will handle that in ParamAccumulator (Will maybe change the name to something else since its handling "job"/"mode" preprocessing as well).

localDataFolder=None,
localResultFolder=None,
env=None,
slurm=None,
verbose=True):
for params in input_params:
param_acc = ParamAccumulator(params)
job = self.create_job(maintainer, hpc, hpcUsername, hpcPassword)
job.set(localExecutableFolder, localDataFolder, localResultFolder, param_acc.params, env, slurm)
job.submit()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense and works (tested it with a couple of jobs), but we may also need to report the status of jobs. I imagine the way this is going to be used, end-users will want to know that all of their jobs ran and didn't throw errors. Maybe next week we can sit down with Drew and get feedback on that.


def get_job_by_id(self, id=None, verbose=True):
"""
Returns Job object with the specified id
Expand Down