User Stories for Requesting Resources

[vsoch]

Here we want to collect user stories for requesting resources, with the goal of designing something that smells like a new Jobspec. The details of that cannot be determined until we understand the current landscape and needs. Specifically, we want to know:

1. How are you defining your jobs and their requirements?

• Are you using the current canonical jobspec v1?
• Flags, whether parsed by the command line or script bindings
• Batch script directives (for example # FLUX: ...)
• Delegated to a workflow tool's format (Maestro, Merlin, Snakemake, ATS, etc.)

2. How are you currently submitting jobs to flux?

E.g., command line flux (run|submit|batch), scripting to our bindings (which ones), or via a workflow tool that integrates with us?

3. What are you trying to do?

E.g., "As a scientist running simulations, I want to ask for 4 nodes with 128 cores each and <with some topology>

4. What would you like to be able to do that you currently either can't or don't know how to?

Possibilities may include things like co-scheduling, inter-node locality or rank layout, intra-node binding/affinity, stage-in/stage-out, dependency types, services, what do you currently have to write a helper script or wrapper or manager for that might better be expressed to the resource manager?

Please feel free to expand on any of the above as it will help us to understand your needs. I (@vsoch) will write an example response in the first thread. Thank you!

[vsoch]

Here are some of my responses (in the format of use cases) to help guidance. I am organizing in "what I currently do" and "what I want to do."

How I am defining jobs

• As a workflow developer, I am writing jobs in other tool input formats (e.g., Snakefile) that gets piped into flux submit from within Python
• As a flux integrations developer, I use flux run|submit|batch typically asking for nodes and tasks
• As a scientific workflow developer (e.g., running lammps) I use run|submit|batch and then topology mapping for cpu and gpu
• As a developer, I struggle when I need to parse a jobspec v1 natively (without supporting Python wrapper, etc).
• As a scientific workflow developer, I have not needed to define anything within the unit of a job that is more complex than homogeneous nodes / tasks / mapping. When there are different components, they get moved up to the level of the workflow, which contains these units of work.

What I'd like to be able to do

• As a workflow developer, I would like a simple, flatter specification file for my workflows (e.g., hcl like nomad) or something that presents the information in the same way I think about it.
• As a flux integrations developer, I would like a language / equivalent specification or ability to define a flux batch session that is more structured than a batch script file with different flux directives. Ideally it would work with my flattened specification file(s).
• As a scientific workflow developer I want to still have simple command line flags to get the basics of what I need

[jedbrown]

As a library/app developer running on pre-acceptance hardware, I need to bind processes to cores or NUMA domains that are close to GPUs, attach job-specific storage volumes (e.g., DAOS), and adjust tracing/debugging settings. It's important to be able to edit job parameters after a job has been submitted to a queue, including adjusting the number of requested nodes (the total number of online nodes changes) and working around nodes that are presumed bad (based on recent jobs failing with logs suggesting hardware failures) but have not yet been taken offline by facilities. If a job hangs, we need to batch collect stack traces for analysis before it times out, again to help with bug reporting to the facility/vendor.

[brandonbiggs]

Hi, following up from Mastodon. I hope this is helpful/relevant.

These answers are what I see as a sysadmin when I help people or I submit my own jobs doing tests or running other codes

How I am defining jobs

• We have slurm and PBS, so many jobs are ran via the command line srun or qsub but the most "in depth" I've seen anyone care to be able to request is the number of cores, memory, or GPUs. I have seen with PBS people requesting mpiprocs in the resource request line. We do also need to submit to different projects for statics purposes.
• We do share nodes between users. So if 4 users each want a GPU, they would be split up on one node each with an allocated GPU. For those types of requests, we usually divide the ncpus by 4 to give everyone equal amounts of memory. If someone requests more memory, we assign them the appropriate number of cores automatically.
• We have a lot of users who are using a web interface/science gateway to submit their jobs. I think this has gotten to be more than half of our users. So they just fill out a small form and then get an interactive session. We do all of the job submissions behind the scenes automatically. Here's the documentation as an example of how we're submitting - https://osc.github.io/ood-documentation/latest/how-tos/app-development/interactive/submit.html?highlight=submit%20yml

What I'd like to be able to do

• We have a lot of people who have workflow tools they want to use with HPC but they don't have ways to integrate easily or they aren't sure of how to do it. We try to work with them, but we're a small team and don't have time to write a lot of code if they need something created. I think there are APIs for PBS/Slurm which could definitely help.
• This is rare, but we have some users who like to port forward or tunnel to compute nodes. Maybe this is just our site and there's already a way to do this, but someone needs to submit a job, open another shell, and then ssh separately to the compute node. It would be convenient if we could do that in one swoop.

Hope this is helpful! If there's any clarification I can do, please let me know!