Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve README #767

Merged
merged 7 commits into from
Oct 15, 2024
Merged

Improve README #767

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7f60e98
Change DOI link to something more stable
Kobzol Oct 14, 2024
d77f4ca
Slightly simplify main description of HQ
Kobzol Oct 14, 2024
3475f02
Expand features section in README
Kobzol Oct 14, 2024
4552271
Simplify installation and usage example
Kobzol Oct 14, 2024
233f203
Move FAQ out of the README
Kobzol Oct 14, 2024
123c9d0
Add license to README
Kobzol Oct 14, 2024
bf58019
Remove distributions mention from FAQ
Kobzol Oct 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 33 additions & 133 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,13 @@
<img src="docs/imgs/hq.png">
</p>

![Tests](https://github.com/it4innovations/hyperqueue/actions/workflows/test.yml/badge.svg) [![DOI paper](https://img.shields.io/badge/Paper-10.1016/j.softx.2024.101814-blue.svg)](https://doi.org/10.1016/j.softx.2024.101814) [![DOI software](https://zenodo.org/badge/349152473.svg)](https://zenodo.org/badge/latestdoi/349152473)
![Tests](https://github.com/it4innovations/hyperqueue/actions/workflows/test.yml/badge.svg) [![DOI paper](https://img.shields.io/badge/Paper-10.1016/j.softx.2024.101814-blue.svg)](https://www.sciencedirect.com/science/article/pii/S2352711024001857) [![DOI software](https://zenodo.org/badge/349152473.svg)](https://zenodo.org/badge/latestdoi/349152473)

**HyperQueue** is a tool designed to simplify execution of large workflows (task graphs) on HPC clusters. It allows you to execute a large number of tasks in a simple way, without having to manually submit jobs into batch schedulers like Slurm or PBS. You just specify what you want to compute – HyperQueue will automatically ask for computational resources and dynamically load-balance tasks across all allocated nodes and cores. HyperQueue can also work without Slurm/PBS as a general task executor.
**HyperQueue** is a tool designed to simplify execution of large workflows (task graphs) on HPC clusters. It allows you
to execute a large number of tasks in a simple way, without having to manually submit jobs into batch schedulers like
Slurm or PBS. You specify what you want to compute and HyperQueue automatically asks for computational resources
and dynamically load-balances tasks across all allocated nodes and resources. HyperQueue can also work without Slurm/PBS
as a general distributed task execution engine.

[Documentation](https://it4innovations.github.io/hyperqueue/)

Expand All @@ -21,24 +25,28 @@ This image shows how HyperQueue can work on a distributed cluster that uses Slur

## Features

- **Resource management**
- **Complex resource management**
- Load balancing of tasks across all available (HPC) resources
- Automatic submission of Slurm/PBS jobs on behalf of the user
- Complex and arbitrary task resource requirements (# of cores, GPUs, memory, FPGAs, ...)
- Fractional resources (task needs `0.5` of a GPU)
- Resource variants (task needs `1 GPU and 4 CPU cores` or `16 CPU cores`)
- Non-fungible resources (tasks are assigned specific resources, e.g. a GPU with ID `1`)
- Fractional resources (tasks can require e.g. `0.5` of a GPU)
- Resource variants (tasks can require e.g. `1 GPU and 4 CPU cores` OR `16 CPU cores`)
- Related resources (tasks can require e.g. `4 CPU cores in the same NUMA node`)

- **Performance**
- Scheduler can scale to hundreds of nodes and workers
- **High performance**
- Scales to hundreds of nodes/workers and millions of tasks
- Overhead per one task is below `0.1ms`
- Allows streaming of stdout/stderr from tasks to avoid creating many small files on distributed filesystems

- **Simple user interface**
- Task graphs can be defined via a CLI, TOML workflow files or a Python API
- Cluster utilization can be monitored with a real-time dashboard

- **Easy deployment**
- Provided as a single, statically linked binary without any dependencies apart from `libc`
- Provided as a single, statically linked binary without any runtime dependencies (apart from `libc`)
- No admin access to a cluster is needed for its usage

# Getting started

## Installation

* Download the latest binary distribution from this [link](https://github.com/It4innovations/hyperqueue/releases/latest).
Expand All @@ -48,12 +56,14 @@ This image shows how HyperQueue can work on a distributed cluster that uses Slur
$ tar -xvzf hq-<version>-linux-x64.tar.gz
```

* That's it! Just use the unpacked `hq` binary.

> If you want to try the newest features, you can also download a nightly
> [build](https://github.com/It4innovations/hyperqueue/releases/nightly).

## Submitting a simple task

* Start a server (e.g. on a login node or in a cluster partition)
* Start a server (e.g. on a login node, a cluster partition, or simply on your PC)

```bash
$ hq server start &
Expand All @@ -65,13 +75,13 @@ This image shows how HyperQueue can work on a distributed cluster that uses Slur
```
* Ask for computing resources

* Start worker manually
* Either start a worker manually

```bash
$ hq worker start &
```

* Automatic submission of workers into PBS/SLURM
* Or configure automatic submission of workers into PBS/SLURM

- PBS:

Expand All @@ -84,138 +94,25 @@ This image shows how HyperQueue can work on a distributed cluster that uses Slur
$ hq alloc add slurm --time-limit 1h -- -p <partition>
```

* Manual request in PBS

- Start worker on the first node of a PBS job

```bash
$ qsub <your-params-of-qsub> -- hq worker start
```
- Start worker on all nodes of a PBS job

```bash
$ qsub <your-params-of-qsub> -- `which pbsdsh` hq worker start
```
* Manual request in SLURM

- Start worker on the first node of a Slurm job

```bash
$ sbatch <your-params-of-sbatch> --wrap "hq worker start"
```
- Start worker on all nodes of a Slurm job

```bash
$ sbatch <your-params-of-sbatch> --wrap "srun --overlap hq worker start"
```

* Monitor the state of jobs
* See the result of the job once it finishes

```bash
$ hq job list --all
$ hq job wait last
$ hq job cat last stdout
```

## What's next?

Check out the [documentation](https://it4innovations.github.io/hyperqueue/).

# FAQ

* **How HQ works?**

You start a HQ server somewhere (e.g. login node, cloud partition of a cluster). Then you can submit your jobs to
the server. You may have hundreds of thousands of jobs; they may have various CPUs and other resource requirements.

Then you can connect any number of HQ workers to the server (either manually or via SLURM/PBS). The server will then
immediately start to assign jobs to them.

Workers are fully and dynamically controlled by server; you do not need to specify what jobs are executed on a
particular worker or preconfigure it in any way.

HQ provides a command line tool for submitting and controlling jobs.

<p align="center">
<img width="600" src="docs/imgs/schema.png">
</p>

* **What is a task in HQ?**

Task is a unit of computation. Currently, it is either the execution of an arbitrary external
program (specified via CLI) or the execution of a single Python function (specified via our Python
API).
You can find FAQ (frequently asked questions) [here](https://it4innovations.github.io/hyperqueue/stable/faq).

* **What is a job in HQ?**

Job is a collection of tasks (a task graph). You can display and manage jobs using the CLI.

* **Do I need to use SLURM or PBS to run HQ?**

No. Even though HQ is designed to smoothly work on systems using SLURM/PBS, they are not required for HQ to work.

* **Is HQ a replacement for SLURM or PBS?**

Definitely not. Multi-tenancy is out of the scope of HQ, i.e. HQ does not provide user isolation. HQ is
light-weight and easy to deploy; on an HPC system each user (or a group of users that trust each other)
may run her own instance of HQ.

* **Do I need an HPC cluster to run HQ?**

No. None of functionality is bound to any HPC technology. Communication between all components is performed using
TCP/IP. You can also run HQ locally.

* **What operating systems does HQ support?**

HyperQueue currently only officially supports Linux (Ubuntu, Debian, CentOS, etc.). It might be possible to
compile it for other operating systems, however we do not provide any support nor promise to fix any bugs for other
operating systems.

* **Is it safe to run HQ on a login node shared by other users?**

Yes. All communication is secured and encrypted. The server generates a secret file and only those users that have
access to it file may submit jobs and connect workers. Users without access to the secret file will only see that the
service is running.

* **How many jobs/tasks may I submit into HQ?**

Our preliminary benchmarks show that the overhead of HQ is around 0.1 ms per task. It should be
thus possible to submit a job with tens or hundreds of thousands tasks into HQ.

Note that HQ is designed for a large number of tasks, not jobs. If you want to perform a lot of
computations, use [task arrays](jobs/arrays.md), i.e. create a job with many tasks, not many jobs
each with a single task.

HQ also supports [streaming](jobs/streaming.md) of task outputs into a single file.
This avoids creating many small files for each task on a distributed file system, which improves
scaling.

* **Does HQ support multi-CPU jobs?**

Yes. You can define an arbitrary amount of cores for each task. HQ is also NUMA aware
and you can select the allocation strategy.

* **Does HQ support job arrays?**

Yes, see [task arrays](https://it4innovations.github.io/hyperqueue/stable/jobs/arrays).

* **Does HQ support dependencies between tasks?**

Yes.

* **How is HQ implemented?**

HQ is implemented in Rust and the Tokio async ecosystem. The scheduler is a work-stealing scheduler
implemented in our project [Tako](https://github.com/spirali/tako/),
which is derived from our previous work [RSDS](https://github.com/It4innovations/rsds).
Integration tests are written in Python, but HQ itself does not depend on Python.

You can find more frequently asked questions [here](https://it4innovations.github.io/hyperqueue/stable/faq).

# HyperQueue team
## HyperQueue team

We are a group of researchers working at [IT4Innovations](https://www.it4i.cz/), the Czech National
Supercomputing Center. We welcome any outside contributions.

# Publications
## Publications

- [HyperQueue: Efficient and ergonomic task graphs on HPC clusters](https://www.sciencedirect.com/science/article/pii/S2352711024001857)
(paper @ SoftwareX journal)
Expand All @@ -239,8 +136,11 @@ If you want to cite HyperQueue, you can use the following BibTex entry:
}
```

# Acknowledgement
## Acknowledgement

* This work was supported by the LIGATE project. This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 956137. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Italy, Sweden, Austria, the Czech Republic, Switzerland.

* This work was supported by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90140).

## License
[MIT](LICENSE)
23 changes: 14 additions & 9 deletions docs/faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# FAQ
Here you can find a list of frequently asked questions about HyperQueue. If you'd like to ask
about anything related to HyperQueue, feel free to ask on our [discussion forum](https://github.com/It4innovations/hyperqueue/discussions) or on our [Zulip server](https://hyperqueue.zulipchat.com/).

## HQ fundamentals

??? question "How does HQ work?"
Expand Down Expand Up @@ -29,6 +32,12 @@

Job is a collection of tasks (a task graph). You can display and manage jobs using the CLI.

??? question "What operating systems does HQ support?"

HyperQueue currently only officially supports Linux. It might be possible to
compile it for other operating systems, however we do not provide any support nor promise to fix any bugs for other
operating systems.

??? question "How to deploy HQ?"

HQ is distributed as a single, self-contained and statically linked binary. It allows you to start
Expand Down Expand Up @@ -93,15 +102,11 @@
those users that have access to that file may submit jobs and connect workers. Users without
access to the secret file will only see that the service is running.

Performance should also not be a concern. Our [experiments](https://kobzol.github.io/phd/#pf96) show that the server
consumes only ~0.3ms of CPU time every second per a thousand tasks executed.

## Relation to other task runtimes

??? question "What is the difference between HQ and Snakemake?"
??? question "How does HQ differ from SnakeMake/Dask/Merlin/...?"

In cluster mode, Snakemake submits each Snakemake job as one HPC job into SLURM/PBS. If your jobs
are too small, you will have to manually aggregate them to avoid exhausting SLURM/PBS resources.
Manual job aggregation is often quite arduous and since the aggregation is static, it might also
waste resources because of missing load balancing.

In the case of HQ, you do not have to aggregate tasks. You can submit millions of small tasks to
HQ and it will take care of assigning them dynamically to individual workers (and SLURM/PBS jobs,
if [automatic allocation](deployment/allocation.md) is used).
You can find a comparison of HQ with similar tools [here](./other-tools.md).
Loading