Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated SLEAP-related slides and slurm scripts #22

Merged
merged 3 commits into from
Oct 1, 2024
Merged

Updated SLEAP-related slides and slurm scripts #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
28d4aff
renamed slurm scripts in slides to match repo
niksirbi Sep 30, 2024
734b1e5
update SLEAP slides and slurm scripts for Oct 2024
niksirbi Sep 30, 2024
7083a9c
remove no longer needed images
niksirbi Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions img/diagrams/array-jobs.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed img/modern_behav_experiment_analysis.png
View file
Open in desktop
Binary file not shown.
61 changes: 19 additions & 42 deletions index.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -502,27 +502,6 @@ scancel -u <USERNAME>

# Example: pose estimation with SLEAP {background-color="#03A062"}

## Modern behavioural analysis {.smaller}

:::: {.columns}

::: {.column width="70%"}
![](img/modern_behav_experiment_analysis.png){fig-align="center" height=400px}
:::

::: {.column width="30%"}
```{mermaid}
%%| file: img/diagrams/video_pipeline_pose.mmd
%%| fig-height: 400px
```
:::

::::

::: aside
Source: [Open-source tools for behavioral video analysis: Setup, methods, and best practices](https://elifesciences.org/articles/79305)
:::

## Pose estimation {.smaller}

![](img/pose_estimation_2D.png){fig-align="center"}
Expand Down Expand Up @@ -596,9 +575,9 @@ see also [SLEAP's guide for remote training](https://sleap.ai/guides/remote.html
Copy the unzipped training package to your scratch space and inspect its contents:

```{.bash code-line-numbers="1|2-3"}
cp -r /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-2023/labels.v001.slp.training_job /ceph/scratch/<USERNAME>/
cd /ceph/scratch/<USERNAME>/labels.v001.slp.training_job
ls
cp -r /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-2023/labels.v001.slp.training_job /ceph/scratch/$USER/
cd /ceph/scratch/$USER/labels.v001.slp.training_job
ls -1
```

::: {.fragment}
Expand Down Expand Up @@ -660,7 +639,7 @@ Suitable for debugging (immediate feedback)
- Execute commands one-by-one, e.g.:
```{.bash code-line-numbers=false}
module load SLEAP
cd /ceph/scratch/<USERNAME>/labels.v001.slp.training_job
cd /ceph/scratch/$USER/labels.v001.slp.training_job
bash train-script.sh

# Stop the session
Expand Down Expand Up @@ -792,7 +771,7 @@ cat slurm.slp_train.gpu-380-18.4232289.err
## View trained models {.smaller}
While you wait for the training job to finish, you can copy and inspect the trained models from a previous run:
```{.bash code-line-numbers="false"}
cp -R /ceph/scratch/sirmpilatzen/labels.v001.slp.training_job/models /ceph/scratch/$USER/labels.v001.slp.training_job/
cp -R /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-2023/labels.v001.slp.training_job/models /ceph/scratch/$USER/labels.v001.slp.training_job/
cd /ceph/scratch/$USER/labels.v001.slp.training_job/models
ls
```
Expand Down Expand Up @@ -839,16 +818,16 @@ see also the SLEAP [model evaluation notebook](https://sleap.ai/notebooks/Model_

## Batch script for inference {.smaller}

```{.bash filename="sleap_inference_slurm.sh" code-line-numbers="1-16|18-22|24-28|30-36"}
```{.bash filename="sleap_infer_slurm.sh" code-line-numbers="1-16|18-22|24-28|30-36"}
#!/bin/bash

#SBATCH -J slp_infer # job name
#SBATCH -p gpu # partition
#SBATCH -N 1 # number of nodes
#SBATCH --mem 64G # memory pool for all cores
#SBATCH -n 32 # number of cores
#SBATCH --mem 32G # memory pool for all cores
#SBATCH -n 8 # number of cores
#SBATCH -t 0-01:00 # time (D-HH:MM)
#SBATCH --gres gpu:rtx5000:1 # request 1 RTX5000 GPU
#SBATCH --gres gpu:1 # request 1 GPU
#SBATCH -o slurm.%x.%N.%j.out # write STDOUT
#SBATCH -e slurm.%x.%N.%j.err # write STDERR
#SBATCH --mail-type=ALL
Expand Down Expand Up @@ -882,12 +861,12 @@ sleap-track $VIDEO_DIR/${VIDEO1_PREFIX}_video.mp4 \

1. Edit and save the batch script
```{.bash code-line-numbers="false"}
nano sleap_inference_slurm.sh
nano sleap_infer_slurm.sh
```

2. Submit the job
```{.bash code-line-numbers="false"}
sbatch sleap_inference_slurm.sh
sbatch sleap_infer_slurm.sh
```

3. Monitor the job
Expand All @@ -897,28 +876,25 @@ squeue --me

## Run inference as an array job {.smaller}

```{mermaid}
%%| file: img/diagrams/array-jobs.mmd
%%| fig-height: 500px
```
![](img/diagrams/array-jobs.svg){fig-align="center"}

## Batch script for array job {.smaller}

```{.bash filename="sleap_inference_slurm_array.sh" code-line-numbers="14|24-28|36-42"}
```{.bash filename="sleap_infer_array_slurm.sh" code-line-numbers="14|24-28|36-42"}
#!/bin/bash

#SBATCH -J slp_infer # job name
#SBATCH -p gpu # partition
#SBATCH -N 1 # number of nodes
#SBATCH --mem 64G # memory pool for all cores
#SBATCH -n 32 # number of cores
#SBATCH --mem 32G # memory pool for all cores
#SBATCH -n 8 # number of cores
#SBATCH -t 0-01:00 # time (D-HH:MM)
#SBATCH --gres gpu:rtx5000:1 # request 1 RTX5000 GPU
#SBATCH --gres gpu:1 # request 1 GPU
#SBATCH -o slurm.%x.%N.%j.out # write STDOUT
#SBATCH -e slurm.%x.%N.%j.err # write STDERR
#SBATCH --mail-type=ALL
#SBATCH [email protected]
#SBATCH --array=1-2
#SBATCH --array=0-1

# Load the SLEAP module
module load SLEAP
Expand All @@ -931,7 +907,7 @@ VIDEO_DIR=/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data/course-hpc-
VIDEO1_PREFIX=sub-01_ses-01_task-EPM_time-165049
VIDEO2_PREFIX=sub-02_ses-01_task-EPM_time-185651
VIDEOS_PREFIXES=($VIDEO1_PREFIX $VIDEO2_PREFIX)
CURRENT_VIDEO_PREFIX=${VIDEOS_PREFIXES[$SLURM_ARRAY_TASK_ID - 1]}
CURRENT_VIDEO_PREFIX=${VIDEOS_PREFIXES[$SLURM_ARRAY_TASK_ID]}
echo "Current video prefix: $CURRENT_VIDEO_PREFIX"

# Go to the job directory
Expand All @@ -952,3 +928,4 @@ sleap-track $VIDEO_DIR/${CURRENT_VIDEO_PREFIX}_video.mp4 \
## Further reading
* [SWC/GCNU Scientific Computing wiki](https://wiki.ucl.ac.uk/display/SSC/High+Performance+Computing)
* [SLURM documentation](https://slurm.schedmd.com/)
* [How to use the SLEAP module on the SWC HPC cluster](https://howto.neuroinformatics.dev/data_analysis/HPC-module-SLEAP.html)
Loading
Loading