diff --git a/docs/source/beginners_guide/linux_shell.rst b/docs/source/beginners_guide/linux_shell.rst index 6729498..efbf79e 100644 --- a/docs/source/beginners_guide/linux_shell.rst +++ b/docs/source/beginners_guide/linux_shell.rst @@ -1,4 +1,4 @@ -Linux Shell +Linux shell =========== Intro @@ -74,10 +74,10 @@ To logout of the shell type ``logout``, ``exit`` or press ``Ctrl + d``. The are a number of shells available to the user. In this tutorial we will be using `Bash `_, the most widely used Linux shell. -Files and Directories +Files and directories --------------------- -Filesystem Organisation +Filesystem organisation ^^^^^^^^^^^^^^^^^^^^^^^ The file system is the component of the operating system that organises data into files. These files are organised into directories. @@ -108,7 +108,7 @@ The output of the ``pwd`` command, ``/usr/researchcomp/elecclust/abs4``, is call The ``cd`` command lets you change your working directory to another location in the file system. ``cd`` with no arguments places you back in your home directory. The special directory ``..`` references the directory above your current directory (known as the parent directory). The is another special direcory ``.`` which references the current directory. These two directories can be viewed as *links*. -Listing Files and Directories +Listing files and directories ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To list the files in a directory use the ``ls`` (list) command. @@ -253,7 +253,7 @@ The next example displays the directory in the long format using the ``-l`` opti Using a directory name as an option causes ``ls`` to list the contents of the directory. To list the attributes of the directory use the ``-d`` option. You can use a pathname as the argument. -Creating, Moving and Copying Files and Directories +Creating, moving and copying files and directories ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can create directories, move or copy files or directories to other locations in the filesystem using the ``mkdir`` (make directory) ``mv`` (move) and ``cp`` (copy) commands. @@ -343,7 +343,7 @@ To delete directories use the ``rmdir`` (remove directory) command. ``rmdir`` will only remove empty directories. To remove a directory and all it's contents use the ``rm -r`` (recursive) option to the ``rm`` command. To be safe and check the files before you remove them use ``rm -ri`` (recursive and interactive) options. -Editing and Displaying the Contents of Files +Editing and displaying the contents of files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Text Editors @@ -410,7 +410,7 @@ The ``cat`` command displays all the text in the users file on the screen. This | ``/pattern`` - search for text in the file -Files and Directory Permissions +Files and directory permissions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Groups are provided to manage sets of users and control access to files and directories. All users belong to a default group and may be a member of other groups. @@ -470,7 +470,7 @@ In the above example the first column of the directory listing shows the permiss This is a ``directory``, the owner ``abs4`` can ``read``, ``write`` and ``access`` the directory. Members of the group, ``csrv``, can ``read`` and ``access`` the directory, they can not create or ``write`` to files in the directory, all ``other`` users do not have any access to the directory. -Changing Permissions +Changing permissions """""""""""""""""""" To change file permissions use the ``chmod`` command. @@ -547,7 +547,7 @@ To change the group of a file use the command ``chgrp ``. -bash-4.1$ -History, Command Line Editing and Job Control +History, command line editing and job control --------------------------------------------- History @@ -570,7 +570,7 @@ The history command lists the last commands you typed. -bash-4.1$ -Command Line Editing +Command line editing ^^^^^^^^^^^^^^^^^^^^ You can select past commands using the ``up`` and ``down`` arrow keys. You can edit the command line using the ``left`` and ``right`` arrow keys and any of the following commands: @@ -593,7 +593,7 @@ ESC-B move back one word =============== ===================================== -Job Control +Job control ^^^^^^^^^^^ Job control deals with managing your programs whilst they are running. Linux uses the name process for a running program. The ``ps`` command list all the processes you have running. @@ -670,16 +670,16 @@ The sleep command does nothing for the number of seconds specified in the argume In this example we put two jobs into the background. The ``fg`` command moves the last job placed in the background into the foreground. ``Ctrl-z`` *stops* (pauses, not kills) the job and returns to the command line. The ``bg`` command places the paused job in the background. ``fg`` can bring specific jobs to the foreground by specifying the job number. -Environment Variables and Shell Scripts +Environment variables and shell scripts --------------------------------------- -Environment Variables +Environment variables ^^^^^^^^^^^^^^^^^^^^^ In the Linux shell a variable is a named object that contains data and which can be used by programs and commands. Environment variables provides a simple way to share configuration settings between multiple applications and processes in Linux. For example the value of an environmental variable can be the default editor that should be used, which can then be used by command to invoke the correct editor when necessary. -Predefined Environment Variables +Predefined environment variables """""""""""""""""""""""""""""""" ========== ========================================================================= Variable Value @@ -710,7 +710,7 @@ To use an environment variable precede its name with a ``$`` character. We can d -bash-4.1$ -Shell Scripts +Shell scripts ^^^^^^^^^^^^^ Shell scripts are files which contain shell commands. You run the script by typing its filename. Things to note: @@ -749,10 +749,10 @@ Shell scripts are files which contain shell commands. You run the script by typi -bash-4.1$ -Input and Output Redirection, Pipes, and Filters +Input and output redirection, pipes, and filters ------------------------------------------------ -Input and Redirection +Input and redirection ^^^^^^^^^^^^^^^^^^^^^ We can change the behaviour of programs to redirect input from a file instead of the keyboard and write to a file instead of the screen. The ``>`` character is used to redirect output to a file and ``<`` to redirect input. @@ -915,7 +915,7 @@ The ``last`` command displays all users and the dates and times they have logged -bash-4.1$ -Useful Commands +Useful commands --------------- To find more information on any command below, type ``man `` which will open up the built-in manual page for that command. diff --git a/docs/source/conf.py b/docs/source/conf.py index 940505d..7ae3919 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -6,8 +6,8 @@ # -- Project information project = 'Viking Documentation' -copyright = '2023, University of York' -author = 'University of York' +copyright = '2023, The University of York' +author = 'The University of York' release = '0.1' version = '0.1.0' diff --git a/docs/source/getting_started/backing_up.rst b/docs/source/getting_started/backing_up.rst index b1116cd..75f8007 100644 --- a/docs/source/getting_started/backing_up.rst +++ b/docs/source/getting_started/backing_up.rst @@ -1,4 +1,4 @@ -Backing Up your Data +Backing up your data ==================== There are two main options depending on how frequently you need to access the data, the University **Filestore** or the **Vault**. @@ -13,7 +13,7 @@ The **Filestore** is more readily accessible, if you need more space please cont Please ensure you **regularly** back up your data, if there is a catastrophic failure all data on Viking could be lost! -The Vault +The vault ---------- Vault is a file archiving service. It can be used to store files that need to be kept, but are unlikely to be accessed again. The main use is for storing research data which needs to be kept for contractual or other reasons, but is probably never going to be used or looked at again. Please see the `Vault User Guide `_ for more details. diff --git a/docs/source/getting_started/code_of_conduct.rst b/docs/source/getting_started/code_of_conduct.rst index 32008c4..c19974c 100644 --- a/docs/source/getting_started/code_of_conduct.rst +++ b/docs/source/getting_started/code_of_conduct.rst @@ -1,4 +1,4 @@ -Code of Conduct +Code of conduct =============== .. FIXME: Needs suggestions @@ -6,7 +6,7 @@ Code of Conduct We hope everyone enjoys using Viking and all its powerful resources. As all of Viking's resources are shared amongst all users, it's up to all of us to do our best to ensure we are using them wisely and not negatively impacting other users. We understand that for some people this may be the first time they have access to a HPC like Viking, or even the first time using a Linux machine, so we have created this page to provide some guidelines for using Viking responsibly and considerately. As always, if you need help with any explanation please get in touch with the team by emailing itsupport@york.ac.uk. -Running Tasks on the Login Nodes +Running tasks on the login nodes -------------------------------- When you first log in to Viking you, and every other user, will be logged into one of the ``login nodes``. These act as gateways to the ``compute nodes``, which is where all the hard work should be done. The login nodes are meant for transferring files, writing code, viewing results and other similar **light work**. If you have a task which crunches through a massive data set, spinning off multiple instances and taking up as many CPUs as it can to get the job done, then running this task on a login node will negatively impact upon other people's ability to do their work. @@ -14,7 +14,7 @@ When you first log in to Viking you, and every other user, will be logged into o Please run all serious work through the ``slurm`` job scheduler, this way they are run on the compute nodes, not the login nodes and you can control their resources. You can read about ``Slurm`` and how to send jobs to it on the :doc:`scheduling jobs page `. -Closing Virtual Desktop Sessions +Closing virtual desktop sessions -------------------------------- :doc:`Virtual desktop sessions <../using_viking/virtual_desktops>` don't close if you simply disconnect, this is to allow you to come back to them at a later time. If you do not ``kill`` the virtual desktop after you have finished and later create new ones, more and more virtual desktops will be running taking up resources. This is why it's required to simply :ref:`kill virtual desktops when finished with `. @@ -26,7 +26,7 @@ Backing up data The ``scratch`` folder, a large area in your home directory where you can store data, is **not backed up**. This is partly due to the sheer size of the filesystem and means that backing up data is each user's responsibility. In a worst case scenario all data could be lost, therefore you should regularly :doc:`back up your data `. If you need any help with this, please get in touch with itsupport@york.ac.uk. -Deleting Unneeded Files +Deleting unneeded files ----------------------- We strongly encourage all users to take time to periodically sort through their data on Viking, back up the data and when it's successfully backed up, delete it from Viking if it's no longer needed. This frees up space for other people's data on the filesystem and helps avoid us reaching storage limits. diff --git a/docs/source/getting_started/connecting_off_campus.rst b/docs/source/getting_started/connecting_off_campus.rst index a972dbf..3375cb0 100644 --- a/docs/source/getting_started/connecting_off_campus.rst +++ b/docs/source/getting_started/connecting_off_campus.rst @@ -1,9 +1,9 @@ .. _connecting-off-campus: -Connecting to Viking off campus +Connecting to viking off campus =============================== -Viking has been configured to only allow connections from the university network. Therefore, in order to access Viking off-campus, you must first connect to either the VPN service, or create an ``ssh`` tunnel. These re-route your traffic through the university network allowing you to connect to Viking as if you were on campus. +Viking has been configured to only allow connections from the University network. Therefore, in order to access Viking off-campus, you must first connect to either the VPN service, or create an ``ssh`` tunnel. These re-route your traffic through the University network allowing you to connect to Viking as if you were on campus. Using the VPN ------------- @@ -11,16 +11,16 @@ Using the VPN Please see the main IT Services page on using the VPN found `here `_. -SSH Gateway +SSH gateway ----------- The University also provides an `SSH gateway service `_ that can be used to allow off-campus access to Viking, as an alternative to the VPN. To use this method, ``ssh`` to ``ssh.york.ac.uk`` (substituting your username for ``abc123``): .. code-block:: console - $ ssh abc123@viking.york.ac.uk + $ ssh abc123@ssh.york.ac.uk -Once you have entered your password and gone through the two-factor authentication, you should then see the following message asking which machine you wish to connect to. Simply enter ``viking`` and press ``Enter``. +Once you have entered your password and gone through the 2FA (two-factor authentication), you should then see the following message asking which machine you wish to connect to. Simply enter ``viking`` and press ``Enter``. .. code-block:: console @@ -41,4 +41,5 @@ To avoid manual entering the hostname when using the SSH gateway, it's also poss ssh -J abc123@ssh.york.ac.uk viking .. hint:: - You will still need to enter your password, and complete the 2FA as before. + + You will still need to enter your password, and then complete the 2FA. diff --git a/docs/source/getting_started/connecting_to_viking.rst b/docs/source/getting_started/connecting_to_viking.rst index aafe459..9f88ccf 100644 --- a/docs/source/getting_started/connecting_to_viking.rst +++ b/docs/source/getting_started/connecting_to_viking.rst @@ -1,4 +1,4 @@ -Connecting to Viking +Connecting to viking ==================== .. hint:: @@ -7,10 +7,10 @@ Connecting to Viking To access Viking you'll need to be on the campus network or using the VPN :doc:`which you can read about here. ` -Terminal Access +Terminal access --------------- -Linux and MacOS +Linux and macOS ^^^^^^^^^^^^^^^ To log in from a terminal emulator, use the following command: @@ -36,7 +36,7 @@ Windows For terminal access to Viking from a Windows desktop, you will need to install `PuTTY `_ (or comparable software). -Configuring PuTTY to Connect to Viking +Configuring PuTTY to connect to viking """"""""""""""""""""""""""""""""""""""" Open PuTTY and configure it to connect to Viking: @@ -48,7 +48,7 @@ Open PuTTY and configure it to connect to Viking: .. image:: ../assets/img/putty1.png -Connecting to Viking +Connecting to viking """"""""""""""""""""" 1. Start PuTTY @@ -57,7 +57,7 @@ Connecting to Viking .. image:: ../assets/img/putty2.png -A terminal window should appear. Log in with your university username and password. +A terminal window should appear. Log in with your University username and password. .. image:: ../assets/img/putty3.png diff --git a/docs/source/getting_started/creating_accounts.rst b/docs/source/getting_started/creating_accounts.rst index baad971..5d7b1cf 100644 --- a/docs/source/getting_started/creating_accounts.rst +++ b/docs/source/getting_started/creating_accounts.rst @@ -1,6 +1,6 @@ .. _creating-an-account: -Creating an Account +Creating an account =================== Getting access to Viking is a simple two-stage process: @@ -9,7 +9,7 @@ Getting access to Viking is a simple two-stage process: 2. Members can then use this ``project code`` to complete the `user application form `_. The PI will be notified each time a user account is assigned to the project. -Project Application +Project application ------------------- In order to use Viking, the project PI (or supervisor) must apply for a Viking project account. These will be used to monitor the usage of Viking, and attribute usage by researchers across the University. We ask that the PI who applies for the account is a permanent member of staff, as this will allow us to have a permanent contact for project reporting purposes. You may also nominate a deputy, who can deal with any project enquiries on your behalf. A PI may request more than one project code, such as in the situation when they have multiple projects. @@ -21,7 +21,7 @@ Once the application has been approved, a new Viking project code will be suppli `The Project Application Form can be found here `_. -User Account Application +User account application ------------------------ Each user on Viking must apply for an individual user account that will be associated with at least one project group. In order to complete the user account application form, **you must already have a valid project code**. If you are unsure what your project code is, you should contact your PI (or supervisor) who should be able to supply you with a suitable project code or request a new one by completing the project application form. diff --git a/docs/source/getting_started/data_management_and_user_quota.rst b/docs/source/getting_started/data_management_and_user_quota.rst index 162ee24..702078f 100644 --- a/docs/source/getting_started/data_management_and_user_quota.rst +++ b/docs/source/getting_started/data_management_and_user_quota.rst @@ -1,4 +1,4 @@ -Data Management and User Quota +Data management and user quota ============================== .. FIXME: This uses OLD information @@ -6,11 +6,11 @@ Data Management and User Quota Viking is a self-contained machine, therefore you will notice your normal UoY home directories are not available. This is intentional for the following reasons: - If the dedicated network link between campus and Viking goes down, it may cause slowness or jobs to fail. Instead, jobs should continue to run until the link is re-established -- If a user tries to read/write from the university filestores using Viking, it is possible that they could bring the entire storage system down for the university +- If a user tries to read/write from the University filestores using Viking, it is possible that they could bring the entire storage system down for the University - UoY home directories are not designed for high-performance computing. Instead Viking has its own filesystem designed for high-performance -Your Area on Viking Explained +Your area on viking explained ----------------------------- When you log in to Viking, you will land in your home directory, specifically: @@ -47,7 +47,7 @@ Additionally you also have access to a ``localtmp`` folder which points to stora -Checking Your Quota +Checking your quota ------------------- To check how much space you have left, run the following command: @@ -76,7 +76,7 @@ Which will produce something similar to the following result: 232K 100G 110G 57 200k 250k -What Happens If You Exceed Your Quota +What happens if you exceed your quota ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When you login to Viking you will be told if you are over quota. If this is in users area with the **100GB** or **200,000** files limit you will need to delete or move your files to your ``scratch`` area. **There is a grace period of 7 days after which you will lose access to Viking.** @@ -87,7 +87,7 @@ If you are over quota in the ``scratch`` area and need more space please email i The most common reason for exceeding your quota in your home directory is by storing ``conda`` environments directly in your home directory. The solution for that is to :ref:`set up your Conda environment first `. -Copying and Moving Your Data to Viking +Copying and moving your data to viking -------------------------------------- There are many ways you can copy data to and from Viking and so we will only go over some general examples here using popular programs as a basic guide. For quick reference here are the important details:: @@ -98,7 +98,7 @@ There are many ways you can copy data to and from Viking and so we will only go .. caution:: - If you are not connected to the campus network, please remember you must be connected to the `university VPN `_ first. + If you are not connected to the campus network, please remember you must be connected to the `University VPN `_ first. Windows @@ -185,12 +185,12 @@ For the username and password, enter your IT Services credentials. After entering these details and connecting to Viking, your Viking area will appear on the right. You will now be able to click and drag files similar to the file manager. More information can be found in the `FileZilla documentation `_. -Moving Data to Google Drive Directly from Viking +Moving data to google drive directly from viking ------------------------------------------------ We know a number of Viking users like to store data on Google Drive. It is possible to copy data directly from Viking to your Google Drive folder. Below we will provide instructions on how to set this up. -Setting up rclone on Viking +Setting up rclone on viking ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to use ``rclone``, you will need a ``client-id``, the steps for which can be found on the `rclone website `_. @@ -247,7 +247,7 @@ If you login into Google Drive you should see the files from ``directory_to_copy 2021/08/16 11:33:23 Fatal error: listing Team Drives failed: googleapi: Error 403: Insufficient Permission: Request had insufficient authentication scopes., insufficientPermissions -DropOff Service +Dropoff service --------------- The York `DropOff Service `_ is a web page that lets you easily and securely exchange files up to 128G with University staff and students or external people. Files are automatically deleted after 14 days and all files are transferred across the network `securely encrypted `_. diff --git a/docs/source/getting_started/quickstart.rst b/docs/source/getting_started/quickstart.rst index 071a9ef..4993985 100644 --- a/docs/source/getting_started/quickstart.rst +++ b/docs/source/getting_started/quickstart.rst @@ -2,10 +2,10 @@ Quickstart ========== .. attention:: - Please ensure you are on the campus network or :ref:`connected to the university VPN `. If you haven't already please :ref:`create an account `. + Please ensure you are on the campus network or :ref:`connected to the University VPN `. If you haven't already please :ref:`create an account `. -Log in to Viking +Log in to viking ---------------- .. code-block:: console @@ -18,7 +18,7 @@ Log in to Viking It's a little bit more involved than the one line above but we have :ref:`nice breakdown here `. -Find the Software You Need +Find the software you need -------------------------- .. code-block:: console @@ -29,7 +29,7 @@ Find the Software You Need .. FIXME: add example output -Load a Module +Load a module -------------- .. code-block:: console @@ -46,13 +46,13 @@ Load a Module To read more about the EasyBuild concept of *common toolchains*, please see the `EasyBuild docs `_. In it's simplest sense, think of it as the compiler version the software was build with. -Develop and Test +Develop and test ---------------- Develop and test the the job you plan to create. Remember not to leave a proper job running on the login node as this can affect other users. If you are testing something and need to kill the command whilst it's running, press ``Ctrl + c``. -Create Job Script +Create job script ----------------- In your favorite text editor, create a jobscript for your job. Save it as something like ``myjobscript.job``. @@ -99,7 +99,7 @@ In your favorite text editor, create a jobscript for your job. Save it as someth echo '\n'Job completed at `date` -Send the Jobscript to the Job Scheduler +Send the jobscript to the job scheduler --------------------------------------- .. code-block:: console @@ -107,13 +107,13 @@ Send the Jobscript to the Job Scheduler $ sbatch myjobscript.job -Check Results +Check results -------------- Depending on what you set for ``#SBATCH --mail-type=`` you should receive some emails as the job progresses. When the job is completed you should have a log file in the directory where you ran the ``sbatch`` command originally. This is a great opportunity to see how efficient your job was. -Adjust the Jobscript +Adjust the jobscript -------------------- If your ``CPU`` or ``memory`` utilisation is very low, it means your settings in the jobscript need adjusting if you are to run the job again. Now is a good time to adjust these down, you should aim to get the actual utilisation close to the requested values, this will mean that Viking can start more jobs quicker and everyone can get their results faster. That's teamwork! ❤️ diff --git a/docs/source/in_depth/program_specific_how_tos.rst b/docs/source/in_depth/program_specific_how_tos.rst index 84b8e60..78f31d4 100644 --- a/docs/source/in_depth/program_specific_how_tos.rst +++ b/docs/source/in_depth/program_specific_how_tos.rst @@ -1,10 +1,10 @@ -Program Specific How-Tos +Program specific how-tos ======================== Here we will share some guides on getting started with some specific applications. In many cases there are multiple ways to accomplish things so you can treat this as a starting point. If you found a good guide elsewhere which you successfully followed or if you would like your own guide added please feel free to `let us know `_. -Jupyter Notebooks +Jupyter notebooks ----------------- The `Jupyter notebook `_ is a web-based notebook environment for interactive computing. Running a ``Jupyter notebook`` server remotely on Viking and connecting to it from your local web browser is certainly possible. @@ -59,7 +59,7 @@ Then, back to the first terminal where the notebook is running, there should be ``Ctrl + left mouse click`` on this link and the browser should load and connect to the notebook running on the compute node! -Jupyter Notebooks using VSCode +Jupyter notebooks using VSCode ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Using some of the above guide as reference, another way to so this is with VSCode and it's similar, but you don't need to make a virtual desktop. If you're interested in this method it's similar to the above in many ways: diff --git a/docs/source/using_viking/apptainer.rst b/docs/source/using_viking/apptainer.rst index a2cc0bc..8b51594 100644 --- a/docs/source/using_viking/apptainer.rst +++ b/docs/source/using_viking/apptainer.rst @@ -10,7 +10,7 @@ To load ``{NAME_APPTAINER}`` on Viking run module load {MOD_APPTAINER} -Using the remote Builder (recommended) +Using the remote builder (recommended) -------------------------------------- You will need to `create an account `_ and then create a ``access token`` for use on Viking. Once your account is set up and you have logged in, click your username in the top right and select ``access tokens``. Give it a name and click ``Create Access Token``. Copy the token and then, on Viking run the following command and then paste the token: @@ -27,7 +27,7 @@ You will need to `create an account `_ and then .. FIXME: Is it worth including this first option? We seem to recommend remote building later on -There are two ways to remote build. +There are two ways to remote build. 1. You can upload or write the recipes on the main page in the web UI 2. Build remotely on Viking using the ``{NAME_APPTAINER} build`` command @@ -50,7 +50,7 @@ An example of the second approach is shown below, this creates a Hello World ima -Example Use Step by Step +Example use step by step ------------------------ Here is an example use of building and running a program through ``{NAME_APPTAINER}`` step-by-step. This example is to allow us to use the `DIA-NN `_ program. I searched `hub.docker.com `_ and found an image with ``dia-nn`` included, this is what we'll use. @@ -98,10 +98,10 @@ This should get you a ``shell`` session within the container, the program ``dian It's important to use the ``--cleanenv`` option when running this container to stop it passing the current environment variables to the container. -Installing Singularity on Your Local System +Installing singularity on your local system -------------------------------------------- -If you are running Linux and would like to install ``Singularity`` locally on your system, ``Singularity`` provide the free, open source `Singularity Community Edition `_. +If you are running Linux and would like to install ``Singularity`` locally on your system, ``Singularity`` provide the free, open source `Singularity Community Edition `_. If you would like to attempt a local install of ``Singularity``, you can find details in the `INSTALL.md `_ file within the ``Singularity`` repository that explains how to install the prerequisites (most notably ``Go``), build, and install the software. If you do not have access to a Linux system where you can build and install ``Singularity`` but you have administrative privileges on another system, you could look at installing a virtualisation tool such as `VirtualBox `_ on which you could run a Linux Virtual Machine (VM) image to install ``Singularity``. diff --git a/docs/source/using_viking/jobscript_examples.rst b/docs/source/using_viking/jobscript_examples.rst index ff46092..706e38c 100644 --- a/docs/source/using_viking/jobscript_examples.rst +++ b/docs/source/using_viking/jobscript_examples.rst @@ -1,4 +1,4 @@ -Jobscript Examples +Jobscript examples ================== @@ -11,7 +11,7 @@ In this section we'll try to give some general purpose examples of different job .. _jobscript_parallelisation: - + Different types of parallelisation ---------------------------------- @@ -31,7 +31,7 @@ For the purposes of simplicity, we'll group the types of parallel execution into - Job arrays -Single Process Jobs +Single process jobs ------------------- For software that does not support any parallelisation, or where single threaded operation is desired, the following example job script can serve as a template. @@ -64,7 +64,7 @@ For software that does not support any parallelisation, or where single threaded .. _threaded-multi-process-jobs: -Multi-Threaded +Multi-threaded -------------- The following job script requests 8 cores on the same node, which could be used in a multi-threaded application. @@ -130,7 +130,7 @@ This job script requests 40 processes that could be split amongst different node .. _jobscript_job_arrays: -Job Arrays +Job arrays ---------- Job arrays are efficient ways of running the same program multiple times. diff --git a/docs/source/using_viking/jobscripts_program_specific.rst b/docs/source/using_viking/jobscripts_program_specific.rst index c86ef0e..4b0a05e 100644 --- a/docs/source/using_viking/jobscripts_program_specific.rst +++ b/docs/source/using_viking/jobscripts_program_specific.rst @@ -1,6 +1,6 @@ .. include:: /global.rst -Jobscripts - Program Specific +Jobscripts - program specific ============================= This section of documentation goes over various programs installed on Viking which are more complicated than average to use. @@ -19,8 +19,8 @@ All of the example files shown on these pages can be downloaded or can be found In each section there may be an example ``module load`` command. Newer versions may be available so please try the command ``module spider NAME`` where 'NAME' is the software to search for, and you will be presented with the currently available list. -Alpha Fold ------------ +AlphaFold +---------- `AlphaFold `_ is an AI system developed by `DeepMind `_ that predicts a protein's 3D structure from its amino acid sequence. The source code for the inference pipeline can be found on the `AlphaFold GitHub `_ page. @@ -111,7 +111,7 @@ Example job scripts alphafold --fasta_paths=T1050.fasta --max_template_date=2020-05-14 --preset=full_dbs --output_dir=$PWD --model_names=model_1,model_2,model_3,model_4,model_5 -Notes for using AlphaFold on Viking +Notes for using AlphaFold on viking ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``AlphaFold`` currently requires access to various genetic databases such as ``UniRef90``, ``MGnify``, ``BFD``, ``Uniclust30``, ``PDB70`` and ``PDB``. @@ -334,7 +334,7 @@ MATLAB .. FIXME: New docs reference a jobscript emailed to Emma - can we get a copy? -Running Interactively +Running interactively ^^^^^^^^^^^^^^^^^^^^^ ``MATLAB`` can be run interactively both with and without a Graphical User Interface (GUI). When running ``MATLAB`` interactively, please ensure that you are doing so inside an :ref:`interactive cluster session `, rather than on :doc:`Viking's login nodes <../getting_started/code_of_conduct>`. @@ -367,7 +367,7 @@ To run ``MATLAB`` interactively with the graphical user interface, you must firs In your virtual desktop session, you should now see the ``MATLAB`` graphical interface which is running on a compute node. -Running in Batch mode +Running in batch mode ^^^^^^^^^^^^^^^^^^^^^ ``MATLAB`` (2019a and newer) can also be run in batch mode, i.e non-interactively. This model of execution fits nicely with HPC systems like Viking, where work can be submitted to the scheduler to be executed. @@ -401,7 +401,7 @@ The following job script could be used to submit a ``MATLAB`` script to the clus **Do not** include the ``.m`` extension, which is part of the ``matlab_batch_example.m`` filename, in the job script when calling ``matlab -batch`` command, as shown. -Standalone MATLAB Programs +Standalone MATLAB programs ^^^^^^^^^^^^^^^^^^^^^^^^^^ It is possible to create standalone ``MATLAB`` programs from your ``MATLAB`` projects, and these can be run on Viking. An advantage of doing this is that when running a standalone program, ``MATLAB`` does not check out a licence from the licence server, which means somebody else who has to run ``MATLAB`` interactively will be able to do so even if your ``MATLAB`` program is running! @@ -455,7 +455,7 @@ If you encounter the following error it is because the compiler has detected tha Certain ``MATLAB`` features are not available in standalone programs, so it is worth being aware of what these are to avoid trouble when running your program. You can find a list of ineligible features here, and comprehensive documentation of supported features `here `_. -Running Standalone Programs +Running standalone programs """"""""""""""""""""""""""" Standalone ``MATLAB`` programs require the ``MATLAB`` Compiler Runtime ``MCR`` to run. This requires the ``MATLAB`` module to be loaded either in your interactive session or in your job script. Make sure that the version you load is the same version that was used when you compiled the program. @@ -604,11 +604,11 @@ The following example shows how to run ``cmdstanr`` using 4 cores, one for each .. note:: - The crucial step in the above job script is setting ``--cpus-per-task=4``, to ensure that you request the same number of cores that you are using in your ``R`` script to parallelize over. + The crucial step in the above job script is setting ``--cpus-per-task=4``, to ensure that you request the same number of cores that you are using in your ``R`` script to parallelise over. .. attention:: - Always explicitly specify the number of cores in your ``R`` code when possible. This is because some ``R`` packages use ``parallel::detect_cores()`` to identify the number of cores on the system to parallelize over. However, this doesn't work on Viking as it returns the number of cores in total on the node, **not** the number of cores you have requested and can result in unexpected behaviour. + Always explicitly specify the number of cores in your ``R`` code when possible. This is because some ``R`` packages use ``parallel::detect_cores()`` to identify the number of cores on the system to parallelise over. However, this doesn't work on Viking as it returns the number of cores in total on the node, **not** the number of cores you have requested and can result in unexpected behaviour. Array jobs @@ -672,7 +672,7 @@ NB: if your ``R`` script also makes use of multi-core parallelisation then you c Rscript --vanilla jobarray.R $SLURM_ARRAY_TASK_ID -Converting A Serial For Loop To Array Job +Converting a serial for loop to array job ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ While array jobs are a very effective way of running trivially parallelisable code on Viking, they require a bit of modification to scripts that you have been running on your personal computer. Take the parameter sweep example from above, this might have started out life as a for loop when running on your computer, as in the example below. This would work well until it takes too long to run, either from increasing the number of iterations or from the model fitting taking longer, until you want to run it on Viking to free up your PC. @@ -720,7 +720,7 @@ This package takes as input: - A location to save a *registry* - A Slurm batch job template file (one provided below) -The registry is just a structured directory where ``batchtools`` saves its environment, which includes items such as the completed Slurm job script, serialized versions of the ``R`` code to run, and outputs from each iteration of the array. +The registry is just a structured directory where ``batchtools`` saves its environment, which includes items such as the completed Slurm job script, serialised versions of the ``R`` code to run, and outputs from each iteration of the array. The ``R`` script below shows how to use ``batchtools`` to convert the for-loop parameter sweep into an array job that runs on Viking. This script will need to be moved onto Viking and run - it can't automatically submit from your PC (yet... watch this space). @@ -791,7 +791,7 @@ The Slurm template that this references is shown below and should be general eno ## Job Resource Interface Definition ## ## ncpus [integer(1)]: Number of required cpus per task, - ## Set larger than 1 if you want to further parallelize + ## Set larger than 1 if you want to further parallelise ## with multicore/parallel within each task. ## walltime [integer(1)]: Walltime for this job, in seconds. ## Must be at least 1 minute. @@ -834,7 +834,7 @@ The Slurm template that this references is shown below and should be general eno #SBATCH --mail-user=<%= email_address %> <%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %> - ## Initialize work environment like + ## Initialise work environment like module add <%= modules %> ## Export value of DEBUGME environemnt var to slave diff --git a/docs/source/using_viking/resource_partitions.rst b/docs/source/using_viking/resource_partitions.rst index efef89e..9f1beea 100644 --- a/docs/source/using_viking/resource_partitions.rst +++ b/docs/source/using_viking/resource_partitions.rst @@ -11,7 +11,7 @@ THIS IS THE OLD PARTITION TABLE -Resource Partitions +Resource partitions ==================== Viking's resources are divided up into various partitions as layed out below, these may change over time as it becomes clear how best to share Viking's resources. @@ -26,7 +26,7 @@ Viking's resources are divided up into various partitions as layed out below, th :header-rows: 1 -Additional Partition Information +Additional partition information -------------------------------- nodes diff --git a/docs/source/using_viking/software_on_viking.rst b/docs/source/using_viking/software_on_viking.rst index df047f1..b79e859 100644 --- a/docs/source/using_viking/software_on_viking.rst +++ b/docs/source/using_viking/software_on_viking.rst @@ -1,4 +1,4 @@ -Software on Viking +Software on viking ================== .. admonition:: Did you know... @@ -7,7 +7,7 @@ Software on Viking We do our best to fulfill all software install requests quickly, however, if it's a small or simple program, it can sometimes be beneficial for a user to download and build the software themselves. 🏗️👷🦺 -Modules and the User Environment +Modules and the user environment -------------------------------- Most software installed on Viking are made available through the module system ``lmod``. This allows multiple versions of the same software to be installed without conflicting or interfering with each other. By loading modules, the user environment is automatically modified to set the appropriate variables (``$PATH`` etc.) to make the corresponding software visible. @@ -24,12 +24,12 @@ For a quick overview of the available ``module`` sub-commands and options, try t The command ``ml`` is a handy front end for the ``module`` command. On Viking, try typing ``ml --help`` to see how you can use it as shorthand for ``module load`` and other commands. We'll continue to use explicit commands like ``module load`` in this guide but know you can use ``ml`` a lot of the time instead. Try it out! 😎 -Using the Module Command +Using the module command ------------------------ Try out the commands below and see their output. Very quickly you will be able to find modules of familiar software, load and unload them. -Searching for Modules +Searching for modules ^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -37,7 +37,7 @@ Searching for Modules $ module spider Python -Loading a Module +Loading a module ^^^^^^^^^^^^^^^^ .. code-block:: console @@ -50,7 +50,7 @@ Loading a Module Always use the **full module name**, including version and toolchain. Being specific means that as new modules are added your work remains the same and reproducible. -Listing Modules in Use +Listing modules in use ^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -58,7 +58,7 @@ Listing Modules in Use $ module list -Unloading a Module +Unloading a module ^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -66,7 +66,7 @@ Unloading a Module $ module unload {MOD_PYTHON} -Purge All Modules +Purge all modules ^^^^^^^^^^^^^^^^^^ .. code-block:: console diff --git a/docs/source/using_viking/submitting_jobs.rst b/docs/source/using_viking/submitting_jobs.rst index 6391483..4254c98 100644 --- a/docs/source/using_viking/submitting_jobs.rst +++ b/docs/source/using_viking/submitting_jobs.rst @@ -1,12 +1,12 @@ -Submitting Jobs +Submitting jobs =============== -Batch Jobs +Batch jobs ---------- Viking can run lots of different jobs in many different ways, but for the most part you'll probably want to submit the job via a **jobscript**. A jobscript is just a text file with various options in it and then a list of the commands to run your desired program. -This is then submitted to the job scheduler called `Slurm `_ which manages all the jobs on Viking and does its best to ensure they all run as fairly and efficiently as possible. +This is then submitted to the job scheduler called `Slurm `_ which manages all the jobs on Viking and does its best to ensure they all run as fairly and efficiently as possible. A typical workflow involves preparing the jobscript for your needs and submitting the job to ``Slurm`` when ready via the ``sbatch`` command. However, this doesn't mean your program will start executing straight away! @@ -49,7 +49,7 @@ Below is an example jobscript to run a Python script, let's save it as ``jobscri Please ensure your project account code is specified through the ``--account`` option so that we can associate your work with your project. Your job won't run without it. It uses ``bash`` syntax and importantly has a set of ``SBATCH`` specific options **before** the commands which need to be run. -Here is a quick summary of the most important ``SBATCH`` options as these determine the computational resources you are requesting, which will impact both on your queueing time (if you request more resources you will experience longer waits), but also on your job performance, however, there are far more possible options than we can go into here. +Here is a quick summary of the most important ``SBATCH`` options as these determine the computational resources you are requesting, which will impact both on your queueing time (if you request more resources you will experience longer waits), but also on your job performance, however, there are far more possible options than we can go into here. The `Slurm documentation `_ is a great place to see them all, and please refer to the :doc:`jobscript for specific applications section ` for more advanced and specialised jobscript examples. - ``--partition``: The compute nodes on Viking are grouped into different *partitions* based on different use cases. For most general computing the ``nodes`` partition will be sufficient. See the :doc:`Resource Partition page ` for more details @@ -70,39 +70,39 @@ If you filled out the ``--mail-user`` option you will get an email when the job This command must be run from the same folder that the Python file ``my_script.py`` is located in, or more generally filepaths should be relative to the directory where ``sbatch`` is run from. -Tips and Best Practices +Tips and best practices ----------------------- -Resource Requests +Resource requests ^^^^^^^^^^^^^^^^^ Whilst you must avoid allocating fewer resources than required for your job to complete, please try to avoid significantly over-allocating resources. In addition to allowing more efficient utilisation of the cluster if job requests are reasonable, smaller jobs are likely to be scheduled quicker, thus improving your personal queue time. The same is true for wall-time; the scheduler assumes that the full duration will be used by the job, and so cannot backfill effectively if jobs are requesting significantly longer wall-times than they actually use. -Job Arrays +Job arrays ^^^^^^^^^^ -When submitting large volumes of jobs with identical resource requests, job arrays offer an efficient mechanism to manage these. +When submitting large volumes of jobs with identical resource requests, job arrays offer an efficient mechanism to manage these. See the :ref:`Job Arrays section ` or the `official Slurm documentation `_ for further details. -Bash Shebang and 'set -e' +Bash shebang and 'set -e' ^^^^^^^^^^^^^^^^^^^^^^^^^ Consider using ``set -e`` after the ``#SBATCH`` section. This has the effect of aborting the job if **any** command within the batch script fails, instead of potentially continuing with an environment that is different to what is expected, or with erroneous data. Furthermore, it ensures that the job displays as ``FAILED`` when querying the status of jobs with ``sacct``. In future versions of Viking this can be done in one line with ``#!/usr/bin/env -S bash -e``. This is the `shebang `_ we were referencing in the title. -Redirecting Output +Redirecting output ^^^^^^^^^^^^^^^^^^ The %j in the ``#SBATCH --output`` line tells ``Slurm`` to substitute the ``job ID`` in the name of the output file. You can also add a ``#SBATCH --error`` with an error file name to separate output and error logs. -Filename Patterns +Filename patterns ^^^^^^^^^^^^^^^^^ There are several useful placeholders similar to ``%j`` that can be used in filenames which will be automatically filled in by ``Slurm``. The full list of these can be found in the ``sbatch`` man page, under the ``filename pattern`` heading or in the `online documentation `_. -Interactive Jobs +Interactive jobs ---------------- If you would like to have a terminal session on a high powered compute node, for example to do interactive data analysis in programs like R, Python or MATLAB, or to run GUI applications, then you need an **interactive job**. @@ -117,7 +117,7 @@ The workflow is: 5. On exit, the allocated resources are automatically released -srun Command +Srun command ^^^^^^^^^^^^ Running the command below will request a Bash shell on a compute node in the interactive partition for 8 hours. diff --git a/docs/source/using_viking/terminal_multiplexing.rst b/docs/source/using_viking/terminal_multiplexing.rst index 8404e64..d3f22ee 100644 --- a/docs/source/using_viking/terminal_multiplexing.rst +++ b/docs/source/using_viking/terminal_multiplexing.rst @@ -1,4 +1,4 @@ -Terminal Multiplexing +Terminal multiplexing ===================== Terminal multiplexing allows you to ``detatch`` from a terminal session, leaving any commands running, and come back to it later - as if you never disconnected. You can ``detatch`` and ``attach`` multiple times, and once ``detatched`` you can even turn off your device, leaving the session running on Viking and come back at any time to pick up where you left off. diff --git a/docs/source/using_viking/virtual_desktops.rst b/docs/source/using_viking/virtual_desktops.rst index dc1024f..91cfc1b 100644 --- a/docs/source/using_viking/virtual_desktops.rst +++ b/docs/source/using_viking/virtual_desktops.rst @@ -1,4 +1,4 @@ -Virtual Desktops +Virtual desktops ================ You can create virtual desktop sessions to run graphical programs. There two main ways you can do this, on the login node and on a compute node. @@ -8,7 +8,7 @@ You can create virtual desktop sessions to run graphical programs. There two mai Remember, the login nodes are only for **light work** as mentioned in the :doc:`code of conduct <../getting_started/code_of_conduct>`, so if you need to use a GUI program for heavy work, then please ensure that is done on a compute node. -Login Node +Login node ---------- .. code-block:: console @@ -42,7 +42,7 @@ And you will be presented with information similar to mine below: If prompted, you should supply the following password: 5jaHqekY -Connect to the Virtual Desktop +Connect to the virtual desktop ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Using the appropriate application for your operating system (examples listed below), log into the virtual desktop. It will ask for a password (in my example this is ``5jaHqekY``), yours will be whatever was displayed in the previous step. @@ -78,7 +78,7 @@ MacOS .. _virtual_desktop: -Use the Virtual Desktop +Use the virtual desktop ^^^^^^^^^^^^^^^^^^^^^^^ You should soon be presented with a virtual desktop running on Viking. Click the ``Applications > System Tools > Terminal`` button to launch a terminal and you can load modules and programs as usual. @@ -90,7 +90,7 @@ You should soon be presented with a virtual desktop running on Viking. Click the it's a virtual desktop, on a remote machine! -List Sessions +List sessions ^^^^^^^^^^^^^ List all the current virtual desktops you have running with the following command: @@ -111,7 +111,7 @@ And you'll be presented with a list similar to mine below: .. _kill_sessions: -Kill Sessions +Kill sessions ^^^^^^^^^^^^^ To kill a session you need to use the ``Identity`` code from the output above, use it with the following command: @@ -129,7 +129,7 @@ Your ``Identity`` code will be different to mine, this is just an example. .. _virtual_session_compute_node: -Compute Node +Compute node ------------- The above method is great for light work like checking results but what if you want to do the heavy work with a GUI application? It's easy, when you get the the virtual desktop :ref:`like above `, then you ask for some resources on a compute node, this is exactly the same as using the ``srun`` command however we use a special wrapper called ``start-interactive-session.sh`` in the terminal in the virtual desktop, for example: diff --git a/docs/source/using_viking/virtual_environments.rst b/docs/source/using_viking/virtual_environments.rst index b30b4ff..7ba3915 100644 --- a/docs/source/using_viking/virtual_environments.rst +++ b/docs/source/using_viking/virtual_environments.rst @@ -1,10 +1,10 @@ -Virtual Environments +Virtual environments ==================== venv (Python) ------------- -For simple virtual environments, where you just need to install packages through ``pip``, ``venv`` can be a good choice. Here is a `really good article `_ about using ``venv`` in Python which is well worth the read. More info can be found in the `documentation `_. Below we'll keep a quickstart guide of the list of commands to get a ``venv`` virtual environment up and running. +For simple virtual environments, where you just need to install packages through ``pip``, ``venv`` can be a good choice. Here is a `really good article `_ about using ``venv`` in Python which is well worth the read. More info can be found in the `documentation `_. Below we'll keep a quickstart guide of the list of commands to get a ``venv`` virtual environment up and running. .. admonition:: Did you know? @@ -59,7 +59,7 @@ Viking provides the ``conda`` utility program, as part of the ``Miniconda`` modu .. _conda_setup: -Initial Setup +Initial setup ^^^^^^^^^^^^^ Before you get started using ``conda`` on Viking, there is a small amount of configuration that you can set up to make working with multiple environments much more straightforward. By default, ``conda`` will create environments and install packages into subdirectories of your ``HOME`` directory, namely: @@ -101,7 +101,7 @@ You will also need to load the Miniconda module, which will enable you to make u in a Viking shell. At this point, you are ready to use the ``conda`` utility with no risk of hitting the 100,000 files quota on your ``HOME`` directory. -Creating an Environment +Creating an environment ^^^^^^^^^^^^^^^^^^^^^^^ There are a few different ways in which environments can be created using the ``conda`` utility, but we are going to describe what is perhaps the most reliable and reproducible method - using an environment file. An environment file is a `YAML `_ file that describes the environment that you would like to create. This allows you to recreate the same environment in multiple places and easily pass on a specification to other users to reproduce your findings. A simple example environment file is shown below. @@ -151,7 +151,7 @@ Here you are telling ``conda`` to create a new environment using the file (``-f` At this point, the environment ``my_first_environment`` has been created, and is ready to be used. Note: the asterisk in the output of ``conda info --envs`` indicates *which* conda environment is currently activated. As you haven't yet activated your new environment, the ``base`` environment (the default environment over which you have no control) is activated. -Using an Environment +Using an environment ^^^^^^^^^^^^^^^^^^^^ Once an environment has been created, you can activate it using the ``source activate`` command. This can be seen clearly in the following example: