Merge pull request #63 from university-of-york/feature/newpages

Feature/newpages

README.md

@@ -15,11 +15,24 @@ The source markdown files are stored in `docs` and the content is deployed via G
 
 ## Development
 
-Clone the repo, install `sphinx` through `pip install sphinx sphinx-rtd-theme` and run `sphinx-build -b html docs/source site/` to compile the web pages and open `site/index.html` in a web browser.
+Clone the repo, install `sphinx` through
+
+`pip install sphinx sphinx-rtd-theme`
+
+### Compile the web pages
+
+`sphinx-build -b html docs/source site/`
+
+### View the website
+
+Open `site/index.html` in a web browser.
+
 There isn't a development server that auto-builds bundled with Sphinx.
 When your changes are ready to be deployed, submit a PR and request approval.
 
-Check links are valid with the following `sphinx-build -b linkcheck docs/source site/`
+### Check links are valid
+
+`sphinx-build -b linkcheck docs/source site/`
 
 ## Replacement
 

docs/source/beginners_guide/linux_shell.rst

@@ -470,7 +470,7 @@ In the above example the first column of the directory listing shows the permiss
 This is a ``directory``, the owner ``abs4`` can ``read``, ``write`` and ``access`` the directory. Members of the group, ``csrv``, can ``read`` and ``access`` the directory, they can not create or ``write`` to files in the directory, all ``other`` users do not have any access to the directory.
 
 
-Chaning Permissions
+Changing Permissions
 """"""""""""""""""""
 
 To change file permissions use the ``chmod`` command.

docs/source/getting_started/backing_up.rst

@@ -1,7 +1,11 @@
 Backing Up your Data
 ====================
 
-There are two main options, the University Filestore or the Vault. Choosing will depend on how frequently you need to access the data. The Vault provides unlimited cold storage and is appropriate for storing sensitive data but retrieving the data can take some time and may incur a charge, the Filestore is more readily accessible, if you need more space please contact [email protected]. If you use the Filestore for backing up then usually, the date will then be transferred to the Vault for archiving.
+There are two main options depending on how frequently you need to access the data, the University **Filestore** or the **Vault**.
+
+The **Vault** provides unlimited cold storage and is appropriate for storing sensitive data but retrieving the data can take some time and may incur a charge
+
+The **Filestore** is more readily accessible, if you need more space please contact [email protected]. If you use the Filestore for backing up, usually the data will be transferred to the Vault for archiving later.
 
 
 .. warning::

docs/source/getting_started/code_of_conduct.rst

@@ -24,7 +24,7 @@ Virtual desktop sessions don't close if you simply disconnect, this is to allow
 Backing up data
 ----------------
 
-The ``scratch`` folder, the main folder in your home directory where you store data, is not backed up. This is partyly due to the sheer size of the filesystem and so this means that backing up your data is every users responsibility. In the worst case scenario all data could be lost, therefore you should regularly `back up your data <getting_started/backing_up>`.
+The ``scratch`` folder, the main folder in your home directory where you store data, is not backed up. This is partyly due to the sheer size of the filesystem and so this means that backing up your data is every users responsibility. In the worst case scenario all data could be lost, therefore you should regularly :doc:`back up your data </getting_started/backing_up>`.
 
 
 Deleting Unneeded Files

docs/source/getting_started/data_management_and_user_quota.rst

@@ -1,6 +1,10 @@
 Data Management and User Quota
 ==============================
 
+.. attention::
+
+    FIXME: This uses OLD information
+
 Viking is a self-contained machine, therefore you will notice your normal UoY home directories are not available. This is intentional for the following reasons:
 
 - If the dedicated network link between campus and Viking goes down, it may cause slowness or jobs to fail. Instead, jobs should continue to run until the link is re-established
@@ -17,7 +21,7 @@ When you log in to Viking, you will land in your home directory, specifically:
 
     /users/abc123
 
-where ``abc123`` will be replaced by your username. This home directory has a size of **50GB** and a file limit of **100,000**. From within the home directory, you can access your ``scratch`` directory at
+Where ``abc123`` will be replaced by your username. This home directory has a size of **50GB** and a file limit of **100,000**. From within the home directory, you can access your ``scratch`` directory at
 
 .. code-block:: console
 
@@ -29,6 +33,12 @@ This is a special high-performance filestore with a default size of **3TB** and
     If you need more ``scratch`` storage space, please email us at [email protected], we are more than happy to accommodate large projects.
 
 
+Additionally, you also have access to the ``warm storage`` area, linked as ``FIXME: warm_storage`` which is **FIXME:??TB** and has no file limit. This is used for FIXME: What is it used for?
+
+.. code-block:: console
+
+    /users/abc123/FIXME: warm
+
 Checking Your Quota
 -------------------
 

docs/source/getting_started/quickstart.rst

@@ -31,7 +31,14 @@ Load a Module
 
 .. code-block:: console
 
-    $ load module lang/Python/3.10.8-GCCcore-12.2.0
+    $ module load lang/Python/3.10.8-GCCcore-12.2.0
+
+
+.. hint::
+
+    The module name scheme on Viking is as follows: ``category`` / ``program_name`` / ``version`` - ``toolchain`` - ``toolchain_version``, where sometimes ``toolchain`` and ``toolchain_version`` not listed.
+
+    To read more about the EasyBuild concept of *common toolchains*, please see the `EasyBuild docs <https://docs.easybuild.io/common-toolchains/>`_. In it's simplest sense, think of it as the compiler version the software was build with.
 
 
 Develop and Test
@@ -45,6 +52,11 @@ Create Job Script
 
 In your favorite text editor, create a jobscript for your job. Save it as something like ``myjobscript.job``.
 
+.. tip::
+
+    Please change the email address **[email protected]** below to your own email address and see the emails it generates!
+
+
 .. code-block:: bash
     :caption: this is just a basic template
     :linenos:
@@ -61,6 +73,7 @@ In your favorite text editor, create a jobscript for your job. Save it as someth
     #SBATCH [email protected]   # Where to send mail
     #SBATCH --output=%x-%j.log              # Standard output log
     #SBATCH --error=%x-%j.err               # Standard error log
+    #SBATCH --partition=test
 
     # Abort if any command fails
     set -e
@@ -99,3 +112,7 @@ Adjust the Jobscript
 --------------------
 
 If your ``CPU`` or ``memory`` utilisation is very low, it means your settings in the jobscript need adjusting if you are to run the job again. Now is a good time to adjust these down, you should aim them pretty close to the end results, this will mean that Viking can start more jobs quicker and everyone can get their results faster. That's teamwork! ❤️
+
+.. note::
+
+    In this example we set which ``partition`` we wanted the jobscript to use as ``test`` as we are just testing. For full jobs generally most users will be happy with the default ``nodes`` partition, you can see more about this on the :doc:`resource partitions </using_viking/resource_partitions>` page.

docs/source/in_depth/program_specific_how_tos.rst

@@ -69,7 +69,7 @@ Using some of the above guide as reference, another way to so this is with VSCod
     4. Once the interactive session is running, load the ``Jupyter`` module and run the notebook, like above
     5. In a **new** remote terminal on Viking, in VSCode, set up the ssh forwarding, like above (noting the ``node`` number from step 4.)
     6. In VSCode, open a new ``Jupyter`` notebook: ``(Ctrl+Shift+P)`` and type ``Jupyter: Create New Jupyter Notebook.``
-    7. In VSCode, press ``select kernel`` in the top right then select ``Exisiting Jupyter server``
+    7. In VSCode, press ``select kernel`` in the top right then select ``Existing Jupyter server``
     8. Paste in the URL of the notebook, just like above, follow the prompts in VSCode to name the notebook and select the available kernel
 
 

docs/source/using_viking/submitting_jobs.rst

@@ -48,7 +48,7 @@ When submitting large volumes of jobs with identical resource requests, job arra
 Bash Shebang and 'set -e'
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Consider using ``set -e`` after the ``#SBATCH`` section. This has the effect of aborting the job if **any** command within the batch script fails, instead of potentially continuing with an environment that is different to what is expected, or with erroneous data. Furthermore, it ensures that the job displays as ``FAILED`` when querying the status of jobs with ``sacct``. In future versions of Viking this can be done in one line with ``#!/usr/bin/env -S bash -e``.
+Consider using ``set -e`` after the ``#SBATCH`` section. This has the effect of aborting the job if **any** command within the batch script fails, instead of potentially continuing with an environment that is different to what is expected, or with erroneous data. Furthermore, it ensures that the job displays as ``FAILED`` when querying the status of jobs with ``sacct``. In future versions of Viking this can be done in one line with ``#!/usr/bin/env -S bash -e``.  This is the `shebang <https://en.wikipedia.org/wiki/Shebang_(Unix)>`_ we were referencing in the title.
 
 
 Batch Jobs
@@ -58,11 +58,15 @@ Viking can run lots of different jobs in many different ways, but for the most p
 
 Below is an example jobscript, let's save it as ``jobscript.job`` for this example:
 
+.. admonition:: Remember
+
+    To change the email address **[email protected]** below to your own email address to receive the emails!
+
+
 .. code-block:: bash
     :caption: jobscript.job
 
     {SHEBANG}
-
     #SBATCH --job-name=my_job               # Job name
     #SBATCH --ntasks=10                     # Number of MPI tasks to request
     #SBATCH --cpus-per-task=1               # Number of CPU cores per MPI task

docs/source/using_viking/terminal_multiplexing.rst

@@ -1,9 +1,9 @@
 Terminal Multiplexing
 =====================
 
-So you're remotely logging into Viking, loading modules and running jobs with the ``sbatch`` or ``srun`` commands, the results are flying in and everything is looking pretty good. But what if you want to monitor the status of something running on Viking, turn off and walk away from your laptop and come back later? Or what if you want to do this with multiple sessions? Well, programs like ``screen`` and ``tmux`` have got you covered.
+So you're remotely logging into Viking, loading modules and running jobs with the ``sbatch`` or ``srun`` commands, the results are flying in and everything is looking pretty good. But what if you want to monitor the status of something running on Viking, turn off and walk away from your laptop and come back later? Or, what if you want to do this with *multiple* sessions? Well, programs like ``screen`` and ``tmux`` have got you covered.
 
-There is a really good beginners guide to ``screen`` and ``tmux`` on `Xiangyu Li's medium page <https://medium.com/@yiskylee/gnu-screen-and-tmux-which-should-you-choose-de325d32fc2a>`_ and well worth the read if you're new to either program. I probably prefer ``tmux`` but ``screen`` is totally fine and on some systems it's the only one available so it's nice to have experience with both.
+Rather than copy and paste a beginners guide here, there is already a really good beginners guide to ``screen`` and ``tmux`` on `Xiangyu Li's medium page <https://medium.com/@yiskylee/gnu-screen-and-tmux-which-should-you-choose-de325d32fc2a>`_ and well worth the read if you're new to either program. I probably prefer ``tmux`` but ``screen`` is totally fine and on some Linux systems it's the only available option so it's nice to have experience with both.
 
 By logging into Viking, running ``tmux`` then running your programs you can ``detach`` from the session, log out of Viking and return at your leisure and the session and with it's history will be there, pretty cool huh? 😎
 

docs/source/using_viking/virtual_environments.rst

@@ -18,6 +18,31 @@ Quickstart
     $ python3 -m pip install package1 package2 package3
 
 
+First we create a new directory with ``mkdir new_project`` to house our project and run ``cd new_project`` to change into that directory. Next we create the ``venv`` virtual environment and tell it to store the files in a folder called ``.venv`` with the ``python3 -m venv .venv`` command. The directory called ``.venv`` can be name anything you like, it doesn't have to be ``.venv``.
+
+.. admonition:: Info
+
+    The full stop ``.`` at the beginning of the directory name makes it ``hidden``. This isn't necessary but can be handy to keep things organised. You can read more about `hidden files and directories here <https://en.wikipedia.org/wiki/Hidden_file_and_hidden_directory>`_.
+
+
+Next we run ``source .venv/bin/activate`` which reads and executes the commands from the file name ``.venv/bin/activate`` - this then activates our environment and you'll see ``(.venv)`` pop up at the beginning of your command prompt like this:
+
+.. code-block:: console
+
+    (.venv) [abc123@login1 [viking] new_project]$
+
+Now we're *inside* our virtual environment, we use Python to install the packages we need and we can start developing. We use ``python3 -m pip install`` command to install the packages, because the ``-m`` option means that particular ``python3`` is going to use *only it's own* ``pip`` to install the packages. This is important because on Viking there are multiple different versions of ``Python`` and ``pip`` installed (and you can even install versions yourself in your home directory) - so this ensures that you use the *correct* ``pip`` associated with the ``python3`` that you are using.
+
+Once all the required packages are installed you can start developing.
+
+.. tip::
+
+    To ``deactivate`` a ``venv`` environment, you simply type ``deactivate`` on the command line.
+
+
+If you need a more complex virtual environment with complex dependencies then ``conda`` is probably more suited to your needs but remember that the best tool is the one that suits *your* needs best and makes the development process smoother for you.
+
+
 Conda
 ------