Skip to content

Commit

Permalink
Fixed typos and minor text tweaks
Browse files Browse the repository at this point in the history
  • Loading branch information
stulacy committed Oct 24, 2023
1 parent bf04286 commit 084339f
Show file tree
Hide file tree
Showing 16 changed files with 150 additions and 133 deletions.
6 changes: 3 additions & 3 deletions docs/source/beginners_guide/linux_shell.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Beginners Guide to the Linux Shell
==================================
Linux Shell
===========

Intro
-----
Expand Down Expand Up @@ -38,7 +38,7 @@ The last line is a ``command prompt`` and it is the means by which the computer

:command:`command [ options ] [ arguments ]`

Options to a command are usually proceeded by a ``-`` or ``--``, this differentiates them from the arguments. The following example shows the ``echo`` command which prints the arguments and the ``ls`` command which take options arguments. The ``ls`` command display the users file. There will be more explanation of files and the ``ls`` command later.
Options to a command are usually proceeded by a ``-`` or ``--`` and are optional, this differentiates them from the arguments which are mandatory. The following example shows the ``echo`` command which prints the arguments, and the ``ls`` command which displays the users files. There will be more explanation of files and the ``ls`` command later.

.. code-block:: console
:caption: example of command execution
Expand Down
1 change: 1 addition & 0 deletions docs/source/getting_started/backing_up.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Vault is a file archiving service. It can be used to store files that need to b
Filestore
---------

Please see the University's `Filestore guide <https://www.york.ac.uk/it-services/filestore/rented/>`_ for more information about accessing and using this service.
To move your data from Viking and to the Filestore ("Storage"), the address you need to use as an example is::

sftp.york.ac.uk:/shared/storage/<filestore>
Expand Down
20 changes: 6 additions & 14 deletions docs/source/getting_started/code_of_conduct.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,38 +3,30 @@ Code of Conduct

.. FIXME: Needs suggestions
We hope everyone enjoys using Viking and all it's powerful resources. As all of Viking's resources are shared amongst the users, it's up to all of us to do our best to ensure we are using them wisely and not negatively impacting other users. We understand that for some people this may be the first time they have access to a HPC like Viking, or even the first time using a Linux machine, so we have created this page to help explain some of the concepts on how to use Viking so we don't impact other users. As always, if you need help with any explanation please get in touch with the team by emailing [email protected].
We hope everyone enjoys using Viking and all its powerful resources. As all of Viking's resources are shared amongst all users, it's up to all of us to do our best to ensure we are using them wisely and not negatively impacting other users. We understand that for some people this may be the first time they have access to a HPC like Viking, or even the first time using a Linux machine, so we have created this page to provide some guidelines for using Viking responsibly and considerately. As always, if you need help with any explanation please get in touch with the team by emailing [email protected].


Running Tasks on the Login Nodes
--------------------------------

When you first log in to Viking you, and every other user, will be logged into one of the ``login nodes``. These act as gateways to the ``compute nodes``, which is where all the hard work should be done. The login nodes are meant for transferring files, writing code, viewing results and other similar **light work**. If you have a task which crunches through a massisve data set, spinning off multiple instances and taking up as many CPUs as it can to get the job done, then running this task on a login node will impact other people.
When you first log in to Viking you, and every other user, will be logged into one of the ``login nodes``. These act as gateways to the ``compute nodes``, which is where all the hard work should be done. The login nodes are meant for transferring files, writing code, viewing results and other similar **light work**. If you have a task which crunches through a massive data set, spinning off multiple instances and taking up as many CPUs as it can to get the job done, then running this task on a login node will negatively impact upon other people's ability to do their work.

Please run all tasks through the ``slurm`` job scheduler, this way they are run on the compute nodes, not the login nodes and you can control their resources. You can read about ``Slurm`` and how to send jobs to it on the :doc:`scheduling jobs page </using_viking/submitting_jobs>`.
Please run all serious work through the ``slurm`` job scheduler, this way they are run on the compute nodes, not the login nodes and you can control their resources. You can read about ``Slurm`` and how to send jobs to it on the :doc:`scheduling jobs page </using_viking/submitting_jobs>`.


Closing Virtual Desktop Sessions
--------------------------------

Virtual desktop sessions don't close if you simply disconnect, this is to allow you to come back to them at a later time. If you do not ``kill`` the virtual desktop after you have finished and later create new ones, more and more virtual desktops will be running taking up resources. This is why it's best practice to simply :ref:`kill any virtual desktops after use <kill_sessions>`.
`Virtual desktop sessions <using_viking/virtual_desktops.html>`_ don't close if you simply disconnect, this is to allow you to come back to them at a later time. If you do not ``kill`` the virtual desktop after you have finished and later create new ones, more and more virtual desktops will be running taking up resources. This is why it's required to simply :ref:`kill virtual desktops when finished with <kill_sessions>`.


Backing up data
----------------

The ``scratch`` folder, a large area in your home directory where you can store data, is **not backed up**. This is partly due to the sheer size of the filesystem and so this means that backing up your data is each users responsibility. In a worst case scenario all data could be lost, therefore you should regularly :doc:`back up your data </getting_started/backing_up>`. If you need any help with this, please get in touch with [email protected].
The ``scratch`` folder, a large area in your home directory where you can store data, is **not backed up**. This is partly due to the sheer size of the filesystem and means that backing up data is each user's responsibility. In a worst case scenario all data could be lost, therefore you should regularly :doc:`back up your data </getting_started/backing_up>`. If you need any help with this, please get in touch with [email protected].


Deleting Unneeded Files
-----------------------

We strongly encourage all users to take time to periodically sort through their data on Viking, back up the data and when it's successfully backed up, delete it from Viking if it's no longer needed. This frees up space for other people's data on the filesystem and helps avoid us reaching the storage limits.








We strongly encourage all users to take time to periodically sort through their data on Viking, back up the data and when it's successfully backed up, delete it from Viking if it's no longer needed. This frees up space for other people's data on the filesystem and helps avoid us reaching storage limits.
2 changes: 1 addition & 1 deletion docs/source/getting_started/connecting_off_campus.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Once you have entered your password and gone through the two-factor authenticati
Jump hosting via the SSH gateway
--------------------------------

To avoid manual entry of the hostname to connect to, it's also possible to connect by "jump hosting" using the SSH gateway:
To avoid manual entering the hostname when using the SSH gateway, it's also possible to directly connect to Viking by "jump hosting":

.. code-block:: console
Expand Down
6 changes: 3 additions & 3 deletions docs/source/getting_started/creating_accounts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,18 @@ Getting access to Viking is a simple two-stage process:

Project Application
-------------------
In order to use Viking, the project PI (or supervisor) must apply for a Viking project account. These will be used to monitor the usage of Viking, and attribute its use by researchers across the University. We ask that the PI who applies for the account is a permanent member of staff, as this will allow us to have a permanent contact for project reporting purposes. You may also nominate a deputy, who can deal with any project enquiries on your behalf. A PI may request more than one project code, for multiple projects.
In order to use Viking, the project PI (or supervisor) must apply for a Viking project account. These will be used to monitor the usage of Viking, and attribute usage by researchers across the University. We ask that the PI who applies for the account is a permanent member of staff, as this will allow us to have a permanent contact for project reporting purposes. You may also nominate a deputy, who can deal with any project enquiries on your behalf. A PI may request more than one project code, such as in the situation when they have multiple projects.

.. attention::

Only the project PI should submit project application forms.

Once the application has been approved, a new Viking project code will be supplied to you. You should then distribute the project code to all research team members, before they apply for user accounts.
Once the application has been approved, a new Viking project code will be supplied to you. You should then distribute the project code to all research team members before they apply for user accounts.

`The Project Application Form can be found here <https://docs.google.com/forms/d/e/1FAIpQLSdyldpc8nBtbsFaY5BDLHo9542-A1Y7segJ7d4Fwjl2yj7mxw/viewform>`_.

User Account Application
------------------------
Each user on Viking must apply for an individual user account and all user accounts are associated with at least one project group. In order to complete the user account application form, **you must have a valid project code**. If you are unsure what your project code is, you should contact your PI (or supervisor) who should be able to supply you with a suitable project code or request a new one by completing the project application form.
Each user on Viking must apply for an individual user account that will be associated with at least one project group. In order to complete the user account application form, **you must already have a valid project code**. If you are unsure what your project code is, you should contact your PI (or supervisor) who should be able to supply you with a suitable project code or request a new one by completing the project application form.

`The user account application form can be found here <https://docs.google.com/forms/d/e/1FAIpQLSfXkL10ypU6EQCBB2jS5oDwTpRMo77ppl7dvdbLnXm5zrKR7Q/viewform>`_.
43 changes: 23 additions & 20 deletions docs/source/getting_started/data_management_and_user_quota.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,19 @@ When you log in to Viking, you will land in your home directory, specifically:
/users/abc123
Where ``abc123`` will be replaced by your username. This home directory has a size of **50GB** and a file limit of **100,000**. From within the home directory, you can access your ``scratch`` directory at
Where ``abc123`` will be replaced by your username. This home directory has a size of **50GB** and a file limit of **100,000**. From within the home directory, you can access your ``scratch`` directory, which is a special high-performance filestore with a default size of **3TB** and **no limit** on the number of files.

.. code-block:: console
/users/abc123/scratch
.. FIXME: needs size
This is a special high-performance filestore with a default size of **3TB** and **no limit** on the number of files. Because of this, you should run all your jobs on Viking from this directory rather than the home directory.
.. attention::

You should run all your jobs on Viking from ``scratch`` as it is faster and has more relaxed file limits than the home directory. However, it is not backed up.


.. tip::
If you need more ``scratch`` storage space, please email us at [email protected], we are more than happy to accommodate large projects.
Expand Down Expand Up @@ -69,8 +73,8 @@ Which will produce the following result:
31384 52428800 78643200 318 100000 150000
What Happens If You Exceed Quota
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
What Happens If You Exceed Your Quota
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When you login to Viking you will be told if you are over quota. If this is in users area with the **50GB** or **100,000** files limit you will need to delete or move your files to your ``scratch`` area. **There is a grace period of 7 days after which you will lose access to Viking.**

Expand Down Expand Up @@ -187,6 +191,7 @@ Setting up rclone on Viking
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In order to use ``rclone``, you will need a ``client-id``, the steps for which can be found on the `rclone website <https://rclone.org/drive/#making-your-own-client-id>`_.
You will also need to either have a local install of `rclone`, or to be doing the setup in a Virtual Desktop.

Login to Viking and navigate to an area on your scratch folder, then load the ``rclone`` module.

Expand All @@ -202,22 +207,20 @@ Next, for the first time using rclone, you will need to configure it using the f
``rclone`` will then ask you a number of questions, including asking for your ``client-id``. We recommend the following answers:

1. ``n`` - new remote
2. Give it a name - don't use spaces, makes it awkward (I used ``gdrive``. Remember this as you will need it later)
3. ``16`` - Google Drive (NB: do not select "google cloud storage")
4. ``Enter`` - Application Client Id (see step above to create an ID)
5. ``Enter`` - Client Secret (see step above to create an ID)
6. ``3`` - Scope
1. Action: ``n`` - new remote
2. Name: Choose something sensible without spaces (I used ``gdrive``)
3. Storage: ``Google Drive`` (NB: do not select "Google Cloud Storage")
4. Application Client Id: See step above to create an ID
5. Client Secret: See step above to create an ID
6. Scope: ``drive.file`` (or ``drive`` if you want to be able to download files from Google Drive onto Viking)
7. ``Enter`` - ID of the root folder
8. ``Enter`` - Service account credentials
9. ``n`` - Edit advanced config
10. ``n`` - Use auto config
11. Copy and paste the URL provided into your browser
12. Authorize app with click through
13. Copy and paste code back into terminal
14. ``n`` - Configure this as Team Drive
15. ``y`` - This is okay
16. ``q``
8. Service account credentials: ``Enter`` to leave empty
9. Edit advanced config: ``n``
10. Web browser authenticate: ``n`` (or Yes if in a Virtual Desktop)
11. Follow the steps to authenticate locally
12. Configure this as a Shared Drive: ``n``
13. ``y`` - This is okay
14. ``q``

You are now ready to transfer files from Viking to your Google Drive. This can be done with the following command

Expand All @@ -234,7 +237,7 @@ You are now ready to transfer files from Viking to your Google Drive. This can
If you login into Google Drive you should see the files from ``directory_to_copy`` inside a folder named ``viking-data``

.. hint::
If you see the following error message please check you are not trying to sync to a **team drive** instead. If you are trying to sync to a team drive you will need to amend question 14 to ``y``.
If you see the following error message please check you are not trying to sync to a **Shared Drive** (previously known as Team Drives) instead. If you are trying to sync to a team drive you will need to amend question 12 to ``y``.

.. code-block:: console
Expand Down
4 changes: 2 additions & 2 deletions docs/source/getting_started/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,8 @@ Depending on what you set for ``#SBATCH --mail-type=`` you should receive some e
Adjust the Jobscript
--------------------

If your ``CPU`` or ``memory`` utilisation is very low, it means your settings in the jobscript need adjusting if you are to run the job again. Now is a good time to adjust these down, you should aim them pretty close to the end results, this will mean that Viking can start more jobs quicker and everyone can get their results faster. That's teamwork! ❤️
If your ``CPU`` or ``memory`` utilisation is very low, it means your settings in the jobscript need adjusting if you are to run the job again. Now is a good time to adjust these down, you should aim to get the actual utilisation close to the requested values, this will mean that Viking can start more jobs quicker and everyone can get their results faster. That's teamwork! ❤️

.. note::

In this example we set which ``partition`` we wanted the jobscript to use as ``test`` as we are just testing. For full jobs generally most users will be happy with the default ``nodes`` partition, you can see more about this on the :doc:`resource partitions </using_viking/resource_partitions>` page.
In this example jobscript we specified the jobs to run on the ``test`` ``partition`` as we are just testing. For full jobs generally most users will be want to use the default ``nodes`` partition, you can see more about this on the :doc:`resource partitions </using_viking/resource_partitions>` page.
Loading

0 comments on commit 084339f

Please sign in to comment.