-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
16 changed files
with
150 additions
and
133 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,38 +3,30 @@ Code of Conduct | |
|
||
.. FIXME: Needs suggestions | ||
We hope everyone enjoys using Viking and all it's powerful resources. As all of Viking's resources are shared amongst the users, it's up to all of us to do our best to ensure we are using them wisely and not negatively impacting other users. We understand that for some people this may be the first time they have access to a HPC like Viking, or even the first time using a Linux machine, so we have created this page to help explain some of the concepts on how to use Viking so we don't impact other users. As always, if you need help with any explanation please get in touch with the team by emailing [email protected]. | ||
We hope everyone enjoys using Viking and all its powerful resources. As all of Viking's resources are shared amongst all users, it's up to all of us to do our best to ensure we are using them wisely and not negatively impacting other users. We understand that for some people this may be the first time they have access to a HPC like Viking, or even the first time using a Linux machine, so we have created this page to provide some guidelines for using Viking responsibly and considerately. As always, if you need help with any explanation please get in touch with the team by emailing [email protected]. | ||
|
||
|
||
Running Tasks on the Login Nodes | ||
-------------------------------- | ||
|
||
When you first log in to Viking you, and every other user, will be logged into one of the ``login nodes``. These act as gateways to the ``compute nodes``, which is where all the hard work should be done. The login nodes are meant for transferring files, writing code, viewing results and other similar **light work**. If you have a task which crunches through a massisve data set, spinning off multiple instances and taking up as many CPUs as it can to get the job done, then running this task on a login node will impact other people. | ||
When you first log in to Viking you, and every other user, will be logged into one of the ``login nodes``. These act as gateways to the ``compute nodes``, which is where all the hard work should be done. The login nodes are meant for transferring files, writing code, viewing results and other similar **light work**. If you have a task which crunches through a massive data set, spinning off multiple instances and taking up as many CPUs as it can to get the job done, then running this task on a login node will negatively impact upon other people's ability to do their work. | ||
|
||
Please run all tasks through the ``slurm`` job scheduler, this way they are run on the compute nodes, not the login nodes and you can control their resources. You can read about ``Slurm`` and how to send jobs to it on the :doc:`scheduling jobs page </using_viking/submitting_jobs>`. | ||
Please run all serious work through the ``slurm`` job scheduler, this way they are run on the compute nodes, not the login nodes and you can control their resources. You can read about ``Slurm`` and how to send jobs to it on the :doc:`scheduling jobs page </using_viking/submitting_jobs>`. | ||
|
||
|
||
Closing Virtual Desktop Sessions | ||
-------------------------------- | ||
|
||
Virtual desktop sessions don't close if you simply disconnect, this is to allow you to come back to them at a later time. If you do not ``kill`` the virtual desktop after you have finished and later create new ones, more and more virtual desktops will be running taking up resources. This is why it's best practice to simply :ref:`kill any virtual desktops after use <kill_sessions>`. | ||
`Virtual desktop sessions <using_viking/virtual_desktops.html>`_ don't close if you simply disconnect, this is to allow you to come back to them at a later time. If you do not ``kill`` the virtual desktop after you have finished and later create new ones, more and more virtual desktops will be running taking up resources. This is why it's required to simply :ref:`kill virtual desktops when finished with <kill_sessions>`. | ||
|
||
|
||
Backing up data | ||
---------------- | ||
|
||
The ``scratch`` folder, a large area in your home directory where you can store data, is **not backed up**. This is partly due to the sheer size of the filesystem and so this means that backing up your data is each users responsibility. In a worst case scenario all data could be lost, therefore you should regularly :doc:`back up your data </getting_started/backing_up>`. If you need any help with this, please get in touch with [email protected]. | ||
The ``scratch`` folder, a large area in your home directory where you can store data, is **not backed up**. This is partly due to the sheer size of the filesystem and means that backing up data is each user's responsibility. In a worst case scenario all data could be lost, therefore you should regularly :doc:`back up your data </getting_started/backing_up>`. If you need any help with this, please get in touch with [email protected]. | ||
|
||
|
||
Deleting Unneeded Files | ||
----------------------- | ||
|
||
We strongly encourage all users to take time to periodically sort through their data on Viking, back up the data and when it's successfully backed up, delete it from Viking if it's no longer needed. This frees up space for other people's data on the filesystem and helps avoid us reaching the storage limits. | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
We strongly encourage all users to take time to periodically sort through their data on Viking, back up the data and when it's successfully backed up, delete it from Viking if it's no longer needed. This frees up space for other people's data on the filesystem and helps avoid us reaching storage limits. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,15 +19,19 @@ When you log in to Viking, you will land in your home directory, specifically: | |
/users/abc123 | ||
Where ``abc123`` will be replaced by your username. This home directory has a size of **50GB** and a file limit of **100,000**. From within the home directory, you can access your ``scratch`` directory at | ||
Where ``abc123`` will be replaced by your username. This home directory has a size of **50GB** and a file limit of **100,000**. From within the home directory, you can access your ``scratch`` directory, which is a special high-performance filestore with a default size of **3TB** and **no limit** on the number of files. | ||
|
||
.. code-block:: console | ||
/users/abc123/scratch | ||
.. FIXME: needs size | ||
This is a special high-performance filestore with a default size of **3TB** and **no limit** on the number of files. Because of this, you should run all your jobs on Viking from this directory rather than the home directory. | ||
.. attention:: | ||
|
||
You should run all your jobs on Viking from ``scratch`` as it is faster and has more relaxed file limits than the home directory. However, it is not backed up. | ||
|
||
|
||
.. tip:: | ||
If you need more ``scratch`` storage space, please email us at [email protected], we are more than happy to accommodate large projects. | ||
|
@@ -69,8 +73,8 @@ Which will produce the following result: | |
31384 52428800 78643200 318 100000 150000 | ||
What Happens If You Exceed Quota | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
What Happens If You Exceed Your Quota | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
When you login to Viking you will be told if you are over quota. If this is in users area with the **50GB** or **100,000** files limit you will need to delete or move your files to your ``scratch`` area. **There is a grace period of 7 days after which you will lose access to Viking.** | ||
|
||
|
@@ -187,6 +191,7 @@ Setting up rclone on Viking | |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
In order to use ``rclone``, you will need a ``client-id``, the steps for which can be found on the `rclone website <https://rclone.org/drive/#making-your-own-client-id>`_. | ||
You will also need to either have a local install of `rclone`, or to be doing the setup in a Virtual Desktop. | ||
|
||
Login to Viking and navigate to an area on your scratch folder, then load the ``rclone`` module. | ||
|
||
|
@@ -202,22 +207,20 @@ Next, for the first time using rclone, you will need to configure it using the f | |
``rclone`` will then ask you a number of questions, including asking for your ``client-id``. We recommend the following answers: | ||
|
||
1. ``n`` - new remote | ||
2. Give it a name - don't use spaces, makes it awkward (I used ``gdrive``. Remember this as you will need it later) | ||
3. ``16`` - Google Drive (NB: do not select "google cloud storage") | ||
4. ``Enter`` - Application Client Id (see step above to create an ID) | ||
5. ``Enter`` - Client Secret (see step above to create an ID) | ||
6. ``3`` - Scope | ||
1. Action: ``n`` - new remote | ||
2. Name: Choose something sensible without spaces (I used ``gdrive``) | ||
3. Storage: ``Google Drive`` (NB: do not select "Google Cloud Storage") | ||
4. Application Client Id: See step above to create an ID | ||
5. Client Secret: See step above to create an ID | ||
6. Scope: ``drive.file`` (or ``drive`` if you want to be able to download files from Google Drive onto Viking) | ||
7. ``Enter`` - ID of the root folder | ||
8. ``Enter`` - Service account credentials | ||
9. ``n`` - Edit advanced config | ||
10. ``n`` - Use auto config | ||
11. Copy and paste the URL provided into your browser | ||
12. Authorize app with click through | ||
13. Copy and paste code back into terminal | ||
14. ``n`` - Configure this as Team Drive | ||
15. ``y`` - This is okay | ||
16. ``q`` | ||
8. Service account credentials: ``Enter`` to leave empty | ||
9. Edit advanced config: ``n`` | ||
10. Web browser authenticate: ``n`` (or Yes if in a Virtual Desktop) | ||
11. Follow the steps to authenticate locally | ||
12. Configure this as a Shared Drive: ``n`` | ||
13. ``y`` - This is okay | ||
14. ``q`` | ||
|
||
You are now ready to transfer files from Viking to your Google Drive. This can be done with the following command | ||
|
||
|
@@ -234,7 +237,7 @@ You are now ready to transfer files from Viking to your Google Drive. This can | |
If you login into Google Drive you should see the files from ``directory_to_copy`` inside a folder named ``viking-data`` | ||
|
||
.. hint:: | ||
If you see the following error message please check you are not trying to sync to a **team drive** instead. If you are trying to sync to a team drive you will need to amend question 14 to ``y``. | ||
If you see the following error message please check you are not trying to sync to a **Shared Drive** (previously known as Team Drives) instead. If you are trying to sync to a team drive you will need to amend question 12 to ``y``. | ||
|
||
.. code-block:: console | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.