Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix/quickfixes #181

Merged
merged 2 commits into from
Jan 31, 2024
Merged

Bugfix/quickfixes #181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b37ca10
update rsync commands, add tip for tmux on data transferrring
nd996 Jan 31, 2024
4998d08
add another tip for tmux on data transferrring
nd996 Jan 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions docs/source/data_management/filestore.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ To move your data from Viking to the filestore ("Storage"), the address you need

You can use ``PuTTY`` on Windows and the command line on Linux and MacOS with commands such as ``rsync`` and ``scp`` much like the examples on the :ref:`Data Management page <transfer_files_linux>`, examples below.

.. tip::

When transferring large amounts of data it can be a good idea to use :doc:`terminal multiplexing </using_viking/terminal_multiplexing>` such as ``tmux``. This would allow you to leave the transfer running (inside a ``tmux`` session) and not have to stay logged in yourself.


Using the ``scp`` command
^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down Expand Up @@ -57,9 +62,9 @@ Replace ``<filename>`` with the name of your file and ``<filestore>`` with the p

.. code-block:: console

$ rsync -P --append <filename> sftp.york.ac.uk:/shared/storage/<filestore>
$ rsync -av <filename> sftp.york.ac.uk:/shared/storage/<filestore>

Option ``-P`` sets the ``--partial`` and ``--progress`` options, which will *keep partially transferred files* and shows *progress*. The ``--append`` option will *append data onto shorter files*. Together this is handy if a large file needs to be transferred but for some reason could get interrupted. Rerunning this command should pick up where it left off.
Options ``a`` set up a number of options useful for *archiving*, ``v`` for *verbose* so you can monitor the process.

To copy a directory to your filestore
"""""""""""""""""""""""""""""""""""""
Expand Down Expand Up @@ -121,7 +126,7 @@ Replace ``<filestore>`` with the path to your filestore and ``<filename>`` with

.. code-block:: console

$ rsync -P --append sftp.york.ac.uk:/shared/storage/<filestore>/<filename> .
$ rsync -av sftp.york.ac.uk:/shared/storage/<filestore>/<filename> .


To copy a directory from your filestore
Expand Down
7 changes: 5 additions & 2 deletions docs/source/data_management/other_locations.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ You can copy your data from any Linux device to Viking using the following comma

For example, you can run the following commands from a terminal running on your device to move files to Viking.

.. tip::

When transferring large amounts of data it can be a good idea to use :doc:`terminal multiplexing </using_viking/terminal_multiplexing>` such as ``tmux``. This would allow you to leave the transfer running (inside a ``tmux`` session) and not have to stay logged in yourself.

scp
"""
Expand Down Expand Up @@ -74,9 +77,9 @@ This is recommended for a large number of files. Rsync can check what is already
$ rsync -av dirname viking.york.ac.uk:~/scratch

.. code-block:: console
:caption: this can be useful for copying a very large file from your device to your scratch area on Viking as it will allow you continue the transfer if the connection breaks for some reason
:caption: this could be useful for copying a very large file from your device to your scratch area on Viking as it will allow you continue the transfer if the connection breaks for some reason

$ rsync -P --append filename viking.york.ac.uk:~/scratch
$ rsync -avP --append filename viking.york.ac.uk:~/scratch

There are many more options you can use with ``rsync``. To view these options run the following command to view the ``rsync`` manual

Expand Down
Loading