From 4a7ffa49527e02d721212e55c9d156049f48cd47 Mon Sep 17 00:00:00 2001 From: Steven Morse Date: Fri, 13 Dec 2024 11:58:51 -0500 Subject: [PATCH] adjusting intro pages --- _data/navigation.yml | 2 +- .../configuring-ssh.md | 2 +- .../{filezilla.md => moving-files.md} | 15 ++++-- .../using-sshfs.md | 52 +++++++++++++++++++ 4 files changed, 66 insertions(+), 5 deletions(-) rename logging-in-and-setting-up-your-hpc-account/{filezilla.md => moving-files.md} (80%) create mode 100644 logging-in-and-setting-up-your-hpc-account/using-sshfs.md diff --git a/_data/navigation.yml b/_data/navigation.yml index 9029c47..7616971 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -8,7 +8,7 @@ - title: 👋 Login & Basic Setup url: /hpc-gitbook/logging-in-and-setting-up-your-hpc-account/login-and-basic-setup.html - title: 📁 Uploading Files - url: /hpc-gitbook/logging-in-and-setting-up-your-hpc-account/filezilla.html + url: /hpc-gitbook/logging-in-and-setting-up-your-hpc-account/moving-files.html - title: 🔒 Using the Jump Server and Configuring SSH url: /hpc-gitbook/logging-in-and-setting-up-your-hpc-account/configuring-ssh.html diff --git a/logging-in-and-setting-up-your-hpc-account/configuring-ssh.md b/logging-in-and-setting-up-your-hpc-account/configuring-ssh.md index 824ba52..dd44dd4 100644 --- a/logging-in-and-setting-up-your-hpc-account/configuring-ssh.md +++ b/logging-in-and-setting-up-your-hpc-account/configuring-ssh.md @@ -5,7 +5,7 @@ Using the clusters from a personal laptop not on the WM network requires you to In this section, we'll summarize how to do this step, and then how to make our SSH lives easier by using SSH keys and setting up `~/.ssh/config`. -## Using `bastion` +## Using the Jump Server (bastion) WM isolates its clusters on an internal private network and controls outside access using a "bastion" host, which acts as a sort of gatekeeper between the outside world and the internal school network. (The term *bastion* originally referred to a critical part of a fortification.) diff --git a/logging-in-and-setting-up-your-hpc-account/filezilla.md b/logging-in-and-setting-up-your-hpc-account/moving-files.md similarity index 80% rename from logging-in-and-setting-up-your-hpc-account/filezilla.md rename to logging-in-and-setting-up-your-hpc-account/moving-files.md index 6ff8be6..69d2f4b 100644 --- a/logging-in-and-setting-up-your-hpc-account/filezilla.md +++ b/logging-in-and-setting-up-your-hpc-account/moving-files.md @@ -2,11 +2,20 @@ ## How to get your files on the HPC -Just like SSH provides us a secure *shell* for working with the cluster, we also need a way to secure *copy* or move files. +Just like SSH provides us a secure *shell* for working with the cluster, we also need a way to secure *copy* or move files from one machine to another (for example, your local computer to the cluster). We'll look at three approaches: + +- **Using command line tools like `scp` or `sftp`.** This is probably the most reliable method and the best choice for large files, but is strictly for file transfer and can be cumbersome for complicated work. + +- **Using a GUI like Filezilla.** This is essentially a wrapper of `sftp`, and makes file transfer a little more user-friendly, but is still strictly for file transfer and probably not a good choice for large files. + +- **Mounting a drive using `sshfs`.** This is the most flexible option, since it allows moving and transferring files with your local, OS-native file explorer, *and* opening/editing files as if they were on your local machine. Still not a great choice for moving large files (due to the risk of mid-transfer disconnect or syncing issues) but a good option for editing scripts --- we cover mounting a drive in a [separate post here](https://d8a-science.github.io/hpc-gitbook/logging-in-and-setting-up-your-hpc-account/using-sshfs.html). + + +## Using the command line The most straightforward and out-of-the-box way is using the command line tool [`scp`](https://haydenjames.io/linux-securely-copy-files-using-scp/). Let's say you're on your local machine and want to move a file in your current working directory to the subfolder `test` in your home directory on the cluster. You can do (for example): -``` +```bash scp myFile.txt @bora.sciclone.wm.edu:test/ ``` @@ -25,7 +34,7 @@ While not appropriate for large files (say, >100Gb), there are also nice GUI opt FileZilla is predominantly a GUI-based way to access FTP and SFTP sites, and as such has a simple graphical installer you can download for any platform here: [https://filezilla-project.org/](https://filezilla-project.org/) -## Basic Use +### Basic Use diff --git a/logging-in-and-setting-up-your-hpc-account/using-sshfs.md b/logging-in-and-setting-up-your-hpc-account/using-sshfs.md new file mode 100644 index 0000000..d664cd1 --- /dev/null +++ b/logging-in-and-setting-up-your-hpc-account/using-sshfs.md @@ -0,0 +1,52 @@ +# Mounting a remote drive locally + +You will likely find it convenient to "mount" a remote drive (like your `/sciclone/home/` directory) on your local, personal computer. This means you add the directory to your local file system in a way that looks just like a local directory to your computer and all the apps on it. Once you do this, you can open your remote folder in your Files/File Explorer/Finder app just like any other folder, move files around, etc -- you can even open scripts on the remote machine in a coding IDE on your local computer, without any plugins/SSH necessary! + +Let's learn how to do this. It is easy on Linux, and fairly easy on Mac and Windows. Instructions for Windows are not included here (if you have done it, send a PR to this repo and we'll add it!) + + +## Mounting on Linux + +For Linux, we'll use the [`sshfs` utility](https://en.wikipedia.org/wiki/SSHFS) ("SSH Filesystem"). First, make sure it's installed: + +```bash +sudo apt update +sudo apt install sshfs +``` + +Then create a directory where the remote directory will be mounted, for example: + +```bash +cd ~ +mkdir mounts +cd mounts +mkdir sciclone +``` + +Now, use `sshfs` to mount the remote directory. Here we'll mount our home directory on SciClone (this uses the shorthand `bora` because we're assuming you've already setup your [SSH config file](https://d8a-science.github.io/hpc-gitbook/logging-in-and-setting-up-your-hpc-account/configuring-ssh.html), if not you'll need to use the full `@bora.sciclone.wm.edu`): + +```bash +sshfs bora:/sciclone/home/ ~/mounts/sciclone +``` + +If successful, it will return silently (no output) and you can try: + +```bash +cd ~/mounts/sciclone +``` + +You should see your folders. Try making a file with `touch test.txt`. Now SSH directly into the remote server and you'll see the file is there. Yay! + +## Mounting at startup + +If you restart your computer, it will disconnect this mount and you'll still see your `~/mounts/sciclone` directory but it'll be empty. Instead of manually reconnecting on startup every time, there are several ways to do this automatically on startup, that we will not cover in detail here but you can look up: + +- Add a line to your `/etc/fstab` file ("filesystems table") which tells Linux how to mount drives on startup. +- Create a bash script `mount.sh` with the sshfs command and then configure Linux to run this script on startup ("Startup Applications") +- Use `systemd` to manage the mount (this is the most robust solution) but also most involved. + +And another option is to just rarely restart your computer and + +## Technical notes + +- \ No newline at end of file