Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document useful way of editing files locally with automatic sync on Jean Zay : (itnotify + rsync), editor extensions, and sshfs #25

Open
lesteve opened this issue Mar 20, 2020 · 2 comments
Labels
help wanted Extra attention is needed

Comments

@lesteve
Copy link
Member

lesteve commented Mar 20, 2020

One usual way people work is to edit their file locally on their computer (they have all the tools they are used to, editor, etc ...) and synchronize on Jean Zay automatically (adding an explicit step like rsync or git push/pull is painful because you will forget it sometimes and when you do it will cost you a lot of time and frustration).

To synchronize a code folder (local -> Jean Zay)

inotify + rsync

The simplest and most generic thing we have found for automatic local -> Jean Zay sync is to use a script like this on your local computer (needs inotify and rsync):

while inotifywait -r -e modify,create,delete ~/folder-to-sync
do
     rsync -avz ~/folder-to-sync jean-zay:~/folder-to-sync --delete
done

If you are on a OSX you need to use fswatch rather than inotifywait, according to this the following command should work (not tested, comment if this issue if it does not work for you!).

fswatch -o ~/path/to/watch | xargs -n1 -I{} \
    rsync -avz ~/folder-to-sync jean-zay:~/folder-to-sync --delete

Editor extensions

Tip from @tomMoral if you are using Visual Studio Code (I also seem to remember there is an equivalent PyCharm functionality as well so this may be accessible in other editors as well):

You might consider using the vscode extension: https://marketplace.visualstudio.com/items?itemName=vscode-ext.sync-rsync

You can define a remote folder and sync your local project files to it.
There is also a on save option that allows you to sync your project on save in vscode.

To view an arbitrary Jean Zay folder on your local computer

An alternative is sshfs with the right options to prevent being automatically disconnected

mkdir -p ~/mnt/jean-zay
sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3 \
    your-jean-zay-login@jean-zay:/your/jean/zay/folder ~/mnt/jean-zay

sshfs can be useful for example to look at TensorBoard logs on Jean Zay with TensorBoard running on your local computer.

For editing files, user feed-back seem to indicate that using sshfs to edit files is a bit laggy so not that useful in practice, if you have tweaks or suggestions to make sshfs a more viable option for editing files, please comment in this issue!

@rstrudel
Copy link
Contributor

rstrudel commented Oct 21, 2020

The command rsync -avz ~/folder-to-sync jean-zay:~/folder-to-sync --delete may be slow when the folder you want to update contains many small files. This command is nice but does not scale well as rsync check every files in the destination folder before uploading any new change. A workaround in my case where I upload code is to add the option --cvs-exclude to avoid copying .git that contains many small files and clean up the folder once in a while to keep it at a reasonnable size (below 1-2 Go).

@lesteve lesteve added the help wanted Extra attention is needed label Apr 4, 2021
@lesteve lesteve changed the title Document folder synchronisation (itnotify + rsync) and sshfs Document useful way of editing files locally with automatic sync on Jean Zay : (itnotify + rsync), editor extensions, and sshfs Jun 3, 2021
@rstrudel
Copy link
Contributor

rstrudel commented Nov 4, 2021

FYI #75 add automatic sync to the set of tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants