Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test and document using sml-sync on Windows #21

Open
pbugnion opened this issue Jun 4, 2018 · 5 comments
Open

Test and document using sml-sync on Windows #21

pbugnion opened this issue Jun 4, 2018 · 5 comments
Milestone

Comments

@pbugnion
Copy link
Contributor

pbugnion commented Jun 4, 2018

We should at least test sml-sync on the Linux subsystem in Windows to see if we can get something working.

Whatever the result, there should be some section in the README on Windows support.

@pbugnion pbugnion added this to the 1.0.0 milestone Jun 4, 2018
@janfreyberg
Copy link
Contributor

Do we have a machine running windows 10?

@pbugnion
Copy link
Contributor Author

pbugnion commented Jun 11, 2018

Yes, in the engineering room. I believe the plan is also to get a Windows laptop.

We can now reproducibly install sml on Cygwin with OpenSSH and rsync. Syncing files work etc. Installing sml-sync is proving trickier because it relies on Paramiko, which relies on libsodium, which is quite hard to compile on Cygwin.

One possible option to make installing more reliable would be to remove paramiko and just rely on rsync and ssh. paramiko is notoriously quite hard to install. By not maintaining a persistent remote session, this would also improve reliability.

@pbugnion
Copy link
Contributor Author

pbugnion commented Jun 29, 2018

We currently use paramiko for:

  1. getting the last updated time for a file during the watch-sync process -- this happens every time a file needs to be synced (to avoid accidentally overwriting files the user has modified on the remote)
  2. checking whether the suggested remote directory exists during initial configuration
  3. listing all remote subdirectories of a given path (to populate the directory suggestion box during the initial configuation)
  4. making directories on the remote as part of the watch-sync process
  5. removing files and directories on the remote as part of the watch-sync process
  6. moving files on the remote as part of the watch-sync process

The only item here where latency is really important is 1. I think the right course of action is to try replacing 1. with shelling out to ssh and see if that materially impacts user experience. If it doesn't, we can press on and do all the others.

@pbugnion
Copy link
Contributor Author

I just validated changing step 1 to use rsync rather than a permanent paramiko connection. This increases the time spent in the watch-sync loop from ~0.4 seconds for tiny files to about 1 second for tiny files. For larger files, the effect is negligible.

I've played around with this while doing actual gmaps development and the extra time doesn't seem to affect the experience, so I think we should remove paramiko everywhere. Besides providing windows support, it will also make sml-sync handle network disconnections much more gracefully, since we wouldn't maintain a persistent connection between the laptop and the SherlockML server.

@janfreyberg
Copy link
Contributor

Sounds good to me; seems like a good decision and that seems like reasonable sync times.

Great idea!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants