handling remote access files inputs and outputs

[JessyBarrette]

In some deployments, the input files and/or the output folder are within remote servers. It would be great to be able to handle those type of deployments:

key aspect to handle

inputs

• input connection parameters in config
• run glob on remote server
• hash files

processing

• load a temporary file locally
• run parser on local temporary file or memory

output

• output connection parameters in config
• upload file to output server

[JessyBarrette]

SLGO is using Paramiko https://www.paramiko.org/ for handling the remote servers

[JessyBarrette]

After further thoughts into this, I'm thinking it would be good to leverage rsync ability to generate a table with a row per file, their hash and time modified all together.

odpy could then:

1. Run that rsync command to the remote server
2. compare to the local registry
3. Download modified files
4. Apply conversion
5. Update registry
6. (Optional) Drop local source copy

[JessyBarrette]

This should simplify a lot of the logic:
https://github.com/PyFilesystem/pyfilesystem2