Mantid Testing Data Should be Available Somewhere Other Than the Linode

[cailafinn]

Contribution to Target

Timeouts occur when linode throttles download. Was seen when many nodes needed to download the data from scratch after being re-generated. Providing a version available inside the STFC network will be far faster to download and far less likely to be throttled.

Issue Description

Initial improvement to allow easier access to the testing data.

Martyn's repos may be useful here as a starting point.

Tasks

• Create a new machine (host) on Openstack with a floating IP address. (Note: this will say external, but it won't be unless the firewall is configured to allow access external to the STFC network.
• Write an ansible script to perform the following:
  • Create a directory on the host to hold the external data.
  • Create a crontab job to rsync the data from the linode into this new directory every few minutes.
    • Will need some way to check if an rsync process is ongoing so it doesn't try multiple times simultaneously.
  • Spin up a docker container with the default nginx image (example here
  • Mount the external data directory from the host inside this docker container.
• Determine why Floating IPs don't seem to be functioning in the openstack mantid project.

Additional Improvements

• Creation of the node included in ansible script by using the Openstack API.

Options for Avoiding rsync conflicts.

• reduce the frequency
• check for existing rsync processes and exit if any are running
• use a "lock file". The rsync script first checks if a lock file exists and if one is found it exits doing nothing assuming another script is running. If no lock files exists, create one, do the rsync and then remove the lock file.