Make sure EFS is being used for file storage

[milanmlft]

#67 seems to indicate that the EFS is not properly used. We would expect the file system to increase in size as needed so we should not run into problems due to insufficient storage space.

[p-j-smith]

I just tried manually pulling the healthbioscienceideas/fastsurfer:gpu image on the Container Service server and had the same error:

[rocky@ip-192-168-56-14 ~]$ sudo docker pull healthbioscienceideas/fastsurfer:gpu
gpu: Pulling from healthbioscienceideas/fastsurfer
80bce60046fa: Pull complete
55a738a15540: Pull complete
e19cf0706c62: Pull complete
de4cdd6c27d1: Pull complete
ec1a9bd0eb8c: Pull complete
8a6a57c93768: Pull complete
ff5439271f3c: Pull complete
5ee84dae1e02: Pull complete
647c4b3e1fdb: Pull complete
9c39f3ebb735: Pull complete
26eda9e73d5b: Pull complete
a6e0b974516e: Extracting  2.765GB/2.765GB
d87a14d7e28a: Download complete
e86eb8ebd148: Download complete
failed to register layer: write /opt/freesurfer/bin/dmri_forrest: no space left on device

So the issue isn't with the XNAT data, but with the size of the docker images we're using. Either we could symlink /var/lib/docker/overlay2 (where images are stored) to somewhere on the EFS, or just keep the larger root block device on the Container Service server?

[milanmlft]

Yeah, putting the images on EFS might be more future proof, if at some point we would ever need even more space.

[milanmlft]

I manually confirmed that EFS is being used for all files stored in /storage, by running df -h on the container service EC2 instance:

[root@ip-192-168-56-14 ~]# df -h
Filesystem                                          Size  Used Avail Use% Mounted on
devtmpfs                                            4.0M     0  4.0M   0% /dev
tmpfs                                               3.8G     0  3.8G   0% /dev/shm
tmpfs                                               1.5G   14M  1.5G   1% /run
/dev/nvme0n1p5                                       29G   22G  7.8G  74% /
/dev/nvme0n1p2                                      994M  250M  744M  26% /boot
/dev/nvme0n1p1                                       99M  7.0M   92M   8% /boot/efi
fs-044dc611742b19193.efs.eu-west-2.amazonaws.com:/  8.0E   36G  8.0E   1% /storage
tmpfs                                               759M     0  759M   0% /run/user/1000

Note that the 36G here is much more than what there normally is, I copied the Docker images over to /storage for testing.

The problem with storing the Docker images on the EFS, is that it is a different file system (nfs), for which the default Docker storage driver is not compatible. It would technically be possible to change the Docker storage driver but that seems to be not recommended and might lead to other problems.

Will close this for now but feel free to reopen if you think we should look further into it.