More storage needed for the web server file system

[milanmlft]

While running the IDEAS IBASH workshop, I'm running into an error when trying to run the Converts NIFTI to 256 conformed MGZ file command:

org.mandas.docker.client.exceptions.ImagePullFailedException: Image pull failed: healthbioscienceideas/fastsurfer:gpu: ProgressMessage{id=null, status=null, stream=null, error=write /var/lib/docker/tmp/GetImageBlob4206791876: no space left on device, progress=null, progressDetail=null, buildImageId=null, digest=null}

In particular, the no space left on device seems to indicate we're running out of storage.

We should probably increase the root_block_device_size in terraform.tfvars_sample:

xnat-aws/provision/terraform.tfvars_sample

Lines 33 to 35 in 2c9a6a1

	"xnat_web" = 10
	"xnat_db" = 10
	"xnat_cserv" = 10

[milanmlft]

Also root_block_device["xnat_db"] isn't used anywhere, so should probably remove that.

[p-j-smith]

that's strange, the EFS should be being used for storing all the data, and it should expand as necessary

[milanmlft]

Hmm yeah not sure yet what's happening, I'll play around with these settings and see if I can get it to work.

[milanmlft]

So after increasing the xnat_web and xnat_cserv root block device sizes to 30 GB, the no space left on device error disappeared, but now I'm getting a new one:

Could not create container from image healthbioscienceideas/fastsurfer:gpu

Note that before, for some reason (can't remember where it came from) I had 30 GB for the xnat_web block size and 10 GB for the xnat_cserv block size. So the original error might have been due to that difference? Maybe we should just use a single root_block_device_size variable and use that for both the web server and the container?

[p-j-smith]

we previously increased the size of the root block device to 30 GB (#32) so we didn't run out of disk space after uploading the data. But now the data should all be stored in the EFS rather than root block device, so there might be an issue with the symlink between /data (root block device) and /storage/data (mounted EFS volume) not being created correctly

[p-j-smith]

Could not create container from image healthbioscienceideas/fastsurfer:gpu

I just checked and the image does exist with that tag (https://hub.docker.com/layers/healthbioscienceideas/fastsurfer/gpu/images/sha256-8110b0a8e4ae31a96ca3993ffee039200bb5f8aa53287ec575b271e884f00608?context=explore), so maybe there's an issue in the Dockerfile for that image. Was there any more info in the error message?

[milanmlft]

Nope, only that, I just tried with deepmi/fastsurfer:latest instead, but same error 😕
The dcm2niix container still works though, so it's not a general Docker container problem.

[p-j-smith]

Nope, only that, I just tried with deepmi/fastsurfer:latest instead, but same error 😕
The dcm2niix container still works though, so it's not a general Docker container problem.

😞 It might be that we do indeed need to run it on a gpu

[milanmlft]

Yeah was thinking something like that as well, so probably makes more sense to track this in #51