[WIP] Adding Docker version of Prism

[danieldeutsch]

Here is what it would look like to add a Dockerized version of a metric.

[danieldeutsch]

The implementation in the GEM metrics library would wrap the one in Repro (here)

[danieldeutsch]

The actual implementation would reformat the input data to the format required by the metric, running predict_batch, and then reformatting the outputs to fit the GEM framework

[danieldeutsch]

If you have a machine with Docker installed, you should be able to directly run test_prism.py. The first time it runs, this Docker image will automatically be downloaded to your machine, which may take 1-2 minutes. After that, you can directly run the metric without downloading the Docker image.

[danieldeutsch]

@sebastianGehrmann @tuetschek @ndaheim Please take a look and let me know what you think.

[sebastianGehrmann]

Hey, given the discussions in the chat, should we merge this so we can proceed with adding the other implementations?

[danieldeutsch]

I think it should be ok. My only concern is how to deal with the GPU device. It seems like the other GPU-based metrics don't manually control the device, but this is necessary for the Dockerized metrics or else they will just use GPU 0.

When you run something in Docker, the code run in Docker runs in its own process, so the CUDA_VISIBLE_DEVICES environment variable you might set from the command line is ignored. My library sets that variable specifically for the Docker processes.

I was actually thinking about this last night. Since the classes are instantiated here without any arguments to the constructor, there isn't an obvious way right now to pass the device ID to the Dockerized metrics.

GEM-metrics/gem_metrics/__init__.py

Lines 124 to 126 in 9435858

	logger.info(f"Computing {metric_class.__name__} for {outs.filename}...")
	metric = metric_class()
	result = metric.compute_cached(cache, outs)

Any thoughts?