Blood cell data benchmark

[schoersch]

I thought it might be good keep track of the discussion related to #50 here.

From today, I noted the following parts having influence on the model results and should thus be addressed in some way in the benchmark:

1. Dataset (this is of course chosen by the user)
2. Used data augmentation
3. Initialization of the network weights
4. Random seed
5. Used Training algorithm
6. Model hyperparameters

[schoersch]

For investigation of the sensitivity to hyperparameters, I would think of considering only point 6. (Model hyperparameters) above. This would reduce the dimension of the parameter space, keeping the task more feasible. Then I see different possibilities:

We can compute first the stochastic variance $var_s$, which we can use to (1) annotate each point with the std and (2) decide on how much model evaluations $M$ of each hyperparameter combination are necessary to estimate the mean performance, since MC error is $var_s / M$. Denote with $P$ the number of hyperparameters.

1. A local sensitivity, i.e. fixing each parameter to the author default and varying one-by-one. So we vary each parameter once across it's range, let's say in $p$ steps. Then this methods leads to $P \times p \times M$ model evaluations.
   Also, the full parameter space is sampled only sparse along 1D lines, intersecting at the mean.
2. We sample $N$ parameter samples uniform/log-uniform from the parameter space. For me these data would lead to the scatter plot suggested in the meeting (Create plots comparing the performance of different models on one dataset #80 2.?), with each point being the mean performance of a parameter sample. This needs $N \times M$ model evaluations.
   In order to explore the performance across the full hyperparameter space, we need $N \sim 10^P$, but it is easier to adjust the overall cost depending on the resources.

[smilesun]

cool! Could you first write a draft yml file defining the space of hyper-parameter that you want to sample?

See here for a concrete running instruction of DIVA:
https://github.com/marrlab/DomainLab/blob/master/examples/yaml/demo.yaml

However, instead of having fixed values of DIVA in the above yaml, we would need an extra yaml indicating which combination of different hyper-parameters to use..

[schoersch]

I thought to be able to continue a benchmark after it crashed for whatever reason, one easy solution would be to sample all parameter beforehand and save them to disk. Thus I'm regarding the hyperparameter sample generation as a first step of the benchmark, that can be separated from the rest.

I think a fully flexible yaml config for the 2. case above, working also with yet unknown algorithms, could be:

# only entries related to parameter sampling
num_param_samples: 30
runs_per_sample: 5    # This could also be dependent on the estimated stochastic variance

Task1Name:
  aname: Name
  p1:
    authorschoice: 1.5
    distribution: uniform    # uniform | loguniform | normal | lognormal
    min: 1
    max: 2
    step: 0    # anything > 0 for discrete variables, 0 for continuous ones.
    
  p2:
    ...
    
  constraints:
    - "p1 < p2"    # any valid Python code evaluating to bool, variables are having the name from above.
    - ...
    
Task2Name:
  ...

from which then a .csv is generated with columns like
index | aname | p1 | ... | pn |
Or if it turns out to be easier to handle later, one file for each algorithm.

For later computation, the parameter names must of course match the ones from the model. However, this does not matter for the sample generation, which thus stays independent from the implemented algorithms.

Also, to keep the flexibility and stay simple, I would consider each parameter by itself. By this I mean, that even if I have two models for which the latent dimension needs to be specified, we do not draw a connection between these two parameters. Instead, the parameters for each model a sampled independently.

Note: I will start a first implementation tomorrow afternoon. Additional thoughts are welcome.

[Car-la-F]

According to the columns in your .csv file.

For the plotting functions I used only one column for all hyperparameters (and save a dictionary for the hyperparam. Setup i.e. {'a': 0, 'b': 6...}). I chose this approach to make the input to the plotting function more general in terms of number of columns.

But I don't know whether this is also a good approach for your application.

[schoersch]

Hmm. If you come from Databases, this is a no go. But this is also not really a database and this seems to be a reasonable way if we want to include all hyperparameters in a single file. And I think a single file is less cluttered to the user.
I will check if the feeding of the parameters into the chain is easier by one of the methods.