Use of MetricPlugins to access last results

[AntonioCarta]

Metric plugins may emit multiple. However, they return a single result. For example:

        exp_acc = ExperienceAccuracy()
        evalp = EvaluationPlugin(main_metric, exp_acc, loggers=None)
        strategy = Naive(...)
        scenario = get_fast_scenario(use_task_labels=True)

        for train_batch_info in scenario.train_stream:
            strategy.train(train_batch_info)
        strategy.eval(scenario.train_stream[:])
        print("TRAIN STREAM ACC: ", main_metric.exp_acc())

here exp_acc returns only the result for the last experience.

How can I access the values for the other experiences? IMO, they are also the last results (for a particular curve), so I expected to be able to retrieve them from the plugin.

[vlomonaco]

mmmh are you asking to keep track of all the metrics results also within the metric_plugin itself? cannot you already access them from the evaluationPlugin main metrics results dict?

[AntonioCarta]

@AndreaCossu You're right, I can use the dictionary. So you don't see any value in providing access directly from the plugin?

What happens if a plugin needs to compute multiple values concurrently? For example, let's say that I want to split EpochAccuracy by task and I have a multitask scenarios (i.e. same mini-batch with different task labels).

[AndreaCossu]

Ok you are referring to the metric plugin, not to the evaluation one 😄 I think that this kind of support should be provided directly from the eval plugin (in your example if you read the dictionary you should be able to get the epoch accuracy grouped by task). This is something we still have to support (we should), currently task metrics work only if task labels are the same within experiences.

[AntonioCarta]

Ok, it makes sense to provide access only via the EvaluationPlugin.

I still don't understand why PluginMetric inherits from Metric. (class PluginMetric(Metric[TResult], StrategyCallbacks['MetricResult'], ABC):), especially if we're not going to support result() for plugins.

[AntonioCarta]

This is something we still have to support (we should), currently task metrics work only if task labels are the same within experiences.

There is really no reason for this limitation to exist.

[AndreaCossu]

I agree, it is simply the way it is right now. I think it's ok if PluginMetric inherits also from Metric. I wouldn't remove support for result, just recommend to use the eval plugin.