MiniBatch Metrics task transition logging gaps

[Mattdl]

Hi, I've noticed that minibatch plugins show large gaps of logging in the TensorboardLogger (Haven't checked for other loggers).
For example, running the examples/tensorboard_logger.py gives large gaps on task transitions during training for the loss_MB and similar for RunningLossEpoch, but also for Minibatch Accuracy metrics like Top1_Acc_MB.
Ideally, I would need a continuous plot where on task transitions only 1 step is taken into account instead of the current multi-step gaps.

Any idea why this is the case and how to fix this?

[AndreaCossu]

Hi @Mattdl , thanks for reaching out. What do you mean by gap exactly? Anyway, I will run the example as soon as possible and take a look at the plots.

[AndreaCossu]

Also, which version of Avalanche are you using? The beta version or the one from the master branch?

[Mattdl]

Hi Andrea, thanks for the help! (: With gaps I mean that the MiniBatch metrics are plotted every single step on the x-axis (which I assume is the global iteration count). But on the task transitions, the 2 subsequent values have about 50 update steps in between that have no values logged.

[Mattdl]

Actually I just found the problem myself. It seems that calling cl_strategy.eval(scenario.test_stream) also increases the counter for the training values on the x-axis (probably some iteration counter). This creates the plot gaps for training metrics where nothing is logged.

[Mattdl]

Note that I am using an older version of Avalanche (just before the beta release), but I haven't seen any issue yet addressing this problem.

[Mattdl]

For information, this is what it should look like (the plot I get when commenting out the cl_strategy.eval(scenario.test_stream):

This is what I get when using the eval after the training on the experience:

[AndreaCossu]

Thanks, I see. The avalanche version you are using is probably a bit outdated. The strategy counters are now managed by the Clock object, which keeps a separate counter for the training iterations, solving your problem. This counter is used as x-value for the metrics.
I attach for reference the plot I obtain running the example, which looks like your first one. I am still seeing small gaps but not of the size you reported. I will investigate a bit further

.

[AndreaCossu]

Zoom-in version

[Mattdl]

That's great, I've updated to the Beta version and indeed it works now as it should! 💯
I think the jumps in your plots could actually be fine if they're only 1 step apart. In my case I have similar results, with the same peaks, now only 1 update-step apart as it should be (:
Thanks a lot for the help!