Speed comparison with other frameworks

[NareshGuru77]

Hello!

I am moving from mammoth (https://github.com/aimagelab/mammoth) repository to avalanche.

I notice than the mammoth repository is generally faster than avalanche under the same experimental settings, cpu and gpu.

In order to understand whether this issue comes from my use of avalanche or there are issues in avalanche itself, I was wondering whether a timing comparison has been made with other existing frameworks.

Thank you!

[vlomonaco]

Dear @NareshGuru77, thanks for considering using Avalanche for your work!

From my understanding Mammoth is more limited than Avalanche in terms of offered functionalities and hence less complex in terms of codebase. It makes sense that for some vertical experiments it may be faster. However, I'm not aware of any benchmark analysis in terms of efficiency.

Maybe if you can share your code or provide more details about the experiment you are running, we may gather more insights and help you out with optimizing your code!

Moving this to the discussion tab since it is not an issue!

[NareshGuru77]

Hi @vlomonaco

Thank you for the quick response. Avalanche is definitely more extensive and offers more functions.

However, it would be nice if avalanche can also take similar time for experimentation. For instance, I ran the naive strategy in avalanche and mammoth for cifar10 dataset using the same gpu (2080Ti). The experimental settings are the same (resnet18, input size 32, batch size 32, 5 epochs, sgd optimizer, 5 tasks with 2 classes each).

In this experiment, mammoth takes roughly 3 mins while avalanche takes roughly 7 mins. I am not entirely sure whether it is a bug in the way i am using avalanche but from understanding this is a simple strategy and I haven't done any major changes. I will try to soon add a self contained example that can show this issue.

Thank you!

[HamedHemati]

Hey @NareshGuru77, can you please try running the Avalanche code with num_workers>0?
For example, when calling strategy's train function, you can pass the number of workers as below:

strategy.train(exp, num_workers=1)

and the same for evaluation, if you are evaluating after each experience:

strategy.eval(exp OR benchmark.test_stream, num_workers=1)

It would be helpful to see the difference in runtime by increasing the number of workers.

Thank you.

[NareshGuru77]

Hey @HamedHemati ,

I ran with different workers. Please find the table below:

Workers	Avalanche	mammoth
0	8m 41	3m 35s
4	7m 9s	2m 34s
8	7m 10s	2m 38s

These numbers are rough values and it different slightly from run to run. However, the difference in speed is there irrespective of workers.

Thank you.

[HamedHemati]

Thank you @NareshGuru77 for the thorough comparison. That was expected to see that increasing the number of workers (up to some extent) can boost the training speed (probably) proportionately for both trainers, but I wanted to see the speed gain by setting a higher number of workers.

A detailed comparison between the two frameworks would be needed to understand the reasons for the discrepancies. As others pointed out some potential reasons, it's important to note that in Avalanche you can technically use plugins that will be triggered before/after any callback (e.g. before/after each experience/epoch/iteration/etc.), for example, the computation of batch accuracy/loss can happen after every model update, therefore it's important to check if the same computation is done on mammoth after each step as well. Checking the metric plugins and other additional steps in both libraries is important.

If you can provide the code examples that you are running for both libraries, we may be able to find those differences ;)

[NareshGuru77]

Thank you @HamedHemati . I did a more detailed comparison and I find that the reason is because I am using the following for reproducibility in avalanche:
torch.backends.cudnn.enabled = False
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

In mammoth, this was not used. Turns out that this slows down computation.

Therefore, avalanche does not have any issues with speed. This discussion can be closed.

[HamedHemati]

Thanks for the update, and good to hear that the problem is fixed now :)