training time

[RyanG41]

Hi,

May I ask about the training time for retrieval task?

I noticed that the code is using 6 GPUs, while I am currently using 4*RTX4090 and it went fast on the 30 epochs of base model but seems very slow on the following 300 epochs. May I know about the total training time on your side?

Thanks a lot.

Best Regard

[auniquesun]

@RyanG41 I think it is normal. The following 300 epochs train a much large Transformer model than previous 30 epochs. I think it takes about 20 hours on 6 2080Ti GPUs if I remember correctly.