What are "Warmup" step requirements?

[Pavelrst]

I'm following the t5 notebook and I have two questions regarding the warmup stage.

1. Which data should be passed to the t5 model during warmup? One sample? One batch? All the data?

2. The example says "IRL, encoder and decoder should be warmed each on their own" - what does that mean and how it should be done?

Thanks!

[pommedeterresautee]

1. any data, it doesn't matter, we just need to capture stuff in cuda graph
2. basically for each encoder length (per step of 8 token usually) you need to do a decoder warmup as we need to capture graphs. TBH it may takes time and is far of a being a nice xp We have recently checked that dynamo is finally able to manage dynamic shape without crashing, opening the way of not having to capture anything (warmup would just be 1-2 mn of autotune on Triton), for that we need to call triton manually instead of the @jit thing to avoid most of the CPU overhead
   CC @ayoub-louati

[Pavelrst]

Thanks! So if I understand you correctly, If each sample in my data is 32 tokens max, I need to make warmup with four sequences: a sequence with 0-8 tokens, 8-16 tokens, 16-24 tokens and a sequence with 24-32 tokens?

[pommedeterresautee]

Yes this is it