Does tensor_parallel support multi-node tensor parallel training?

[liguodongiot]

No description provided.

[zhangjunyi111]

I want to konw too.

[longday1102]

@BlackSamorez Hope you can answer this question 😄😄

[longday1102]

@BlackSamorez I have 2 servers with a total of 16 GPUs, so I would love to be able to use multi-nodes tensor-parallel to train a large language model, for example Bloom 176B. So I hope you can answer how to use multi-nodes tensor-parallel.
Thank you very much

[PieterZanders]

Is this solved? if so, how?

[deema-A]

same question.

[Tezcan98]

ahaha everybody have same problem but I think there is no feature like this but we absolutely need it.
Recently I tried DeepSpeed which is developing from microsoft, maybe it has but Microsoft's code doesn't suppport Windows 😄