Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

单机8卡A100-80G deepspeed ZERO3 或者 非ZERO3 pretrain LLaMA-7B时，不能充分利用显卡 #56

Open
ShadowTeamCN opened this issue Apr 25, 2023 · 1 comment
Open

单机8卡A100-80G deepspeed ZERO3 或者 非ZERO3 pretrain LLaMA-7B时，不能充分利用显卡 #56

ShadowTeamCN opened this issue Apr 25, 2023 · 1 comment

Comments

@ShadowTeamCN
Copy link

不用deepspeed会爆显存，
有没有推荐的预训练参数设置，可以全程高效率的跑GPU
@MinhuiWan
Copy link

根据megatron框架的对比测试，zero3策略，megatron使用from apex.optimizers import FusedAdam as Adam 比 TencentPretrain中使用的deepspeed.ops.adam.DeepSpeedCPUAdam，GPU利用率高

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ShadowTeamCN @MinhuiWan