Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nan in loss #6

Open
zyzwhdx opened this issue Aug 29, 2023 · 2 comments
Open

Nan in loss #6

zyzwhdx opened this issue Aug 29, 2023 · 2 comments

Comments

@zyzwhdx
Copy link

zyzwhdx commented Aug 29, 2023

Hi, many thanks for the great work.

But when I was training this model, the training log showed Nan in loss, and this happened even in your uploaded exp folder. I guess maybe it is caused by the smooth loss calculation (pred_distance).

pred_distance = pred_distance[:args.N_rgb], pred_distance_coarse = pred_distance_coarse[:args.N_rgb], pred_distance_smooth = pred_distance[args.N_rgb:]

Do you have any suggestion for this problem? Thank you!

@booker-max
Copy link

I also encountered the same problem, args.N_rgb = 4096, and len(pred_distance) = 4096, hence pred_distance_smooth = [], and ultimately caused NAN loss.

@booker-max
Copy link

I don't know how to solve it, Do you have any advice?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants