Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Unet3d] - Add infinite data loader to align epochs->samples transition #697

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

mmarcinkiewicz
Copy link
Contributor

@mmarcinkiewicz mmarcinkiewicz commented Jan 17, 2024

The main change is to replace standard dataloaders (finishing at each epoch) to "infinite" ones - uniformly sampling from the dataset till the program terminates.
This technically might change the order of samples, but:

  1. Everyone is using fairly large batch size (i.e. 56, which is 1/3 of the dataset)
  2. I checked the distribution of samples and throughout the training it is indistinguishable from the previous behavior
  3. The new RCPs are very similar to the old ones - I'll open a PR soon. Indeed some RCPs are a bit faster than previously, not sure if it's related to the new behavior, a bug in the old behavior when we switched to samples, or just due to the variance

There is no need to modify the submission code.

@mmarcinkiewicz mmarcinkiewicz requested a review from a team as a code owner January 17, 2024 16:54
Copy link

github-actions bot commented Jan 17, 2024

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant