Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fully distributed training #252

Open
JimCircadian opened this issue Apr 12, 2024 · 1 comment
Open

Fully distributed training #252

JimCircadian opened this issue Apr 12, 2024 · 1 comment
Assignees
@JimCircadian
Labels
enhancement New feature or request
Milestone
v0.2.9

Comments

@JimCircadian
Copy link
Member

Description

Multi-node multi-*PU training. This is required for really scaling our use of the data pipeline for big predictions and given the construction of the pipeline as it exists, we just need some library changes to ensure that we can utilise resources as they're available. This will track additional development required to ensure that we scale to the HPC capabilities in question.
@JimCircadian JimCircadian self-assigned this Apr 12, 2024
@JimCircadian JimCircadian added this to the v0.3.0 milestone Apr 12, 2024
@JimCircadian JimCircadian added the enhancement New feature or request label Apr 12, 2024
@JimCircadian JimCircadian modified the milestones: v0.4.0, v0.3.0 Apr 15, 2024
@JimCircadian
Copy link
Member Author

The structure of the library facilitates some usage of distributed mechanisms, this is definitely not a CLI workflow. Some additional scripts are being added under icenet-pipeline for the moment

JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 23, 2024
@JimCircadian
Fixes icenet-ai#263: implemented a basic extension check to allow ful…
5f56a1f 
…ly qualified dataset filenames and Dev icenet-ai#252: refactoring of existing training functionality to allow extension to use horovod for fully distributed training as a child implementation of the original tensorflow
@JimCircadian JimCircadian modified the milestones: v0.3.0, v0.2.9 May 23, 2024
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 23, 2024
@JimCircadian
Dev icenet-ai#252: testing with horovod
39ccc6c
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 23, 2024
@JimCircadian
Dev icenet-ai#252: trying to solve some kind of deadlocking?
72f4b53
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 23, 2024
@JimCircadian
Dev icenet-ai#252: attempting to restructure more appropriately
e1421b9
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 23, 2024
@JimCircadian
Dev icenet-ai#252: small runtime fixes
347fbc6
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 24, 2024
@JimCircadian
Dev icenet-ai#252: erroneous import
808557d
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 24, 2024
@JimCircadian
Dev icenet-ai#252: adding further amendments to stabilise MPI based t…
babef09 
…raining with horovod
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 27, 2024
@JimCircadian
Dev icenet-ai#252: mapping error on args
7ae2904
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 27, 2024
@JimCircadian
Dev icenet-ai#252: missing constructor call
6abaae5
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 27, 2024
@JimCircadian
Dev icenet-ai#252: fixing constructor call, was actually that variabl…
680fdd2 
…es weren't available for callbacks, not a missing call
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 27, 2024
@JimCircadian
Dev icenet-ai#252: adding some logic for checkpoint callbacks
61714fa
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 27, 2024
@JimCircadian
Dev icenet-ai#252: wandb init error fix
63d5e03
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 28, 2024
@JimCircadian
Dev icenet-ai#252: fixing missing validation steps and learning rate …
77f211c 
…decay defaulting incorrectly
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 28, 2024
@JimCircadian
Dev icenet-ai#252: rank specific callbacks
c05e022
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 28, 2024
@JimCircadian
Dev icenet-ai#252: avoiding race condition with save / evaluate on MP…
36a1842 
…I jobs
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 28, 2024
@JimCircadian
Dev icenet-ai#252: looking at test set fixes
0ebcdcb
JimCircadian added a commit to JimCircadian/icenet that referenced this issue May 28, 2024
@JimCircadian
Dev icenet-ai#252: trying to load_model rather than weights
6bd2cfa
JimCircadian added a commit to JimCircadian/icenet that referenced this issue Jun 7, 2024
@JimCircadian
Dev icenet-ai#252: couple of notes
7a519cd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JimCircadian JimCircadian

Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.2.9
Development

No branches or pull requests
1 participant
@JimCircadian