Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REQUEST] Allow dynamic neural architectures to be used #383

Open
johnHostetter opened this issue Mar 6, 2024 · 2 comments
Open

[REQUEST] Allow dynamic neural architectures to be used #383

johnHostetter opened this issue Mar 6, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@johnHostetter
Copy link

Hi there! Wonderful library and want to say thank you for all that you do by maintaining this repository.

Is your feature request related to a problem? Please describe.
I research self-organizing and morphetic neural architectures, and apply these to offline RL settings. I enjoy using your library but find it is difficult to incorporate the dynamic nature of these networks. Mainly, when you dynamically add new tunable parameters to a (policy/target) network, their associated optimizer does not reference these newly added tunable parameters.

Describe the solution you'd like
A clean solution to re-initialize the optimizer if new weights/parameters have been added during the .fit() call. Currently, my existing method is very "hack-y" and messy. I also have to use a separate fork where I override some code that prevents modification (such as frozen dataclasses). Further, if the policy and target networks' parameters do not perfectly align, the hard_sync method will throw an error; instead in my work I use a "sync if possible" approach, where the intersection of these parameters is synced, otherwise the networks' weights are unioned together.

Additional context
Although dynamically adding new weights/parameters to a network may see a drop in performance, it is possible to do so without causing this: https://proceedings.mlr.press/v48/wei16.html

I can also try to do this by submitting a PR if you could point me in the right direction or suggest changes you would be comfortable with me making. As it stands right now, my current code that does this is not too great and likely breaks features in the library I do not use in my work. I was hoping you might be able to suggest ideas how to easily and cleanly do this instead - thanks for your time!

@johnHostetter johnHostetter added the enhancement New feature or request label Mar 6, 2024
@takuseno
Copy link
Owner

@johnHostetter Hi, thank you for your request! Supporting the dynamically changing architecture is tricky indeed. One direction I can propose is that we could design a dedicated algorithm to support this use case so that we can reduce the shared parts of codes.

@johnHostetter
Copy link
Author

Hi @takuseno, thank you for your reply! Sorry for my delayed response - I am preparing to propose my dissertation work in the coming month.

I think a dedicated algorithm could work but may cause verbose or redundant algorithm definitions. Perhaps it might be possible to instead delicate it such that if the model (e.g., DQN) is identified as an instance of DynamicModule (or whatever name is appropriate), the algorithm would then allow the relevant behavior to dynamically modify its architecture; this could make use of the existing callbacks in the algorithms' .fit implementations. More specifically, the library's existing EncoderFactory could possibly be expanded to produce these dynamic networks automatically. Otherwise, if it is simply just a regular torch.nn.Module, then no special dynamic architecture handling is required (keep the implementation as-is now).

So my proposed solution would be that we create a DynamicEncoderFactory that generates a special torch.nn.Module which is then recognized as triggering certain default callbacks to enable the growing/shrinking of the model's parameters recognized by the optimizer. If this sounds like something you would be interested in, I can work on an implementation for this with minimal code footprint so as to not disrupt the library's structure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants