Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optionalized multistage module, add linear incrementing stage #50

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

Purg
Copy link
Member

@Purg Purg commented Nov 29, 2024

What does this PR do?

Add a version of the multistage module that can do a few additional optional things, otherwise it should allow the same functionality of the existing multistage module.

  • Allow optional use of a single stage that linearly increases dilation value.
  • Allow optional use of a single stage that is set to a constant dilation value.
    • This can be set to 1 to effectively disable dilation.
  • Allow optional usage of softmax going into single stage layers (defaulted to False)
  • Allow optional addition of previous single stage logit outputs to successive single stage logit outputs.

Added new experiment config for M2 to make use of this new model, linear single stages, no internal softmaxing and residual single stage combination.

@Purg Purg force-pushed the dev/network-experiment branch from decb561 to ed1737b Compare December 1, 2024 20:20
@Purg Purg force-pushed the dev/network-experiment branch from ed1737b to beaaa59 Compare December 1, 2024 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants