Adding PairNorm support #418
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: achiverram28 <[email protected]>
| # Implementation of normalization layers for GraphNeuralNetworks | ||
|
|
||
| @doc raw""" | ||
| PairNorm(scale_value; [scale_individually]) |
There was a problem hiding this comment.
Suggested change
| PairNorm(scale_value; [scale_individually]) | |
| PairNorm(scale_value; scale_individually=false) |
| @doc raw""" | ||
| PairNorm(scale_value; [scale_individually]) | ||
|
|
||
| PairNorm layer from paper [PairNorm: Tackling Oversmoothing in GNNs](https://arxiv.org/abs/1909.12223) |
There was a problem hiding this comment.
Suggested change
| PairNorm layer from paper [PairNorm: Tackling Oversmoothing in GNNs](https://arxiv.org/abs/1909.12223) | |
| PairNorm layer from paper [PairNorm: Tackling Oversmoothing in GNNs](https://arxiv.org/abs/1909.12223). |
|
|
||
| PairNorm layer from paper [PairNorm: Tackling Oversmoothing in GNNs](https://arxiv.org/abs/1909.12223) | ||
|
|
||
| Performs the operation(normalization) |
There was a problem hiding this comment.
Suggested change
| Performs the operation(normalization) | |
| Performs the operation |
Comment on lines
+10
to
+15
| \mathbf{x}_i^c &= \mathbf{x}_i - \frac{1}{n} | ||
| \sum_{i=1}^n \mathbf{x}_i \\ | ||
|
|
||
| \mathbf{x}_i^{\prime} &= s \cdot | ||
| \frac{\mathbf{x}_i^c}{\sqrt{\frac{1}{n} \sum_{i=1}^n | ||
| {\| \mathbf{x}_i^c \|}^2_2}} |
There was a problem hiding this comment.
Suggested change
| \mathbf{x}_i^c &= \mathbf{x}_i - \frac{1}{n} | |
| \sum_{i=1}^n \mathbf{x}_i \\ | |
| \mathbf{x}_i^{\prime} &= s \cdot | |
| \frac{\mathbf{x}_i^c}{\sqrt{\frac{1}{n} \sum_{i=1}^n | |
| {\| \mathbf{x}_i^c \|}^2_2}} | |
| \mathbf{x}_i^c &= \mathbf{x}_i - \frac{1}{n} | |
| \sum_{i=1}^n \mathbf{x}_i \\ | |
| \mathbf{x}_i^{\prime} &= s \cdot | |
| \frac{\mathbf{x}_i^c}{\sqrt{\frac{1}{n} \sum_{i=1}^n | |
| {\| \mathbf{x}_i^c \|}^2_2}} |
Comment on lines
+18
to
+19
| The input to this layer is the output from GNN layers | ||
|
|
There was a problem hiding this comment.
Suggested change
| The input to this layer is the output from GNN layers |
|
|
||
| # Arguments | ||
|
|
||
| - `scale_value`: Scaling factor `s` used in normalisation. Default `1.0` |
There was a problem hiding this comment.
Suggested change
| - `scale_value`: Scaling factor `s` used in normalisation. Default `1.0` | |
| - `scale_value`: Scaling factor `s` used in normalisation. Default `1.0`. |
| ``` | ||
| Default `false` | ||
|
|
||
| - `ϵ` : Small value added in the denominator for numerical stability. Default `1f-5` |
There was a problem hiding this comment.
Suggested change
| - `ϵ` : Small value added in the denominator for numerical stability. Default `1f-5` | |
| - `ϵ` : Small value added in the denominator for numerical stability. Default `1f-5`. |
This should be mentioned in the first line fo the docstring.
| @functor PairNorm | ||
|
|
||
| function PairNorm(scale_value::Real=1.0f0; scale_individually::Bool=false, eps::Real=1f-5, ϵ=nothing) | ||
| ε = _greek_ascii_depwarn(ϵ => eps, :BatchNorm, "ϵ" => "eps") |
There was a problem hiding this comment.
Suggested change
| ε = _greek_ascii_depwarn(ϵ => eps, :BatchNorm, "ϵ" => "eps") |
|
|
||
| @functor PairNorm | ||
|
|
||
| function PairNorm(scale_value::Real=1.0f0; scale_individually::Bool=false, eps::Real=1f-5, ϵ=nothing) |
There was a problem hiding this comment.
Suggested change
| function PairNorm(scale_value::Real=1.0f0; scale_individually::Bool=false, eps::Real=1f-5, ϵ=nothing) | |
| function PairNorm(scale_value::Real=1.0f0; scale_individually::Bool=false, eps::Real=1f-5) |
| return PairNorm(scale_value, ε, scale_individually) | ||
| end | ||
|
|
||
| function (PN::PairNorm)(x::AbstractArray) |
There was a problem hiding this comment.
Suggested change
| function (PN::PairNorm)(x::AbstractArray) | |
| function (pn::PairNorm)(x::AbstractArray) | |
| eps = ofeltype(x, pn.ϵ) | |
| s = ofeltype(pn.scale_value) |
| end | ||
|
|
||
| function (PN::PairNorm)(x::AbstractArray) | ||
| xm = mean(x, dims=1) |
There was a problem hiding this comment.
all dimensions are wrong here and belowe. The node dimension is the secnd dimension, the feature dimension is the first
Suggested change
| xm = mean(x, dims=1) | |
| xm = mean(x, dims=2) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addressing #405
Adding a new
normalise.jl