Distinct init for kernel and recurrent

[MartinuzziFrancesco]

Adding different initializers options for input matrix and recurrent matrix in recurrent cells as described in #2514. Similar approaches are available in:

• Keras/TF
• Flax

An example of the implementation is as follows:

function RNNCell((in, out)::Pair, σ=relu;
    kernel_init = glorot_uniform,
    recurrent_kernel_init = glorot_uniform,
    bias = true)
    Wi = kernel_init(out, in)
    U = recurrent_kernel_init(out, out)
    b = create_bias(Wi, bias, size(Wi, 1))
    return RNNCell(σ, Wi, U, b)
end

PR Checklist

• Tests are added (should not be necessary)
• Entry in NEWS.md
• Documentation, if applicable (updated docstrings)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.37%. Comparing base (c51e6fb) to head (bd2d257).
Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2522       +/-   ##
===========================================
+ Coverage   33.40%   60.37%   +26.96%     
===========================================
  Files          31       31               
  Lines        1907     1938       +31     
===========================================
+ Hits          637     1170      +533     
+ Misses       1270      768      -502     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[CarloLucibello]

Suggested change

	σ::F
	σ::F

We have two spaces indent in Flux, unfortunately

[MartinuzziFrancesco]

yeha I figured, although https://github.com/FluxML/Flux.jl/blob/master/.JuliaFormatter.toml#L1

indent = 4

kinda confused me

[CarloLucibello]

ah, that should be fixed, or even better we should move to 4 spaces indent

[MartinuzziFrancesco]

With 0.15 coming up that could be a good opportunity! Also at the moment if you do follow https://github.com/FluxML/Flux.jl/blob/master/CONTRIBUTING.md

julia dev/flux_format.jl --verbose .

results in formatting 80 files, so I guess it's formatting everything with 4 spaces indent

[CarloLucibello]

can we use init_kernel and init_recurrent_kernel instead?

[MartinuzziFrancesco]

The naming at the moment follows Flax's, while TF has kernel_initializer and recurrent_initializer. But I have nothing against changing it to init_kernel and init_recurrent_kernel if that's preferred!

[CarloLucibello]

Lux has init_weight, init_bias, .... I think the init_* convention is better as the argument stand out more clearly as similar.

[MartinuzziFrancesco]

makes sense then, I'll change it