Parameter transformation #943
-
|
Can someone explain why |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hi Zeel ... I think (if I remember well) logexp is implementing what people (nowadays) call the "soft relu" so log(1+exp(x)) It's nice because it's monotonic ... and it behaves linearly in the positive half space. But exp() or square() are also valid. Square is not monotonic, so if it's used (e.g.) in a Bayesian model ... then it causes a bimodality that wasn't there in the original model. exp() is good too but often will cause numerical overflow problems. If I remember well, the implementation is likely clipped to avoid this, but it's a little bit of a sludgy solution. Neil |
Beta Was this translation helpful? Give feedback.
Hi Zeel ... I think (if I remember well) logexp is implementing what people (nowadays) call the "soft relu" so log(1+exp(x))
It's nice because it's monotonic ... and it behaves linearly in the positive half space. But exp() or square() are also valid. Square is not monotonic, so if it's used (e.g.) in a Bayesian model ... then it causes a bimodality that wasn't there in the original model. exp() is good too but often will cause numerical overflow problems. If I remember well, the implementation is likely clipped to avoid this, but it's a little bit of a sludgy solution.
Neil