Support all the ONNX ops

[ShigekiKarita]

Still no plan to impl onnx conversion but list here

https://github.com/onnx/onnx/blob/master/docs/Operators.md

Unary

• Abs
• Acos
• ArgMax
• ArgMin
• Asin
• Atan
• AveragePool
• Cast
• Ceil
• Clip
• Cos
• Elu (NOTE already wrapped in cudnn.activationForward/Backward)
• Exp
• Floor
• GlobalAveragePool
• GlobalLpPool
• GlobalMaxPool
• HardSigmoid
• Identity
• LRN
• LeakyRelu (NOTE already wrapped in cudnn.activationForward/Backward)
• Log
• LogSoftmax
• LpPool
• MaxPool
• MaxRoiPool
• Mean
• Multinomial
• Neg
• Pow
• Reciprocal
• Relu
• Selu
• Sigmoid
• Sin
• Softmax (NOTE already wrapped in cudnn.softmaxForward/Backward)
• Softplus
• Softsign
• Sqrt (NOTE: as pow)
• Sum
• Tan
• Tanh
• TopK
• Upsample

Binary (NOTE: some ops are partially supported via CUDNN opTensor)

• Add

• And

• Conv (NOTE CPU supports arbitrary shapes but CUDNN supports only 4/5 dim and limited shapes)

• ConvTranspose

• Div

• Equal

• Gemm

• Greater

• Less

• MatMul (NOTE: no batch matmul support yet)

• Max

• Min

• Mul

• Not

• Or

• Sub

• Xor

• Random

  • Dropout
  • RandomNormal
  • RandomNormalLike
  • RandomUniform
  • RandomUniformLike

Reduction (NOTE: some ops are partially supported via CUDNN opTensor)

• Gather
• ReduceL1
• ReduceL2
• ReduceLogSum
• ReduceLogSumExp
• ReduceMax
• ReduceMean
• ReduceMin
• ReduceProd
• ReduceSum
• ReduceSumSquare

Topology

• Concat
• Flatten
• Pad
• Reshape
• Shape
• Size
• Slice
• Split
• Squeeze
• Tile
• Transpose
• Unsqueeze

Learnable/Parametric

• BatchNormalization
• RNN
• GRU
• LSTM
• PRelu (NOTE already wrapped in cudnn.activationForward/Backward)

what?

• Constant
• SpaceToDepth
• LpNormalization
• DepthToSpace
• Hardmax
• InstanceNormalization

non onnx ops

• CTC Loss https://github.com/baidu-research/warp-ctc
• FFT using conv
  • https://github.com/locuslab/pytorch_fft
  • https://github.com/diggerdu/pytorch_audio/blob/master/time_frequence.py
  • add torch.stft and torch.fft pytorch/pytorch#3775