RFC for aligning StableHLO and TOSA arithmetic #1149
Conversation
Initial version of an RFC to discuss aligning StableHLO and TOSA arithmetic operation. Signed-off-by: Eric Kunze <[email protected]>
eric-k256
force-pushed
the
tosa_align_rfc
branch
from
e8d168b
to
e38a4b6
Compare
There was a problem hiding this comment.
Hi folks! Apologies for the late reply - there have been out of band conversations about this RFC, and now I'd like to summarize them here on GitHub.
As the RFC mentions, StableHLO dialect currently has support for quantization via:
- Supporting
quant.uniformelement types. - Having dedicated ops like
uniform_quantize/uniform_dequantize. - Allowing regular ops like
add/convolutionto take quantized tensors.
This support was inherited from MHLO when StableHLO was bootstrapped, and MHLO's support was motivated by mobile use cases and inherited from TFLite. TFLite quantization has a specification, but StableHLO quantization does not.
One key aspect of TFLite quantization spec is that it uses a floating-point scale and an integer zero point as quantization parameters. In comparison, the quantization parameters proposed in this RFC involve an integer multiplier and shift. Harmonizing this or coming up with some kind of a compromise solution looks like the main open question at the moment.
Towards that end, I would like to propose for us to collaborate on pull requests to StableHLO specification. @sdasgup3 has created an initial PR #1352 that drafts a specification for QuantizedType and for semantics of quantized add, modelled after TFLite quantization semantics. Let's get together as a community and discuss the details, with the plan to progress to more involved ops like convolution in the future pull requests.
StableHLO dialect currently supports quantization via: 1) Supporting `quant.uniform` element types. 2) Having dedicated ops like `uniform_quantize` / `uniform_dequantize`. 3) Allowing regular ops like `add` / `convolution` to take quantized tensors. This support was inherited from MHLO when StableHLO was bootstrapped, and MHLO support was motivated by mobile use cases and inherited from TFLite. As pointed out in #1149, StableHLO specification doesn't support quantization at the moment, and this is an important gap that we would like to fix before StableHLO v1.0 (see #588). To continue the discussion started in #1149 and to make progress towards v1.0, this pull request: A) Adds QuantizedType to the StableHLO specification, modelled after [TFLite quantization spec](https://www.tensorflow.org/lite/performance/quantization_spec). B) To start a conversation about the applications of QuantizedType and the semantics of quantized ops, proposes semantics for quantized `add`. TFLite quantization spec doesn't cover everything. It specs constraints on types (which we captured accordingly in this pull request), but it doesn't go into describing semantics of quantized ops. As a result, the proposed semantics for quantized `add` is intentionally naive, as compared with the much more involved implementations in the TensorFlow repository, e.g.: * [tfl.add](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/kernels/add.cc). * [tf.UniformQuantizedAdd](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/uniform_quant_ops/uniform_quantized_add_op.cc). upd: After community discussion, we removed the spec for quantized `add` leaving that for future work, since further alignment is required. --------- Co-authored-by: Eugene Burmako <[email protected]>
Initial version of an RFC to discuss aligning StableHLO and TOSA arithmetic operation.
Signed-off-by: Eric Kunze [email protected]