support for non-blocking prediction API like polling

[withsmilo]

Hi, BentoML team.
I have an interesting use-case where I need to use a non-blocking API like polling because the caller cannot wait longer than a fixed timeout. Any good ideas for BentoML to support this case?

[parano]

Thank you for the suggestion @withsmilo! As we discussed in a different thread, this is best done via Yatai managed deployments. I created a feature request for Yatai project here: https://github.com/bentoml/Yatai/issues/156

[withsmilo]

Thanks a lot! 😄

[ssheng]

@withsmilo do you think something like gRPC bidirectional streaming can be an acceptable solution ?

[withsmilo]

Thanks for kind comment, @ssheng .
That approach could be OK, but I'm not sure if it will satisfy all of our requirements, which are:

1. The range of change in response time of prediction API is very large, and it may take longer than the fixed timeout. To support this case, we need a non-blocking prediction API.
2. The order of requests doesn't really matter. We can randomly generate a request id and use it as a communication approach between client and server.