Make ayon-shotgrid services multi-threaded and asynchronous

[robin-ynput]

Is there an existing issue for this?

• I have searched the existing issues.

Please describe the feature you have in mind and explain what the current shortcomings are?

Our current services implementation is quite monobloc with a big while True that does everything at once.
This is functional but in my opinion this is a bit fragile and hard to test.

We could make all of those process asynchronous by reworking a bit of the initial design there.

Stability gains:

• Currently if an intermediary crashes then everything goes down and all events in the batch are lost, with this design we can imagine, pause and recover features.
• Currently the whole performance is driven by the slower bricks, splitting into dedicated thread might also leads the way to parallelization later on
• Each business "task" will have a small dedicated scope, this will be easiest to furnish unit tests and extend/re-use.

How would you imagine the implementation of the feature?

Quick example for leecher service (very quick thoughts, my be ending slightly different):

• split gathering FLOW event process in its own thread
• split event AYON translator process in its own thread
• introduce a queue between those 2 threads

Are there any labels you wish to add?

• I have added the relevant labels to the enhancement request.

Describe alternatives you've considered:

No response

Additional context:

No response