Handle orders that have been processing for a long time

[SandGrainOne]

Description

Implement a new mechanism to pick up/detect orders that have the processing status for too long. Orders like that was picked up, but something must have gone wrong during the processing. The most obvious cause is the case of getting an exception when trying to write orders to a Kafka Topic.

Development

Beta Give feedback

1. Create a new TriggerController endpoint that will start the processing orders that are stuck in a temporary state for too long
2. Write necessary data access logic to identify possible orders to handle
3. Write necessary bussiness logic to handle the stuck orders. Use existing logic in OrderProcessingService.RetryOrderProcessing
4. Define a kubernetes job that triggers a check at regular intervals

Acceptance criteria

Beta Give feedback

1. Notification orders with the processing status for longer than x minutes are picked up

[acn-sbuad]

Cannot see that this is a realistic issue. With the changes made to the producer to ensure that we return false if publishing to a topic fails, we will always identify the cases where issues have occurred and change the status in the DB for the order to be picked up next round. I believe we can push this issue further down the line to see if it occurs again during testing

[SandGrainOne]

We will need to rewrite the logic around this when removing Kafka. Closing as not fixing.