Diagnosis on the issue of no availability of instances in the RP_IVR Callback service

[Sachinbisht27]

Describe the bug
Currently, we are facing some error logs on the Monitoring Dashboard that there are a few instances when the Google Cloud function is not able to handle the sudden load and fails to scale up in certain cases.
Here is the error log:

The request was aborted because there was no available instance.
Additional troubleshooting documentation can be found at: https://cloud.google.com/functions/docs/troubleshooting#scalability

To Reproduce
Check the error logs of the Cloud function

Expected behavior/Action Items:

• Check the root cause and come up with the required solution.
• As the request load started increasing, need to come up with a plan for how we are going to handle this will increase in the future.

Screenshots

Additional context
NA

Acceptance Criteria

• The auto-scaling should work smoothly and should handle the upcoming load

Documentation
Add whatever documentation will be required here.

[Sachinbisht27]

Analysis of the issue

Diagnosis:
Sometimes, our existing infrastructure faced challenges in managing the application load. There were instances where we received webhooks from Kookoo, but unfortunately, our infrastructure is insufficient to capture and process them.

Current Observations:
On crossing the 10 instance count in the cloud function, we are getting the error on the monitoring dashboard and the failure for the webhook processing is occurring.

Impact:
Approx. 500-600 webhook failures in a month.

Solution:
Ways in which we can enhance the capability of current infrastructure. A way is to implement the queue-based approach.

Action Item:

• Integrate the New URL to the KooKoo webhook handler #11

[Sachinbisht27]

Closing this issue as the diagnosis is completed and created a new issue for the Implementation of the Queue service in the RP_IVR callback. DostEducation/rp-ivr-webhook-handler#11