Add sidekiq backoff service to handle OS Places outages #272
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KludgeKML
force-pushed
the
sidekiq-slow-down-on-error
branch
2 times, most recently
from
752c651 to
f5d6713
Compare
KludgeKML
force-pushed
the
sidekiq-slow-down-on-error
branch
3 times, most recently
from
efe704f to
395bd75
Compare
KludgeKML
force-pushed
the
sidekiq-slow-down-on-error
branch
from
1f9c839 to
6f9a7d0
Compare
1pretz1
approved these changes
Oct 12, 2023
|
|
||
| def record_success | ||
| initial_interval = current_interval | ||
| target_interval = [initial_interval - 1, @min_interval].max |
There was a problem hiding this comment.
Could we use attr_reader vs calling instance vars direct
- When OS Places has a slowdown or outage, we can avoid our own alerting problems by backing off calls (since they're not time-critical). - Add a service which is initialised on startup (relevant mainly to worker processes) which adjusts the scheduled interval of PostcodeProcessWorker creation (currently once per second). We record OS Places API failures, and with each failure we double the interval, until we reach a max of 180s. When we record a successful call, we reduce the interval by 1s, so it will quickly back off if many errors occur, and slowly creep back to full speed when the errors are over.
KludgeKML
force-pushed
the
sidekiq-slow-down-on-error
branch
from
6f9a7d0 to
fb6f7e6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://trello.com/c/8P4LRqvl/2171-investigate-how-we-can-handle-os-places-api-being-down