Handle AI Model Exceptions [Anthropic, OpenAI, etc]

[ItWasEnder]

There is a risk of the AI SDKs generating unhandled exceptions, such as 429 and overload exceptions. To prevent fatal crashes, we need to implement handling these errors in a more graceful way.

Example

[pearai-server-api2] [2024-09-01 21:03:53] [2024-09-01 21:03:53 +0000] [19] [ERROR] Error handling request
[pearai-server-api2] [2024-09-01 21:03:53] Traceback (most recent call last):
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 183, in handle_request
[pearai-server-api2] [2024-09-01 21:03:53]     for item in respiter:
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/werkzeug/wsgi.py", line 256, in __next__
[pearai-server-api2] [2024-09-01 21:03:53]     return self._next()
[pearai-server-api2] [2024-09-01 21:03:53]            ^^^^^^^^^^^^
[pearai-server-api2] [2024-09-01 21:17:25]     return self._request(
[pearai-server-api2] [2024-09-01 21:17:25]            ^^^^^^^^^^^^^^
[pearai-server-api2] [2024-09-01 21:17:25]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/anthropic/_base_client.py", line 1040, in _request
[pearai-server-api2] [2024-09-01 21:17:25]     raise self._make_status_error_from_response(err.response) from None
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
[pearai-server-api2] [2024-09-01 21:03:53]     for item in iterable:
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/flask/helpers.py", line 113, in generator
[pearai-server-api2] [2024-09-01 21:03:53]     yield from gen
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/app/services/llm/anthropic_service.py", line 61, in generate
[pearai-server-api2] [2024-09-01 21:03:53]     for chunk in response:
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/anthropic/_streaming.py", line 68, in __iter__
[pearai-server-api2] [2024-09-01 21:03:53]     for item in self._iterator:
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/anthropic/_streaming.py", line 110, in __stream__
[pearai-server-api2] [2024-09-01 21:03:53]     raise self._client._make_status_error(
[pearai-server-api2] [2024-09-01 21:03:53] anthropic.APIStatusError: {'type': 'error', 'error': {'details': None, 'type': 'overloaded_error', 'message': 'Overloaded'}}

[nang-dev]

we have model swtiching

[ItWasEnder]

@nang-dev Rewrote. We should be handling the exceptions in some way instead of letting them just be caught by flask.

[nang-dev]

tru

[cqdev-co]

I got this

[nang-dev]

goat @cqdev-co

[cqdev-co]

Done and merged

Source: trust me bro