Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow workers to specify redis Retry Helpers settings #386

Closed
mernmic opened this issue Jan 17, 2023 · 2 comments
Closed

Allow workers to specify redis Retry Helpers settings #386

mernmic opened this issue Jan 17, 2023 · 2 comments

Comments

@mernmic
Copy link
Contributor

mernmic commented Jan 17, 2023

Currently the retry settings conn_retries, conn_timeout, conn_retry_delay are only applied to the initial connection and any connection issues while running, eg: temporary network hiccup, disconnected by host, will cause the worker to crash.

Proposing extending for the following settings:
retry: Retry instance with a Backoff strategy and the max number of retries
retry_on_error: list of Exceptions to retry on
retry_on_timeout: if True, retry on TimeoutError only

redis-py documentation on retry helpers:
https://redis-py.readthedocs.io/en/stable/retry.html

@mernmic
Copy link
Contributor Author

mernmic commented Jan 17, 2023

example stack trace:

File "/project_dir/venv/lib/python3.9/site-packages/arq/worker.py", line 868, in run_worker
    worker.run()
  File "/project_dir/venv/lib/python3.9/site-packages/arq/worker.py", line 301, in run
    self.loop.run_until_complete(self.main_task)
  File "/opt/homebrew/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/project_dir/venv/lib/python3.9/site-packages/arq/worker.py", line 354, in main
    await self._poll_iteration()
  File "/project_dir/fc_tq/utils/arq_utils.py", line 181, in custom_poll_iteration
    job_ids = await worker.pool.zrangebyscore(
  File "/project_dir/venv/lib/python3.9/site-packages/redis/asyncio/client.py", line 487, in execute_command
    return await conn.retry.call_with_retry(
  File "/project_dir/venv/lib/python3.9/site-packages/redis/asyncio/retry.py", line 62, in call_with_retry
    await fail(error)
  File "/project_dir/venv/lib/python3.9/site-packages/redis/asyncio/client.py", line 476, in _disconnect_raise
    raise error
  File "/project_dir/venv/lib/python3.9/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
    return await do()
  File "/project_dir/venv/lib/python3.9/site-packages/redis/asyncio/client.py", line 463, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
  File "/project_dir/venv/lib/python3.9/site-packages/redis/asyncio/client.py", line 505, in parse_response
    response = await connection.read_response()
  File "/project_dir/venv/lib/python3.9/site-packages/redis/asyncio/connection.py", line 939, in read_response
    raise ConnectionError(
redis.exceptions.ConnectionError: Error while reading from host : (60, 'Operation timed out')

@mernmic
Copy link
Contributor Author

mernmic commented Jan 17, 2023

proposed fix: #387

@mernmic mernmic closed this as completed Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant