Only log outcome from client library retries to fallback hosts

[mattheworiordan]

A customer has reported seeing timeouts in their logs and has therefore assumed publishes are failing with his client. However, this is not always the case as client libraries may have succeeded with the publish to an alternative endpoint.

We should consider adding a new spec item that states that when a request is retried by the client library, once the outcome is determined (success/failure), then the log entry should reflect what happened along with the final outcome. This should help to avoid ambiguity in the logs.

┆Issue is synchronized with this Jira Task by Unito

[mattheworiordan]

I have now implemented this in Ruby as a convenience.

Example warnings emitted for a request that fails but then succeeds is now:

warn: Ably::Rest::Client - Retry 1 for get /time {} as initial attempt failed (seq #5g-tSQ): getaddrinfo: nodename nor servname provided, or not known (SocketError)
warn: Ably::Rest::Client - Request SUCCEEDED after 1 retry for get /time {} (seq #5g-tSQ, time elapsed 5.11s)

And a request that never succeeds:

warn: Ably::Rest::Client - Retry 1 for get /time {} as initial attempt failed (seq #KyefoA): getaddrinfo: nodename nor servname provided, or not known (SocketError)
warn: Ably::Rest::Client - Retry 2 for get /time {} as initial attempt failed (seq #KyefoA): getaddrinfo: nodename nor servname provided, or not known (SocketError)
error: Ably::Rest::Client - Request FAILED after 2 retries for get /time {} (seq #KyefoA, time elapsed 15.02s)

Note that:

• I have included a sequence ID for the retries. This allows someone to filter through logs later and find the retries from one client. We could use request_id, but we don't yet support that, so I decided against that.
• Success or failure conditions still log a result i.e. either way, the user knows when the retry sequence is over.

[sync-by-unito]

➤ Automation for Jira commented:

The link to the corresponding Jira issue is https://ably.atlassian.net/browse/SDK-2824