Custom Retry policy not triggering in PollyResiliencePipelineProvider

[aftabkajal]

I've implemented a custom retry mechanism using Polly in the PollyResiliencePipelineProvider. Despite configuring the ShouldHandle property and providing retry strategies, retries are not triggered for retry-able exceptions or HTTP status codes.

1. Configure Ocelot with a downstream service returning 503 status.
2. Use the provided custom PollyResiliencePipelineProvider.
3. Observe that the retry policy does not trigger as expected.

Code Snippet:

public class PollyResiliencePipelineProvider : PollyQoSResiliencePipelineProvider
{
    private readonly OcelotGlobalConfiguration _options;
    private readonly IOcelotLogger _logger;
    public PollyResiliencePipelineProvider(
        IOptions<OcelotGlobalConfiguration> options,
        IOcelotLoggerFactory loggerFactory,
        ResiliencePipelineRegistry<OcelotResiliencePipelineKey> registry)
        : base(loggerFactory, registry)
    {
        _options = options.Value;
        _logger = loggerFactory.CreateLogger<PollyResiliencePipelineProvider>();
    }

    protected override ResiliencePipelineBuilder<HttpResponseMessage> ConfigureCircuitBreaker(ResiliencePipelineBuilder<HttpResponseMessage> builder, DownstreamRoute route)
    {
        ConfigureRetry(builder, route); // Add Retry strategy first

        return base.ConfigureCircuitBreaker(builder, route);
    }

    protected ResiliencePipelineBuilder<HttpResponseMessage> ConfigureRetry(ResiliencePipelineBuilder<HttpResponseMessage> builder, DownstreamRoute route)
    {
        var isQosOptionsExist = route.MetadataOptions.Metadata.TryGetValue(nameof(FileQoSOptions), out var retryOptions);

        if (!isQosOptionsExist || string.IsNullOrWhiteSpace(retryOptions))
        {
            _logger.LogWarning($"QoSOptions not found or empty for route: {GetRouteName(route)}.");
            return builder;
        }

        var retryMetadata = JsonSerializer.Deserialize<OcelotQoSOptions>(retryOptions);

        if (retryMetadata == null)
        {
            _logger.LogWarning($"Failed to deserialize QoSOptions for route: {GetRouteName(route)}.");
            return builder;
        }

        // Add Retry strategy only if the retry count is defined in QoSOptions and greater than 0
        if (retryMetadata.RetryCount <= 0)
        {
            _logger.LogWarning($"RetryCount {retryMetadata.RetryCount} for route: {GetRouteName(route)}.");
            return builder;
        }

        _logger.LogInformation($"Retry options for route: {GetRouteName(route)}: RetryCount={retryMetadata.RetryCount}, RetryDelay={retryMetadata.RetryDelay}");

        var info = $"Retry for the route: {GetRouteName(route)}: ";
        var strategyOptions = new RetryStrategyOptions<HttpResponseMessage>
        {
            MaxRetryAttempts = retryMetadata.RetryCount,
            Delay = TimeSpan.FromMilliseconds(retryMetadata.RetryDelay),
            ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                         .Handle<Exception>(ex =>
                         {
                             var shouldRetry = RetryableExceptions.Contains(ex.GetType());

                             if (shouldRetry)
                             {
                                 _logger.LogDebug($"Retry triggered by exception: {ex.GetType().Name}");
                             }

                             return shouldRetry;
                         })
                        .HandleResult(response =>
                        {
                            var shouldRetry = ServerErrorCodes.Contains(response.StatusCode);

                            if (shouldRetry)
                            {
                                _logger.LogDebug($"Retry triggered by HTTP status code: {response.StatusCode}");
                            }

                            return shouldRetry;
                        }),
            BackoffType = DelayBackoffType.Constant,
            OnRetry = args =>
            {
                _logger.LogWarning(info +
                    $"Retry attempt {args.AttemptNumber} after {args.RetryDelay.TotalMilliseconds} ms due to {args.Outcome.Exception?.Message ?? args.Outcome.Result?.StatusCode.ToString()}");
                return ValueTask.CompletedTask;
            },
        };

        return builder.AddRetry(strategyOptions);
    }

    private static readonly ImmutableArray<Type> RetryableExceptions =
         [
            typeof(SocketException),
            typeof(HttpRequestException),
            typeof(TimeoutRejectedException),
            typeof(BrokenCircuitException),
            typeof(RateLimitRejectedException),
            typeof(TimeoutException),
         ];
}

.AddPolly<PollyResiliencePipelineProvider>();

Log:

Specifications

• Version: 23.4.2
• .NET version: 8.0
• Hosting: Docker and Local

Expected Behavior:
The retry logic should trigger on HTTP 503 status codes or HttpRequestException.

Actual Behavior:
The retry mechanism is not invoked, and no retry attempts are logged.

[raman-m]

Dear Aftab,
Usually we don't advise on custom logic in user's gateways. This is totally up to you how to setup custom logic.
Your log shows the first call to the downstream service, but to understand a complete log is required to review.
Another issue is absent ocelot.json. You must attach complete configuration file to review ❗

Actual Behavior:
The retry mechanism is not invoked, and no retry attempts are logged.

I guess you have a problem with timeout settings... Look, the request was triggered at 1:17:46 and then the request was finished in 2 seconds at 1:17:48. So, total processing time is around 2 seconds! This is abnormally short timeout, because the default timeout is 90 seconds (see Notes).
Again, what's your TimeoutValue in QoSOptions? Please show your ocelot.json!

Expected Behavior:
The retry logic should trigger on HTTP 503 status codes or HttpRequestException.

I can't recommend right now, because firstly we have to solve the problem with Timeout event!
Another issue is abnormal status code which is 102 (Processing) ❗ Seems your downstream service is overloaded and put the request into a processing queue waiting until previous requests will be processed. So, that is why you have timeout exception. Seems your downstream service can't process a request in 2 seconds.
You have to increase timeout in 2-3 times, and watch for what will happen. Better to get some statistics of processing time by downstream services and specify maximum processing time as a TimeoutValue in milliseconds in QoSOptions.

P.S. FYI

Please note that currently Ocelot settings are limited in control of timeout. So you can define only route timeouts in QoS.
We have the open PR #2073 which gives more control over timeouts. And this PR is prioritized for merging in the next release.

[aftabkajal]

This is my QoS configuration

{
    "TimeoutValue": 1000,
    "DurationOfBreak": 900000,
    "ExceptionsAllowedBeforeBreaking": 4
}

[aftabkajal]

I guess you have a problem with timeout settings... Look, the request was triggered at 1:17:46 and then the request was finished in 2 seconds at 1:17:48. So, total processing time is around 2 seconds! This is abnormally short timeout, because the default timeout is 90 seconds (see Notes).

Got it, the issue was with the timeout. I'll proceed to test the remaining components now. Thanks for your guidance!

[raman-m]

This is my QoS configuration

{
    "TimeoutValue": 1000,
    "DurationOfBreak": 900000,
    "ExceptionsAllowedBeforeBreaking": 4
}

Your timeout value is 1 second! It is an abnormally low. Please note, the option has a dimension of milliseconds, not seconds! If you increase the value to at least 5000 ms, then I believe your retry logic will start functioning properly.

Secondly, the duration the break is specified in. So, 900000 means 900 seconds! This is an abnormally high value. The duration of the break should be comparable to the timeout. It shouldn't be higher the timeout!

Finally, realized that your retry logic doesn't make much sense because it should analyze status code first. If your downstream service is overloaded, that you will always get a 102 response. Rewrite the retry logic status code analysis. Disable exceptions analysis and decide based on status only.

[aftabkajal]

I have implemented the configuration, and both Retry and Timeout are functioning as expected. However, the CircuitBreaker doesn't seem to be working as intended.

QosOptions:

{
    "RetryCount": 5,
    "RetryDelay": 3000,
    "TimeoutValue": 40000,
    "DurationOfBreak": 5000,
    "ExceptionsAllowedBeforeBreaking": 3
}

Code

public class PollyResiliencePipelineProvider : PollyQoSResiliencePipelineProvider
{
    private readonly OcelotGlobalConfiguration _options;
    private readonly IOcelotLogger _logger;
    public PollyResiliencePipelineProvider(
        IOptions<OcelotGlobalConfiguration> options,
        IOcelotLoggerFactory loggerFactory,
        ResiliencePipelineRegistry<OcelotResiliencePipelineKey> registry)
        : base(loggerFactory, registry)
    {
        _options = options.Value;
        _logger = loggerFactory.CreateLogger<PollyResiliencePipelineProvider>();
    }

    protected override void ConfigureStrategies(ResiliencePipelineBuilder<HttpResponseMessage> builder, DownstreamRoute route)
    {
        ConfigureRetryStrategies(builder, route);

        ConfigureCircuitBreaker(builder, route);

        ConfigureTimeout(builder, route);
    }

    protected override ResiliencePipelineBuilder<HttpResponseMessage> ConfigureCircuitBreaker(ResiliencePipelineBuilder<HttpResponseMessage> builder, DownstreamRoute route)
    {
        return base.ConfigureCircuitBreaker(builder, route);
    }

    protected override ResiliencePipelineBuilder<HttpResponseMessage> ConfigureTimeout(ResiliencePipelineBuilder<HttpResponseMessage> builder, DownstreamRoute route)
    {
        return base.ConfigureTimeout(builder, route);
    }

    protected ResiliencePipelineBuilder<HttpResponseMessage> ConfigureRetryStrategies(ResiliencePipelineBuilder<HttpResponseMessage> builder, DownstreamRoute route)
    {
        var isQosOptionsExist = route.MetadataOptions.Metadata.TryGetValue(nameof(FileQoSOptions), out var retryOptions);

        if (!isQosOptionsExist || string.IsNullOrWhiteSpace(retryOptions))
        {
            _logger.LogWarning($"QoSOptions not found or empty for route: {GetRouteName(route)}.");
            return builder;
        }

        var retryMetadata = JsonSerializer.Deserialize<OcelotQoSOptions>(retryOptions);

        if (retryMetadata == null)
        {
            _logger.LogWarning($"Failed to deserialize QoSOptions for route: {GetRouteName(route)}.");
            return builder;
        }

        // Add Retry strategy only if the retry count is defined in QoSOptions and greater than 0
        if (retryMetadata.RetryCount <= 0)
        {
            _logger.LogWarning($"RetryCount {retryMetadata.RetryCount} for route: {GetRouteName(route)}.");
            return builder;
        }

        _logger.LogInformation($"Retry options for route: {GetRouteName(route)}: RetryCount={retryMetadata.RetryCount}, RetryDelay={retryMetadata.RetryDelay}");

        var info = $"Retry for the route: {GetRouteName(route)}: ";
        var strategyOptions = new RetryStrategyOptions<HttpResponseMessage>
        {
            MaxRetryAttempts = retryMetadata.RetryCount,
            Delay = TimeSpan.FromMilliseconds(retryMetadata.RetryDelay),
            ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                         .Handle<Exception>(ex =>
                         {
                             var shouldRetry = RetryableExceptions.Contains(ex.GetType());

                             if (shouldRetry)
                             {
                                 _logger.LogDebug($"Retry triggered by exception: {ex.GetType().Name}");
                             }

                             return shouldRetry;
                         })
                        .HandleResult(response =>
                        {
                            var shouldRetry = this.ServerErrorCodes.Contains(response.StatusCode);

                            if (shouldRetry)
                            {
                                _logger.LogDebug($"Retry triggered by HTTP status code: {response.StatusCode}");
                            }

                            return shouldRetry;
                        }),

            OnRetry = args =>
            {
                _logger.LogWarning(info +
                    $"Retry attempt {args.AttemptNumber} after {args.RetryDelay.TotalMilliseconds} ms due to {args.Outcome.Exception?.Message ?? args.Outcome.Result?.StatusCode.ToString()}");
                return ValueTask.CompletedTask;
            },
        };

        return builder.AddRetry(strategyOptions);
    }

    private static readonly ImmutableArray<Type> RetryableExceptions =
         [
            typeof(SocketException),
            typeof(HttpRequestException),
            typeof(TimeoutRejectedException),
            typeof(BrokenCircuitException),
            typeof(RateLimitRejectedException),
            typeof(TimeoutException),
         ];

    protected override HashSet<HttpStatusCode> ServerErrorCodes => DefaulRetrytServerErrorCodes;

    protected static readonly HashSet<HttpStatusCode> DefaulRetrytServerErrorCodes = new HashSet<HttpStatusCode>
            {
                HttpStatusCode.InternalServerError,
                HttpStatusCode.NotImplemented,
                HttpStatusCode.BadGateway,
                HttpStatusCode.ServiceUnavailable,
                HttpStatusCode.GatewayTimeout,
                HttpStatusCode.HttpVersionNotSupported,
                HttpStatusCode.VariantAlsoNegotiates,
                HttpStatusCode.InsufficientStorage,
                HttpStatusCode.LoopDetected,
            };
}

[aftabkajal]

I have tested Retry, Timeout, and Circuitbreaker. All of that works fine for me.

@raman-m thanks for your endless support and cooperation.