First request to downstream service extremely slow

[hkoestin]

Expected Behavior

All calls to the downstream services should be similarly fast.

Actual Behavior

• First request takes around 2secs
• Following requests take only a few ms
• After some period of time, e.g. 20secs, the next request will again take around 2secs
• This issue seems to be similar to httpclient is slow without rules #1618 and Why slow response in .NET 6 vs IIS ? #1591

Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000001, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/orders/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000001, previousRequestId: no previous request id, message: No authentication needed for /api/orders/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000001, previousRequestId: no previous request id, message: /api/orders/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000001, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56522/api/orders/Get?id=1236
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - - - 200 472 application/json;+charset=UTF-8 2242.4820ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000002, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/invoices/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000002, previousRequestId: no previous request id, message: No authentication needed for /api/invoices/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000002, previousRequestId: no previous request id, message: /api/invoices/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000002, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56521/api/invoices/Get?id=P1234
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - - - 200 812 application/json;+charset=UTF-8 2091.1109ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000003, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/orders/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000003, previousRequestId: no previous request id, message: No authentication needed for /api/orders/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000003, previousRequestId: no previous request id, message: /api/orders/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000003, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56522/api/orders/Get?id=1236
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - - - 200 472 application/json;+charset=UTF-8 8.2941ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000004, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/invoices/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000004, previousRequestId: no previous request id, message: No authentication needed for /api/invoices/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000004, previousRequestId: no previous request id, message: /api/invoices/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000004, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56521/api/invoices/Get?id=P1234
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - - - 200 812 application/json;+charset=UTF-8 9.0330ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000005, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/orders/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000005, previousRequestId: no previous request id, message: No authentication needed for /api/orders/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000005, previousRequestId: no previous request id, message: /api/orders/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000005, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56522/api/orders/Get?id=1236
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - - - 200 472 application/json;+charset=UTF-8 5.1499ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000006, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/invoices/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000006, previousRequestId: no previous request id, message: No authentication needed for /api/invoices/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000006, previousRequestId: no previous request id, message: /api/invoices/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000006, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56521/api/invoices/Get?id=P1234
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - - - 200 812 application/json;+charset=UTF-8 9.9788ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000007, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/invoices/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000007, previousRequestId: no previous request id, message: No authentication needed for /api/invoices/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000007, previousRequestId: no previous request id, message: /api/invoices/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000007, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56521/api/invoices/Get?id=P1234
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - - - 200 812 application/json;+charset=UTF-8 2089.3012ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000008, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/orders/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000008, previousRequestId: no previous request id, message: No authentication needed for /api/orders/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000008, previousRequestId: no previous request id, message: /api/orders/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000008, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56522/api/orders/Get?id=1236
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - - - 200 472 application/json;+charset=UTF-8 2092.2855ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000009, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/orders/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000009, previousRequestId: no previous request id, message: No authentication needed for /api/orders/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000009, previousRequestId: no previous request id, message: /api/orders/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:00000009, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56522/api/orders/Get?id=1236
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/orders/Get?id=1236 - - - 200 472 application/json;+charset=UTF-8 4.5739ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - -
info: Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware[0]
      requestId: 0HMNPD4PNATD2:0000000A, previousRequestId: no previous request id, message: EndpointRateLimiting is not enabled for /api/invoices/{route}
info: Ocelot.Authentication.Middleware.AuthenticationMiddleware[0]
      requestId: 0HMNPD4PNATD2:0000000A, previousRequestId: no previous request id, message: No authentication needed for /api/invoices/Get
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HMNPD4PNATD2:0000000A, previousRequestId: no previous request id, message: /api/invoices/{route} route does not require user to be authorized
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HMNPD4PNATD2:0000000A, previousRequestId: no previous request id, message: 200 (OK) status code, request uri: http://localhost:56521/api/invoices/Get?id=P1234
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/api/invoices/Get?id=P1234 - - - 200 812 application/json;+charset=UTF-8 8.7457ms

Steps to Reproduce the Problem

1. Installed Ocelot for a default ASP.NET Core (net6) project
2. Configured the defaults in Program.cs etc.
3. Adding just two routes to the configuration

Specifications

• Version: tested with various combinations of 17.0.1 and 18.0 with net5.0, net6.0, net7.0
• Platform: win10x64
• Subsystem:

[hkoestin]

It seems, that it is related to the DNS.
When you have configurations using the IP addresses of the upstreams and downstreams, calls would be much faster.

[jorosa21]

Is this issue resolved? I have an issue in which on first gateway call to services it took around 2 secs then on second call it will be much faster but after sometime it will again go 2 secs and then next call would be much faster again. This is currently deployed on IIS. I did setup the IIS like on this article
https://www.leansentry.com/guide/reset-restart-recycle-iis/max-application-pool-warmup#install-the-application-initialization-module

[hkoestin]

After doing some debugging, I figured out, that it seems to be related to the DNS.

Also compare these threads:

• https://stackoverflow.com/questions/71638441/how-to-improve-slow-dns-with-httpclient
• https://stackoverflow.com/questions/37384945/how-to-force-ipv6-or-ipv4-for-httpwebrequest-or-webrequest-c-sharp/70475741#70475741

You could give it a try and use the SocketsHttpHandler to solve that issue.

When I started to configure all downstreams using the IP, calls were much faster.