bug: reqwest always reuses http/2 connections

[lucacasonato]

What is the issue?

When quickly creating a large amount of requests (150) in a short amount of time (within a few 100ms) to a single origin that serves HTTP/2, some requests will fail with the error http2 error: protocol error: refused stream before processing any application logic. I assume this is because all requests are multiplexed onto a single HTTP/2 connection, which causes the connection to exceed the maximum amount of concurrent streams that it can handle (SETTINGS_MAX_CONCURRENT_STREAMS). This would explain why this only happens at more than ~100 requests, because most clients and servers set SETTINGS_MAX_CONCURRENT_STREAMS to ~100 as recommended by the HTTP/2 spec.

What do I expect?

Before reaching the maximum amount of concurrent streams for a connection, reqwest starts another HTTP/2 connection to that origin to handle requests with.

or

Reqwest buffers requests until the single HTTP/2 connection has free stream capacity again to handle the request.

How to reproduce?

A full reproduction can be found here: https://gist.github.com/lucacasonato/3dd98a7eb81516d20fd464a9c52a299f. Clone the gist, move main.rs to src/main.rs and run cargo run

What this does:

1. Create a reqwest::Client
2. Create many requests (about 120) to a single origin in a very short amount of time
3. Check the response for each request and log it

You will see that out of 120 requests, about 15 will fail with the error http2 error: protocol error: refused stream before processing any application logic. You might need to increase the amount of requests to the origin if you don't see this request.

Meta

Reqwest: 0.16.4
OS: Ubuntu 20.04
Rust: 1.45.0 stable

cc @ry

[seanmonstar]

most clients and servers set SETTINGS_MAX_CONCURRENT_STREAMS to ~100 as recommended by the HTTP/2 spec.

It's unfortunate that many things do that by default. The spec warns people should try to never set it lower than 100, but not to default cap at 100 XD

Before reaching the maximum amount of concurrent streams for a connection, reqwest starts another HTTP/2 connection to that origin to handle requests with.

The HTTP2 spec claims that a client shouldn't open multiple connections. I know some clients do this anyway, but I'm not sure that's the best option.

Reqwest buffers requests until the single HTTP/2 connection has free stream capacity again to handle the request.

This could possibly work, but has a problem of an unbounded buffer. Those are typically not great, but if we use a bounded buffer, then eventually requests won't be able to be buffered and backpressure will be needed by the user anyways...

One option is to use something like tower::retry and define the retry policy to look for errors with the REFUSED_STREAM error code.

[RoXuS]

any news on this issue?

[dswij]

See hyperium/h2#731.

To summarize:

1. the http/2 spec allows a client to open as many stream as possible before getting max_concurrent_stream.
2. most servers implement 100 max as a default, although this is the minimum recommended by the http/2 spec
3. hyper, and subsequently reqwest does not have a way for client to configure this at the moment

After hyperium/h2#731:

1. The default will be set in hyper to 100. The version that this will be shipped with is still TBD.
2. Any user of the hyper client can configure this themselves.

[tristan-morris]

This is shipped in hyper 1.2.0.

Just realised this is likely causing me problems interacting with various Google APIs hyper::Error(Http2, Error { kind: GoAway(b"client_misbehavior", PROTOCOL_ERROR, Remote) })

[seanmonstar]

Thanks for the reminder, yea so this should be fixed in 0.12 (released today or tomorrow, I hope).