Resource exhaustion produces misleading error message

[callionica]

--- EDIT ---
See this comment for the real issue:
#8155 (comment)

--- ORIGINAL ---
A client app that that does many fetches to a web server eventually hits "error trying to connect: dns error: failed to lookup address information: nodename nor servname provided, or not known". Once this error is hit, no future fetch will succeed until the app is restarted. Restarting the app allows fetches to succeed instantly so it's definitely not a persistent problem in the network - it's limited to the Deno app - and the app includes carefully written retry code, so it's definitely not a case of getting lucky or unlucky with the timing.

As an example, the client app started at 19:46, did one fetch every 4-6 seconds, and hit the error at 20:00.
The length of time to see the error seems variable.

Clues or red herrings that could explain why you haven't see in your own testing:
The server is a CDN
The client is on a high latency satellite connection

deno 1.5.0
v8 8.7.220.3
typescript 4.0.3

(Also present on Deno 1.4.6)

[lucacasonato]

You are probably being rate limited by your DNS provider. We do not cache DNS queries at the moment. It might sound counter intuitive, but can you try increase your request rate? That might keep the HTTP connection open for longer, reducing the amount of DNS queries needed. Also are you using Deno.HttpClient? If so are you creating a new one for each request?

[callionica]

I'm using the fetch function and I'm just giving it a URL, so not touching Deno.HttpClient directly.

If the DNS is getting hit at the same rate as the fetches being issued, that's still only 1 request every 4 seconds. Not saying that someone wouldn't rate limit that, but I wonder. It also makes me wonder why closing the app and restarting wouldn't still be affected by the DNS rate limit if one had been imposed.

Can't really increase the request rate because while the client is self-limiting currently, it would likely get rate limited by the server on the actual request if I increase the rate.

[lucacasonato]

Makes sense. I can't think of why this may happen other than the dns caching.

[callionica]

Do you have a long-running test that exercises fetch that I can look at to compare with my code?

[lucacasonato]

Not really no.

[callionica]

One more potentially useful piece of info: the server supports HTTP2.

Are there any long-running Deno tests?

I suspect the bug is in the bowels of reqwest or even lower. If we assume that the reported error of a DNS problem is accurate, it's clearly not right that there's no way to get past it except to kill the process. Feels like a pooled connection becomes tainted by an initial error.

I'm working around the issue by letting the client app crash on the first error and adding a monitor app that respawns it.

Do you know if there's an environment variable or something that I can set that will force a Deno app to use only HTTP 1.1 so I can see if it's HTTP2-related?

[callionica]

New info: is failing consistently at exactly 120 consecutive requests to same server.

[callionica]

@lucacasonato Maybe you already found this bug back in July: seanmonstar/reqwest#976

Looks pretty similar: A limit of 120 on HTTP2 requests. Doesn't appear to require the requests to be made quickly though.

[lucacasonato]

@callionica they are different errors. The already reported one definitely requires it to happen quickly (it happens with ~100 simultaneous connections).

[callionica]

Just doing a run that isn't showing the error after I added calls to Process.close and Process.stderr.close. Maybe it was a handle/resource limit showing up in a random place. Is the Deno resource table limited in size?

[callionica]

Here's the final bug report with a small repro:

Title:
Resource exhaustion (file handle exhaustion) produces misleading error message from fetch ("error: Uncaught (in promise) Http: error sending request for url (https://doc.deno.land/builtin/stable#fetch): error trying to connect: dns error: failed to lookup address information: nodename nor servname provided, or not known")

Summary:
Putting a Deno app at the edge of resource exhaustion by failing to close Process handles and then calling fetch causes fetch to fail with a misleading error message pointing the finger at a DNS error.

Steps to reproduce:

Run the code below on Deno 1.50 on Mac OS X 10.13.6

const url = "https://doc.deno.land/builtin/stable#fetch";

for (let i = 0; i < 3408; ++i) {
    try {
        const p = Deno.run({
            cmd: ["echo", `${i}`],
            stdout: "piped",
            stderr: "piped",
        });


        const { success } = await p.status();
    } catch (e) {
        console.log(i);
        throw e;
    }
}

console.log(Deno.resources());
const response = await fetch(url);

Actual result:
The call to fetch results in the following error message: dns error: failed to lookup address information: nodename nor servname provided, or not known

error: Uncaught (in promise) Http: error sending request for url (https://doc.deno.land/builtin/stable#fetch): error trying to connect: dns error: failed to lookup address information: nodename nor servname provided, or not known
    at processResponse (core.js:226:13)
    at Object.jsonOpAsync (core.js:244:12)
    at async fetch (deno:op_crates/fetch/26_fetch.js:1273:29)
    at async file:///Users/user/Documents/github/project/repro.ts:20:18

Desired result:
Either the call to fetch should complete successfully or, if it must fail, it should not produce an error message that points you in the wrong direction.

If it is impractical to change fetch to produce a better error (or in addition), here are some further suggestions:

1. Perhaps Deno could check for resource exhaustion on process exit and report the issue.
2. The API for Deno.run could be designed so that process handles cannot be leaked (for example, requiring Process.detach to be called or requiring an additional argument to Deno.run if the developer's intention is for the process to live beyond the lifetime of the variable, and otherwise cleaning up the resources automatically).
3. The documentation for Deno.run and for Process could describe how to fully clean up the resources. (Related bug Deno.Process close()/status()/kill() #6373)

Notes:
If a later version of Deno or a different operating system does not reproduce the error, replace the value 3408 with a large number until you hit the catch that logs i then replace 3408 in the repro code with the number for i that was logged and rerun the repro.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Jason5Lee]

I met a similar error when using a tool I wrote in Rust with tokio, any idea about how could I fix it?