[🐛 Bug]: IWebDriver.Manage().Network.StartMonitoring() causes file uploads to fail

[MJB222398]

What happened?

I have a project that uploads files into the web app under test on the Chrome browser. With Selenium 4.23 this was working flawlessly. I tried to update to Selenium 4.27 and with no changes other than this NuGet package version this stopped working.

The implementation of the web app under test is such that when files are uploaded this is done in 4MB chunks. If the file size is less than 4MB then the upload still works as it used to with 4.23. However if the size is greater than 4MB such that it is chunked then the upload will not work anymore.

Narrowing it down I eventually find that it is a call to IWebDriver.Manage().Network.StartMonitoring() that appears to be the cause. If I remove this line it works fine, with it it will not do the upload.

How can we reproduce the issue?

Here is a simple example to see the issue. Note that this is not the site I was testing, I cannot give you access to that, but this one appears to show the issue too. Note here I don't know the chunking size for this site but I was observing failure using a file that was 80MB and seeing success with a file that was 1KB.

IWebDriver driver = null;
try
{
    driver = new ChromeDriver();
    driver.Manage().Network.StartMonitoring().GetAwaiter().GetResult();
    driver.Navigate().GoToUrl("https://tus.io/demo");
    Thread.Sleep(1000);
    driver.FindElement(By.Id("P0-0")).SendKeys("C:\\Users\\YourName\\file.file");
    Thread.Sleep(22000);
}
finally
{
    driver?.Quit();
}

Relevant log output

Trace level driver logs are 200MB so cannot upload or paste in here. Please download from: https://www.filemail.com/d/xbkqrdjzsxgdvms

Operating System

Windows 10

Selenium version

4.27.0

What are the browser(s) and version(s) where you see this issue?

Chrome 131.0.6778.140

What are the browser driver(s) and version(s) where you see this issue?

ChromeDriver 131.0.6778.140

Are you using Selenium Grid?

I was but seems not necessary to reproduce the issue

[github-actions]

@MJB222398, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[nvborisenko]

Hi @MJB222398 , I remember you were involved in internal logs (if my memory doesn't lye me): https://www.selenium.dev/documentation/webdriver/troubleshooting/logging/ - can you please provide?

[MJB222398]

@nvborisenko apologies I can provide yes in a couple of hours. From the driver logs or the selenium container logs? However it should be straightforward for you to replicate. But yes that was me I think

[nvborisenko]

I have executed your code, what is expected behavior? It was success on my side.

[MJB222398]

I have executed your code, what is expected behavior? It was success on my side.

So if you use large file size (100mb say) and use the code given the upload should fail. If you comment out the network monitoring line then it should do the upload successfully

[nvborisenko]

I tried to SendKeys(...) the file 1.42 GB size - success.

[MJB222398]

That is very strange. If I run this:

IWebDriver driver = null;
try
{
    driver = new ChromeDriver();
    driver.Manage().Network.StartMonitoring().GetAwaiter().GetResult();
    driver.Navigate().GoToUrl("https://tus.io/demo");
    Thread.Sleep(1000);
    driver.FindElement(By.Id("P0-0")).SendKeys("C:\\Users\\xxxxx\\Downloads\\goldfinch4_Trim.mp4");
    Thread.Sleep(22000);
}
finally
{
    driver?.Quit();
}

Then 100% of the time it will fail to do the upload. The screen will be stuck like this:

If instead I run this:

IWebDriver driver = null;
try
{
    driver = new ChromeDriver();
    //driver.Manage().Network.StartMonitoring().GetAwaiter().GetResult();
    driver.Navigate().GoToUrl("https://tus.io/demo");
    Thread.Sleep(1000);
    driver.FindElement(By.Id("P0-0")).SendKeys("C:\\Users\\xxxxx\\Downloads\\goldfinch4_Trim.mp4");
    Thread.Sleep(22000);
}
finally
{
    driver?.Quit();
}

Then 100% of the time the upload succeeds:

I have tried this many times on 2 machines on 2 different networks and the results are always as above.

[MJB222398]

@nvborisenko Trace level driver logs uploaded to https://www.filemail.com/d/xbkqrdjzsxgdvms

[nvborisenko]

Got it, reproduced locally. It doesn't depend on file size. I used 300KB file and it failed. Silently...

The last remote message was CDP Fetch.requestPaused event (got it on http level, with content of our file), and then just nothing. I guess something happened at broker level, when probably we cannot process this event (probably deserialize?).

[MJB222398]

Ok nice, glad I'm not crazy haha. So what can be done about it? You want to add some defensive code in the event handling/deserialisation?

[nvborisenko]

I caught it:

21:20:15.221 TRACE DevToolsSession: CDP RCV << {"method":"Fetch.requestPaused","params":{"requestId":"interception-job-27.0","request":{"url":"https://tusd.tusdemo.net/files/d98a9ddd6c3ff97e11702ac5de5834fa+jWEe0kWhtgB76yK8H8sOMJTKe8YkgD.YdEsgHbGinT8yu5T4_y06otqVdQ4CpcYu9naKLFXjZJrPceKYXORvQsNjy6Jh1HQLR9JabneigVwYIzvDJdmKVtesHSxTNFZH","method":"PATCH","headers":{"Accept":"*/*","Content-Type":"application/offset+octet-stream","Origin":"https://tus.io","Referer":"https://tus.io/","Tus-Resumable":"1.0.0","Upload-Offset":"0","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36","X-Request-ID":"65500868-cb54-4da1-91ea-8b20d48e15db","sec-ch-ua":"\"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\""},"postData":"MZ...truncated...300KB...sBhlSvzaYd6AA=="}],"initialPriority":"High","referrerPolicy":"strict-origin-when-cross-origin"},"frameId":"89D7C96F5D38D5A7D3B241D276D0F2C7","resourceType":"XHR","networkId":"28664.36"},"sessionId":"4BA6A1A6EAF683025962B3925A89C493"}
21:20:15.242 ERROR DevToolsSession: Unexpected error occured while processing message: System.InvalidOperationException: Cannot read incomplete UTF-16 JSON text as string with missing low surrogate.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_ReadIncompleteUTF16()
   at System.Text.Json.JsonReaderHelper.TryUnescape(ReadOnlySpan`1 source, Span`1 destination, Int32 idx, Int32& written)
   at System.Text.Json.JsonReaderHelper.Unescape(ReadOnlySpan`1 source, Span`1 destination, Int32& written)
   at System.Text.Json.JsonDocument.WriteString(DbRow& row, Utf8JsonWriter writer)
   at System.Text.Json.JsonDocument.WriteComplexElement(Int32 index, Utf8JsonWriter writer)
   at System.Text.Json.JsonDocument.WriteElementTo(Int32 index, Utf8JsonWriter writer)
   at System.Text.Json.Nodes.JsonObject.WriteTo(Utf8JsonWriter writer, JsonSerializerOptions options)
   at System.Text.Json.Nodes.JsonNode.WriteToPooledBuffer(JsonSerializerOptions options, JsonWriterOptions writerOptions, Int32 bufferSize)
   at System.Text.Json.Nodes.JsonNode.ToString()
   at OpenQA.Selenium.DevTools.DevToolsSession.ProcessMessage(String message) in D:\SeleniumHQ\selenium\dotnet\src\webdriver\DevTools\DevToolsSession.cs:line 593
   at OpenQA.Selenium.DevTools.DevToolsSession.MonitorMessageQueue() in D:\SeleniumHQ\selenium\dotnet\src\webdriver\DevTools\DevToolsSession.cs:line 541

(modified postData property).

[nvborisenko]

The issue is not reproducible in Selenium v4.23 where Newtonsoft.Json is used for deserialization, and begins to be reproducible in v4.24 where System.Text.Json is used.

[nvborisenko]

The remote version is still the same, it is chrome 131. So it is really related to deserialization. I blame UTF-16 mentioned in the exception, most likely it should be UTF-8. We should understand clearly this diff, and most likely how to force System.Text.Json to use UTF-8.

[MJB222398]

Ah yes been bitten by that migration myself. Had to write a lot of unit tests using Verify to check that the serialisation did not change. But it is hard to cover everything.

[nvborisenko]

@MJB222398 please share your experience, it would be helpful.

When I see string representation of json in VS Code, it cannot be formatted properly.

[nvborisenko]

Perfect, seems https://learn.microsoft.com/en-us/dotnet/api/system.text.encodings.web.javascriptencoder.unsaferelaxedjsonescaping?view=net-9.0&viewFallbackFrom=net-6.0-pp should help.

[nvborisenko]

Didn't help, it works with textual files, but not with binary files.

BUT
postData property is deprecated: https://chromedevtools.github.io/devtools-protocol/tot/Network/#type-Request where postDataEntries is suggested to be used instead. And seems we can deserialize it (I should double check).

What options I see:

1. Hard: understand how Newtonsoft.Json encoding works and try to mimic it in System.Text.Json.
2. Easy: Don't generate deprecated postData property at all and migrate to new.

[MJB222398]

That migration was quite a few years ago now I'm afraid , all I remember is the importance and usefulness of the unit tests rather than what the differences actually were and how to resolve them.

Seems odd that postData is deprecated while postDataEntries is still labeled as 'Experimental'.