Frequent transport-level SqlClient exception on Linux Rocky 9.5 Minimal ISO - A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35

[mbozier]

Describe the bug

Running a simple C# .NET 8.0 Console program on a newly installed VMware VM running Rocky9.5 Minimal ISO with prerequisites
dotnet-runtime-8.0, aspnetcore-runtime-8.0 and mssql-server-selinux x86_64 16.0.4165.4-7 all installed via a script we see frequent
occurrences of the following exception.

Exception: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35 - An internal exception was caught)

Note that the simple console program simply attempts to provide command text of increasing length to the SqlCommand ExecuteNonQuery() method.
Since it is not the first request which fails, the basic SqlClient -> SQL Server communication appears to be configured correctly.

I have attached a log of the execution of the console program on my Rocky9.5 VM - sqlclient_test_script_log_28-11-2024-17-29-11.log

Exception message:
Exception: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35 - An internal exception was caught

Stack trace:

Stack    at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean isAsync, Int32 timeout, Boolean asyncWrite)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Mfb.SqlClientError.Program.TestGeneratedScript(SqlConnection connection, Int32 numIntProps)
   at Mfb.SqlClientError.Program.TestGeneratedScript(SqlConnection connection, Int32 numIntProps)
Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35 - An internal exception was caught)
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.Read(Span`1 buffer)
   at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Read(Span`1 buffer)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at Microsoft.Data.SqlClient.SNI.SNITCPHandle.Receive(SNIPacket& packet, Int32 timeoutInMilliseconds)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean isAsync, Int32 timeout, Boolean asyncWrite)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()

To reproduce

1. Install a new VWware VM with Rocky9.5 Minimal ISO.
   VM configuration used was VMware VM with 8 CPUs, 8 Cores per Socket, 32 GB Memory, 100GB Hard Disk Thin Provisioned, EFI Boot
   As part of the Linux installation, create a user 'administrator' and select the 'Make this user administrator' option. Ensure the VM has internet access to allow subsequent install of SQL Server, .NET etc.

2. Use the supplied SqlTest Visual Studio Solution to produce a release build and publish as a single file, self-contained linux-x64 executable for .NET8.0

3. Once installation is complete, log into the new VM using the administrator user. Enter the command nmcli to determine the IP address allocated if using DHCP.
   Copy the SqlTest executable built in step 2 along with the uploaded bash script sqlclient_test.sh to the home directory of the administrator user on the Rocky 9.5 VM

4. Run following commands from a terminal on the VM
   chmod +x ./sqlclient_test.sh
   ./sqlclient_test.sh

5. This script performs the following
   Installs dotnet-runtime-8.0, aspnetcore-runtime-8.0 and mssql-server-selinux x86_64 16.0.4165.4-7
   Configures sa password on SQL Server installed to match that used in SqlTest executable
   Opens the required firewall ports.
   Runs SqlTest program in a loop.
   Each execution of SqlTest creates a new DB MFB_TESTDB
   Sends SQL requests of increasing length to attempt to create a table in that DB.
   Deletes DB MFB_TESTDB if the program executes without error.

6. Results should be logged in a file in the administrator home directory sqlclient_test_script_log*.log

Attached
SqlTest.zip - Simple C# .NET8.0 Console solution using Sql Server System Administrator user to send text of increasing length to the SqlCommand ExecuteNonQuery() method
sqlclient_test.zip - Bash script to install dotnet-runtime-8.0, aspnetcore-runtime-8.0 and mssql-server-selinux x86_64 16.0.4165.4-7 and configure firewall ports
sqlclient_test_script_log_28-11-2024-17-29-11.log - Results of execution on my Rocky9.5 VM

sqlclient_test.zip
sqlclient_test_script_log_28-11-2024-17-29-11.log
SqlTest.zip

Expected behavior

Program should run without transport- error

Further technical details

Microsoft.Data.SqlClient version: (5.2.2)
.NET target: (.NET 8.0)
SQL Server version: (mssql-server-selinux x86_64 16.0.4165.4-7)
Operating system: (Rocky Linux 9.5 Minimal ISO running on VMware VM with 8 CPUs, 8 Cores per Socket, 32 GB Memory, 100GB Hard Disk Thin Provisioned, EFI Boot)

Additional context

[benrr101]

Hi @mbozier Sorry for the delay in getting to back to you on this issue. We're currently in release mode so it will likely be a little while before I can dedicate time to looking into this. Technically Rocky Linux isn't officially supported for .NET framework, but I'm still going to look into this for you. Since Rocky is supposed to be bug-for-bug compatible with RHEL, I don't see any reason why we can't support it. (I'm also a Rocky Linux user, so I'd hope the same courtesy would be extended if I were in your situation). The repro situation looks really robust, thank you for that.

I'll let you know when I have another update 👍

[mbozier]

Hi @benrr101 Many thanks for your response. Actually, we do believe this issue also happens in RHEL 9.5

To check this out, since I don't have a RHEL subscription, I downloaded RHEL 9.5 Binary DVD ISO and then installed packages for dotnet-runtime-8.0, aspnetcore-runtime-8.0 and mssql-server-selinux x86_64 16.0.4165.4-7 which I had pulled from the Rocky Repos.
Again, trying to install our product, which runs a number of SQL scripts as part of the installation, produced the same exception.

Other relevant information is that we have not seen this issue on Rocky Linux 9.4, only on Rocky Linux 9.5