The MARS TDS header contained errors using ASP.NET Core and EF Core connecting to Azure SQL Server

[AfsanehR-zz]

Copied from here:

I have an ASP.NET Core app [Microsoft.AspNetCore.App 2.1.4] with EF Core 2.1.4 [DbContext pooling enabled] and data stored on an Azure SQL database.

Occasionally [once in 1-2 days] I get unrecoverable error stating System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

Once I restart the app, everything resumes working until it happens again.

Probably import to note is that I don't have MultipleActiveResultSets enabled in my connection string at all, which makes the issue even more strange.

Has anyone else noticed something similar? Are there any ways I can trace the problem?

Stack trace:

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteReader(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Inviton.Web.Portal.Controllers.EventListController.GetEvents(BL bl, Int32 venueId, Int32 promoterId) 

[brunolau]

Regarding the stackoverflow issue addiotional questions asked by Afsaneh:

Unfortunately can't post the csproj file contents...

• I'm using EF Core 2.1.4 with Linq2Db provider (https://github.com/linq2db/linq2db.EntityFrameworkCore) for some of the queries that are poorly translated by the EF Core.., I'm however also referencing the System.Data.SqlClient 4.5.1 package in on of the solution's projects

• I'm hosting on Azure Websites, so I assume it's some sort of IIS

• I'll have to log this more explicitly. Is logging the "State" field sufficient for you ?

[brunolau]

Here's the Dump of the SqlError object:

Class:
16

Errors:
System.Data.SqlClient.SqlError: The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

Error code:
-2146232060

Line number:
1

Number:
4011

SQL Procedure:
None

Source:
Core .Net SqlClient Data Provider

State:
3

[stap123]

I had a random spate of these errors appear in my ASP.NET application today. I use ASP.NET Core 2.1, EF Core 2.1 and Azure SQL Database. They occurred in a small time window and once I stopped and started the web app the errors seemed to stop.

I think I queried successfully from SSMS while the errors were happening but couldn't say for sure that it was at the same time. If it happens again I'll query in SSMS to confirm that I can.

Is there anything I can provide that will help with a solution to this? (I have application insights and database logging turned on for my application)

NOTES:
MARS is explicitly set to off in my connection string (I use the one Azure Provides via the portal UI)

[AfsanehR-zz]

@stap123 @brunolau Would you be able to provide us a minimal repro sample application? Thanks.

[stap123]

@afsanehr I wouldn't even know where to start, I've so far only seen it once today and restarting my application made it stop happening. I will keep my eyes open and if it starts happening in a pattern of some kind I'll try and put one together but at the moment it seems random so I've got no starting point for a repro.

I realise that's useless to you as well though haha

I have got about 130x exceptions logged in my App Insights at the moment but I doubt the stack trace would be enough for you?

It also looks like the error occurred on ALL queries no matter where they originated from so it doesn't appear to be a query specific issue.

[brunolau]

I confirm the same - errors occurring at random queries, random time. I've tried to stress-test the application by running 30 threads that was making DB queries in parallel, but that didn't reproduce the issue either.

I've tried to use the SSMS query while the error was occurring and it worked, I've also tried to deploy to another azure release slot at the time the issue was occurring and it worked.

What I've temporarily ended up with was writing a middleware that once detects this particular error [based on the err message], automatically restarts the web app. As the issue occurs 1-2 times a day, it's acceptable, but much better would be having the issue solved.

What connects both mine and Adam's issue is the ASP.NET Core 2.1 + EF Core 2.1 + Azure SQL DB combo, but I cannot even say what part of the chain to blame.

[AfsanehR-zz]

ok thanks @brunolau @stap123 . We will investigate this with the configurations you mentioned, although I think it would be highly unlikely we hit the same issue as we need to know what queries are running. Will get back to you on this. Could you also please give us the output of dotnet --info?

[brunolau]

Here you go, obtained from the Azure console

.NET Core SDK (reflecting any global.json):
Version: 2.2.100
Commit: 51868761f2

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x86
Base Path: D:\Program Files (x86)\dotnet\sdk\2.2.100\

Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed

.NET Core SDKs installed:
1.1.10 [D:\Program Files (x86)\dotnet\sdk]
2.1.403 [D:\Program Files (x86)\dotnet\sdk]
2.1.500 [D:\Program Files (x86)\dotnet\sdk]
2.2.100 [D:\Program Files (x86)\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.12 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[stap123]

Here's mine 👍

.NET Core SDK (reflecting any global.json):
Version: 2.2.100
Commit: 51868761f2

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x86
Base Path: D:\Program Files (x86)\dotnet\sdk\2.2.100\

Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed

.NET Core SDKs installed:
1.1.10 [D:\Program Files (x86)\dotnet\sdk]
2.1.403 [D:\Program Files (x86)\dotnet\sdk]
2.1.500 [D:\Program Files (x86)\dotnet\sdk]
2.2.100 [D:\Program Files (x86)\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.12 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[hackfaq]

Hello, we have experiensed same problem:

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
at System.Data.SqlClient.SqlCommand.<>c.b__122_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in C:\projects\dapper\Dapper\SqlMapper.Async.cs:line 419

We use

• Dapper 1.50.5
• .NETCOREAPP 2.1.2
• Runtime in docker: FROM microsoft/dotnet:2.1-aspnetcore-runtime AS final

How to reproduce:
When we experienced problem: our database log file was full, and we got exception: the transaction log for database '' is full due to "LOG_BACKUP", after that exception all queries in .net core app was with errors.

When we increased log file, .net 4.7 apps starts to work, but .net core app continue to throw exception. Only restart of application helped

[stap123]

I got this again this morning between 07:38:09 and 07:39:22.

Not sure if that's useful to help you track anything down or not. (I did nothing to any of my services to stop it happening but I then have successful requests being processed at 07:40:06)

[brunolau]

I managed to temporarily auto-mitigate the issue by creating custom Entity framework execution strategy that once encounters the MARS error, closes the connection, clears the pool and reopens. Since I've introduced this workaround, I've been running errorless for 3 days.

To sum it up. Once the error is encountered, closing & reopening the error connection + clearing the connection pool seems to re-establish working order

[PolitovArtyom]

We are getting the same problem.
Errors starts occuring in same moment on different servers. Application pool restart helps,

Application Server
Microsoft Windows Server 2012 R2 Standard 6.3.9600 Build 9600
dotnet host Version: 2.1.5 Commit: 290303f510
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore
Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore
Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Database
Microsoft SQL Server 2017 (RTM-CU11) (KB4462262) - 14.0.3038.14 (X64) Sep 14 2018 13:53:44 Copyright (C) 2017 Microsoft Corporation Enterprise Edition (64-bit) on Windows Server 2012 R2 Standard 6.3 (Build 9600: )

Application
Dapper 1.50.5
Microsoft.AspNetCore.App 2.1.5

[sepehr1014]

We're having the same issue. Any solutions?

[PolitovArtyom]

@afsanehr
Do you need some additional data for investigation?
I can try to get application and tcp dump while errors occuring

[AfsanehR-zz]

@PolitovArtyom Yes. That would be helpful.
@sepehr1014 @PolitovArtyom do you have the MARS enabled in your connection? Also if possible a repro would be useful.

[PolitovArtyom]

MARS is not enabled on client and disabled on server

[wboevink]

We're also having the same issue. With NetCore, EFCore and an Azure SQL db

[mshenoy83]

@brunolau could you maybe create a gist with the workaround.

[brunolau]

Sure, I will post it in a couple of days, I'm away from the PC for a couple of days and as soon as I get back, I'll do it

[mshenoy83]

Just 1 more thing. I've been facing this error for a few days now. I have a durable function which calls my Sql Server VM. The version of Sql Server configured was Sql Server 2017
(Microsoft SQL Server 2017 (RTM-CU13) (KB4466404) - 14.0.3048.4 (X64) Nov 30 2018 12:57:58 Copyright (C) 2017 Microsoft Corporation Standard Edition (64-bit) on Windows Server 2016 Datacenter 10.0 (Build 14393: ) (Hypervisor)

On a hunch I fired up a new VM, this time with Sql Server 2016. And so far my durable function seems to be working fine. I really throttled it because I was worried maybe the number of connections was causing the problem. So the function is running very slowly now but still it has no errors, so far.

[brunolau]

@mshenoy83 Sorry it took a bit longer, here goes my dirty hack. In order to use it, please check the "Custom execution strategy" section of this documentation https://docs.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency

class CustomExecutionStrategy:

using Brulasoft.Inviton.Data.Internal.SQL;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Storage;
using System;
using System.Data.SqlClient;

namespace Brulasoft.Inviton.Data.Internal
{
    /// <summary>
    /// Custom SQL execution
    /// </summary>
    internal class CustomExecutionStrategy : SqlServerRetryingExecutionStrategy
    {
        public CustomExecutionStrategy(ExecutionStrategyDependencies dependencies) : base(dependencies)
        {
        }

        protected override bool ShouldRetryOn(Exception exception)
        {
            if (base.ShouldRetryOn(exception))
            {
                return true;
            }

            if (exception is SqlException sqlException)
            {
                if (SqlRecoveryAttemptor.ShouldTryRecover(exception, ExceptionsEncountered.Count) && Dependencies != null && Dependencies.CurrentDbContext != null)
                {
                    var context = Dependencies.CurrentDbContext.Context;
                    if (context != null)
                    {
                        var sqlConn = context.Database.GetDbConnection() as SqlConnection;
                        if (sqlConn != null)
                        {
                            SqlRecoveryAttemptor.TryRecover(sqlConn, ExceptionsEncountered.Count);
                            return true;
                        }
                    }
                }
            }

            return false;
        }
    }
}

class SqlRecoveryAttemptor

using System;
using System.Collections.Generic;
using System.Data.Common;
using System.Data.SqlClient;
using System.Text;

namespace Brulasoft.Inviton.Data.Internal.SQL
{
    public class SqlRecoveryAttemptor
    {
        public const int RetryCount = 4;


        public static bool ShouldTryRecover(Exception ex, int retryAttemptsCount)
        {
            return retryAttemptsCount < RetryCount && IsMARSError(ex);
        }

        public static bool IsMARSError(Exception ex)
        {
            return ex.Message.Contains("MARS TDS header");
        }

        public static void TryRecover(SqlConnection sqlConn, int retryAttemptsCount)
        {
            if (retryAttemptsCount > 2)
            {
                System.Threading.Thread.Sleep(1500);
            }

            if (sqlConn.State != System.Data.ConnectionState.Closed)
            {
                TryClose(sqlConn);
                TryClearPool(sqlConn);
                TryOpen(sqlConn);
            }
            else
            {
                TryOpen(sqlConn);
                TryClose(sqlConn);
                TryClearPool(sqlConn);
                TryOpen(sqlConn);
                TryClose(sqlConn);
            }
        }

        private static void TryClearPool(SqlConnection conn)
        {
            try
            {
                SqlConnection.ClearPool(conn);
            }
            catch (Exception)
            {
            }
        }

        private static void TryClose(DbConnection conn)
        {
            try
            {
                conn.Close();
            }
            catch (Exception)
            {
            }
        }

        private static void TryOpen(DbConnection conn)
        {
            try
            {
                conn.Open();
            }
            catch (Exception)
            {
            }
        }
    }
}

[rcvink]

+1 Am experiencing the same problem, not sure how to fix it besides repeatedly restarting (based on comments above).

[mscrivo]

Same issue here since migrating to EF Core from EF6 in Dec 2018. We see this happen randomly on a single server (among many that are hitting the DB). We have gone 10-15 days without seeing it, then it just pops up and doesn't recover until the server is restarted.

Currently using EF Core 2.2.3 on Azure SQL DB

[brunolau]

I second to that - for us it started once we migrated from ASP.NET to ASP.NET Core

[dkershner6]

Just to chime in, I just experienced this issue in Azure Functions v2 (.Net Core 2.1). I am using EFCore v2.2.3 to connect to Azure SQL. The call I am making to the DB that was spawning the error was as basic as it gets, essentially a SELECT * into a table with 2 rows and 8 columns.

Since my deployment is a little different from the above, maybe this will help:
For this app, I have 3 different deployments that use the same shared library, that contains my EF Core Data Model.

• "External" deployment calls the DB very frequently to get basic info and input data. It errored 1,433 times over the past 12 hours (every call, so far as I can tell) on the first, most basic call, stopping the rest. A restart fixed the error, have not seen it in 20 minutes.
• "Data Manipulation" deployment calls the DB fairly frequently (100's per hour), but only errored 5 times in 12 hours, no need to restart (I looked closer, there were zero errors (but a separate error that contained the word "mars" - definitely unrelated)).
• "API" deployment never errored. It calls the DB on a sporadic, on demand basis.

Up until recently, this app lived on a Sql Server Web Deployment on a Ubuntu VM. I had never seen this particular error (but had another odd SSL error instead that exhibited similar behavior, with restart fixing).

I will also say that at that time I was using EF Core 2.1.4 as well, and only recently upgraded to 2.2.3.

Connection string is the same for all 3 deployments, the MARS section is: MultipleActiveResultSets=False;

It appears to have started during a period of high DTU use (about 40 min at 90%+). This is not an infrequent occurrence, however, and has actually happened far less over the past day.

Hopefully that helps.

[divega]

As recently announced in the .NET Blog, focus on new SqlClient features an improvements is moving to the new Microsoft.Data.SqlClient package. For this reason, we are moving this issue to the new repo at https://github.com/dotnet/SqlClient. We will still use https://github.com/dotnet/corefx to track issues on other providers like System.Data.Odbc and System.Data.OleDB, and general ADO.NET and .NET data access issues.

[nurhat]

We have same problem for the following run time env, currently solved the problem solution provided by @brunolau

Application Server
Microsoft Windows Server 2016
.NET Core runtimes installed:
Dotnet core runtime 2.1.5
Database
Microsoft SQL Server 2017

Application
Microsoft.EntityFrameworkCore 2.1.4
Microsoft.AspNetCore.App 2.1.5

[mdhalgara]

I managed to temporarily auto-mitigate the issue by creating custom Entity framework execution strategy that once encounters the MARS error, closes the connection, clears the pool and reopens. Since I've introduced this workaround, I've been running errorless for 3 days.

To sum it up. Once the error is encountered, closing & reopening the error connection + clearing the connection pool seems to re-establish working order

What happens to the underlying original request (say for example, it was a SELECT statement)?
Sounds like this strategy would merely give up and not let the error bubble up and just clear out the connection pool so it doesn't happen again.

[brunolau]

As far as I'm not mistaken the execution strategy should ensure that the request is retried after the cleanup is done so no harm happening other than pool cleanup afaik

[DVAD7427]

Hi,
we are facing the same issue, I have .Net core 2.1 application using Dapper 2.0.30 version as ORM and Sql databaseto store the data. my code is running on windows servers, occasionally (once in a week) getting an exception System.Data.SqlClient.SqlException
ex.Message : The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors
and it is getting auto resolved with out restarting the servers. Please let me know if you need any additional information.

[cheenamalhotra]

Hi @DVAD7427

Did you try setting AppContext switch as mentioned here: #85 (comment)

[cheenamalhotra]

Closing as issue has been fixed and backported to all versions.

[bohms27]

This issue started occurring for us on 8/12, which is well after these fixes have been backported to previous versions. It started occurring in a current release, that wasnt previously having the issue. The company did VMWare upgrades it appears, could this be something that is causing this?

[DavidKlempfner]

I'm getting this error too.
I'm using System.Data.SqlClient, and looks like the OP is too. But isn't https://github.com/dotnet for Microsoft.Data.SqlClient?
cheenamalhotra wrote above "Closing as issue has been fixed and backported to all versions" however is this for System.Data.SqlClient or Microsoft.Data.SqlClient?

I've submitted an issue here:
#1262

[cheenamalhotra]

This repository only provides update to Microsoft.Data.SqlClient, however System.Data.SqlClient was updated with this fix too for v4.8.2. Please use v4.8.2+ to get this fix.

[brewfreedom]

Just saw this in the wild running .net 4.8/SQL 2019/Server 2019 using system.data.sqlclient\4.8.5\runtimes\win\lib\netstandard2.0\System.Data.SqlClient.dll which is supposed to be fixed for this. Just recently installed KB5023702 and Servicing Stack 10.0.17763.4121 We are actually in the process of moving to microsoft.data.sqlclient/latest for all code. Assuming this is an entirely separate issue? Recycling the impacted application pool was the only way we were able to recover and assume at least for now mitigation would be ensure when we catch this to clear the pool for that connection? No reproduction code yet.

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest) at System.Data.SqlClient.TdsParser.GetDTCAddress(Int32 timeout, TdsParserStateObject stateObj) at System.Data.SqlClient.SqlInternalConnectionTds.GetDTCAddress() at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx) at System.Data.SqlClient.SqlInternalConnection.Enlist(Transaction tx) at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction) at System.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction) at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource1 retry) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()

Failing on both execute reader as well as opening connetions.
The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)

[brewfreedom]

I think I missed the part where we need to also add MakeReadAsyncBlocking = true for .net framework clients. Can you guys confirm this bug is still possible with System.Data.SqlClient if we did not set that? Probably moot point since we are about to shift to Microsoft.Data.SqlClient.

[rehman22]

did any one fixed the issue ?
because i have same problem

[DavidKlempfner]

did any one fixed the issue ? because i have same problem

@rehman22 See #85 (comment)

[stephan-dantas]

I am currently using System.Data.SqlClient version 4.8.5 and facing the same issue.

[nicksav]

Still have the same issue with System.Data.SqlClient version 4.8.6

[JRahnama]

System.Data.SqlClient is currently in servicing mode and only receives security updates. To ensure you benefit from the latest updates and features, I highly recommend transitioning to Microsoft.Data.SqlClient.

[Thorium]

This appeared as a new issue after patching System.Data.SqlClient to version 4.8.6. Must be some regression issue after the latest security fixes?

[Thorium]

The real fix seems to be adding to your SQL connection string:
MultipleActiveResultSets=False

I have no idea what it does, but whatever it is, it can't be worse than currently having random SQL connection failures in production.

[Thorium]

It seems even "MultipleActiveResultSets=False" is not fixing this totally reliably.

[ErikEJ]

@Thorium This issue is closed. I you encounter issues, I suggest create a new github issue with a runnable repro.