5.3.4 Header size exceeds fixed maximum size

[chrismcv]

Describe the bug
We upgraded from 2.0.3 to 2.3.0 and had issues with attachments when sending through smtp.office365.com

In the logs we would see the final attachment boundary, and then a vague error.

C: --=-DdAczYjEkl/ABnOTLAA2UQ==--
S: 552 5.3.4 Header size exceeds fixed maximum size
C: RSET
S: 250 2.0.0 Resetting

See the screenshot for a comparison of the 2 versions. It seems that message size, and BDAT options have changed between the versions, which is enough to trigger our issues.

If possible, I'd like to know:

1. Is there a mail server setting we can change to remediate this issue for 2.3.0?
2. Is there setting in MailKit 2.3.0 to use the older style of message?

Screenshots
If applicable, add screenshots to help explain your problem.

Server (please complete the following information):

• .NET Core 2.2

[jstedfast]

1. I don't know anything about server settings for any server, so I can't help with that.
2. Yes, you can disable both SIZE and BDAT:

client.Connect ("smtp.office365.com", 465, true);
client.Authenticate ("username", "password");

client.Capabilities &= ~(SmtpCapabilities.Chunking | SmtpCapabilities.Binary | SmtpCapabilities.Size);

It seems to me that office365.com accepted the SIZE= parameter in the MAIL FROM command, so I'm curious to know if you need to disable that.

I also find it curious that office365.com is failing with the BDAT command. Why?

The error message you are getting (try googling it) suggests that you've got a really long header that it doesn't like and that it has nothing to do with BDAT or SIZE=. Are you sure that's not the issue?

[chrismcv]

client.Capabilities &= ~(SmtpCapabilities.Chunking | SmtpCapabilities.Binary | SmtpCapabilities.Size);

Thanks for this. I'll try it out and see if it makes a different on 2.3.0.

I'm not in control of the server (or office 365 account) so have limited insight into what is happening there. I think it's sending on to various mail filtering services - I don't know if this happens inline, or as a subsequent phase.

I'm reasonably confident the header lengths are good. Unless there is a CR/LF type issue which is causing them to look right, but not actually be right.

The header size in the 365 account is set to 256KB. My attachment is slightly larger than that (300KB) - so the only possibility that would make sense to me is that its treating the attachment as a header.

[ghost]

1. I don't know anything about server settings for any server, so I can't help with that.
2. Yes, you can disable both SIZE and BDAT:

client.Connect ("smtp.office365.com", 465, true);
client.Authenticate ("username", "password");

client.Capabilities &= ~(SmtpCapabilities.Chunking | SmtpCapabilities.Binary | SmtpCapabilities.Size);

It seems to me that office365.com accepted the SIZE= parameter in the MAIL FROM command, so I'm curious to know if you need to disable that.

I also find it curious that office365.com is failing with the BDAT command. Why?

The error message you are getting (try googling it) suggests that you've got a really long header that it doesn't like and that it has nothing to do with BDAT or SIZE=. Are you sure that's not the issue?

I encountered the OP issue as well, after upgrading to 2.3.0 from 2.2.0. After upgrading, emails with large attachments would no longer send and would fail with the error "header size exceeds fixed maximum size."

SOLUTION:
Disabling both SIZE and BDAT as specified above.

After disabling, the same email with large attachments that wouldn't send prior, now sends! Due to this test case, I agree with @chrismcv that it's possible the attachments are being treated as a header, and thus contributing to the header limit.

[jstedfast]

I might be missing something important, but I can't help but think this must be a bug in Exchange because I don't see what MailKit is doing wrong :-\

Do either of you have a sample message that you can share with me that fails to send unless you disable SIZE & BDAT?

I think I have an office365 account, I just need to remember what my credentials are since I haven't used it in years

[jstedfast]

Okay, so I can reproduce this as long as the message is large enough.

Just for my own notes, here's the EHLO response:

C: EHLO [192.168.1.152]
S: 250-MN2PR20CA0003.outlook.office365.com Hello [72.94.35.232]
S: 250-SIZE 157286400
S: 250-PIPELINING
S: 250-DSN
S: 250-ENHANCEDSTATUSCODES
S: 250-AUTH LOGIN XOAUTH2
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250 SMTPUTF8

Here's the MAIL FROM and RCPT TO commands:

C: MAIL FROM:<me@me.com> SIZE=6156842
C: RCPT TO:<me@me.com>
S: 250 2.1.0 Sender OK
S: 250 2.1.5 Recipient OK

Here's the BDAT command:

C: BDAT 6156842 LAST
...
S: 552 5.3.4 Header size exceeds fixed maximum size

What follows is a message that is exactly 615842 bytes including the trailing CRLF sequence. I was worried that the byte count would be off somehow and maybe that was why the error was happening, but that's clearly not the issue. My other concern was that perhaps one (or more) lines might not have ended with CRLF (e.g. perhaps just LF was used), but I have also verified that this is also not the case.

615842 is also smaller than 157286400 which is the maximum size advertised by the server in the EHLO response.

The only other idea I have is that Office365.com expects smaller BDAT chunks, e.g.:

C: BDAT 4096
{4096 bytes}
C: BDAT 4096
{4096 bytes}
...
C: BDAT 0 LAST
S: 250 OK
S: 250 OK
...
S: 250 OK

[jstedfast]

Ended up writing logic to chunk the message via multiple BDAT commands. Started with 4096 bytes at a time and worked my way up to a chunk of up to 8MB and they all worked fine (test message is only ~6MB which failed w/o the new BDAT logic).

Since even the 8MB BDAT command worked, it suggests to me that the problem is that MimeKit flushes the output stream in the MimeMessage.WriteTo() logic which is causing outlook365.com to freak out.

I already know that this Flush exists because I ran into that problem early on when implementing this new BDAT logic (I was expecting Flush to happen only once the full message had been written out).

I suspect the problem is MimeKit.IO.FilteredStream.Flush() will flush the underlying stream and I need that to not happen, or at the very least, I need to make it optional so that I can disable it when sending BDAT.

[jstedfast]

Confirmed. I just commented out the code in FilteredStream that flushed the source stream in its own implementations of Flush and FlushAsync and sending via office365.com worked fine.

So it seems that there was a bug in MimeKit and also a bug in office365.com's SMTP server implementation because it should not be freaking out when it receives smaller packets.

[jstedfast]

Anyone hitting this bug:

Grab a MimeKit nuget package >= 2.3.0.11 from https://www.myget.org/feed/mimekit/package/nuget/MimeKit

if I have time this weekend, I might try to publish a new release (2.3.1), but I've spent too long on this so far this morning and need to get closing my pool.

[ghost]

@jstedfast Personally, I am not. The solution for me was to disable size and BDAT, so I'm not waiting on the fix to be released. When it is released though, I will update to the latest version of the nuget package, remove the line of code that disables size and BDAT, and try sending that message again that had failed prior. Thank you for all of your guidance, troubleshooting, and fixing of the issue!