Memory usage when using Mailkit in Docker Container #1105
Comments
|
That sounds excessive. My first inclination is that you could use the following code to manually Dispose all of the streams used by a MimeMessage to see if that helps: static void DisposeMimeMessage (MimeMessage message)
{
foreach (var bodyPart in message.BodyParts) {
if (bodyPart is MessagePart rfc822) {
DisposeMimeMessage (rfc822.Message);
} else {
var part = (MimePart) bodyPart;
part.Content.Stream.Dispose ();
}
}
} |
|
Thanks, I'll test this I don't see and implementation of DisposeMessagePart in your example and I can't find one in MailKit. Did you mean the following? So recursively calling the method DisposeMimeMessage? ` } |
|
Sorry, yes, I meant DisposeMimeMessage(). I updated the code in my previous comment. Thanks! |
|
I have tried your suggestion, but it is not solving the problem. I did some further analysis and the issue seems to be related to the SSL or StartTls (encrypted connection). What I observe is that the memory increases every time I call using The memory increase but does not decrease. So it seems that some memory is allocated on this call by Mailkit or maybe the OS that is not released for some reason. |
|
Let me make sure I understand your scenario correctly... Are you re-using the same ImapClient to connect to multiple sources over the lifespan of the object? In other words: var client = new ImapClient ();
client.Connect ("one.host.com", 993, SecureSocketOptions.SslOnConnect);
// do some stuff...
client.Disconnect (true);
client.Connect ("two.host.com", 993, SecureSocketOptions.SslOnConnect);
// do some stuff...
client.Disconnect (true); |
|
This is what I am basically doing, this is part of the code I do dispose the message. message.From.Add(new MailboxAddress("From name", "from@email.test"));
message.To.Add(new MailboxAddress("To name", "to@email.test"));
message.Subject = "How you doin'?";
message.Body = new TextPart("plain")
{
Text = @"Hey, hello this is a test"
};
using (var client = new SmtpClient())
{
client.Connect("one.host.com", 465, SecureSocketOptions.SslOnConnect);
client.Authenticate("username", "password");
client.Send(message);
client.Disconnect(true);
} |
|
I'm going to need a memory profiler for this... I'm not seeing any obvious cause in the code 😦 |
|
I created a small project (console app) to reproduce the issue here https://github.com/kvanderbok/MailkitMemIssue When I run this I see that the memory grows until 300 MB then levels of. So it is not as dramatic as in my WebApi project. |
|
I might be wrong, but doesn't that kinda sound like the GC is just allowing the heap to grow to a certain size? If that's not the case, what I'll need to debug this is a tool that can show me what objects are still dangling after the SmtpClient has been disposed. As far as I can tell, the SmtpClient correctly disposes the SslStream, so I'm not sure what MailKit could be leaking. |
|
I just confirmed by stepping thru everything with a debugger that the SslStream's Dispose() method gets called and that the underlying NetworkStream's Dispose() method also gets called, so that eliminates the theory that the SslStream isn't getting disposed properly. |
|
@kvanderbok I don't suppose you've made any discoveries on this since December? |
|
We must have observed the same memory leak in our case. I have send with "swaks" with every SMTP Server Mails. It could be that this output can help: The SMPT Server that not work: |
|
Just so I make sure I understand correctly, the server that produced the first log did not have a leak but the second server did have a leak? Or did both of them leak? |
|
Yes you understand correct. |
|
@FelixMarek Thanks. I see that the second server supports the PIPELINING extension that the first server does not. What if you modify your code to disable that? client.Connect (...);
client.Authenticate (...);
// This will disable PIPELINING
client.Capabilities &= ~SmtpCapabilities.Pipelining;
client.Send (message);Does this stop the leak? |
|
Maybe I'm reading it wrong, but isn't that reversed: the first smtp server supports pipelining? |
|
D'oh, yes, you are right. |
|
I'm also experiencing something similar to what I'm seeing in this issue. I've tried the suggestions I saw, but so far no luck. Definitely willing to help test if able. |
|
Do you guys provide your own ServerCertificateValidationCallback on the SmtpClient? |
|
I do not. We let it validate on it's own. We also let it default for the SecureSocketOptions |
|
if there was a leak, it sounds like it'd have to be related to upgrading the connection to an SSL connection but that code is so simple it's basically just wrapping the NetworkStream in an SslStream and calling sslStream.AuthenticateAsClient() so I can't see how there could possibly be a leak. |
|
The leak would have to exist in both of the following blocks of code: and/or |
|
I'm trying to get some heap dumps today to see if I can see anything. I'll let you know if I find anything useful. |
|
The only potential leak I can find is https://github.com/jstedfast/MailKit/blob/master/MailKit/MailService.cs#L533 which would suggest you'd have to get an exception, but you guys don't appear to be getting an SslHandshakeException so I don't think that's it. (this potentially leaks only because .NET >= 4.6 introduced a Dispose() method for X509Certificates) |
|
I'm struggling to recreate this on my machine, and don't have access to the SMTP server or containers where I can recreate the issue. I'm still trying to work with the teams that do to see if I can be of any help with this, but it's much slower going than I had hoped for. |
|
Oops, let me reopen this since you guys are clearly still having issues. |
|
I ended up modifying the app a bit to comment out MimeMessage creation (just to rule that out) and making the code just Connect & Disconnect w/o any SSL or anything and here's what I got: The amount of memory leaked each loop iteration makes me think this isn't in MailKit or I would expect it to be consistent. It's also a HUGE amount (248K, 264K, 312K, 384K, ...) - I don't think the SmtpClient allocates that much memory in total, nevermind could it leak that much per loop... not with just a Connect/Disconnect cycle. Based on this, I added: GC.Collect ();
GC.WaitForPendingFinalizers ();
GC.Collect ();to a few locations - at the end of each loop and before taking a snapshot of the current process memory. This change made it so that nearly every loop had a 0B increase in memory. Out of 10 loops, on the first loop had a memory increase (makes sense) and 1 other loop iteration (why? don't know.). This was pretty consistent, though, as I ran it several times. Then I changed the settings to use STARTTLS and I got this: Keep in mind that if the NetworkStream is getting disposed, that also proves that the container SslStream is also getting disposed. So where is that 1.3MB memory increase in loop 5 coming from? Or the 644KB increase in loop 10? Or the 260KB in loop 3? If there is a leak, it seems to pretty clearly be a leak in .NET core libraries like SslStream. |
|
Ended up using the memory profiler that comes as part of Visual Studio Enterprise and the only MailKit objects "leaked" (iow still alive after 100 iterations of the loop creating/connecting/disconnecting/disposing the SmtpClient) were 37 instances of Since TaskCompletionSource does not seem to be disposable, not sure how I can be "leaking" that so I suspect it's just that the GC didn't get around to it. Also "leaked" are:
Conclusion: unless the 37 instances of the TaskCOmpletionSource are actually a leak, there is no leak in MailKit. |
|
Is this something we need the .net team to investigate? They must be doing something different because using the legacy SMTP component (against their wishes) doesn't seem to have the same issue. |
|
Yes, this will need to be investigated by the .NET team but I suspect it's not a leak at all, but rather just the runtime allowing the GC to grow since it has so much RAM available to it. |
|
|
|
I'm closing this because I can't find a leak in the test case I've been using which effectively boils down to this (netcoreapp3.1): using System;
using MailKit.Security;
using MailKit.Net.Smtp;
namespace MailKitMemoryLeak
{
class Program
{
static void Main (string[] args)
{
for (int i = 0; i < 100; i++) {
using (var client = new SmtpClient ()) {
client.Connect ("smtp.gmail.com", 587, SecureSocketOptions.Auto);
client.Disconnect (true);
}
}
Console.ReadKey ();
}
}
}I've been putting a breakpoint before and after the for-loop so that I can take memory snapshots and then compare the heaps. There was a theory that the leak only happens with some servers, so maybe... but I would need someone to provide me with one of these SMTP servers so that I can try this test case against one of those. That said, MailKit wouldn't treat those different servers differently - it'd still run all of the exact same code-paths, so I don't see how there would be a leak unless the SslStream (or something that the SslStream uses) is leaking(?). |
|
What you guys are seeing is exactly the same as this: https://stackoverflow.com/questions/18739892/finding-memory-leaks-in-c-sharp |
|
Hello, I had a similar problem. When using Mailkit in the docker container, the memory of the container would increase by about 150-300 Mbs for each email sent. When using Before the Connect, I've added I find it weird that the emails were still sending without any exception before.
I think this was the issue in my case. |
|
Where did you find the LetsEncrypt CRL location quote from? Also, if setting It's totally plausible that these CRLs are significant in size and that would certainly explain the memory usage increasing. |
|
Oh, that quote is from my StackOverflow answer for an SslHandshakeException question here: https://stackoverflow.com/questions/61894037/mailkit-gets-an-sslhandshakeexception-with-letsencrypt-ssl-certificates |
|
Yes, sorry I lost the tab and only had the quote left. So, it would mean that there is no problem checking if the certificate is revoked, but the problem is that after doing it, the memory is not freed and after a couple checks, there will be a huge memory leak (in Gbs), thus crashing the container. Also, many users on the application simultaneously sending mails would crash it even if the memory was not leaked if you use .Connect on each mail sent. |
|
Someone should probably submit a bug report about this to the .NET Core team https://github.com/dotnet/core/issues with specific runtime versions, OS, etc. I think everyone is seeing this on Linux, right? |
|
I was on Linux yes. |
|
Linux too, yes. |
We also had the issue on Linux . |
|
Just as heads up, the dotnet/runtime issue was just closed:
|
Hi jstedfast,
We are using your Mailkit library for sending emails from our dotnet core 3.1 web api. Actually we developed a netstandard 2.0 library that includes Mailkit as a dependency. This netstandard library is used by our webapi. We run our application in a docker container using mcr.microsoft.com/dotnet/aspnet:3.1 as base.
What we notice is that when using either SSL (SslOnConnect) or STartTls (StartTls), the container uses an excesive amount of memory making our container crash. The memory increases with each email sent, but seems to level at some point. We tested this in a kubernetes cluster and on a dev machine. On the cluster the memory goes up to about 2GB and levels of. On our localmachine the memory goes up to 1,6GB and then levels off. When setting the the SecureSocketOptions to None, we don't see a dramatic increase in memory is use around 200 MB.
What we are wondering is whether this memory usage is typical or whether this is as we think too much.
Regards, Kees
The text was updated successfully, but these errors were encountered: