Frequent OOM issues when importing a mailbox #395
Comments
This one most likely indicates the server encountered a malformed MIME message. Does the error happen multiple times? |
|
07c8495 fixes the resource leak caused by message parse errors. The remaining problem is to figure out whether it is okay for message parse to fail here and how it can be improved or fixed. Most likely that |
|
That was happening while I was moving about 2000 emails so I'm not exactly sure which ones are causing the issue. I will try to find one and send it to you. |
|
It does't seem to be any one message but seems like it might have something to do with messages with no subject. |
|
I was seemingly able to cause a crash and get that error by copying individual messages with no subject. But then when copying the same one again it didn't happen. |
|
I just tried again from master and am still getting the OOM issues. |
|
Apparently minio-go client we use for S3 turned out to also have severe problems with streaming API causing a lot more memory allocation than necessary.
I will see what I can do to workaround this. |
|
Interesting. Thanks again! |
Blob size would be unknown ahead of time if message store compression is used (e.g. in imapsql). Part of #395 fix
|
Just rebuilt and tested this and wow what a difference. No memory issues and performance is way better for importing that mailbox. I'm still getting the EOF errors but at this point it is quite clear that they are unrelated. With the performance gain it will make it easier for me to debug that issue. |
|
Got messages you sent. Strangely, after removing Apple Mail's XML I was able to successfully add them to a IMAP mailbox without any errors. I guess something is off with the way Apple Mail's saved messages. I would need some more help from you. Enable I/O tracing: This should save to log everything IMAP server receives as-is along with any errors it causes. Then try adding messages that cause problems and email me the log (it can contain auth. credentials - you might want to remove these, look for AUTHENTICATE PLAIN or LOGIN lines). |
The message as printed in the log seems to be oddly truncated for unknown reason. To be honest, I have no ideas why this might happen. |
|
I'll try another email client in case it's an issue with Apple mail. I'll let you know what I find. |
|
The same thing happens when transferring the email using thunderbird. So it's not the client mangling it. Looking at the source in gmail, that's how the email ends. It ends with an unfinished img tag. Is maddy parsing the contents of the email? How does it decided when the EOF is expected or not? In gmail it displays the email fine except that there is a partial img tag printed out in plain text at the end. |
maddy expects a well-formed MIME structure. I guess BodyStructure parser should be modified to accept malformed messages and handle them in some way. CC @emersion |
|
i was able to partially reproduce this issue (the debug log |
|
I've got many hundred spam (and a few valid) emails triggering this unexpected EOF while syncing over some old mail from dovecot. You probably have what you need, but if you do need more sample emails, I am happy to share. |
|
The original issue is resolved, the "unexpected EOF" part is tracked in #466. |
Describe the bug
I am copying the Sent mailbox, via mac Mail, from my gmail (downloaded over imap) and am getting frequent OOM crashes.
Steps to reproduce
Not really sure. I can't really tell if it is particular messages that are causing the issue or the quantity of them.
Log files
Mac Mail error:
syslog
Configuration file
maddy.conf
Environment information
The text was updated successfully, but these errors were encountered: