Quoted strings in email addresses are incorrectly parsed by lookup->normaliseHeaders

[GiovanniSalmeri]

If I understand well, there is a little problem in the method lookup->normaliseHeaders in core.php. By looking at the code and at the way this method is used in toolbox->mail, I understand that the purpose of this function is to format an almost-RFC2822 headers list as a MIME-compliant headers list (with non-ASCII characters properly encoded, and folded). If this is so, lookup->normaliseHeaders fails to decode properly in email addresses quoted strings (and also, less important, address groups). I know that RFC2822 describes a thousand forms 😱 that never occur in the wild and that can be safely ignored, but quoted strings are indeed common enough: for example "Salmeri, Giovanni" <g.salmeri@example.org>. This string is presentely translated by lookup->normaliseHeaders in <"Salmeri>, Giovanni" <g.salmeri@example.org>.

After studying this issue, I found that there are two PHP functions with the aim of parsing address lists:
https://www.php.net/manual/en/function.imap-rfc822-parse-adrlist.php
https://www.php.net/manual/en/function.mailparse-rfc822-parse-addresses.php
Unfortunately neither is in the standard installation.
There are also PHP functions:
symfony/symfony#40205
Unfortunately, they intend to follow fully RFC2822 and are huge.

In order to adapt Mailer to the new API, I have written a very light alternative parseAddresses (lines 240-292) which recognises quoted strings (and flattens addresses groups). After some tests it seems to work well with all the common cases. Perhaps it can be useful? (In the original Mailer I did not face the problem, because following the steps of PHPmailer and the like, the function that builds the headers receives the single components of addresses [display name and address for each mailbox]: so no parsing was ever needed.)

[annaesvensson]

Can you please clarify which cases of address strings do you consider a "must have" to be supported? I ask so we can get a good understanding of what is required to normalise fields in MIME headers. As is so often the case, we only aim to support a useful subset of features, we don't want to support everything that is technically possible in RFC2822.

[GiovanniSalmeri]

As is so often the case, we only aim to support a useful subset of features, we don't want to support everything that is technically possible in RFC2822.

I agree, RFC2822 is a monstrosity!

Can you please clarify which cases of address strings do you consider a "must have" to be supported?

In my experience I have seen only three common cases:

1. name@site.com
2. Display Name <name@site.com>
3. "Display name, possibly with commas, possibly with \"escaped\" quotes" <name@site.com>

(These are also the only forms used as examples in the Annexes of RFC2822.) The first two are presently supported by normaliseHeaders, the third one is not. This form is compulsory when the display name contains for example a comma (or other forbidden characters), in my mailbox there are many such addresses.

Instead, I do not consider "addresses groups" a "must have": never seen in the wild (I added them to my function only because I realised that they required only one line more 😊).

[markseuffert]

I'm very sorry it took so long to fix the bug. We talked a few months ago about a suitable software design, thank you Giovanni for your feedback and your ideas. Here's what we came up with. The mail handling in the core extension has been updated. The new API for developers offers a couple of normalisation methods. To make a developer's life easier, but also to make the source code less error-prone. You don't have to use them... but I would say it's highly recommended... unless you have your own normalisation methods.

• Developers can use $this->yellow->lookup->normaliseAddress() on an individual address. Which makes sure that an individual address has special characters removed, at the moment this includes removing quotes.
• Developers can build a string with multiple addresses separated by comma. Just for clarification, this is possible, but then you would have to use the normalisation multiple times. Use once on each address.
• Mail handlers can normalise MIME headers with $this->yellow->lookup->normaliseHeaders(). Any address that is not one of the supported formats will be replaced by "error-mail-address". Which makes sure no unsanitised/unnormalised input data slips trough. This may come handy for debugging.
• We preserve and encode european special characters such as ÄÅÖŒ. The really troublesome characters that you mentioned in the bug report(s) have been removed, but we should run more tests to be on the safe side.

Should be fixed now, please let me know if this solution works for you.

[markseuffert]

What do you think, @GiovanniSalmeri? Can you take a look and test the current solution?

[GiovanniSalmeri]

Sorry for the veeeeeeery late answer. Yes, all seems fine now. Thank you very much!

[markseuffert]

Great to have a working solution. Hope the API now has everything you need for your email extensions.

Thank you for fixing this together and happy coding. 😆