Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix mail parsing in collector #7652

Merged
merged 1 commit into from
Jul 13, 2020
+225 −83
Merged

Fix mail parsing in collector #7652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 94 additions & 77 deletions inc/mailcollector.class.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@
}

use LitEmoji\LitEmoji;
use Laminas\Mail\Address;
use Laminas\Mail\Header\AbstractAddressList;
use Laminas\Mail\Storage\Message;
use Laminas\Mime\Mime as Laminas_Mime;

/**
Expand Down Expand Up @@ -717,9 +720,7 @@ function collect($mailgateID, $display = 0) {
$rejinput = [];
$rejinput['mailcollectors_id'] = $mailgateID;

$req_field = $this->getRequesterField();
$h_requester = $message->getHeader($req_field)->getAddressList();
$requester = $h_requester->current()->getEmail();
$requester = $this->getRequesterEmail($message);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reply-to was always defined when using the imap extension (see #4554 (comment)), but now this header is not available if not set in the message.


if (!$tkt['_blacklisted']) {
global $DB;
Expand Down Expand Up @@ -951,9 +952,7 @@ function buildTicket($uid, \Laminas\Mail\Storage\Message $message, $options = []
}

// Who is the user ?
$req_field = $this->getRequesterField();
$h_requester = $message->getHeader($req_field)->getAddressList();
$requester = $h_requester->current()->getEmail();
$requester = $this->getRequesterEmail($message);

$tkt['_users_id_requester'] = User::getOrImportByEmail($requester);
$tkt["_users_id_requester_notif"]['use_notification'][0] = 1;
Expand Down Expand Up @@ -1320,8 +1319,10 @@ function getAdditionnalHeaders(\Laminas\Mail\Storage\Message $message) {
$head = [];
$headers = $message->getHeaders();

foreach ($headers as $key => $value) {
foreach ($headers as $header) {
// is line with additional header?
$key = $header->getFieldName();
$value = $header->getFieldValue();
Comment on lines +1322 to +1325
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$key was a numeric key here, so following preg_match() calls were always returning false.

if (preg_match("/^X-/i", $key)
|| preg_match("/^Auto-Submitted/i", $key)
|| preg_match("/^Received/i", $key)) {
Expand Down Expand Up @@ -1354,79 +1355,71 @@ function getAdditionnalHeaders(\Laminas\Mail\Storage\Message $message) {
**/
function getHeaders(\Laminas\Mail\Storage\Message $message) {

$h_sender = $message->getHeader('from')->getAddressList();
$sender = $h_sender->current();
$sender_email = $this->getEmailFromHeader($message, 'from');

$h_to = $message->getHeader('to')->getAddressList();
$h_to->rewind();
$to = $h_to->current();

$reply_to_addr = null;
if (isset($message->reply_to)) {
$h_reply_to = $message->getHeader('reply_to')->getAddressList();
$reply_to = $h_reply_to->current();
$reply_to_addr = Toolbox::strtolower($reply_to->getEmail());
if (preg_match('/^(mailer-daemon|postmaster)@/i', $sender_email) === 1) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With imap extension, we were comparing $sender->mailbox, which was not containing the host (i.e. postmaster@domain.org was returning postmater).
Now, we have to deal with the full email address.

return [];
}

$to = $this->getEmailFromHeader($message, 'to');

$reply_to_addr = Toolbox::strtolower($this->getEmailFromHeader($message, 'reply-to'));

$date = date("Y-m-d H:i:s", strtotime($message->date));
$mail_details = [];

if ((Toolbox::strtolower($sender->getEmail()) != 'mailer-daemon')
&& (Toolbox::strtolower($sender->getEmail()) != 'postmaster')) {

// Construct to and cc arrays
$h_tos = $message->getHeader('to');
$tos = [];
foreach ($h_tos->getAddressList() as $address) {
$mailto = Toolbox::strtolower($address->getEmail());
if ($mailto === $this->fields['name']) {
$to = $address;
}
$tos[] = $mailto;
// Construct to and cc arrays
$h_tos = $message->getHeader('to');
$tos = [];
foreach ($h_tos->getAddressList() as $address) {
$mailto = Toolbox::strtolower($address->getEmail());
if ($mailto === $this->fields['name']) {
$to = $mailto;
}
$tos[] = $mailto;
}

$ccs = [];
if (isset($message->cc)) {
$h_ccs = $message->getHeader('cc');
foreach ($h_ccs->getAddressList() as $address) {
$ccs[] = Toolbox::strtolower($address->getEmail());
}
$ccs = [];
if (isset($message->cc)) {
$h_ccs = $message->getHeader('cc');
foreach ($h_ccs->getAddressList() as $address) {
$ccs[] = Toolbox::strtolower($address->getEmail());
}
}

// secu on subject setting
try {
$subject = $message->getHeader('subject')->getFieldValue();
} catch (Laminas\Mail\Storage\Exception\InvalidArgumentException $e) {
$subject = '';
}
// secu on subject setting
try {
$subject = $message->getHeader('subject')->getFieldValue();
} catch (Laminas\Mail\Storage\Exception\InvalidArgumentException $e) {
$subject = '';
}

$mail_details = [
'from' => Toolbox::strtolower($sender_email),
'subject' => $subject,
'reply-to' => $reply_to_addr,
'to' => Toolbox::strtolower($to),
'message_id' => $message->getHeader('message_id')->getFieldValue(),
'tos' => $tos,
'ccs' => $ccs,
'date' => $date
];

$mail_details = [
'from' => Toolbox::strtolower($sender->getEmail()),
'subject' => $subject,
'reply-to' => $reply_to_addr,
'to' => Toolbox::strtolower($to->getEmail()),
'message_id' => $message->getHeader('message_id')->getFieldValue(),
'tos' => $tos,
'ccs' => $ccs,
'date' => $date
];

if (isset($message->references)) {
if ($reference = $message->getHeader('references')) {
$mail_details['references'] = $reference->getFieldValue();
}
if (isset($message->references)) {
if ($reference = $message->getHeader('references')) {
$mail_details['references'] = $reference->getFieldValue();
}
}

if (isset($message->in_reply_to)) {
if ($inreplyto = $message->getHeader('in_reply_to')) {
$mail_details['in_reply_to'] = $inreplyto->getFieldValue();
}
if (isset($message->in_reply_to)) {
if ($inreplyto = $message->getHeader('in_reply_to')) {
$mail_details['in_reply_to'] = $inreplyto->getFieldValue();
}
}

//Add additional headers in X-
foreach ($this->getAdditionnalHeaders($message) as $header => $value) {
$mail_details[$header] = $value;
}
//Add additional headers in X-
foreach ($this->getAdditionnalHeaders($message) as $header => $value) {
$mail_details[$header] = $value;
}

return $mail_details;
Expand Down Expand Up @@ -1953,21 +1946,46 @@ function cleanDBonPurge() {
Rule::cleanForItemCriteria($this, '_mailgate');
}

/**
* Get the requester email address.
*
* @param Message $message
*
* @return string|null
*/
private function getRequesterEmail(Message $message): ?string {
$email = null;

if ($this->fields['requester_field'] === self::REQUESTER_FIELD_REPLY_TO) {
// Try to find requester in "reply-to"
$email = $this->getEmailFromHeader($message, 'reply-to');
}

if ($email === null) {
// Fallback on default "from"
$email = $this->getEmailFromHeader($message, 'from');
}

return $email;
}

/**
* Get the requester field
* Get the email address from given header.
*
* @return string
**/
private function getRequesterField() {
switch ($this->fields['requester_field']) {
case self::REQUESTER_FIELD_REPLY_TO:
return "reply-to";

default:
return "from";
* @param Message $message
* @param string $header_name
*
* @return string|null
*/
private function getEmailFromHeader(Message $message, string $header_name): ?string {
if (!$message->getHeaders()->has($header_name)) {
return null;
}

$header = $message->getHeader($header_name);
$address = $header instanceof AbstractAddressList ? $header->getAddressList()->rewind() : null;

return $address instanceof Address ? $address->getEmail() : null;
}


Expand Down Expand Up @@ -1996,10 +2014,9 @@ public function getDecodedContent(\Laminas\Mail\Storage\Part $part) {
case '7bit':
case '8bit':
case 'binary':
default:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In GLPI 9.4, we were returning verbatim content if no transfer encoding was defined. I put back this behaviour as it seems that emails sent by Outlook are not containing this header for text parts (see attached email from #7649).

// returned verbatim
break;
default:
throw new \UnexpectedValueException("$encoding is not known");
}

$contentType = $part->getHeader('contentType');
Expand Down
34 changes: 34 additions & 0 deletions tests/emails-tests/09-reply-to-tech.eml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
Delivered-To: REDACTED-EMAIL
Received: by 2002:a50:cfc3:0:0:0:0:0 with SMTP id i3csp1241712edk;
Thu, 9 Jul 2020 00:07:53 -0700 (PDT)
Return-Path: <REDACTED-EMAIL>
Received: from REDACTED-SERVER (REDACTED-SERVER [REDACTED-IP])
by mx.google.com with ESMTPS id ck17si1268608edb.508.2020.07.09.00.07.53
for <REDACTED-EMAIL>
(version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
Thu, 09 Jul 2020 00:07:53 -0700 (PDT)
Received: from mail-lf1-f69.google.com ([REDACTED-IP])
by REDACTED-SERVER with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jul 2020 08:07:53 +0100
Received: by mail-lf1-f69.google.com with SMTP id f8so1572484lfh.22
for <REDACTED-EMAIL>; Thu, 09 Jul 2020 00:07:52 -0700 (PDT)
MIME-Version: 1.0
From: Tech (alt address) <alternative.email@glpi-project.org>
Reply-To: Tech Ni Cian <tech@glpi-project.org>
Date: Thu, 9 Jul 2020 08:07:40 +0100
Message-ID: <CAJjtjRxMt1T5Zv1SR6sr+XKK9-U8C4M3cpPGP5fVwvWintxwdQ@mail.gmail.com>
Subject: Re: [GLPI #0038927] Update - Issues with new Windows 10 machine
To: GLPI debug <unittests@glpi-project.org>
Content-Type: multipart/alternative; boundary="000000000000f50df905a9fce128"

--000000000000f50df905a9fce128
Content-Type: text/plain; charset="UTF-8"
This message have reply to header, requester should be get from this header.
--000000000000f50df905a9fce128
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

This message have reply to header, requester should be get from this header.

--000000000000f50df905a9fce128--
85 changes: 85 additions & 0 deletions tests/emails-tests/10-missing-content-transfer-encoding.eml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
From: Normal User <normal@glpi-project.org>
To: GLPI debug <unittests@glpi-project.org>
Subject: Test Email from Outlook
Thread-Topic: Test Email from Outlook
Thread-Index: AdZW2I0jVbjPDha9QGyhAGwBB3ep4A==
Date: Fri, 10 Jul 2020 16:47:41 +0000
Message-ID:
<DM6PR05MB5865518283B3393AA31FAEDFBC650@DM6PR05MB5865.namprd05.prod.outlook.com>
Content-Language: en-US
X-MS-Has-Attach:
X-MS-Exchange-Organization-SCL: -1
X-MS-TNEF-Correlator:
X-MS-Exchange-Organization-RecordReviewCfmType: 0
Content-Type: multipart/alternative;
boundary="_000_DM6PR05MB5865518283B3393AA31FAEDFBC650DM6PR05MB5865namp_"
MIME-Version: 1.0

--_000_DM6PR05MB5865518283B3393AA31FAEDFBC650DM6PR05MB5865namp_
Content-Type: text/plain; charset="us-ascii"
With only text in the body, the email will not be imported and the following error will appear when forcing the retrieval by clicking the 'Get email tickets now' button on the receiver.
'Uncaught Exception UnexpectedValueException: is not known in /var/www/glpi/inc/mailcollector.class.php at line 2002'
--_000_DM6PR05MB5865518283B3393AA31FAEDFBC650DM6PR05MB5865namp_
Content-Type: text/html; charset="us-ascii"

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">With only text in the body, the email will not be imported and the following error will appear when forcing the retrieval by clicking the &#8216;Get email tickets now&#8217; button on the receiver.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;color:#24292E;background:white">'Uncaught Exception UnexpectedValueException: is not known in /var/www/glpi/inc/mailcollector.class.php at line 2002'</span><o:p></o:p></p>
</div>
</body>
</html>
--_000_DM6PR05MB5865518283B3393AA31FAEDFBC650DM6PR05MB5865namp_--
Loading