Improving visibility into SMTP issues / robustness

[DavidAnderson684]

In our experience of using Freescout over the last couple of years, probably the weakest "link in the chain" has been issues with outgoing SMTP. If Freescout itself goes down, then people know instantly (because they're trying to use it in real time, so they notice). However, if a reply is sent, but doesn't actually reach the customer, this usually is significantly less visible. Freescout will show a notice after a while if the outgoing queue gets too long, but this takes a while to manifest. By the time it does manifest, a lot of conversations may be affected, depending on the size of the install. Also, it is not easy to investigate problems with the queue - in System -> Tools you can see what's in the queue, but there is little visibility into why something is still there after a long time.

As some improvements to making Freescout more robust, I'd suggest:

• Allowing multiple outgoing SMTP servers to be configured. Currently, the outgoing SMTP server is a single point of failure. For users who don't have round-robin DNS for their SMTP server, this cannot be avoided. Allowing configuration of one or more further SMTP servers for a mailbox would mean that support replies can go out when their main SMTP server is down.
• In the "Outgoing Emails" pop-up for a sent reply, when the status is "Accepted for delivery", show the ID that the receiving SMTP server gave when accepting it. This makes it much easier to trace, especially on busy SMTP servers. Plus, if the SMTP ID is known, then custom code can be written that hooks into the mailserver in order to show the log entries from there.
• If the status is not yet that the mail has been accepted for delivery, then this should be shown in the main conversation view (not just in the "Outgoing Emails" pop-up) - this helps delivery problems to be discovered earlier. If a delivery attempt was made that did not succeed, this information should be shown (either directly or in "Outgoing Emails" pop-up) - e.g. that an SMTP 500 error was returned.
• In the queue display in System -> Tools in general, jobs should display more information - in particular, the results of past attempts to process the job. Currently all you can really see is that the job is in the queue, but you can't know why it has been there a long time.

[naturaw-canine-food]

I was thinking the same thing. We're BCC'ing all emails right now to ensure they are going out. It's not feasible at 50 convo's per minute.

[lawfulsoftware]

You can add a vote to this: https://feedback.userreport.com/25a3cb5f-e4bd-4470-b6f3-79fcfaa8e90f/#idea/416731

You should add the rest as separate feature requests on the same site.

[freescout-helpdesk]

1. Allowing multiple outgoing SMTP servers to be configured. Currently, the outgoing SMTP server is a single point of failure. For users who don't have round-robin DNS for their SMTP server, this cannot be avoided. Allowing configuration of one or more further SMTP servers for a mailbox would mean that support replies can go out when their main SMTP server is down.

https://feedback.userreport.com/25a3cb5f-e4bd-4470-b6f3-79fcfaa8e90f/#idea/416731

2. In the "Outgoing Emails" pop-up for a sent reply, when the status is "Accepted for delivery", show the ID that the receiving SMTP server gave when accepting it. This makes it much easier to trace, especially on busy SMTP servers. Plus, if the SMTP ID is known, then custom code can be written that hooks into the mailserver in order to show the log entries from there.

#3330

3. If the status is not yet that the mail has been accepted for delivery, then this should be shown in the main conversation view (not just in the "Outgoing Emails" pop-up) - this helps delivery problems to be discovered earlier. If a delivery attempt was made that did not succeed, this information should be shown (either directly or in "Outgoing Emails" pop-up) - e.g. that an SMTP 500 error was returned.

This is partially implemented, see #3268 (comment)

4. In the queue display in System -> Tools in general, jobs should display more information - in particular, the results of past attempts to process the job. Currently all you can really see is that the job is in the queue, but you can't know why it has been there a long time.

#3331

[freescout-helpdesk]

3. If a delivery attempt was made that did not succeed, this information should be shown (either directly or in "Outgoing Emails" pop-up) - e.g. that an SMTP 500 error was returned.

This is beyond FreeScout possibilities - on "delivery" step only bounce emails can tell you that an email could not be delivered to the recipient.

Here is how it all works, there are two steps:

• Step 1. FreeScout passes an email to the mail server for delivery. If there is an error on this step FreeScout shows this error in Outgoing Emails window, in "Manage > Logs > Send Errors" and to the customer in the conversation (after 1 hour of attempts).
• Step 2. Mail server delivers an email to the recipient. If there is an error on this step the only way for FreeScout to know about it - is to receive a bounce email from the mail server responsible for delivering the email. If FreeScout receives a bounce email it tries to link it to the conversation it's related and show corresponding info in the related conversation and thread.

[lawfulsoftware]

@freescout-helpdesk Perhaps this tracking functionality could be enabled if the emails are sent via API rather than SMTP. If you are open to developing it, here are some suggestions:

1. Module that supports all prominent services. This will be harder to develop and maintain.
2. Allow the user to deactivate the SMTP server settings on a mailbox but still create a reply. The webhook module event "convo.agent.reply.created" could call a webhook that the user can configure to work with their own mail service.

With #2, the user can do their own error handling and, if a message fails, post a note in the conversation with the appropriate details.

[DavidAnderson684]

Thank you.

Concerning #3, I should have been clearer - I meant 500 errors on the initial delivery step, i.e. failures that occur on "Step 1" (#3327 (comment)). For example, SMTP server has a problem which results on giveing 5XX errors immediately on connection (or indeed at any stage before the final acceptance at the end of the conversation).

If I understand rightly, this is now covered. However, even a 1 hour delay before notification is a long time (and after many retries), especially on a large setup. A 5XX code from SMTP is intended to indicate a permanent failure according to the SMTP RFCs; by not showing it for an hour, Freescout effectively decides to treat it as non-permanent, i.e. equivalent to 4XX. In my view it'd be better if the ticket indicates that a 5XX has occurred immediately.

I suppose that everyone thinks their idea is the most important one ever, but, for me, lack of ability for redundant outgoing SMTP capability is the #1 robustness issue in Freescout. Anyway, I've voted, and can just hope that enough users will agree with me. Many thanks for all your work on Freescout.

[freescout-helpdesk]

Concerning #3, I should have been clearer - I meant 500 errors on the initial delivery step, i.e. failures that occur on "Step 1" (#3327 (comment)). For example, SMTP server has a problem which results on giveing 5XX errors immediately on connection (or indeed at any stage before the final acceptance at the end of the conversation).

#3339