Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation: add recommendation that jvm-option and :SystemEmail options match to avoid spam filters #4210

Closed
donsizemore opened this issue Oct 17, 2017 · 18 comments · Fixed by #9939
Labels
Milestone

Comments

@donsizemore
Copy link
Contributor

IQSS and Odum have each run into trouble with e-mail verification messages sent from Dataverse instances getting chomped by spam filters. In Odum's case, the token-èd link was getting stripped so end users only received "Please contact us if you did not intend this change or if you need assistance." Changing the mail-resource "from" field in domain.xml to match the value set for :SystemEmail seemed to help IQSS.

Though dealing with junk mail and mail filters is outside the scope of Dataverse's documentation, it might be helpful for the installation/configuration guide to note the above. Consider this self-assigned (pull request forthcoming; the links in Odum's outgoing messages are still getting stripped so Odum may have more to tweak).

@pdurbin
Copy link
Member

pdurbin commented Oct 17, 2017

In Odum's case, the token-èd link was getting stripped so end users only received "Please contact us if you did not intend this change or if you need assistance."

This reminds me a lot of #3407.

@donsizemore
Copy link
Contributor Author

Can I claim that I hadn't read the entire title of #3407? I promise I did a search. Anyway, UNC has us on three Office365 whitelists (spam, spoof, fraud) but the link is still viewed as evil. My Gmail accounts receive the e-mail in its entirety. Still working with UNC to resolve this as Thu-Mai hopes to require e-mail verification as Dataverse makes more use of it.

@pdurbin
Copy link
Member

pdurbin commented Oct 17, 2017

Thu-Mai hopes to require e-mail verification as Dataverse makes more use of it

#3300 represents our plans to have email verification mean something. Maybe you aren't allowed to publish if your email hasn't been verified or something. Or maybe we simply don't send you notifications if you haven't verified your email. We can discuss what the consequences should be there.

@donsizemore you and @kcondon were working on email trouble in #4206 and I guess this issue #4210 is simply about documenting the fix. Do you feel like making a pull request? Here's what it says as of 4.8.1: https://github.com/IQSS/dataverse/blob/v4.8.1/doc/sphinx-guides/source/installation/config.rst#systememail

@donsizemore
Copy link
Contributor Author

@pdurbin oh. whoops. so i duplicated #3407 and #4206. I'm still working with UNC/Microsoft on our end; if I learn anything more regarding Dataverse configuration I'll include it in the PR.

@djbrooke
Copy link
Contributor

@donsizemore - thanks for taking a look at this!

@pdurbin
Copy link
Member

pdurbin commented Oct 23, 2017

mail-resource "from" field in domain.xml

Just to make this more concrete, here's what the "from" filed looks like on my test server:

grep @ /usr/local/glassfish4/glassfish/domains/domain1/config/domain.xml

<mail-resource host="localhost" from="do-not-reply@dev1.dataverse.org" user="dataversenotify" jndi-name="mail/notifyMailSession"></mail-resource>

It looks like "do-not-reply" comes from scripts/installer/glassfish-setup.sh:

./asadmin $ASADMIN_OPTS create-javamail-resource --mailhost "$SMTP_SERVER" --mailuser "dataversenotify" --fromaddress "do-not-reply@${HOST_ADDRESS}" mail/notifyMailSession

It sounds like the installer shouldn't put that "do-not-reply" address in there in the first place. According to http://guides.dataverse.org/en/4.8/installation/installation-main.html#running-the-dataverse-installer the installer asks for the "Administrator email address for this Dataverse" so I assume that is the value that should be put in both the JVM mail-resource and then database :SystemEmail. Someone please tell me if I'm confused. Thanks.

@donsizemore
Copy link
Contributor Author

@pdurbin excellent point. Feeding the jvm-option the administrator e-mail address might only be a problem if said administrator's personal address were to become Shibbolized, but I imagine neither option's address would show up in the authenticateduser table (no other conflicts)?

@pdurbin
Copy link
Member

pdurbin commented Oct 23, 2017

@donsizemore part of the confusion here is the meaning of "Administrator email address for this Dataverse" from http://guides.dataverse.org/en/4.8/installation/installation-main.html#running-the-dataverse-installer and https://github.com/IQSS/dataverse/blob/v4.8.1/scripts/installer/install#L103

Which of the following are we talking about?

These are separate email addresses.

From a quick look at https://github.com/IQSS/dataverse/blob/v4.8.1/scripts/installer/install#L1278 it looks like

"Administrator email address for this Dataverse"

should be changed to

"System/Support email address for this Dataverse"

Otherwise, it's confusing.

I guess we could have the installer prompt for both email addresses but don't let me turn a small chunk into a big chunk. @donsizemore if you just want to make a small tweak to the docs to explain the situation, it would be incremental progress!

@landreev
Copy link
Contributor

Joining the discussion a bit late, but a couple of things:

The "from=" parameter, in the "mail-resource" section of the domain.xml; that in turn is set by the installer to do-not-reply@${HOST_ADDRESS}:

It's not clear if we need this parameter at all. It is a leftover from some old days, when we specifically didn't want people to be able to reply to these email. Note that, currently, this parameter is NOT actually being used for the "From:" field of the email. Instead we are setting it in the Java code, in MailServiceBean.java:

msg.setFrom(systemAddress);

and systemAddress is the value of the systemEmail setting.

The parameter set in the domain.xml is not completely ignored - it appears that it ends up being used at least for generating the id of the email (just from looking at the raw headers of a Dataverse-sent email). But I don't think it has any effect on whether the email gets relayed and delivered, or rejected and/or assigned spam points by filters... So it may be safe to just get rid of the parameter completely (I'll test that). Once again, for all practical purposes it is the value of the systemEmail that's important - that's what the recipients of the email see, etc.

@landreev
Copy link
Contributor

landreev commented Oct 25, 2017

Kevin and I were talking about/experimenting with these email settings last week; some of the results we were seeing were somewhat counter-intuitive. It appears that it's more complicated, than just saying that the systemEmail setting must match the HOST_ADDRESS in the domain.xml. I mean, if the host name in systemEmail matches the address of your server, than yes, it should work - your email will be coming from where your "from:" header says, and all the relays and spam filters should be happy. But this straightforward case is not going to satisfy everybody. Some dataverse owners will always want to use outside mail servers. So the full answer that we document in the guide should say something like, if the systemEmail you want to use is not in the same domain as your server, it should be in a domain that your mail/smtp server (specified in the mail-resource section of the domain.xml) is configured to relay mail for.

For example: We run Dataverse on dataverse.org; we configure it to use "mail.hmdc.harvard.edu" as the smtp server. mail.hmdc.harvard.edu is the registered MX record for dataverse.org. So we should be able to use support@hmdc.harvard.edu or support@dataverse.org as our systemEmail, both should work.

Explaining it all in the guide clearly may be a bit tricky.

@pdurbin
Copy link
Member

pdurbin commented Oct 27, 2017

We discussed this issue in standup yesterday and decided to move it out of Development at https://waffle.io/IQSS/dataverse . @donsizemore you are very welcome to make a pull request and we'd move this issue straight into Code Review. Probably anything you add would be an improvement but we hope that all the discussion here hasn't confused the matter. Please let us know if you need anything. Thanks!

@donsizemore
Copy link
Contributor Author

@pdurbin I'm still going back and forth with our campus mail admins =)

@pdurbin
Copy link
Member

pdurbin commented Oct 27, 2017

@donsizemore ok, no rush. At least we've started to figure out this mail delivery issue. Thanks.

@pdurbin
Copy link
Member

pdurbin commented Jun 27, 2018

@donsizemore we haven't talked about this issue in a while. Should it still be assigned to you? If not we can throw it back on the open issue heap. 😄

@mheppler
Copy link
Contributor

Annual community mtg check in on this issue... @donsizemore? @landreev? @pdurbin? Bueller?

giphy

@donsizemore
Copy link
Contributor Author

I'm happy to make a pull request to address the ticket title? The original issue I was hitting was a bug in Dataverse which apparently won't be fixed now that Shibboleth logins are assumed to be verified (though most of Odum's aren't... separate issue).

@poikilotherm
Copy link
Contributor

poikilotherm commented Feb 8, 2023

@pdurbin asked me to create a backlink to #7424 here.

Looking at https://github.com/eclipse-ee4j/angus-mail/blob/3b3df4f82fc29c64bfc8fb4e510539b6ad12fbc2/providers/smtp/src/main/java/org/eclipse/angus/mail/smtp/SMTPTransport.java#L1749-L1763, it is pretty clear that if we send every of our messages with a from set in them, this will always have precedence over anything we do with "mail.from".

With #7424 I will simply go ahead and remove that setting, as we have been requiring people to set the database settings since ever.

@pdurbin
Copy link
Member

pdurbin commented Mar 26, 2024

Resolved by this pull request:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants