Verify if cloud.gov catalog can send email #3798

jbrown-xentity · 2022-04-26T21:27:55Z

mogul · 2022-04-27T16:03:34Z

The SMTP service is already registered and available in the appropriate spaces, so there's no need to mess with the broker deployment to complete this.

That said there might still be a step to move a provisioned SES instance into the approved status on the AWS side (via console or in future via Terraform); can't remember where I left that!

nickumia-reisys · 2022-05-02T15:36:14Z

Successfully created SMTP service and bound to catalog. Need help with the following,

How to configure ckan with SMTP credentials
How to change all of the production emails in staging to dummy emails so real emails don't go out.

jbrown-xentity · 2022-05-02T15:37:23Z

Email settings can be set in the .profile:

As to replacing the URL's in staging, it would be something like:

Connect to catalog DB (either with SSH or with tunneling)
Examine the organizations table, there should be an extras field that contains emails
Set all emails to datagovhelp, or some email that we can verify without spamming others

nickumia-reisys · 2022-05-02T15:38:06Z

Going to follow the instructions here... https://docs.ckan.org/en/2.9/maintaining/email-notifications.html

nickumia-reisys · 2022-05-02T15:48:56Z

Is it just me or should the smtp service output what the actual email is? (outside of the domain_arn)

  "datagov-smtp": [
    {
      "credentials": {
        "domain_arn": "<fancyawssutff>/ses-2e4380c03ce8656f.ssb.data.gov",
        "email_receipt_error": "datagovhelp@gsa.gov",
        "region": "blahblahblah",
        "smtp_password": "blahblahblah",
        "smtp_server": "blahblahblah",
        "smtp_user": "blahblahblah"
      },

nickumia-reisys · 2022-05-04T00:56:20Z

Was able to get error from SMTP setup... long story short, we have to figure out what the real email address is 😞

nickumia-reisys · 2022-05-04T01:12:14Z

Okay.. doing <anything>@domain_name seems to have worked, but there's a formatting issues somewhere in our pipeline,

   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR Please do not reply to this email as it was sent from a non-monitored address.
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR 2022-05-04 01:09:31,124 ERROR [ckanext.harvest.logic.action.update] 'NoneType' object has no attribute 'strip'
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR Traceback (most recent call last):
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/bin/ckan", line 8, in <module>
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR sys.exit(ckan())
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 829, in __call__
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return self.main(*args, **kwargs)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 782, in main
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR rv = self.invoke(ctx)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return _process_result(sub_ctx.command.invoke(sub_ctx))
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return _process_result(sub_ctx.command.invoke(sub_ctx))
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return ctx.invoke(self.callback, **ctx.params)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 610, in invoke
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return callback(*args, **kwargs)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return f(get_current_context(), *args, **kwargs)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/cli.py", line 264, in run
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR utils.run_harvester()
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/utils.py", line 364, in run_harvester
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR tk.get_action("harvest_jobs_run")(context, {})
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/logic/__init__.py", line 504, in wrapped
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR result = _action(context, data_dict, **kw)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 715, in harvest_jobs_run
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR send_summary_email(context, job_obj.source.id, status)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 810, in send_summary_email
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR send_mail(recipients, subject, body)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 828, in send_mail
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR mailer.mail_recipient(**email)
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 125, in mail_recipient
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR body_html=body_html, headers=headers)
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 107, in _mail_recipient
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR smtp_connection.sendmail(mail_from, [recipient_email], msg.as_string())
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 877, in sendmail
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR (code, resp) = self.rcpt(each, rcpt_options)
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 547, in rcpt
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR self.putcmd("rcpt", "TO:%s%s" % (quoteaddr(recip), optionlist))
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 153, in quoteaddr
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR if addrstring.strip().startswith('<'):
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR AttributeError: 'NoneType' object has no attribute 'strip'

nickumia-reisys · 2022-05-04T01:15:48Z

Okay, I know, this is a lot of logs, but maybe python isn't the error, I found this in the logs too... Definitely an SMTP broker issue and I don't know where aaron's email came from... @mogul any insights?

   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR 2022-05-04 01:09:30,896 ERROR [ckan.lib.mailer] SMTPDataError(554, b"Access denied: User `arn:aws:iam::821341638715:user/cf/ses-b7a1cd34ddf3b0cf-csb--e3a67950-0aa6-4cb0-b272-76cfcf93cb8a' is not authorized to perform `ses:SendRawEmail' on resource `arn:aws:ses:us-west-2:821341638715:identity/aaron.borden@gsa.gov'")
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR Traceback (most recent call last):
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 107, in _mail_recipient
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR smtp_connection.sendmail(mail_from, [recipient_email], msg.as_string())
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 893, in sendmail
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR raise SMTPDataError(code, resp)
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR smtplib.SMTPDataError: (554, b"Access denied: User `arn:aws:iam::821341638715:user/cf/ses-b7a1cd34ddf3b0cf-csb--e3a67950-0aa6-4cb0-b272-76cfcf93cb8a' is not authorized to perform `ses:SendRawEmail' on resource `arn:aws:ses:us-west-2:821341638715:identity/aaron.borden@gsa.gov'")

P.S. This is a freshly created service today. So there's nothing tying this particular instance to anything aaron was doing before.

P.S.2. This is the code that adds the SendRawEmail permission, https://github.com/GSA/datagov-brokerpak-smtp/blob/main/terraform/bind/main.tf#L15-L35

mogul · 2022-05-04T02:23:22Z

Not a clue and the HCL is very simple. This is really surprising!

nickumia-reisys · 2022-05-04T13:21:32Z

I'll try and investigate then. Thanks for taking a look @mogul!

nickumia-reisys · 2022-05-12T14:03:20Z

It was determined that the datagov-brokerpak-smtp service could not actually send email. Which lead to Debug and create test for datagov-smtp broker #3810

FuhuXia · 2022-08-11T19:18:27Z

Error sending emails:

[APP/TASK/05ccfb8b/0] OUT 
INFO  [ckanext.datagovcatalog.harvester.notifications] Extra recipients for source found: [{'name': 'fuhu.xia@gsa.gov', 'email': 'fuhu.xia@gsa.gov'}]                                                                                                                         
[APP/TASK/05ccfb8b/0] OUT 
ERROR [ckanext.harvest.logic.action.update] 'NoneType' object has no attribute 'strip'          
[APP/TASK/05ccfb8b/0] ERR 
ERROR [ckanext.harvest.logic.action.update] 'NoneType' object has no attribute 'strip'          
ERR Traceback (most recent call last):                                                                                      
ERR File "/home/vcap/deps/1/bin/ckan", line 8, in <module>                                                                  
ERR sys.exit(ckan())                                                                                                        
ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 829, in __call__                        
ERR return self.main(*args, **kwargs)                                                                                       
...
ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/logic/__init__.py", line 504, in wrapped                
ERR result = _action(context, data_dict, **kw)                                                                              
ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 715, in harvest_jobs_run      
ERR send_summary_email(context, job_obj.source.id, status)                                                                  
ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 810, in send_summary_email    
ERR send_mail(recipients, subject, body)                                                                                    
ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 828, in send_mail             
ERR mailer.mail_recipient(**email)                                                                                          
ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 125, in mail_recipient             
ERR body_html=body_html, headers=headers)                                                                                   
ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 107, in _mail_recipient            
ERR smtp_connection.sendmail(mail_from, [recipient_email], msg.as_string())                                                 
ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 877, in sendmail                                         
ERR (code, resp) = self.rcpt(each, rcpt_options)                                                                            
ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 547, in rcpt                                             
ERR self.putcmd("rcpt", "TO:%s%s" % (quoteaddr(recip), optionlist))
ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 153, in quoteaddr                                        
ERR if addrstring.strip().startswith('<'):                                                                                  
ERR AttributeError: 'NoneType' object has no attribute 'strip'                                                              
OUT Exit status 1

nickumia-reisys · 2022-08-12T14:24:56Z

To reiterate: the instructions in this comment needs to be checked/done everytime there's a questionable email added to the DB or if the DB is restored with bad emails.

FuhuXia · 2022-08-12T14:28:24Z

This is a harvest extension bug. Did a PR upstream to fix it, checking email address exist before sending notifications, so we don't have to run any manual steps.

[Update]
Upstream PR merged.

nickumia-reisys self-assigned this May 2, 2022

nickumia-reisys mentioned this issue May 2, 2022

Add SMTP Service GSA/catalog.data.gov#457

Merged

nickumia-reisys changed the title ~~cloud.gov catalog can send email~~ Verify if cloud.gov catalog can send email May 4, 2022

nickumia-reisys mentioned this issue May 4, 2022

Update smtp broker to 1.1.3 GSA/datagov-ssb#139

Merged

nickumia-reisys added this to the Sprint 20220512 milestone May 12, 2022

nickumia-reisys closed this as completed May 12, 2022

nickumia-reisys added component/catalog Related to catalog component playbooks/roles component/ssb Notifications labels Oct 9, 2023

nickumia-reisys added this to data.gov team board Oct 9, 2023

nickumia-reisys moved this to 🗄 Closed in data.gov team board Oct 9, 2023

nickumia-reisys added the Testing label Oct 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Verify if cloud.gov catalog can send email #3798

Verify if cloud.gov catalog can send email #3798

jbrown-xentity commented Apr 26, 2022 •

edited by nickumia-reisys

Loading

mogul commented Apr 27, 2022

nickumia-reisys commented May 2, 2022 •

edited

Loading

jbrown-xentity commented May 2, 2022 •

edited

Loading

nickumia-reisys commented May 2, 2022

nickumia-reisys commented May 2, 2022

nickumia-reisys commented May 4, 2022

nickumia-reisys commented May 4, 2022

nickumia-reisys commented May 4, 2022 •

edited

Loading

mogul commented May 4, 2022

nickumia-reisys commented May 4, 2022

nickumia-reisys commented May 12, 2022

FuhuXia commented Aug 11, 2022

nickumia-reisys commented Aug 12, 2022

FuhuXia commented Aug 12, 2022 •

edited

Loading

Verify if cloud.gov catalog can send email #3798

Verify if cloud.gov catalog can send email #3798

Comments

jbrown-xentity commented Apr 26, 2022 • edited by nickumia-reisys Loading

User Story

Acceptance Criteria

Background

Security Considerations (required)

Sketch

mogul commented Apr 27, 2022

nickumia-reisys commented May 2, 2022 • edited Loading

jbrown-xentity commented May 2, 2022 • edited Loading

nickumia-reisys commented May 2, 2022

nickumia-reisys commented May 2, 2022

nickumia-reisys commented May 4, 2022

nickumia-reisys commented May 4, 2022

nickumia-reisys commented May 4, 2022 • edited Loading

mogul commented May 4, 2022

nickumia-reisys commented May 4, 2022

nickumia-reisys commented May 12, 2022

FuhuXia commented Aug 11, 2022

nickumia-reisys commented Aug 12, 2022

FuhuXia commented Aug 12, 2022 • edited Loading

jbrown-xentity commented Apr 26, 2022 •

edited by nickumia-reisys

Loading

nickumia-reisys commented May 2, 2022 •

edited

Loading

jbrown-xentity commented May 2, 2022 •

edited

Loading

nickumia-reisys commented May 4, 2022 •

edited

Loading

FuhuXia commented Aug 12, 2022 •

edited

Loading