Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Mail is not triggered when configured with AWS SES #184

Open
Errahulaws opened this issue Sep 12, 2024 · 18 comments
Open

[BUG] Mail is not triggered when configured with AWS SES #184

Errahulaws opened this issue Sep 12, 2024 · 18 comments
Assignees
Labels

Comments

@Errahulaws
Copy link

Describe the bug
Configured the SMTP settings in switchboard.env with all required details, still when we use the AWS token where mail is set for alert is not triggered.

I have deployed the setup in my own EC2 instance.

Expected behavior
When I access the Token generated from UI, it should have trigger the mail.

Desktop (please complete the following information):

  • OS: MacOS
  • Browser: chrome

Additional context
Here is my switchboard.env
`# Required Settings
CANARY_PUBLIC_DOMAIN=<SERVER_IP> #str
CANARY_WG_PRIVATE_KEY_SEED=<PRIVATE_KEY_SEED_ENCODED> #str
LOG_FILE=switchboard.log #str

Optional Settings (these have sane defaults)

#CANARY_CHANNEL_DNS_IP= #str
#CANARY_CHANNEL_DNS_PORT= #int
#CANARY_CHANNEL_HTTP_PORT= #int
#CANARY_CHANNEL_SMTP_PORT= #int
#CANARY_CHANNEL_MYSQL_PORT= #int
#CANARY_CHANNEL_MTLS_KUBECONFIG_PORT= #int
#CANARY_CHANNEL_WIREGUARD_PORT= #int
#CANARY_SWITCHBOARD_SCHEME= #["https", "http"]
#CANARY_FORCE_HTTPS= #bool
#CANARY_REAL_IP_HEADER= #str

#CANARY_WG_PRIVATE_KEY_N= #str

Email Alert Settings

CANARY_ALERT_EMAIL_FROM_ADDRESS=<MY_EMAIL> #str
CANARY_ALERT_EMAIL_FROM_DISPLAY="CanartAlert" #str
CANARY_ALERT_EMAIL_SUBJECT="CanaryAlert-Token-Used" #str

Alert Settings

#CANARY_MAX_ALERTS_PER_MINUTE= #int
#CANARY_MAX_ALERT_FAILURES= #int

3rd Party Settings

#CANARY_IPINFO_API_KEY= #str

Mailgun Required Settings

#CANARY_MAILGUN_API_KEY= #str
#CANARY_MAILGUN_BASE_URL= #str
#CANARY_MAILGUN_DOMAIN_NAME= #str

Sendgrid Required Settings

#CANARY_SENDGRID_API_KEY= #str
#CANARY_SENDGRID_SANDBOX_MODE= #str

SMTP Required Settings

CANARY_SMTP_USERNAME="<SMTP_USERNAME>" #str
CANARY_SMTP_PASSWORD="<SMTP_PASSWORD>" #str
CANARY_SMTP_SERVER=email-smtp.ap-south-1.amazonaws.com #str
CANARY_SMTP_PORT=587 #str

Sentry Settings

#CANARY_SENTRY_DSN= #str
#CANARY_SENTRY_ENVIRONMENT= #["prod", "staging", "dev", "ci", "local"]
#CANARY_SENTRY_ENABLE= #bool

Logging Settings

#CANARY_SWITCHBOARD_LOG_SIZE= #int
#CANARY_SWITCHBOARD_LOG_COUNT= #int
#ERROR_LOG_WEBHOOK= #str

#CANARY_TOKEN_RETURN= #["gif", "fortune"]`

@Errahulaws Errahulaws added the bug label Sep 12, 2024
@jayjb jayjb self-assigned this Sep 16, 2024
@jayjb
Copy link
Contributor

jayjb commented Sep 16, 2024

Hi @Errahulaws,

Thanks for writing in. Lets see if we can get to the bottom of this. Just for initial clarification, you are trying to trigger an AWS key Canarytoken that you have created on your own Canarytokens server. This private Canarytokens server is configured with AWS SES SMTP settings for email notifications? Is all this information correct?

We can start to check where the break is by determining whether the AWS key Canarytoken is reporting to your server and your server is failing to send through the notification?

Or, is the AWS key Canarytoken trigger not making it to your server?

For this we can do a couple quick checks;

  1. Create a web bug token, trigger it and see if you get an email notification.
  2. Create a DNS token, trigger it and see if you get an email notification.

Please let me know how those tests go

@Errahulaws
Copy link
Author

@jayjb Apology for the late reply.
Will test this and update you.

@Errahulaws
Copy link
Author

@jayjb I did tried with the Web Bug and it does not work.

I tried direct access but no alert has been triggered.

@jayjb
Copy link
Contributor

jayjb commented Sep 25, 2024

Hi @Errahulaws.

Thanks for doing that test. The final thing we need to confirm is whether:

  1. the requests are actually getting to your server (i.e. are the DNS records correct? Are the firewall rules correct etc?), or
  2. the Canarytokens server is failing to actually send the alert when it receives the request.

For 1,
a) if you run the Canarytokens server in the foreground (docker compose without the -d), and you navigate to the web bug url, do you see the request coming in on the Canarytokens server?
b) You can also quickly check the DNS records are somewhat correct by doing host <canarytokens-server> and confirming the IP address that gets returned matches your Canarytokens server IP.

For 2, we have already confirmed 1. So now we can try check the logging of the Canarytokens server running in the foreground once the server receives the request. It should give you some indication of why it does not send the alert - perhaps an exception or missing settings warning.

@Errahulaws
Copy link
Author

Errahulaws commented Sep 25, 2024

@jayjb testing all these over private IP and accessing url using private IP.

here is the logs when I try to access the Web Bug URL


frontend | INFO: :46602 - "POST /d3aece8093b71007b5ccfedad91ebb11/generate HTTP/1.0" 200 OK
nginx | 192.168.60.168 - - [25/Sep/2024:07:24:52 +0000] "POST /d3aece8093b71007b5ccfedad91ebb11/generate HTTP/1.1" 200 378 "http:///nest/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
redis | 1:M 25 Sep 2024 07:25:21.082 * 1 changes in 60 seconds. Saving...
redis | 1:M 25 Sep 2024 07:25:21.083 * Background saving started by pid 19
redis | 19:C 25 Sep 2024 07:25:21.088 * DB saved on disk
redis | 19:C 25 Sep 2024 07:25:21.088 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
redis | 1:M 25 Sep 2024 07:25:21.183 * Background saving terminated with success
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.queries#info] Saved canarydrop for token: 69lpi5c1yowcxr0txa0fb1hhy
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.channel#info] reactor is running?: True
switchboard | 2024-09-25T07:25:42+0000 [twisted.python.log#info] "" - - [25/Sep/2024:07:25:41 +0000] "GET /stuff/69lpi5c1yowcxr0txa0fb1hhy/payments.js HTTP/1.0" 200 55 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
nginx | 192.168.60.168 - - [25/Sep/2024:07:25:42 +0000] "GET /stuff/69lpi5c1yowcxr0txa0fb1hhy/payments.js HTTP/1.1" 200 66 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.channel_http#info] HTTP b'GET' on path b'/favicon.ico' did not correspond to a token. Error: /favicon.ico
switchboard | 2024-09-25T07:25:42+0000 [twisted.python.log#info] "" - - [25/Sep/2024:07:25:42 +0000] "GET /favicon.ico HTTP/1.0" 200 55 "http:///stuff/69lpi5c1yowcxr0txa0fb1hhy/payments.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
nginx | 192.168.60.168 - - [25/Sep/2024:07:25:42 +0000] "GET /favicon.ico HTTP/1.1" 200 66 "http:///stuff/69lpi5c1yowcxr0txa0fb1hhy/payments.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.channel_http#info] HTTP b'GET' on path b'/favicon.ico' did not correspond to a token. Error: /favicon.ico
switchboard | 2024-09-25T07:25:42+0000 [twisted.python.log#info] "" - - [25/Sep/2024:07:25:42 +0000] "GET /favicon.ico HTTP/1.0" 200 55 "http:///stuff/69lpi5c1yowcxr0txa0fb1hhy/payments.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
nginx | 192.168.60.168 - - [25/Sep/2024:07:25:42 +0000] "GET /favicon.ico HTTP/1.1" 200 66 "http:///stuff/69lpi5c1yowcxr0txa0fb1hhy/payments.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.channel_output_email#error] A smtp error occurred: <class 'smtplib.SMTPDataError'> - (554, b"Transaction failed: User name is missing: 'CanartAlert'.")
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.queries#info] Saved canarydrop for token: 69lpi5c1yowcxr0txa0fb1hhy
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.channel_output_email#error] Failed to send email for token 69lpi5c1yowcxr0txa0fb1hhy.
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.channel#info] Dispatched alert: Dispatched to output channel for: 69lpi5c1yowcxr0txa0fb1hhy


I do have username and password defined in switchboard.env

@jayjb
Copy link
Contributor

jayjb commented Sep 25, 2024

Thanks for the logs @Errahulaws. I just want to check the verified AWS user for your SES is CanartAlert? i.e. that is not a typo?

@Errahulaws
Copy link
Author

@jayjb No, I am completely unsure from where its coming. We have not defined such username in switchboard or frontend env file.

Usually SES username is AccessKey which we get while configuring it.

@jayjb
Copy link
Contributor

jayjb commented Sep 26, 2024

@Errahulaws: in your initial post about your Switchboard.env you have

CANARY_ALERT_EMAIL_FROM_DISPLAY="CanartAlert" #str

Just as an fyi. I'm going to have to setup SES to test what could be going wrong.

There is an opportunity to create a very useful PR where we handle AWS SES on its own (without trying to use it through general SMTP).

@Errahulaws
Copy link
Author

@jayjb yes, but I guess that is just the Display name of From address. not username.

@Errahulaws
Copy link
Author

Hi @jayjb Any luck on this.

@OlesYudin
Copy link

Hi @Errahulaws ! Are you able to create an AWS token through self-hosted version? Because each time I try to create AWS canary token I get a 500 Internal Server Error.

@Errahulaws
Copy link
Author

@OlesYudin Yes, You need to deploy the AWS infra like internet Gateways and Lambda to get it work separately.

This does not get deployed when you run the docker

@OlesYudin
Copy link

@OlesYudin Yes, You need to deploy the AWS infra like internet Gateways and Lambda to get it work separately.

This does not get deployed when you run the docker

I think that the AWS token works on Thinkst infrastructure, doesn't it? So to get it to work, I need to manually deploy some resource using Terraform and than canarytoken will automatically create IAM creds in my infrastructure, am I correct?

@OlesYudin
Copy link

@Errahulaws Recently we have very similar issue with AWS SES.
I have SES in production mode and an identity like security@my-domain.com to send emails from this user. I manually created IAM User for SES SMTP and it didn't work as expected. To get it to work I open the AWS SES console and on SMTP settings I click "Create SMTP credentials" (NOT "Manage my existing SMTP credentials"), after that AWS generates a user for me with valid SMTP credentials. In the IAM user console, I can see that docker uses credentials to send emails.

So, I would recommend you try to create an SES user with the help of AWS and try to avoid a "managed user".
image

@Errahulaws
Copy link
Author

@OlesYudin Yes, You need to deploy the AWS infra like internet Gateways and Lambda to get it work separately.
This does not get deployed when you run the docker

I think that the AWS token works on Thinkst infrastructure, doesn't it? So to get it to work, I need to manually deploy some resource using Terraform and than canarytoken will automatically create IAM creds in my infrastructure, am I correct?

@OlesYudin yes that'c correct.

@Errahulaws
Copy link
Author

@jayjb I have been trying hard on this. There are two key points I would like to highlight

  1. When we deploy the AWS infra for Key management and Token logging. Lambda ProcessUserAPITokensLogs needs to be in the VPC in case we are using the private instance.

  2. In switchboard.env where CANARY_ALERT_EMAIL_FROM_DISPLAY= seems like this is just for the display name, but looks like in backend it required same as the from email address.

The error
switchboard | 2024-09-25T07:25:42+0000 [canarytokens.channel_output_email#error] A smtp error occurred: <class 'smtplib.SMTPDataError'> - (554, b"Transaction failed: User name is missing: 'CanartAlert'.")

Got fixed after I put actual email of main.

Can we have this issues fixed.

@Errahulaws
Copy link
Author

@jayjb I have raised PR for Display Name issue.
Can you please have a look and help in merge.

@jayjb
Copy link
Contributor

jayjb commented Oct 29, 2024

Hi @Errahulaws,

Thanks so much for figuring out the issue.

Looking at the PR, there is a slight issue. The error you are hitting is AWS SES specific. So the PR change would mean that everyone using the canarytokens build would have their emails (from display) changed under their feet. I'll comment in the PR and lets see if we can find something that works for everyone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants