Bcc, cc and multiple to fields in Personalization don't work

[swapnilt]

Issue Summary

I'm using sendgrid-python version 5.6.0. I'm able to send emails as long as there are only the following fields in the email

• from
• to
• subject
• content/body
  Apart from this any extra field if added to personalization object, I get a HTTP 400 BadRequest. Moreover, the response body is empty so there is no specific error message. I'm using mail helper function but you can clearly see the data structure which get posted in the final call. See below examples which I've tried out. Actual email ids have been masked out.

bcc field doesn't work

In [118]: pprint(m.get())
{'content': [{'type': 'text/plain', 'value': 'This is a test email'}],
 'from': {'email': 'abc@mydomain.com', 'name': 'ABC'},
 'personalizations': [{'bcc': [{'email': 'xyz@gmail.com'}],
                       'subject': 'Test email 2',
                       'to': [{'email': 'def@gmail.com'}]}],
 'subject': 'Test email 2'}

In [119]: response = sg.client.mail.send.post(request_body=mail.get())
---------------------------------------------------------------------------
BadRequestsError                          Traceback (most recent call last)
<ipython-input-119-32b16380a934> in <module>
----> 1 response = sg.client.mail.send.post(request_body=mail.get())

~/env/lib/python3.5/site-packages/python_http_client/client.py in http_request(*_, **kwargs)
    250                 request.get_method = lambda: method
    251                 timeout = kwargs.pop('timeout', None)
--> 252                 return Response(self._make_request(opener, request, timeout=timeout))
    253             return http_request
    254         else:

~/env/lib/python3.5/site-packages/python_http_client/client.py in _make_request(self, opener, request, timeout)
    174             exc = handle_error(err)
    175             exc.__cause__ = None
--> 176             raise exc
    177
    178     def _(self, name):

BadRequestsError: HTTP Error 400: Bad Request

Note that although the subject field is repeated above in mail object as well as personalization object, I've tried having it only in mail object and only in personalization object as well. It doesn't matter. It's a bit weird that subject can get added to two places in the object even while using helper functions which is not the case with to field. Irrespective of whether you add to field using Mail object constructor or via Personalization.subject, it always correctly adds it under personalization field in the resulting object. That is not the case with subject.

cc does not work

In [127]: pprint(m.get())
{'content': [{'type': 'text/plain', 'value': 'This is a test email'}],
 'from': {'email': 'abc@mydomain.com', 'name': 'ABC'},
 'personalizations': [{'cc': [{'email': 'abc@mydomain.com',
                               'name': 'ABC'}],
                       'to': [{'email': 'xyz@gmail.com'}]}],
 'subject': 'Test email 2'}

In [128]: response = sg.client.mail.send.post(request_body=mail.get())
---------------------------------------------------------------------------
BadRequestsError                          Traceback (most recent call last)
<ipython-input-128-32b16380a934> in <module>
----> 1 response = sg.client.mail.send.post(request_body=mail.get())

~/env/lib/python3.5/site-packages/python_http_client/client.py in http_request(*_, **kwargs)
    250                 request.get_method = lambda: method
    251                 timeout = kwargs.pop('timeout', None)
--> 252                 return Response(self._make_request(opener, request, timeout=timeout))
    253             return http_request
    254         else:

~/env/lib/python3.5/site-packages/python_http_client/client.py in _make_request(self, opener, request, timeout)
    174             exc = handle_error(err)
    175             exc.__cause__ = None
--> 176             raise exc
    177
    178     def _(self, name):

BadRequestsError: HTTP Error 400: Bad Request

multiple to fields also do not work

In [134]: pprint(m.get())
{'content': [{'type': 'text/plain', 'value': 'This is a test email'}],
 'from': {'email': 'abc@mydomain.com', 'name': 'ABC'},
 'personalizations': [{'to': [{'email': 'xyz@gmail.com'}]},
                      {'to': [{'email': 'abc@mydomain.com',
                               'name': 'ABC'}]}],
 'subject': 'Test email 2'}

In [135]: response = sg.client.mail.send.post(request_body=mail.get())
---------------------------------------------------------------------------
BadRequestsError                          Traceback (most recent call last)
<ipython-input-135-32b16380a934> in <module>
----> 1 response = sg.client.mail.send.post(request_body=mail.get())

~/env/lib/python3.5/site-packages/python_http_client/client.py in http_request(*_, **kwargs)
    250                 request.get_method = lambda: method
    251                 timeout = kwargs.pop('timeout', None)
--> 252                 return Response(self._make_request(opener, request, timeout=timeout))
    253             return http_request
    254         else:

~/env/lib/python3.5/site-packages/python_http_client/client.py in _make_request(self, opener, request, timeout)
    174             exc = handle_error(err)
    175             exc.__cause__ = None
--> 176             raise exc
    177
    178     def _(self, name):

BadRequestsError: HTTP Error 400: Bad Request

In some cases above the from email id is also added as cc or bcc since I have that requirement. But I've also tried adding some other email id there and it doesn't matter. In all the above cases the response body is empty.

In [136]: response.body
Out[136]: b''

In all the above cases if I remove the the additional cc or bcc or to field from the personalization, I'm able to successfully send the email.

Technical details:

• OS : Ubuntu 16.04
• sendgrid-python Version: 5.6.0
• Python Version: 3.7.0
• Django: 2.1.0

[thinkingserious]

Hello @swapnilt,

Could you please try to determine the error message, like so?

Also, could you please supply the source code you used to generate the bcc and cc examples?

Thank you!

With Best Regards,

Elmer

[swapnilt]

Hi @thinkingserious ,

Here is a simple script which reproduces the error

import urllib
import sendgrid
from pprint import pprint
from sendgrid.helpers.mail import *

SENDGRID_API_KEY = 'SENDGRID_API_KEY'

def send_email(from_email, to_email, subject, text_body, html_body = None):
    sg = sendgrid.SendGridAPIClient(apikey=SENDGRID_API_KEY)
    from_email = Email(from_email)
    to_email = Email(to_email)
    text_content = Content("text/plain", text_body)
    mail = Mail(from_email, subject, to_email)
    mail.add_content(text_content)
    personalization = Personalization()
    personalization.add_bcc(Email("def@gmail.com"))
    mail.add_personalization(personalization)
    if html_body:
        html_content = Content("text/html", html_body)
        mail.add_content(html_content)
    try:
    	print("sending mail post request: ")
    	pprint(mail.get())
    	print()
    	response = sg.client.mail.send.post(request_body=mail.get())
    except Exception as e:
    	pprint(e.to_dict)
    	return
    if response.status_code > 299 or response.status_code < 200:
    	print("Error while sending email: ", response.body)

    return response


send_email("abc@mydomain.com", 
			"xyz@gmail.com",
			"Test email 2",
			"This is a test email")

If you replace the SENDGRID_API_KEY string value with your API Key, the script can be run as-is without any modification to reproduce the error. Only dependency is Python 3.7.0 and sendgrid-python 5.6.0. After running the script, I can see the below error message -

(env) swapnil@Swapnils-Macbook:
└─ $ ▶ python test_sendgrid.py 
sending mail post request: 
{'content': [{'type': 'text/plain', 'value': 'This is a test email'}],
 'from': {'email': 'abc@mydomain.com'},
 'personalizations': [{'to': [{'email': 'xyz@gmail.com'}]},
                      {'bcc': [{'email': 'def@gmail.com'}]}],
 'subject': 'Test email 2'}

{'errors': [{'field': 'personalizations.1.to',
             'help': 'http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#message.personalizations.to',
             'message': 'The to array is required for all personalization '
                        'objects, and must have at least one email object with '
                        'a valid email address.'}]}
(env) swapnil@Swapnils-Macbook:
└─ $ ▶ 

As per the error message, the to field is missing while you can clearly see in the data structure which gets posted, that the to field is present. Changing all email ids to real email addresses doesn't help.

[swapnilt]

Hi @thinkingserious is there any update on this?

[cipy]

Hi there,

I have the same issue as above: once BCC field is set, the mail.send.post returns with:

python_http_client.exceptions.BadRequestsError: HTTP Error 400: Bad Request

Could you please elevate this "question" to an actual bug priority?

Thanks!

[cipy]

OK,

the curl example from @thinkingserious original link

curl --request POST --url https://api.sendgrid.com/v3/mail/send --header "Authorization: Bearer $SENDGRID_API_KEY" --header 'Content-Type: application/json' --data @json.ok | python -m json.tool

gives one hint:

{
"errors": [{
"field": "personalizations.0",
"help": "http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#message.recipient-errors",
"message": "Each email address in the personalization block should be unique between to, cc, and bcc. We found the first duplicate instance of [your.duplicate@email.com] in the personalizations.0.bcc field."
}]}

/CC @swapnilt

[swapnilt]

@cipy although your curl example is giving hint regarding duplicate email ids, as I've confirmed above even providing different email addresses gives me the same error.

[kylearoberts]

@swapnilt

Looking at your script that produces the issue it looks like you added the TO address outside of the personalization block. This looks like it resulted in two personalization blocks. One with a TO and one with a BCC. To get things working you need the personalization block to have the TO and BCC in one personalization block. I think what you are looking for is something like this:

personalization = Personalization()
personalization.add_to(Email("xyz@gmail.com"))
personalization.add_bcc(Email("def@gmail.com"))
mail.add_personalization(personalization)

@cipy

Your issue looks different from the first, but to be sure we would likely need to see your code to get a better idea of what is happening. We also might be able to get this figured out using the JSON in your cURL call if you can provide that.

[swapnilt]

@kylearoberts I've taken a lot of time to verify this issue and explain it in as much detail as possible. Request you to please go through everything I've posted. Please see the output yourself by running the script I've posted. Kindly verify that the suggestion which you've given actually works by making those modifications in the script. I've posted the output of the script too and you can clearly see that there is only a single personalization block in the request. If you just try it yourself, you'll see that making the changes which you suggested don't work.

If this was just a sendgrid-python library issue, I'd have been happy to submit a PR myself but this looks like server side issue. So the actual issue is THERE IS NO WAY TO SEND CC/BCC MAILS WITH V3 API. Atleast for me. I urge you to try sending one yourself using curl, python, whatever. That sounds like a pretty major issue to me and I hope the team takes it seriously.

[kylearoberts]

@swapnilt

Sorry for not being clearer in my response. I do want to say that you did a great job of explaining and provided some great information. I have two examples for you below, one is a modification of your code and the other is an example I created. These examples have been tested and shown to be working as expected.

Your code modified:

import urllib
import sendgrid
from pprint import pprint
from sendgrid.helpers.mail import *

SENDGRID_API_KEY = 'SENDGRID_API_KEY'

def send_email(from_email, to_email, subject, text_body, html_body = None):
    sg = sendgrid.SendGridAPIClient(apikey=SENDGRID_API_KEY)
    from_email = Email(from_email)
    to_email = Email(to_email)
    text_content = Content("text/plain", text_body)
    mail = Mail(from_email, subject, to_email)
    mail.add_content(text_content)
    mail.personalizations[0].add_bcc(Email("def@gmail.com"))
    if html_body:
        html_content = Content("text/html", html_body)
        mail.add_content(html_content)
    try:
    	print("sending mail post request: ")
    	pprint(mail.get())
    	print()
    	response = sg.client.mail.send.post(request_body=mail.get())
    except Exception as e:
    	pprint(e.to_dict)
    	return
    if response.status_code > 299 or response.status_code < 200:
    	print("Error while sending email: ", response.body)

    return response


send_email("abc@mydomain.com", 
			"xyz@gmail.com",
			"Test email 2",
			"This is a test email")

My code example:

import sendgrid
from sendgrid.helpers.mail import *


def build_customer_testing():

    mail = Mail()

    mail.from_email = Email("from_email@domain.com", "From Name")
    mail.subject = "Hello World from the SendGrid Python Library"

    personalization = Personalization()
    personalization.add_to(Email("to_email@domain.com"))
    personalization.add_bcc(Email("bcc_email@domain.com"))
    mail.add_personalization(personalization)

    mail.add_content(Content("text/plain", "some text here"))
    mail.add_content(Content("text/html", ("<html><body>some text "
                             "here</body></html>")))

    mail.reply_to = Email("reply_to@domain.com")

    return mail.get()

def send_customer_testing():

    sg = sendgrid.SendGridAPIClient(apikey='API KEY')
    data = build_customer_testing()
    print(data)
    try:
      response = sg.client.mail.send.post(request_body=data)
    except exceptions.BadRequestsError as e:
      print(e.body)
      exit()
    print(response.status_code)
    print(response.body)
    print(response.headers)

send_customer_testing()

To improve our libraries we would love to have your feedback on the helpers here. This is not the only case we have run into where there has been confusion about how to structure things like this and we would like to get feedback to help with improvements that we will make.

If this does not solve your issue please let us know and we will try our best to meet your needs.

[swapnilt]

@kylearoberts Thank you so much for giving the right solution. I can confirm that my examples are now working.

[fswaleh]

Hi @thinkingserious ,

Here is a simple script which reproduces the error

import urllib
import sendgrid
from pprint import pprint
from sendgrid.helpers.mail import *

SENDGRID_API_KEY = 'SENDGRID_API_KEY'

def send_email(from_email, to_email, subject, text_body, html_body = None):
    sg = sendgrid.SendGridAPIClient(apikey=SENDGRID_API_KEY)
    from_email = Email(from_email)
    to_email = Email(to_email)
    text_content = Content("text/plain", text_body)
    mail = Mail(from_email, subject, to_email)
    mail.add_content(text_content)
    personalization = Personalization()
    personalization.add_bcc(Email("def@gmail.com"))
    mail.add_personalization(personalization)
    if html_body:
        html_content = Content("text/html", html_body)
        mail.add_content(html_content)
    try:
    	print("sending mail post request: ")
    	pprint(mail.get())
    	print()
    	response = sg.client.mail.send.post(request_body=mail.get())
    except Exception as e:
    	pprint(e.to_dict)
    	return
    if response.status_code > 299 or response.status_code < 200:
    	print("Error while sending email: ", response.body)

    return response


send_email("abc@mydomain.com", 
			"xyz@gmail.com",
			"Test email 2",
			"This is a test email")

If you replace the SENDGRID_API_KEY string value with your API Key, the script can be run as-is without any modification to reproduce the error. Only dependency is Python 3.7.0 and sendgrid-python 5.6.0. After running the script, I can see the below error message -

(env) swapnil@Swapnils-Macbook:
└─ $ ▶ python test_sendgrid.py 
sending mail post request: 
{'content': [{'type': 'text/plain', 'value': 'This is a test email'}],
 'from': {'email': 'abc@mydomain.com'},
 'personalizations': [{'to': [{'email': 'xyz@gmail.com'}]},
                      {'bcc': [{'email': 'def@gmail.com'}]}],
 'subject': 'Test email 2'}

{'errors': [{'field': 'personalizations.1.to',
             'help': 'http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#message.personalizations.to',
             'message': 'The to array is required for all personalization '
                        'objects, and must have at least one email object with '
                        'a valid email address.'}]}
(env) swapnil@Swapnils-Macbook:
└─ $ ▶ 

As per the error message, the to field is missing while you can clearly see in the data structure which gets posted, that the to field is present. Changing all email ids to real email addresses doesn't help.

Hi Swapnilt

I have the same problem, can you help me out as how I can get the above error details.

Thanks a lot.

[swapnilt]

@shopiex Its actually very simple and annoying. In my case, I wasn't adding the to field the way it was expecting. If you're constructing the Mail object by passing a to field to the constructor, then a personalization block is automatically added to your object so you just edit that instead of creating a new one with Personalization(), like this -

mail = Mail(from_email, subject, to_email)
mail.personalizations[0].add_bcc(Email("def@gmail.com"))

The other way to create Mail object is what kyle has shown -

    # Since no *to* field is passed here, no personalization block gets added
    mail = Mail()   
    mail.from_email = Email("from_email@domain.com", "From Name")
    mail.subject = "Hello World from the SendGrid Python Library"

    # So now you can create one here and pass
    personalization = Personalization()     
    personalization.add_to(Email("to_email@domain.com"))
    personalization.add_bcc(Email("bcc_email@domain.com"))
    mail.add_personalization(personalization)

[codal-npanchal]

Hi All,
can we send email to BCC email list without TO email in SnedgripAPI client? because without TO email i am getting BadRequestError.
can someone please help me on this?
Thanks