fail send utf-8 email

[alex_po]

When i tried to send email i got error:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 274-277: ordinal not in range(128)

Traceback:

---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/management/commands/shell.py in <module>()
----> 1 EmailMessage('тест', 'тест', 'admin@ya.ru', ['user@yandex.ru']).send()

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/message.py in send(self, fail_silently)
    253             # send to.
    254             return 0
--> 255         return self.get_connection(fail_silently).send_messages([self])
    256 
    257     def attach(self, filename=None, content=None, mimetype=None):

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/backends/smtp.py in send_messages(self, email_messages)
     89             num_sent = 0
     90             for message in email_messages:
---> 91                 sent = self._send(message)
     92                 if sent:
     93                     num_sent += 1

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/backends/smtp.py in _send(self, email_message)
    105         try:
    106             self.connection.sendmail(from_email, recipients,
--> 107                     email_message.message().as_string())
    108         except:
    109             if not self.fail_silently:

/usr/local/lib/python3.2/smtplib.py in sendmail(self, from_addr, to_addrs, msg, mail_options, rcpt_options)
    733         esmtp_opts = []
    734         if isinstance(msg, str):
--> 735             msg = _fix_eols(msg).encode('ascii')
    736         if self.does_esmtp:
    737             # Hmmm? what's this? -ddm

UnicodeEncodeError: 'ascii' codec can't encode characters in position 274-277: ordinal not in range(128)

[Luke Plant]

I haven't fully reproduced it, but can confirm that EmailMessage.message().as_string is returning a unicode object not a bytestring object, on Python 3.2 and 3.3, as follows.

>>> EmailMessage('тест', 'тест', 'admin@ya.ru', ['user@yandex.ru']).message().as_string()

On Python 2 it is returning a bytestring, and I think that smtplib is expecting a bytestring, hence the problem.

Logically, this method ought to produce a bytestring, since it applies the encoding specified and returns a series of bytes fir for sending over the network (which includes in it a declaration of the charset).

However, our SafeMIMEText class that is doing this inherits from the stdlib MIMEText class, and it follows the implementation of MIMEText. MIMEText.as_string() also produces a unicode string under Python 3. This seems crazy to me, since the entire job of MIMEText is to encode the input data as an email message ready to be sent over SMTP. But it's difficult to believe they did this without thought.

It may be that we are not supposed to be using the as_string() method to render to a string. But it's not clear what we should be using.

[Luke Plant]

Just to confirm: stdlib MIMEText encodes unicode objects to bytestrings under Python 2.7:

In [4]: MIMEText(u"€", "plain", "utf-8").as_string()
Out[4]: 'Content-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\n\n\xe2\x82\xac'

But not under Python 3.2 or 3.3:

In [3]: MIMEText("€", "plain", "utf-8").as_string()
Out[3]: 'Content-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\n\n€'

[Claude Paroz]

From #11212, we decided to not use any transfer encoding (base64/quote-printable), base64 being the default in the standard lib. That may explain why this issue has not appeared elsewhere before.

Now if we decide to encode the payload transmitted to MIMEText, it will fail in Generator._handle_text which checks that input is a string (not bytestring).

[Claude Paroz]

Encode message before submitting it to sendmail

[Aymeric Augustin]

Assuming that the two last chunks of the diff (changes to existing tests) don't have any effect under Python 2, and are only necessary to make the tests meaningful under Python 3, this is ready for commit.

[Claude Paroz <claude@…>]

In 1620c27936718a87bbc8acce7886ef20fee2b3fe:

Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.

[Claude Paroz <claude@…>]

In 8967906e0a38884665cf542dfb2d84b087d23d99:

[1.5.x] Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.
Backport of 1620c27936 from master.

[Preston Holmes <preston@…>]

In f0044618e7e8de122d721670b2f1b77abeabcf27:

Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.