Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#19186 closed Bug (fixed)

fail send utf-8 email

Reported by: alex_po Owned by: nobody
Component: Core (Mail) Version: dev
Severity: Release blocker Keywords: mail utf8 python3
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When i tried to send email i got error:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 274-277: ordinal not in range(128)

Traceback:

---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/management/commands/shell.py in <module>()
----> 1 EmailMessage('тест', 'тест', 'admin@ya.ru', ['user@yandex.ru']).send()

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/message.py in send(self, fail_silently)
    253             # send to.
    254             return 0
--> 255         return self.get_connection(fail_silently).send_messages([self])
    256 
    257     def attach(self, filename=None, content=None, mimetype=None):

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/backends/smtp.py in send_messages(self, email_messages)
     89             num_sent = 0
     90             for message in email_messages:
---> 91                 sent = self._send(message)
     92                 if sent:
     93                     num_sent += 1

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/backends/smtp.py in _send(self, email_message)
    105         try:
    106             self.connection.sendmail(from_email, recipients,
--> 107                     email_message.message().as_string())
    108         except:
    109             if not self.fail_silently:

/usr/local/lib/python3.2/smtplib.py in sendmail(self, from_addr, to_addrs, msg, mail_options, rcpt_options)
    733         esmtp_opts = []
    734         if isinstance(msg, str):
--> 735             msg = _fix_eols(msg).encode('ascii')
    736         if self.does_esmtp:
    737             # Hmmm? what's this? -ddm

UnicodeEncodeError: 'ascii' codec can't encode characters in position 274-277: ordinal not in range(128)

Attachments (1)

19186-1.diff (3.7 KB ) - added by Claude Paroz 11 years ago.
Encode message before submitting it to sendmail

Download all attachments as: .zip

Change History (10)

comment:1 by Luke Plant, 11 years ago

Keywords: python3 added
Triage Stage: UnreviewedAccepted

I haven't fully reproduced it, but can confirm that EmailMessage.message().as_string is returning a unicode object not a bytestring object, on Python 3.2 and 3.3, as follows.

>>> EmailMessage('тест', 'тест', 'admin@ya.ru', ['user@yandex.ru']).message().as_string()

On Python 2 it is returning a bytestring, and I think that smtplib is expecting a bytestring, hence the problem.

Logically, this method ought to produce a bytestring, since it applies the encoding specified and returns a series of bytes fir for sending over the network (which includes in it a declaration of the charset).

However, our SafeMIMEText class that is doing this inherits from the stdlib MIMEText class, and it follows the implementation of MIMEText. MIMEText.as_string() also produces a unicode string under Python 3. This seems crazy to me, since the entire job of MIMEText is to encode the input data as an email message ready to be sent over SMTP. But it's difficult to believe they did this without thought.

It may be that we are not supposed to be using the as_string() method to render to a string. But it's not clear what we should be using.

comment:2 by Luke Plant, 11 years ago

Just to confirm: stdlib MIMEText encodes unicode objects to bytestrings under Python 2.7:

In [4]: MIMEText(u"€", "plain", "utf-8").as_string()
Out[4]: 'Content-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\n\n\xe2\x82\xac'

But not under Python 3.2 or 3.3:

In [3]: MIMEText("€", "plain", "utf-8").as_string()
Out[3]: 'Content-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\n\n€'

comment:3 by Claude Paroz, 11 years ago

From #11212, we decided to not use any transfer encoding (base64/quote-printable), base64 being the default in the standard lib. That may explain why this issue has not appeared elsewhere before.

Now if we decide to encode the payload transmitted to MIMEText, it will fail in Generator._handle_text which checks that input is a string (not bytestring).

by Claude Paroz, 11 years ago

Attachment: 19186-1.diff added

Encode message before submitting it to sendmail

comment:4 by Claude Paroz, 11 years ago

Has patch: set

comment:5 by Claude Paroz, 11 years ago

Severity: NormalRelease blocker

comment:6 by Aymeric Augustin, 11 years ago

Triage Stage: AcceptedReady for checkin

Assuming that the two last chunks of the diff (changes to existing tests) don't have any effect under Python 2, and are only necessary to make the tests meaningful under Python 3, this is ready for commit.

comment:7 by Claude Paroz <claude@…>, 11 years ago

Resolution: fixed
Status: newclosed

In 1620c27936718a87bbc8acce7886ef20fee2b3fe:

Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.

comment:8 by Claude Paroz <claude@…>, 11 years ago

In 8967906e0a38884665cf542dfb2d84b087d23d99:

[1.5.x] Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.
Backport of 1620c27936 from master.

comment:9 by Preston Holmes <preston@…>, 11 years ago

In f0044618e7e8de122d721670b2f1b77abeabcf27:

Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.

Note: See TracTickets for help on using tickets.
Back to Top