Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#19186 closed Bug (fixed)

fail send utf-8 email

Reported by: alex_po Owned by: nobody
Component: Core (Mail) Version: master
Severity: Release blocker Keywords: mail utf8 python3
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When i tried to send email i got error:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 274-277: ordinal not in range(128)

Traceback:

---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/management/commands/shell.py in <module>()
----> 1 EmailMessage('тест', 'тест', 'admin@ya.ru', ['user@yandex.ru']).send()

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/message.py in send(self, fail_silently)
    253             # send to.
    254             return 0
--> 255         return self.get_connection(fail_silently).send_messages([self])
    256 
    257     def attach(self, filename=None, content=None, mimetype=None):

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/backends/smtp.py in send_messages(self, email_messages)
     89             num_sent = 0
     90             for message in email_messages:
---> 91                 sent = self._send(message)
     92                 if sent:
     93                     num_sent += 1

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/backends/smtp.py in _send(self, email_message)
    105         try:
    106             self.connection.sendmail(from_email, recipients,
--> 107                     email_message.message().as_string())
    108         except:
    109             if not self.fail_silently:

/usr/local/lib/python3.2/smtplib.py in sendmail(self, from_addr, to_addrs, msg, mail_options, rcpt_options)
    733         esmtp_opts = []
    734         if isinstance(msg, str):
--> 735             msg = _fix_eols(msg).encode('ascii')
    736         if self.does_esmtp:
    737             # Hmmm? what's this? -ddm

UnicodeEncodeError: 'ascii' codec can't encode characters in position 274-277: ordinal not in range(128)

Attachments (1)

19186-1.diff (3.7 KB) - added by Claude Paroz 4 years ago.
Encode message before submitting it to sendmail

Download all attachments as: .zip

Change History (10)

comment:1 Changed 4 years ago by Luke Plant

Keywords: python3 added
Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Triage Stage: UnreviewedAccepted

I haven't fully reproduced it, but can confirm that EmailMessage.message().as_string is returning a unicode object not a bytestring object, on Python 3.2 and 3.3, as follows.

>>> EmailMessage('тест', 'тест', 'admin@ya.ru', ['user@yandex.ru']).message().as_string()

On Python 2 it is returning a bytestring, and I think that smtplib is expecting a bytestring, hence the problem.

Logically, this method ought to produce a bytestring, since it applies the encoding specified and returns a series of bytes fir for sending over the network (which includes in it a declaration of the charset).

However, our SafeMIMEText class that is doing this inherits from the stdlib MIMEText class, and it follows the implementation of MIMEText. MIMEText.as_string() also produces a unicode string under Python 3. This seems crazy to me, since the entire job of MIMEText is to encode the input data as an email message ready to be sent over SMTP. But it's difficult to believe they did this without thought.

It may be that we are not supposed to be using the as_string() method to render to a string. But it's not clear what we should be using.

comment:2 Changed 4 years ago by Luke Plant

Just to confirm: stdlib MIMEText encodes unicode objects to bytestrings under Python 2.7:

In [4]: MIMEText(u"€", "plain", "utf-8").as_string()
Out[4]: 'Content-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\n\n\xe2\x82\xac'

But not under Python 3.2 or 3.3:

In [3]: MIMEText("€", "plain", "utf-8").as_string()
Out[3]: 'Content-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\n\n€'

comment:3 Changed 4 years ago by Claude Paroz

From #11212, we decided to not use any transfer encoding (base64/quote-printable), base64 being the default in the standard lib. That may explain why this issue has not appeared elsewhere before.

Now if we decide to encode the payload transmitted to MIMEText, it will fail in Generator._handle_text which checks that input is a string (not bytestring).

Changed 4 years ago by Claude Paroz

Attachment: 19186-1.diff added

Encode message before submitting it to sendmail

comment:4 Changed 4 years ago by Claude Paroz

Has patch: set

comment:5 Changed 4 years ago by Claude Paroz

Severity: NormalRelease blocker

comment:6 Changed 4 years ago by Aymeric Augustin

Triage Stage: AcceptedReady for checkin

Assuming that the two last chunks of the diff (changes to existing tests) don't have any effect under Python 2, and are only necessary to make the tests meaningful under Python 3, this is ready for commit.

comment:7 Changed 4 years ago by Claude Paroz <claude@…>

Resolution: fixed
Status: newclosed

In 1620c27936718a87bbc8acce7886ef20fee2b3fe:

Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.

comment:8 Changed 4 years ago by Claude Paroz <claude@…>

In 8967906e0a38884665cf542dfb2d84b087d23d99:

[1.5.x] Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.
Backport of 1620c27936 from master.

comment:9 Changed 4 years ago by Preston Holmes <preston@…>

In f0044618e7e8de122d721670b2f1b77abeabcf27:

Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.

Note: See TracTickets for help on using tickets.
Back to Top