Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#19186 closed Bug (fixed)

fail send utf-8 email

Reported by: alex_po Owned by: nobody
Component: Core (Mail) Version: master
Severity: Release blocker Keywords: mail utf8 python3
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When i tried to send email i got error:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 274-277: ordinal not in range(128)

Traceback:

---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/management/commands/shell.py in <module>()
----> 1 EmailMessage('тест', 'тест', 'admin@ya.ru', ['user@yandex.ru']).send()

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/message.py in send(self, fail_silently)
    253             # send to.
    254             return 0
--> 255         return self.get_connection(fail_silently).send_messages([self])
    256 
    257     def attach(self, filename=None, content=None, mimetype=None):

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/backends/smtp.py in send_messages(self, email_messages)
     89             num_sent = 0
     90             for message in email_messages:
---> 91                 sent = self._send(message)
     92                 if sent:
     93                     num_sent += 1

/home/trosna/.virtualenvs/dj_yourpics/src/django/django/core/mail/backends/smtp.py in _send(self, email_message)
    105         try:
    106             self.connection.sendmail(from_email, recipients,
--> 107                     email_message.message().as_string())
    108         except:
    109             if not self.fail_silently:

/usr/local/lib/python3.2/smtplib.py in sendmail(self, from_addr, to_addrs, msg, mail_options, rcpt_options)
    733         esmtp_opts = []
    734         if isinstance(msg, str):
--> 735             msg = _fix_eols(msg).encode('ascii')
    736         if self.does_esmtp:
    737             # Hmmm? what's this? -ddm

UnicodeEncodeError: 'ascii' codec can't encode characters in position 274-277: ordinal not in range(128)

Attachments (1)

19186-1.diff (3.7 KB) - added by claudep 3 years ago.
Encode message before submitting it to sendmail

Download all attachments as: .zip

Change History (10)

comment:1 Changed 3 years ago by lukeplant

  • Keywords python3 added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

I haven't fully reproduced it, but can confirm that EmailMessage.message().as_string is returning a unicode object not a bytestring object, on Python 3.2 and 3.3, as follows.

>>> EmailMessage('тест', 'тест', 'admin@ya.ru', ['user@yandex.ru']).message().as_string()

On Python 2 it is returning a bytestring, and I think that smtplib is expecting a bytestring, hence the problem.

Logically, this method ought to produce a bytestring, since it applies the encoding specified and returns a series of bytes fir for sending over the network (which includes in it a declaration of the charset).

However, our SafeMIMEText class that is doing this inherits from the stdlib MIMEText class, and it follows the implementation of MIMEText. MIMEText.as_string() also produces a unicode string under Python 3. This seems crazy to me, since the entire job of MIMEText is to encode the input data as an email message ready to be sent over SMTP. But it's difficult to believe they did this without thought.

It may be that we are not supposed to be using the as_string() method to render to a string. But it's not clear what we should be using.

comment:2 Changed 3 years ago by lukeplant

Just to confirm: stdlib MIMEText encodes unicode objects to bytestrings under Python 2.7:

In [4]: MIMEText(u"€", "plain", "utf-8").as_string()
Out[4]: 'Content-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\n\n\xe2\x82\xac'

But not under Python 3.2 or 3.3:

In [3]: MIMEText("€", "plain", "utf-8").as_string()
Out[3]: 'Content-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\n\n€'

comment:3 Changed 3 years ago by claudep

From #11212, we decided to not use any transfer encoding (base64/quote-printable), base64 being the default in the standard lib. That may explain why this issue has not appeared elsewhere before.

Now if we decide to encode the payload transmitted to MIMEText, it will fail in Generator._handle_text which checks that input is a string (not bytestring).

Changed 3 years ago by claudep

Encode message before submitting it to sendmail

comment:4 Changed 3 years ago by claudep

  • Has patch set

comment:5 Changed 3 years ago by claudep

  • Severity changed from Normal to Release blocker

comment:6 Changed 3 years ago by aaugustin

  • Triage Stage changed from Accepted to Ready for checkin

Assuming that the two last chunks of the diff (changes to existing tests) don't have any effect under Python 2, and are only necessary to make the tests meaningful under Python 3, this is ready for commit.

comment:7 Changed 3 years ago by Claude Paroz <claude@…>

  • Resolution set to fixed
  • Status changed from new to closed

In 1620c27936718a87bbc8acce7886ef20fee2b3fe:

Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.

comment:8 Changed 3 years ago by Claude Paroz <claude@…>

In 8967906e0a38884665cf542dfb2d84b087d23d99:

[1.5.x] Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.
Backport of 1620c27936 from master.

comment:9 Changed 3 years ago by Preston Holmes <preston@…>

In f0044618e7e8de122d721670b2f1b77abeabcf27:

Fixed #19186 -- Fixed sending mail with unicode content on Python 3

Thanks alex_po for the report and Luke Plant for the analysis.

Note: See TracTickets for help on using tickets.
Back to Top