Opened 11 years ago

Closed 9 years ago

#19107 closed Bug (fixed)

Can't send unicode e-mail

Reported by: carneiro.be@… Owned by: nobody
Component: Core (Mail) Version: 1.6
Severity: Release blocker Keywords: mail unicode
Cc: Ben Kornrath, Tim Graham Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Hello guys,

I'm on django dev and having troubles sending unicode emails. Here's the relevant code:

# person is a normal django user
subject = u"Welcome to Odeio - %s" % person.name
message = render_to_string('accounts/registration_email.txt', ctx_dict)
person.email_user(subject, message)

Error:

'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

Stacktrace:

File "/odeio/django/core/mail/__init__.py", line 62, in send_mail
    connection=connection).send()
 
  File "/odeio/django/core/mail/message.py", line 251, in send
    return self.get_connection(fail_silently).send_messages([self])
 
  File "/odeio/django/core/mail/backends/smtp.py", line 91, in send_messages
    sent = self._send(message)
   
  File "/odeio/django/core/mail/backends/smtp.py", line 106, in _send
    self.connection.sendmail(from_email, recipients, email_message.message().as_string())
 
  File "/odeio/django/core/mail/message.py", line 141, in as_string
    g.flatten(self, unixfrom=unixfrom)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 84, in flatten
    self._write(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 109, in _write
    self._dispatch(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 135, in _dispatch
    meth(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 178, in _handle_text
    self._fp.write(payload)
 
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

Thanks in advance for the help!

Attachments (2)

19107.diff (777 bytes) - added by Claude Paroz 11 years ago.
Workaround for python bug 1368247
19107-django-1.6.5.patch (550 bytes) - added by Ben Kornrath 9 years ago.
Patch to fix this issue on Django 1.6.5.

Download all attachments as: .zip

Change History (23)

comment:1 Changed 11 years ago by Claude Paroz

Can you isolate if the encoding error is due to unicode in subject, in message, or both? If it is related to message, can you also give us the output of the render_to_string call?

comment:2 Changed 11 years ago by carneiro.be@…

Hello claudep,
thank you for your reply. I believe the encoding error is due to unicode in the message, because the character at position 2 in the message is "á", which is not ASCII. It seems like the subject is ok, even though it might contain unicode characters (person.name might be unicode). Here's the output of render_to_string:

Olá My name,

seja bem-vindo ao sistema de administração jurídica online Odeio. O escritório Departamento Jurídico cadastrou-lhe no sistema. Os seus dados são:

Nome: My name
Usuário / Email: my-email@gmail.com

Para validar o seu cadastro e criar uma senha, entre no seguinte link:
http://odeio.com.br/pessoas/validar/130281cba7106776845efc1ab274776b02ff5206/

Qualquer dúvida, sugestão ou crítica não hesite em entrar em contato. Seu feedback é muito bem-vindo.

Obrigado,
Equipe Odeio

Thanks again for your help.

comment:3 Changed 11 years ago by carneiro.be@…

I have just double checked the subject and message. Removing all unicode characters from the message results in the message being sent, even when I added unicode characters to the subject.

comment:4 Changed 11 years ago by Claude Paroz

Thanks, can you still tell us the exact version of Python, the encoding of the file, the value of settings.FILE_CHARSET, settings.DEFAULT_CHARSET and sys.getdefaultencoding()?

comment:5 Changed 11 years ago by carneiro.be@…

Hi claudep,

thanks for the quick reply. My python version is Python 2.6.5. I haven't set FILE_CHARSET neither DEFAULT_CHARSET, so I assume it's utf-8. I'm going to set it manually and try again. sys.getdefaultencoding() returns 'ascii'. For the file I did this: file -i registration_email.txt
registration_email.txt: text/plain; charset=utf-8. So I guess it should be okay.

I've tried again setting FILE_CHARSET and DEFAULT_CHARSET to utf-8 but it still fails with the same stacktrace.

If you want I can change the code to introduce any tests you wish. Many thanks again for the help.

comment:6 Changed 11 years ago by Claude Paroz

I think i get it. This issue is probably related to http://bugs.python.org/issue1368247 that has been fixed in Python 2.6.6 (http://hg.python.org/cpython/rev/dd378b885b94/). I will attach a patch with a workaround for Python < 2.6.6. Can you test it?

Changed 11 years ago by Claude Paroz

Attachment: 19107.diff added

Workaround for python bug 1368247

comment:7 Changed 11 years ago by carneiro.be@…

OH. MY. GOD. Seriously, are you a wizard? Because that's some serious black magic you just performed. That did fix it. How did you manage to find such bug in python? Wow... many many many many many thanks. That'd have taken me 10 years to figure out.

comment:8 Changed 11 years ago by Claude Paroz

Great :-) It is not so magic. I couldn't reproduce the failure. The only obvious difference between your setup and mine was the Python version. So I looked for the NEWS file of recent Python 2.6 releases and searched for issues related to MIME. I admit I was lucky to find a match so quickly.

comment:9 Changed 11 years ago by Claude Paroz

Has patch: set
Triage Stage: UnreviewedReady for checkin

comment:10 Changed 11 years ago by Claude Paroz <claude@…>

Resolution: fixed
Status: newclosed

In 501d793398c3ecc3c8a54475d07609cf34643553:

Fixed #19107 -- Workarounded message-encoding bug on Python < 2.6.6

Thanks Bernardo Pires for the report.

comment:11 Changed 9 years ago by Ben Kornrath

The fix seems to have been removed from the latest version of Django as I'm getting this problem with Django 1.6.5 on Python 2.6.5.

Code that demonstrates the problem (replace with valid emails):

from core.models import Setting
from django.core.mail import send_mail
send_mail('hello', u'マウス', 'noreply@example.com', ('test@example.com',))

System information:

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.4 LTS"
$ python --version
Python 2.6.5

I'll attach the patch I'm using to fix the issue. Should I re-open this issue? Should I create a pull request for this? Thanks.

Changed 9 years ago by Ben Kornrath

Attachment: 19107-django-1.6.5.patch added

Patch to fix this issue on Django 1.6.5.

comment:12 Changed 9 years ago by Ben Kornrath

Cc: Ben Kornrath added

comment:13 Changed 9 years ago by Tim Graham

Resolution: fixed
Severity: NormalRelease blocker
Status: closednew
Version: master1.6

Looks like the code was removed in 35a447a263115b22a7040a9f4a16ff4bd2078466. I'm not sure it was intentional, but I think we should probably add it back since Ubuntu 10.04 server with that version of Python is supported until April 2015. I will check with Florian.

comment:14 Changed 9 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In e4b2bea743944b67a55d0ef3d17982a2319f866d:

[1.6.x] Fixed #19107 -- Restored bug fix for sending unicode email with Python 2.6.5 and below.

comment:15 Changed 9 years ago by Greg Humphreys

I'm still encountering this bug on Ubuntu 10.04 with Django 1.6.6, even though your patch has landed there. In fact, upgrading to Django 1.6.6 (from 1.4.1) was what caused the issue.

The stack trace is the same as in the OP. What information would be useful to help reproduce?

comment:16 Changed 9 years ago by Greg Humphreys

Resolution: fixed
Status: closednew

comment:17 Changed 9 years ago by Greg Humphreys

Here's the portion of the email that appears to be causing issues (printed with 'print email.dict'):

Hi\r\n\r\nWhen browsing with chrome on my Android tablet (Samsung Galaxy 12 Note Pro), the suit symbols don\u2019t show *in the diagrams*, which is of course only a problem for the bidding (it\u2019s ok in the text, although without colors).

comment:18 Changed 9 years ago by Tim Graham

Cc: Tim Graham added
Has patch: unset
Triage Stage: Ready for checkinAccepted

A working, tested patch would be needed as I don't have motivation to reproduce and fix the issue myself.

comment:19 Changed 9 years ago by Claude Paroz

Our FAQ is clear: https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django.
You should really try to install the most recent 2.6 Python version. Unfortunately, it seems Ubuntu doesn't care much (https://bugs.launchpad.net/ubuntu/+source/python2.6/+bug/788525).

comment:20 Changed 9 years ago by Ben Kornrath

The fix in Django 1.6.6 using Python 2.6.5 in Ubuntu 10.04 is working for me. @Humper: You could try updating to the latest version of the Python 2.6.5 package. I think the Ubuntu devs have back-ported some fixes from later versions of Python to the 2.6.5 package and that might be why things are working for you. Just an idea though.

comment:21 Changed 9 years ago by Tim Graham

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.
Back to Top