Opened 11 years ago

Closed 10 years ago

#19107 closed Bug (fixed)

Can't send unicode e-mail

Reported by: carneiro.be@… Owned by: nobody
Component: Core (Mail) Version: 1.6
Severity: Release blocker Keywords: mail unicode
Cc: Ben Kornrath, Tim Graham Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Hello guys,

I'm on django dev and having troubles sending unicode emails. Here's the relevant code:

# person is a normal django user
subject = u"Welcome to Odeio - %s" % person.name
message = render_to_string('accounts/registration_email.txt', ctx_dict)
person.email_user(subject, message)

Error:

'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

Stacktrace:

File "/odeio/django/core/mail/__init__.py", line 62, in send_mail
    connection=connection).send()
 
  File "/odeio/django/core/mail/message.py", line 251, in send
    return self.get_connection(fail_silently).send_messages([self])
 
  File "/odeio/django/core/mail/backends/smtp.py", line 91, in send_messages
    sent = self._send(message)
   
  File "/odeio/django/core/mail/backends/smtp.py", line 106, in _send
    self.connection.sendmail(from_email, recipients, email_message.message().as_string())
 
  File "/odeio/django/core/mail/message.py", line 141, in as_string
    g.flatten(self, unixfrom=unixfrom)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 84, in flatten
    self._write(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 109, in _write
    self._dispatch(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 135, in _dispatch
    meth(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 178, in _handle_text
    self._fp.write(payload)
 
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

Thanks in advance for the help!

Attachments (2)

19107.diff (777 bytes ) - added by Claude Paroz 11 years ago.
Workaround for python bug 1368247
19107-django-1.6.5.patch (550 bytes ) - added by Ben Kornrath 10 years ago.
Patch to fix this issue on Django 1.6.5.

Download all attachments as: .zip

Change History (23)

comment:1 by Claude Paroz, 11 years ago

Can you isolate if the encoding error is due to unicode in subject, in message, or both? If it is related to message, can you also give us the output of the render_to_string call?

comment:2 by carneiro.be@…, 11 years ago

Hello claudep,
thank you for your reply. I believe the encoding error is due to unicode in the message, because the character at position 2 in the message is "á", which is not ASCII. It seems like the subject is ok, even though it might contain unicode characters (person.name might be unicode). Here's the output of render_to_string:

Olá My name,

seja bem-vindo ao sistema de administração jurídica online Odeio. O escritório Departamento Jurídico cadastrou-lhe no sistema. Os seus dados são:

Nome: My name
Usuário / Email: my-email@gmail.com

Para validar o seu cadastro e criar uma senha, entre no seguinte link:
http://odeio.com.br/pessoas/validar/130281cba7106776845efc1ab274776b02ff5206/

Qualquer dúvida, sugestão ou crítica não hesite em entrar em contato. Seu feedback é muito bem-vindo.

Obrigado,
Equipe Odeio

Thanks again for your help.

comment:3 by carneiro.be@…, 11 years ago

I have just double checked the subject and message. Removing all unicode characters from the message results in the message being sent, even when I added unicode characters to the subject.

comment:4 by Claude Paroz, 11 years ago

Thanks, can you still tell us the exact version of Python, the encoding of the file, the value of settings.FILE_CHARSET, settings.DEFAULT_CHARSET and sys.getdefaultencoding()?

comment:5 by carneiro.be@…, 11 years ago

Hi claudep,

thanks for the quick reply. My python version is Python 2.6.5. I haven't set FILE_CHARSET neither DEFAULT_CHARSET, so I assume it's utf-8. I'm going to set it manually and try again. sys.getdefaultencoding() returns 'ascii'. For the file I did this: file -i registration_email.txt
registration_email.txt: text/plain; charset=utf-8. So I guess it should be okay.

I've tried again setting FILE_CHARSET and DEFAULT_CHARSET to utf-8 but it still fails with the same stacktrace.

If you want I can change the code to introduce any tests you wish. Many thanks again for the help.

comment:6 by Claude Paroz, 11 years ago

I think i get it. This issue is probably related to http://bugs.python.org/issue1368247 that has been fixed in Python 2.6.6 (http://hg.python.org/cpython/rev/dd378b885b94/). I will attach a patch with a workaround for Python < 2.6.6. Can you test it?

by Claude Paroz, 11 years ago

Attachment: 19107.diff added

Workaround for python bug 1368247

comment:7 by carneiro.be@…, 11 years ago

OH. MY. GOD. Seriously, are you a wizard? Because that's some serious black magic you just performed. That did fix it. How did you manage to find such bug in python? Wow... many many many many many thanks. That'd have taken me 10 years to figure out.

comment:8 by Claude Paroz, 11 years ago

Great :-) It is not so magic. I couldn't reproduce the failure. The only obvious difference between your setup and mine was the Python version. So I looked for the NEWS file of recent Python 2.6 releases and searched for issues related to MIME. I admit I was lucky to find a match so quickly.

comment:9 by Claude Paroz, 11 years ago

Has patch: set
Triage Stage: UnreviewedReady for checkin

comment:10 by Claude Paroz <claude@…>, 11 years ago

Resolution: fixed
Status: newclosed

In 501d793398c3ecc3c8a54475d07609cf34643553:

Fixed #19107 -- Workarounded message-encoding bug on Python < 2.6.6

Thanks Bernardo Pires for the report.

comment:11 by Ben Kornrath, 10 years ago

The fix seems to have been removed from the latest version of Django as I'm getting this problem with Django 1.6.5 on Python 2.6.5.

Code that demonstrates the problem (replace with valid emails):

from core.models import Setting
from django.core.mail import send_mail
send_mail('hello', u'マウス', 'noreply@example.com', ('test@example.com',))

System information:

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.4 LTS"
$ python --version
Python 2.6.5

I'll attach the patch I'm using to fix the issue. Should I re-open this issue? Should I create a pull request for this? Thanks.

by Ben Kornrath, 10 years ago

Attachment: 19107-django-1.6.5.patch added

Patch to fix this issue on Django 1.6.5.

comment:12 by Ben Kornrath, 10 years ago

Cc: Ben Kornrath added

comment:13 by Tim Graham, 10 years ago

Resolution: fixed
Severity: NormalRelease blocker
Status: closednew
Version: master1.6

Looks like the code was removed in 35a447a263115b22a7040a9f4a16ff4bd2078466. I'm not sure it was intentional, but I think we should probably add it back since Ubuntu 10.04 server with that version of Python is supported until April 2015. I will check with Florian.

comment:14 by Tim Graham <timograham@…>, 10 years ago

Resolution: fixed
Status: newclosed

In e4b2bea743944b67a55d0ef3d17982a2319f866d:

[1.6.x] Fixed #19107 -- Restored bug fix for sending unicode email with Python 2.6.5 and below.

comment:15 by Greg Humphreys, 10 years ago

I'm still encountering this bug on Ubuntu 10.04 with Django 1.6.6, even though your patch has landed there. In fact, upgrading to Django 1.6.6 (from 1.4.1) was what caused the issue.

The stack trace is the same as in the OP. What information would be useful to help reproduce?

comment:16 by Greg Humphreys, 10 years ago

Resolution: fixed
Status: closednew

comment:17 by Greg Humphreys, 10 years ago

Here's the portion of the email that appears to be causing issues (printed with 'print email.dict'):

Hi\r\n\r\nWhen browsing with chrome on my Android tablet (Samsung Galaxy 12 Note Pro), the suit symbols don\u2019t show *in the diagrams*, which is of course only a problem for the bidding (it\u2019s ok in the text, although without colors).

comment:18 by Tim Graham, 10 years ago

Cc: Tim Graham added
Has patch: unset
Triage Stage: Ready for checkinAccepted

A working, tested patch would be needed as I don't have motivation to reproduce and fix the issue myself.

comment:19 by Claude Paroz, 10 years ago

Our FAQ is clear: https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django.
You should really try to install the most recent 2.6 Python version. Unfortunately, it seems Ubuntu doesn't care much (https://bugs.launchpad.net/ubuntu/+source/python2.6/+bug/788525).

comment:20 by Ben Kornrath, 10 years ago

The fix in Django 1.6.6 using Python 2.6.5 in Ubuntu 10.04 is working for me. @Humper: You could try updating to the latest version of the Python 2.6.5 package. I think the Ubuntu devs have back-ported some fixes from later versions of Python to the 2.6.5 package and that might be why things are working for you. Just an idea though.

comment:21 by Tim Graham, 10 years ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.
Back to Top