Opened 3 years ago

Closed 12 months ago

#19107 closed Bug (fixed)

Can't send unicode e-mail

Reported by: carneiro.be@… Owned by: nobody
Component: Core (Mail) Version: 1.6
Severity: Release blocker Keywords: mail unicode
Cc: benkonrath, timgraham Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Hello guys,

I'm on django dev and having troubles sending unicode emails. Here's the relevant code:

# person is a normal django user
subject = u"Welcome to Odeio - %s" % person.name
message = render_to_string('accounts/registration_email.txt', ctx_dict)
person.email_user(subject, message)

Error:

'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

Stacktrace:

File "/odeio/django/core/mail/__init__.py", line 62, in send_mail
    connection=connection).send()
 
  File "/odeio/django/core/mail/message.py", line 251, in send
    return self.get_connection(fail_silently).send_messages([self])
 
  File "/odeio/django/core/mail/backends/smtp.py", line 91, in send_messages
    sent = self._send(message)
   
  File "/odeio/django/core/mail/backends/smtp.py", line 106, in _send
    self.connection.sendmail(from_email, recipients, email_message.message().as_string())
 
  File "/odeio/django/core/mail/message.py", line 141, in as_string
    g.flatten(self, unixfrom=unixfrom)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 84, in flatten
    self._write(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 109, in _write
    self._dispatch(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 135, in _dispatch
    meth(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 178, in _handle_text
    self._fp.write(payload)
 
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

Thanks in advance for the help!

Attachments (2)

19107.diff (777 bytes) - added by claudep 3 years ago.
Workaround for python bug 1368247
19107-django-1.6.5.patch (550 bytes) - added by benkonrath 13 months ago.
Patch to fix this issue on Django 1.6.5.

Download all attachments as: .zip

Change History (23)

comment:1 Changed 3 years ago by claudep

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Can you isolate if the encoding error is due to unicode in subject, in message, or both? If it is related to message, can you also give us the output of the render_to_string call?

comment:2 Changed 3 years ago by carneiro.be@…

Hello claudep,
thank you for your reply. I believe the encoding error is due to unicode in the message, because the character at position 2 in the message is "á", which is not ASCII. It seems like the subject is ok, even though it might contain unicode characters (person.name might be unicode). Here's the output of render_to_string:

Olá My name,

seja bem-vindo ao sistema de administração jurídica online Odeio. O escritório Departamento Jurídico cadastrou-lhe no sistema. Os seus dados são:

Nome: My name
Usuário / Email: my-email@gmail.com

Para validar o seu cadastro e criar uma senha, entre no seguinte link:
http://odeio.com.br/pessoas/validar/130281cba7106776845efc1ab274776b02ff5206/

Qualquer dúvida, sugestão ou crítica não hesite em entrar em contato. Seu feedback é muito bem-vindo.

Obrigado,
Equipe Odeio

Thanks again for your help.

comment:3 Changed 3 years ago by carneiro.be@…

I have just double checked the subject and message. Removing all unicode characters from the message results in the message being sent, even when I added unicode characters to the subject.

comment:4 Changed 3 years ago by claudep

Thanks, can you still tell us the exact version of Python, the encoding of the file, the value of settings.FILE_CHARSET, settings.DEFAULT_CHARSET and sys.getdefaultencoding()?

comment:5 Changed 3 years ago by carneiro.be@…

Hi claudep,

thanks for the quick reply. My python version is Python 2.6.5. I haven't set FILE_CHARSET neither DEFAULT_CHARSET, so I assume it's utf-8. I'm going to set it manually and try again. sys.getdefaultencoding() returns 'ascii'. For the file I did this: file -i registration_email.txt
registration_email.txt: text/plain; charset=utf-8. So I guess it should be okay.

I've tried again setting FILE_CHARSET and DEFAULT_CHARSET to utf-8 but it still fails with the same stacktrace.

If you want I can change the code to introduce any tests you wish. Many thanks again for the help.

comment:6 Changed 3 years ago by claudep

I think i get it. This issue is probably related to http://bugs.python.org/issue1368247 that has been fixed in Python 2.6.6 (http://hg.python.org/cpython/rev/dd378b885b94/). I will attach a patch with a workaround for Python < 2.6.6. Can you test it?

Changed 3 years ago by claudep

Workaround for python bug 1368247

comment:7 Changed 3 years ago by carneiro.be@…

OH. MY. GOD. Seriously, are you a wizard? Because that's some serious black magic you just performed. That did fix it. How did you manage to find such bug in python? Wow... many many many many many thanks. That'd have taken me 10 years to figure out.

comment:8 Changed 3 years ago by claudep

Great :-) It is not so magic. I couldn't reproduce the failure. The only obvious difference between your setup and mine was the Python version. So I looked for the NEWS file of recent Python 2.6 releases and searched for issues related to MIME. I admit I was lucky to find a match so quickly.

comment:9 Changed 3 years ago by claudep

  • Has patch set
  • Triage Stage changed from Unreviewed to Ready for checkin

comment:10 Changed 3 years ago by Claude Paroz <claude@…>

  • Resolution set to fixed
  • Status changed from new to closed

In 501d793398c3ecc3c8a54475d07609cf34643553:

Fixed #19107 -- Workarounded message-encoding bug on Python < 2.6.6

Thanks Bernardo Pires for the report.

comment:11 Changed 13 months ago by benkonrath

The fix seems to have been removed from the latest version of Django as I'm getting this problem with Django 1.6.5 on Python 2.6.5.

Code that demonstrates the problem (replace with valid emails):

from core.models import Setting
from django.core.mail import send_mail
send_mail('hello', u'マウス', 'noreply@example.com', ('test@example.com',))

System information:

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.4 LTS"
$ python --version
Python 2.6.5

I'll attach the patch I'm using to fix the issue. Should I re-open this issue? Should I create a pull request for this? Thanks.

Changed 13 months ago by benkonrath

Patch to fix this issue on Django 1.6.5.

comment:12 Changed 13 months ago by benkonrath

  • Cc benkonrath added

comment:13 Changed 13 months ago by timo

  • Resolution fixed deleted
  • Severity changed from Normal to Release blocker
  • Status changed from closed to new
  • Version changed from master to 1.6

Looks like the code was removed in 35a447a263115b22a7040a9f4a16ff4bd2078466. I'm not sure it was intentional, but I think we should probably add it back since Ubuntu 10.04 server with that version of Python is supported until April 2015. I will check with Florian.

comment:14 Changed 13 months ago by Tim Graham <timograham@…>

  • Resolution set to fixed
  • Status changed from new to closed

In e4b2bea743944b67a55d0ef3d17982a2319f866d:

[1.6.x] Fixed #19107 -- Restored bug fix for sending unicode email with Python 2.6.5 and below.

comment:15 Changed 13 months ago by Humper

I'm still encountering this bug on Ubuntu 10.04 with Django 1.6.6, even though your patch has landed there. In fact, upgrading to Django 1.6.6 (from 1.4.1) was what caused the issue.

The stack trace is the same as in the OP. What information would be useful to help reproduce?

comment:16 Changed 13 months ago by Humper

  • Resolution fixed deleted
  • Status changed from closed to new

comment:17 Changed 13 months ago by Humper

Here's the portion of the email that appears to be causing issues (printed with 'print email.dict'):

Hi\r\n\r\nWhen browsing with chrome on my Android tablet (Samsung Galaxy 12 Note Pro), the suit symbols don\u2019t show *in the diagrams*, which is of course only a problem for the bidding (it\u2019s ok in the text, although without colors).

comment:18 Changed 13 months ago by timgraham

  • Cc timgraham added
  • Has patch unset
  • Triage Stage changed from Ready for checkin to Accepted

A working, tested patch would be needed as I don't have motivation to reproduce and fix the issue myself.

comment:19 Changed 13 months ago by claudep

Our FAQ is clear: https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django.
You should really try to install the most recent 2.6 Python version. Unfortunately, it seems Ubuntu doesn't care much (https://bugs.launchpad.net/ubuntu/+source/python2.6/+bug/788525).

comment:20 Changed 12 months ago by benkonrath

The fix in Django 1.6.6 using Python 2.6.5 in Ubuntu 10.04 is working for me. @Humper: You could try updating to the latest version of the Python 2.6.5 package. I think the Ubuntu devs have back-ported some fixes from later versions of Python to the 2.6.5 package and that might be why things are working for you. Just an idea though.

comment:21 Changed 12 months ago by timgraham

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.
Back to Top