django crashes on email address that passed validate_email() (utf8-tld)

[harm]

description

It seems validate_email() and send_email() in django don't agree on the validness of email addresses whenever utf8 is used in the tld.

I'm not sure what is currently allowed the domain part, there is an experimental RFC 5335, but at least django should agree with itself on this matter.

steps to reproduce

python manage.py shell
>>> from django.core.validators import validate_email
>>> from django.core.mail import send_mail
>>> email = u'chw08820@nyc.odn.ne.j\uff43'
>>> validate_email(email)
>>> send_mail('subject','message','from@example.com',[email])

result

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/python2.5/lib/python2.5/site-packages/django/core/mail/__init__.py", line 61, in send_mail
    connection=connection).send()
  File "/opt/python2.5/lib/python2.5/site-packages/django/core/mail/message.py", line 175, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/opt/python2.5/lib/python2.5/site-packages/django/core/mail/backends/smtp.py", line 85, in send_messages
    sent = self._send(message)
  File "/opt/python2.5/lib/python2.5/site-packages/django/core/mail/backends/smtp.py", line 101, in _send
    email_message.message().as_string())
  File "/opt/python2.5/lib/python2.5/site-packages/django/core/mail/message.py", line 147, in message
    msg['To'] = ', '.join(self.to)
  File "/opt/python2.5/lib/python2.5/site-packages/django/core/mail/message.py", line 86, in __setitem__
    name, val = forbid_multi_line_headers(name, val, self.encoding)
  File "/opt/python2.5/lib/python2.5/site-packages/django/core/mail/message.py", line 70, in forbid_multi_line_headers
    result.append(formataddr((nm, str(addr))))
UnicodeEncodeError: 'ascii' codec can't encode character u'\uff43' in position 21: ordinal not in range(128)

expected result

Either

1. validate_email() should reject the email address,

or

2. send_email() should handle the message and attempt to deliver it.

version

django 1.2

[anonymous]

I added such test to regressiontests/mail.py and it worked fine.

[Andi Albrecht]

For the record, support for IDN in validate_email() was added in r12474 (#9764).

[Jannis Leidel]

Hm, I can't seem to reproduce the error, tbh. Can someone confirm the test breakage?

[Chris Beaven]

Is the try/except even needed? If the name gets always encoded to str using the charset, why not just assume we should do this for the email address to?

[Jannis Leidel]

(In [14216]) Fixed #14301 -- Handle email validation gracefully with email addresses containing non-ASCII characters. Thanks, Andi Albrecht.

[Jannis Leidel]

(In [14217]) [1.2.X] Fixed #14301 -- Handle email validation gracefully with email addresses containing non-ASCII characters. Thanks, Andi Albrecht.

Backport from trunk (r14216).

[philomat]

While one part of the problem is fixed, the SMTP backend still needs to be fixed since it is not expecting non-ASCII characters.

      File "/Sites/vendor/Django-1.3-alpha-1/django/core/mail/__init__.py", line 61, in send_mail
        connection=connection).send()
      File "/Sites/vendor/Django-1.3-alpha-1/django/core/mail/message.py", line 186, in send
        return self.get_connection(fail_silently).send_messages([self])
      File "/Sites/vendor/Django-1.3-alpha-1/django/core/mail/backends/smtp.py", line 85, in send_messages
        sent = self._send(message)
      File "/Sites/vendor/Django-1.3-alpha-1/django/core/mail/backends/smtp.py", line 101, in _send
        email_message.message().as_string())
      File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/smtplib.py", line 703, in sendmail
        (code,resp)=self.rcpt(each, rcpt_options)
      File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/smtplib.py", line 457, in rcpt
        self.putcmd("rcpt","TO:%s%s" % (quoteaddr(recip),optionlist))
      File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/smtplib.py", line 318, in putcmd
        self.send(str)
      File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/smtplib.py", line 305, in send
        self.sock.sendall(str)
      File "<string>", line 1, in sendall
    
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 16: ordinal not in range(128)

[Jannis Leidel]

Without giving specific details what actually spawned this traceback, I don't see a way to see the reason. Providing a test case would be the best way.

[philomat]

The exact same steps to reproduce as listed by the original poster will spawn this traceback *if* you are using the SMTP backend. The email tests are probably not using it, which is why the error went unnoticed.

So, for the following example would show the problem:

   >>> email = u'foo@bär.com'
   >>> send_mail('subject','message','from@example.com',[email])

[Andi Albrecht]

Maybe we need something like this in the SMTP backend:

>>> email = u'foo@bär.com'
>>> name, domain = email.split('@', 1)
>>> email = '@'.join([name, domain.encode('idna')])
>>> email
<<< u'foo@xn--br-eda5w.com'

But I'm not very familiar with IDN and the depths of SMTP, so this approach could be wrong.

[philomat]

The relevant Wikipedia article (German only, ​http://de.wikipedia.org/wiki/E-Mail-Adresse#Der_Dom.C3.A4nenteil_.28Domain_Part.29) says that with the introduction of IDN, nothing changes technically with regard to the SMTP protocol: Characters above ASCII code #127 are illegal. It is the client's responsibility to convert to an IDNA string.

Hence, andialbrechts approach seems to be absolutely correct.

[Adam Nelson]

@philomat is right. Even in gmail, they won't accept IDN domain names like: test@façonnable.com

[Adam Nelson]

One problem that this ticket exposes is that now we allow unicode email addresses for user accounts (#9764) but those people will never be able to receive email through Django - until it supports send() to IDN email addresses. I almost wonder if #9764 should be undone with regards to the email address until Django can send to IDN email addresses.

[Jannis Leidel]

Replying to philomat:

The relevant Wikipedia article (German only, ​http://de.wikipedia.org/wiki/E-Mail-Adresse#Der_Dom.C3.A4nenteil_.28Domain_Part.29) says that with the introduction of IDN, nothing changes technically with regard to the SMTP protocol: Characters above ASCII code #127 are illegal. It is the client's responsibility to convert to an IDNA string.

Hence, andialbrechts approach seems to be absolutely correct.

Indeed, and the actual culprit seems to be the fact that to, cc and bcc attributes of an EmailMessage are only idna encoded when calling its message() method (and blankly passed by the SMTP backend by using the recipients() method (see 1)). In other words, the encoding with idna needs to happen earlier in the life of an EmailMessage, say in the __init__ when the different recipient attributes are handled anyway (see 2).

1: ​http://code.djangoproject.com/browser/django/trunk/django/core/mail/message.py?rev=14216#L173

2: ​http://code.djangoproject.com/browser/django/trunk/django/core/mail/message.py?rev=14216#L121

[philomat]

Replying to jezdez:

the encoding with idna needs to happen earlier in the life of an EmailMessage, say in the __init__ when the different recipient attributes are handled anyway (see 2).

I think encoding during __init__ is a bad idea since you might want to change those object variables at a later time. Maybe set private object vars during __init__ and make to, cc, bcc properties delivering encoded values?

[Claude Paroz]

IDN encode in smtp backend

[Claude Paroz]

Just attached a patch using the idea on comment 12 from andialbrecht. I choose to use the encode-at-the-last-moment approach. Note that only the addresses really used by SMTP to send the messages are idna encoded, not the addresses in the message headers.

[Jannis Leidel]

(In [15006]) Fixed #14301 -- Further refine changes made in r14216 to support non-ASCII characters in email addresses. Thanks, Claude Peroz and Andi Albrecht.

[Jannis Leidel]

(In [15007]) [1.2.X] Fixed #14301 -- Further refine changes made in r14217 to support non-ASCII characters in email addresses. Thanks, Claude Peroz and Andi Albrecht.

Backport from trunk (r15006).

[Jacob]

Milestone 1.3 deleted