Improve django.core.mail.messages EAI processing

[j-bernard]

This ticket is the third and last of a list of tickets aiming at bringing Email Address Internationalization (EAI) compliance to Django by supporting International Domain Name (IDN) with regards to the latest standard (IDNA 2008) and fixing some processing on internationalized domains or email addresses.
Previous tickets: #33967, #33968

sanitize_address transforms the email address domain in Punycode regardless of the email server's compliance with EAI.

The main issue here is that the conversion is performed with the deprecated IDNA 2003 standard instead of IDNA 2008 (see the previous ticket for more information) but this conversion should also be skipped for consistency with the user input and only performed if the server does not support Unicode email addresses.

The logic (with the backend using Python smtplib) would then be:

• try to send the message, regardless of the presence of Unicode in the address (Python smtplib will add the right options if you use the send_message method else the SMTPUTF8 option should be provided)
• if smtplib.SMTPNotSupportedError is raised
  • if the local-part is ASCII only, convert the domain to A-Label using and IDNA 2008 compliant library and retry
  • else return failure

[Florian Apolloner]

try to send the message, regardless of the presence of Unicode in the address (Python smtplib will add the right options if you use the send_message method else the SMTPUTF8 option should be provided)

Can you link the code in question?

if smtplib.SMTPNotSupportedError is raised

What does this error tell us? Given a standard setup your "mail server" is often a simple relay running on localhost which probably doesn't give you errors often. Whether or how the server it relays to does domain internationalization is unknown to it.

I am not deep into mail standards, so any details you could provide here would be helpful. I also do think that we have some leeway when changing stuff here because international domain names are not used often. We just should make clear to not introduce any security issues (like sending mail to a different domain all of a sudden -- though I am not sure if we can prevent that realistically because for instance ß encodes differently in the standards iirc).

[j-bernard]

Replying to Florian Apolloner:

try to send the message, regardless of the presence of Unicode in the address (Python smtplib will add the right options if you use the send_message method else the SMTPUTF8 option should be provided)

Can you link the code in question?

​This line transforms the domain to A-Label.

if smtplib.SMTPNotSupportedError is raised

What does this error tell us? Given a standard setup your "mail server" is often a simple relay running on localhost which probably doesn't give you errors often. Whether or how the server it relays to does domain internationalization is unknown to it.

This error means the mail server does not support the SMTPUTF8 option and won't be able to process a to address with Unicode.

I am not deep into mail standards, so any details you could provide here would be helpful. I also do think that we have some leeway when changing stuff here because international domain names are not used often. We just should make clear to not introduce any security issues (like sending mail to a different domain all of a sudden -- though I am not sure if we can prevent that realistically because for instance ß encodes differently in the standards iirc).

IDNA 2008 was created to lower some security issues with the old IDNA 2003 standard. And as you mentioned, keeping both standards active creates even more issues. Many already moved to the new standard and Django could clearly make a difference by being also compliant. Without that kind of move, IDNs will keep being "not used often".

[Mike Edmunds]

(I've opened #35714 as a request to support SMTPUTF8 in the SMTP EmailBackend.)