Make EmailValidator accept non-ASCII characters

[Ramin Farajpour Cami]

from django.core.validators import validate_email
validate_email('うえあいお@email.com')

if you check this url email chacker with うえあいお@address.com , this is not valid email address ,

Thanks,
Ramin

[Claude Paroz]

Sure thing!
Duplicate of #26423

[Claude Paroz]

Reopening as I have a patch which targets this specific issue.

[Ramin Farajpour Cami]

Replying to claudep:

Reopening as I have a patch which targets this specific issue.

Hi,

Thanks a lot,

[Claude Paroz]

I'm just wondering, is there still a use case to keep ASCII-only validation (and hence provide validate_email_ascii)?

[Tim Graham]

Not sure, maybe you want to ask on the DevelopersMailingList. I guess usage might be a bit difficult until #25594 is fixed.

[Claude Paroz]

I just tested Firefox and Chrome email validation, and they don't accept non-ASCII in the local part.

In any case, I think we should provide both validators (ASCII-only and Unicode). It might be a bit too soon to unconditionally allow Unicode in emails.

[Tim Graham]

See #21859 for a documentation request to clarify the state of ASCII/Unicode email addresses in Django.

[Ramin Farajpour Cami]

Hi Tim,

Please merge PR ,
Thanks

[Claude Paroz]

Hey RaminFP,
I plan to come with a new patch, with two versions of the email validator. I don't think that non-ASCII local parts of email addresses are widespread enough to set it by default. The idea is that you could easily opt-in for the Unicode validator of your choice when you define the field in your models.

[Ramin Farajpour Cami]

Replying to claudep:

Hey RaminFP,
I plan to come with a new patch, with two versions of the email validator. I don't think that non-ASCII local parts of email addresses are widespread enough to set it by default. The idea is that you could easily opt-in for the Unicode validator of your choice when you define the field in your models.

Will be fix in new version 1.11? i can make suggestions to fix?

[Claude Paroz]

Yes, the plan is clearly to include that in 1.11. We still have some months ahead :-)

[Ramin Farajpour Cami]

Replying to claudep:

Yes, the plan is clearly to include that in 1.11. We still have some months ahead :-)

Owesome, i have one question, you have ​CONTRIBUTORS LIST i can add this list? or no i should try for send a lot of report for added to list contributors?

[Claude Paroz]

Yes, you are supposed to have done significant work for Django to be listed there. Of course, that's very subjective, but filling a couple of reports isn't sufficient for that.

[Ramin Farajpour Cami]

Very good, i like working with Django community always in security and see code issue for fix it,i see you write PR for this report so I can't add my name to ​CONTRIBUTORS LIST, :((

Thanks,

[Ramin Farajpour Cami]

Hi,
I see here way Contributing to Django
​https://docs.djangoproject.com/en/dev/internals/contributing/
You write PR for patch issue ,this means my name not added?

[Tim Graham]

Yes, we add to AUTHORS based on code contributions not bug reports.

[Wout De Puysseleir]

​PR

I've added a new patch for this.

[Florian Apolloner]

I am against this patch, adding more regular expressions is the wrong way to go. I'd like to propose to change the current email validator to just check if "@" is in the address and be done with it. See also ​https://davidcel.is/posts/stop-validating-email-addresses-with-regex/ -- I think this is something which should have a bit of discussion on the mailing list.

[Tim Graham]

Ideas about simplification are discussed in #26423 and ​on the django-developers mailing list.

[Collin Anderson]

if we do allow non-ascii, I wonder if we should be sure the email is "printable" (not allow hidden characters like '\u200b') ​https://docs.python.org/3/library/stdtypes.html#str.isprintable

[j-bernard]

Commenting here since #33967 has been closed as a duplicate.

Unicode in local-part is allowed by the latest standards, therefore EmailValidator is preventing valid email addresses to be used in Django. Making the current regex allow Unicode characters instead of [0-9A-Z] would do the trick.

#26423 won't solve this as HTML5 validator does not allow Unicode in local-part either.

[j-bernard]

I submitted this ​PR. I made it change as little as possible to at least get Unicode local-part valid.

[Jacob Walls]

Improvement flag was set on prior PR proposing additional regular expressions. Current ​PR simplifies the existing one per comment.

[Carlton Gibson]

The new PR seems OK™ — for ​strings `\w` is equivalent to [a-zA-Z0-9_] with ASCII, and the unicode examples then pass.

I worry slightly about bringing in a host of lookalike address vulnerabilities. 🤔

I think this needs a discussion to decide the way forward.

1. I'm not convinced this is really a distinct issue to #26423.
2. ​The mailing list discussion was essentially unanimous to radically simplify here (rather than continue to tweak).

Florian's comment:21 more or less sums it up:

...propose to change the current email validator to just check if "@" is in the address and be done with it.

We've said similar with URLValidator a number of times.

I'm not sure we shouldn't (again) mark this as a duplicate of #26423, re-purpose that to simplify the validation, make sure How to customise validation shows the way forward clearly, and then close everything else in this area as wontfix. 🤔

[Claude Paroz]

Different projects have different requirements. What about providing different validators: a simple one where only <somechar>@<somechar> presence is checked, a more elaborate one like the current one, and an equivalent to the previous allowing unicode. The question then is to decide which would be the default.

[Carlton Gibson]

I think that sounds quite reasonable Claude.

[j-bernard]

Here is a little context for my use case. In general, I create my own validator whenever it's needed to override the default Django behavior but I have a particular case where django-allauth app is used and is using the EmailValidator. In that case, I cannot easily override it.
My suggestion would then be to make the default validator more permissive to get some flexibility in the kind of use case that I have. One can still include another validation layer on top of that.

If you don't mind implementing a more complex specific validator for internationalized email addresses it would be better to avoid using only a regex. I kept it the simplest as I could in my PR because I'm aware that changing the validator is touchy.

[Carlton Gibson]

My suggestion would then be to make the default validator more permissive to get some flexibility in the kind of use case that I have. One can still include another validation layer on top of that.

I don't think we can just swap out the current validation for a looser one. Folks will be depending on the existing behaviour.

Maybe we can ship a couple of variants, but I'm not sure what switching method we might allow. I think we need a story there in order to proceed. 🤔

I looked at django-allauth — it's using the validate_email instance, in various, deeply-nested places — I think an issue over there, to look at making that pluggable, is needed really. (In the meantime, one could monkey patch validate_email with whatever validator you wanted to adjust that… — again, not something I think we can just swap out from beneath it.)

[Collin Anderson]

I'm just noticing today that unicode.org has recommendations on email security:

​https://www.unicode.org/reports/tr39/#Email_Security_Profiles

• It must be in NFKC format
• It must have level = <restriction level> or less, from Restriction_Level_Detection
• It must not have mixed number systems according to Mixed_Number_Detection
• It must satisfy dot-atom-text from RFC 5322 §3.2.3, where atext is extended as follows:
  • Where C ≤ U+007F, C is defined as in §3.2.3. (That is, C ∈ [!#-'*+\-/-9=?A-Z\^{-~]. This list copies what is already in §3.2.3, and follows HTML5 for ASCII.)}
  • Where C > U+007F, both of the following conditions are true:
    • C has Identifier_Status=Allowed from General Security Profile
    • If C is the first character, it must be XID_Start from Default Identifier_Syntax in [UAX31]

It doesn't recommend which "restriction level" to use, and maybe we should allow the user to decide what level to use (defaulting to 1: ASCII-Only).

(Also, it would be nice if Python implemented "Mixed-Script Detection", "Restriction-Level Detection" and "Mixed-Number Detection" as part of unicodedata.)