Opened 3 months ago

Closed 3 weeks ago

Last modified 2 weeks ago

#35378 closed Bug (invalid)

Incorrect folding of long address headers with special characters when using 7bit Content-Transfer-Encoding in EmailMessage

Reported by: andres Owned by:
Component: Core (Mail) Version: dev
Severity: Normal Keywords: email, compat32
Cc: Florian Apolloner, Joachim Jablon, Mike Edmunds Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Hi!

I'm getting errors when trying to send email messages when the "to" header has non-ASCII chars.
The problem seems to happen when all these conditions are true at the same time:

  • the readable name part of the recipient is big (e.g. "A very long and big name for this recipient" <to@…>)
  • there is at least one non-ASCII char
  • there is at least one "special char" (only seems to happen with comma or parenthesis)
  • the special char isn't close to or between no-ASCII chars

Code example:

import email
from django.core.mail import EmailMessage

recipient = '"A véry long name with non-ASCII char and, comma" <to@example.com>'

msg = EmailMessage(from_email='from@example.com', to=[recipient]).message()
msg.policy = email.policy.default.clone(cte_type='7bit')
print(msg.as_bytes())

That prints a bytes string with:

\nTo: A =?utf-8?q?v=C3=A9ry?= long name with non-ASCII char and, comma\n <to@example.com>\n

It has an unprotected comma, and the message is rejected by ESPs.
Using default policy fix this problem, but possibly causes other bad consequences:

\nTo: =?utf-8?q?A_v=C3=A9ry_long_name_with_non-ASCII_char_and=2C_comma?=\n <to@example.com>\n

More details here: https://github.com/anymail/django-anymail/issues/369

Change History (12)

comment:1 by Sarah Boyce, 3 months ago

Cc: Florian Apolloner Joachim Jablon added
Summary: Bad UTF-8 "To" header encoding in EmailMessageIncorrect folding of long address headers with special characters when using 7bit Content-Transfer-Encoding in EmailMessage
Triage Stage: UnreviewedAccepted
Version: 5.0dev

Thank you for the report!
I can replicate and the discussion in https://github.com/anymail/django-anymail/issues/369 is also very useful, thank you.

I'm pretty sure the problem only occurs with Python email's legacy email.message.Message, which uses a different folding algorithm (via email.policy.compat32) than modern email.message.EmailMessage (email.policy.default) uses.
My understanding is the Compat32 legacy layer is there specifically to replicate Python 2's email behavior (including any bugs), so there's not much point in reporting bugs against it.

Agree that #31784 is also somewhat related.

comment:2 by Lufafa Joshua, 3 months ago

Owner: changed from nobody to Lufafa Joshua
Status: newassigned

comment:3 by Lufafa Joshua, 3 months ago

Has patch: set
Version 1, edited 3 months ago by Lufafa Joshua (previous) (next) (diff)

comment:4 by andres, 3 months ago

Thanks, Lufafa Joshua! I manually reproduced your fix here, did a quick test and it seems to work!

comment:5 by Sarah Boyce, 3 months ago

Patch needs improvement: set

comment:6 by Lufafa Joshua, 2 months ago

Has patch: unset

comment:7 by Lufafa Joshua, 2 months ago

Owner: Lufafa Joshua removed
Status: assignednew

comment:8 by Mike Edmunds, 4 weeks ago

[I'm responsible for the example cited above. Setting an existing message's policy attribute like that isn't documented anywhere, so probably shouldn't be considered a valid test case. Apologies for that.]

I've come to the conclusion that any attempt to use modern email.policy.default to serialize an email.message.Message (which is constructed with legacy email.policy.compat32) is prone to errors, and probably just isn't supported by the Python email package. Additional information in https://github.com/anymail/django-anymail/issues/369#issuecomment-2184252097

I would suggest not trying to fix this particular issue within Django. Long term, there would be a lot of advantages to upgrading django.core.mail to use modern email.message.EmailMessage (and email.policy.default). But that's a separate issue, and I think needs to happen all at once; trying to mix legacy and modern email code seems problematic.

Here's an updated test, using only documented Python email package features (generator policy override). It has the same problem, but I think clarifies this is more of a Python email bug than a Django problem:

import email.generator
import email.policy
import io
import django.core.mail

to = '"Người nhận a very very long, name" <to@example.com>'
mime_message = django.core.mail.EmailMessage(to=[to]).message()

fp = io.BytesIO()
g = email.generator.BytesGenerator(
    fp, 
    policy=email.policy.default.clone(cte_type="7bit"),
)
g.flatten(mime_message)
print(fp.getvalue().decode("ascii"))
# [... other headers ...]
# To: =?utf-8?b?TmfGsOG7nWkgbmjhuq1u?= a very very long, name <to@example.com>

comment:9 by Mike Edmunds, 4 weeks ago

Cc: Mike Edmunds added

comment:10 by Mike Edmunds, 4 weeks ago

Keywords: email compat32 added

comment:11 by Sarah Boyce, 3 weeks ago

Resolution: invalid
Status: newclosed

Thank you for the update Mike, updated the ticket accordingly
Also following the discussion of https://groups.google.com/u/2/g/django-developers/c/2zf9GQtjdIk which is nice to see 👍

comment:12 by Mike Edmunds, 2 weeks ago

(Reported Python email bug as python/cpython#121284 )

Note: See TracTickets for help on using tickets.
Back to Top