Handle non-UTF-8 bytes objects for text/* attachments

[Michael Schwarz]

I've noticed that that attach() from django.core.mail.EmailMessage allows a bytes object to be passed as content even when the MIME type is set to text/plain. SafeMIMEText will then try to decode the text in the bytes object as UTF-8, which is necessary because the MIME specifications require text/* parts to be decodable with the encoding specified in the Content-Type header. But decoding the bytes object will fail if the bytes object does not contain valid UTF-8 encoded text.

OTOH, attach_file() will in that case first try to decode the file's content but fall back on treating the attachment as binary, setting the Content-Type to application/octet-stream.

I think the fallback provided by attach_file() is useful and I would like to extend attach() to that same behavior.

[Michael Schwarz]

I will shortly create a pull request with my proposed changes.

[Tim Graham]

Might be a duplicate of #26802 which was committed just a couple weeks ago.

[Michael Schwarz]

Pull request: ​https://github.com/django/django/pull/7014

[Michael Schwarz]

Replying to timgraham:

Sorry for not mentioning #26802. That change did improve the situation, allowing bytes objects. But it tries to unconditionally decode it as UTF-8 (compared to the logic in attach_file(), which falls back to treating it as a binary file). This means that if a bytes object is passed which does not contain valid UTF-8, a UnicodeDecodeError will be thrown.

With the changes in the pull request, the behavior is the same as when attaching a non-UTF-8 file.

[Claude Paroz <claude@…>]

In 6fe391d4:

Refs #27007 -- Enhanced mail text attachment test

The test now also checks whether the sent message's attachment has the expected
name, content and mime type.

[Tim Graham]

Pending some small documentation edits.

[Tim Graham <timograham@…>]

In 72d541b:

Fixed #27007 -- Handled non-UTF-8 bytes objects for text/* attachments.

The fallback logic which allows non-UTF-8 encoded files to be passed to
attach_file() even when a text/* mime type has been specified is
moved to attach(). Both functions now fall back to a content type of
application/octet-stream.

A side effect is that a file's content is decoded in memory instead of
opening it in text mode and reading it into a string.

Some mimetype-related logic in _create_attachment() has become
obsolete as the code moved from attach_file() to attach() already
handles this.

[Sarah Boyce <42296566+sarahboyce@…>]

In 23529b66:

Fixed #36478 -- Fixed inconsistent mail attachment handling.

Fixed an inconsistency between EmailMessage.attach() and .attachments
when attaching bytes content with a text/* mimetype. The attach()
function decodes UTF-8 bytes if possible and otherwise changes the
mimetype to application/octet-stream to preserve the content's unknown
encoding (refs #27007). Providing equivalent content directly in
EmailMessage.attachments did not apply the same logic, leading
to an "AttributeError: 'bytes' object has no attribute 'encode'"
in SafeMIMEText.set_payload().

Updated EmailMessage._create_mime_attachment() to match attach()'s
handling for text/* mimetypes with bytes content. Updated test cases
to accurately cover behavior on both paths.