Add RFC 3834 Auto-Submitted header to emails by default

[Tobias Bengfort]

RFC 3824 (​https://www.rfc-editor.org/rfc/rfc3834) defines the Auto-Submitted header for emails to avoid mail loops. The vast majority of mails sent by Django should use Auto-Submitted: auto-generated. The only exceptions I can think of are:

• Django is used in a bigger system that also receives emails. In that case it may also be appropriate to use Auto-Submitted: auto-replied in some cases.
• Django is used to implement an email client. In that case Auto-Submitted should not be used.

Since these are rare exceptions, I think Django should use Auto-Submitted: auto-generated by default. Users who need more control should have to explicitly disable this behavior.

I did not do a larger survey, but just from the mails I currently have in my inbox I noticed that gitlab and unattended-upgrades both use Auto-Submitted.

[Sarah Boyce]

Hi Tobias, thank you for this!
Accepting as this sounds like the right thing to do to me. Added a couple of people in cc in case they have any concerns and can update the ticket accordingly.

[Adam Johnson]

I found very few search results about the header (​DuckDuckGo, ​Google). Notably, I don’t see any “Email best practices” articles discussing this header.

But on GitHub code search there are ​11.4k results for "auto-submitted" "auto-generated", among which I found these tools:

• Wagtail since 2018: ​https://github.com/wagtail/wagtail/blob/1ec62a6c2d99365886e41c517d386d8d36fa5d50/docs/releases/2.3.rst?plain=1#L39)
• Mastodon sets this header: ​https://github.com/mastodon/mastodon/blob/1ad119941ff672b93f2b04dc29f82443349bb69c/app/mailers/application_mailer.rb#L21
• Bugzilla: ​https://github.com/mozilla-bteam/bmo/blob/8192fe39ea8b93a7f3bd16fec4807e5d349ce223/Bugzilla/Mailer.pm#L100

It seems reasonable that Django would set it. But there’s some risk since some search results are people trying to *remove* the header (​1, ​2). I think we should at least document a way to opt-out, possibly by subclassing EmailMessage and overriding message() to delete the header.

[cgracin]

Hello everyone! I'm a new contributor here and just submitted a PR for this feature. I went ahead and added the default behavior to the EmailMessage class to attach the "Auto-Submitted : auto-generated" header. I took the advice of Adam and.created a subclass of EmailMessage named NoAutoSubmittedHeaderEmailMessage that removes the "Auto-Submitted : auto-generated" header to allow a user to opt-out of this default behavior.

[Tobias Bengfort]

Thanks for the patch! I am not sure if NoAutoSubmittedHeaderEmailMessage is the best option though. The way I understood Adam we should document how to create such a class, not provide it in Django itself. Either way, having to use a different Message class is a bit awkward because you can no longer use send_mail(). My proposal would be to add a setting DEFAULT_EMAIL_HEADERS which would be more flexible. I cannot think of any other headers for which this could be useful right now, but who knows. I am not sure whether we want another setting though.

[Sarah Boyce]

I am also not a fan of NoAutoSubmittedHeaderEmailMessage as a user would need to make many updates to their code (including overwriting the EmailMultiAlternatives, mail_admins etc).
I can see the appeal of having something like a DEFAULT_EMAIL_HEADERS setting, however we try to avoid adding new setting to Django when we can. We'd need very strong agreement that this is the best way forward here.
I would recommend creating a discussion on the ​forum to try and get input from a wider audience as to what would be the best approach.

[Tobias Bengfort]

The documentation already contains this line:

Not all features of the EmailMessage class are available through the send_mail() and related wrapper functions. If you wish to use advanced features, such as BCC’ed recipients, file attachments, or multi-part email, you’ll need to create EmailMessage instances directly.
​https://docs.djangoproject.com/en/5.0/topics/email/#the-emailmessage-class

For consistency I think we should also add the Auto-Submitted header in the wrapper functions.

The special thing here would be that dropping down to EmailMessage allows you to remove a header rather than add one. I am not sure how best to explain that in the docs. I see that most parameters are only documented once for send_mail() and not repeated for the other wrapper functions. So maybe it would be sufficient to add a note only to send_mail(). The note could be something like this:

send_mail() uses the Auto-Submitted mail header to indicate that the mail was created by software rather than a human.

[cgracin]

Thank you guys for the comments, I'll work on implementing these requests and submit an updated PR.

[David Smith]

Does the header need to be removed entirely? Could we advise folk to set the header to "no" to disable it?
​https://www.iana.org/assignments/auto-submitted-keywords/auto-submitted-keywords.xhtml

The current patch has this to add the header:

if "Auto-Submitted" not in self.extra_headers: 
     # Default to adding the Auto-Submitted: auto-generated header 
     self.extra_headers["Auto-Submitted"] = "auto-generated"

So could we documentat this by something like...

By default EmailMessage sets the Auto-Submitted header to auto-generated to indicate that the mail was created by software rather than a human. The value of the Auto-Submitted header can be customised by the headers option. To disable the header the value of Auto-Submitted can be set to "no".

[Florian Apolloner]

I don't feel good about this change. If we are to add this header by default we need a way to disable it by default as well imo (especially since one cannot patch 3rd party apps easily). This probably ties into making smtp backends configurable and for instance add options to set headers there…

[Mike Edmunds]

[​django-anymail maintainer here]

This will break sending for many users working with an Email Service Provider.

Most ESPs restrict the headers that can be used. Some will silently strip headers they don't allow, but others will reject the message entirely. (The restrictions are sometimes different for the ESP's SMTP endpoint vs. their HTTP API. A few ESPs don't allow any extra headers.)

I would suggest, instead, documenting how users can set this header themselves when sending a message. For users who want it on all messages, one option is a "wrapper" email backend that adds the header:

# Add this to your project, and in settings.py set:
#   EMAIL_BACKEND="path.to.this.file.AutoSubmittedSMTPBackend"
class AutoSubmittedSMTPBackend(django.core.mail.backends.smtp.EmailBackend):
    def send_messages(messages):
        for message in messages:
            message.extra_headers.setdefault("Auto-Submitted", "auto-generated")
        return super().send_messages(messages)

Long term, allowing configurable defaults for each email backend could simplify this. Looks like #35514 is starting down that path.

FWIW, django-anymail email backends ​already support defaults. If you're using an Anymail backend (with an ESP that allows this header), you can use:

# settings.py
ANYMAIL = {
    ...,
    "SEND_DEFAULTS": {
        "extra_headers": {
            "Auto-Submitted": "auto-generated",
        },
    },
}

[Mike Edmunds]

Took a closer look at the proposed patch, and as currently implemented this will only affect email backends that call EmailMessage.message(). Many non-SMTP email backends don't, so my earlier concern wouldn't apply to them. But this will still break anyone using an ESP's SMTP endpoint if their ESP doesn't allow the header. (And if the docs recommend setting "Auto-Submitted": "no" in headers to disable the feature, that will break both SMTP and non-SMTP backends for those ESPs.)

I appreciate the thought and effort that went into this change (and that it attracted a new contributor!), but I also don't feel good about it. After re-reading RFC 3834 section 5 a few times, it's not at all clear to me that Auto-Submitted: auto-generated is an appropriate generalization for the wide variety of apps that are implemented with Django, and the emails they might send.

For example, if you're using something like django-helpdesk to reply to a user's support ticket, you are creating what the RFC calls a "manually generated message" (it's a direct reply to another message, and not automatic). The RFC says auto-generated "MUST NOT be used on manually generated messages." In practical terms, you might want to see vacation autoresponses or challenge messages necessary for your support response to be delivered. At best, this seems like it should be a django-helpdesk setting, not something Django applies by default to all apps built with the framework.

I suppose django-helpdesk might be a case where "Django is used to implement an email client" of sorts. My concern is, I'm not sure a general-purpose framework like Django can really be sure that sort of use is a "rare exception."

[Also, to be clear, django-helpdesk needs to (and does) set Auto-Submitted: auto-replied on any automatic responses it sends, to prevent mail loops. And preventing mail loops is the primary point of RFC 3834. But "auto-replied" is always going to be app specific logic, and isn't covered by this proposed change.]

[Florian Apolloner]

I think Mikes analysis is great. It makes sense for applications to set those headers as needed and we can audit our usage of send_mail in Django itself (ie admin error emails), but it doesn't seem to make sense for a framework. Even if it were possible to set this header manually to "no" it would be backwards incompatible enough that it would be hard to get in without a transitional setting to opt into the behavior.

[Florian Apolloner]

Sorry didn't mean to already close the ticket (wontfix is indeed what I am leaning towards, but that is just my opinion).

[Tobias Bengfort]

I am fine with closing this as wontfix as long as

1. There is an option available to add the header if desired. Mike already provided a backend, but also mentioned that it does not work in all cases.
2. The documentation explains how to do this. This could use more work.

[Mike Edmunds]

Mike already provided a backend, but also mentioned that it does not work in all cases.

My AutoSubmittedSMTPBackend example above is untested code, but should work in every case where this proposed change would have worked. (It won't work if your ESP doesn't allow this header, but neither would this PR. My other example above was specific to using the third-party django-anymail library, so wouldn't belong in the Django docs.)

I'd be very much in favor of Florian's "making smtp backends configurable and for instance add options to set headers there" idea. That should be a separate ticket. This Auto-Submitted header use case would make a great example for that feature's documentation.

[Sarah Boyce]

Replying to Mike Edmunds:

I'd be very much in favor of Florian's "making smtp backends configurable and for instance add options to set headers there" idea. That should be a separate ticket. This Auto-Submitted header use case would make a great example for that feature's documentation.

Agreed 👍 if anyone would like to raise this ticket, I will accept it
I think we are in agreement that we will pivot from the original request here and can mark this as wontfix. Thank you all for this discussion

[Adam Johnson]

Thank you for the all the input Mike! I agree with wontfixing this.