#35365 closed New feature (wontfix)
Add RFC 3834 Auto-Submitted header to emails by default
Reported by: | Tobias Bengfort | Owned by: | cgracin |
---|---|---|---|
Component: | Core (Mail) | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Florian Apolloner, Russell Keith-Magee, Mike Edmunds | Triage Stage: | Accepted |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | yes |
Easy pickings: | no | UI/UX: | no |
Description
RFC 3824 (https://www.rfc-editor.org/rfc/rfc3834) defines the Auto-Submitted
header for emails to avoid mail loops. The vast majority of mails sent by Django should use Auto-Submitted: auto-generated
. The only exceptions I can think of are:
- Django is used in a bigger system that also receives emails. In that case it may also be appropriate to use
Auto-Submitted: auto-replied
in some cases. - Django is used to implement an email client. In that case
Auto-Submitted
should not be used.
Since these are rare exceptions, I think Django should use Auto-Submitted: auto-generated
by default. Users who need more control should have to explicitly disable this behavior.
I did not do a larger survey, but just from the mails I currently have in my inbox I noticed that gitlab and unattended-upgrades both use Auto-Submitted
.
Change History (20)
comment:1 by , 6 months ago
Cc: | added |
---|---|
Triage Stage: | Unreviewed → Accepted |
Type: | Uncategorized → New feature |
Version: | 5.0 → dev |
comment:2 by , 6 months ago
I found very few search results about the header (DuckDuckGo, Google). Notably, I don’t see any “Email best practices” articles discussing this header.
But on GitHub code search there are 11.4k results for "auto-submitted" "auto-generated"
, among which I found these tools:
- Wagtail since 2018: https://github.com/wagtail/wagtail/blob/1ec62a6c2d99365886e41c517d386d8d36fa5d50/docs/releases/2.3.rst?plain=1#L39)
- Mastodon sets this header: https://github.com/mastodon/mastodon/blob/1ad119941ff672b93f2b04dc29f82443349bb69c/app/mailers/application_mailer.rb#L21
- Bugzilla: https://github.com/mozilla-bteam/bmo/blob/8192fe39ea8b93a7f3bd16fec4807e5d349ce223/Bugzilla/Mailer.pm#L100
It seems reasonable that Django would set it. But there’s some risk since some search results are people trying to *remove* the header (1, 2). I think we should at least document a way to opt-out, possibly by subclassing EmailMessage
and overriding message()
to delete the header.
comment:3 by , 5 months ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:4 by , 5 months ago
Has patch: | set |
---|
comment:5 by , 5 months ago
Hello everyone! I'm a new contributor here and just submitted a PR for this feature. I went ahead and added the default behavior to the EmailMessage class to attach the "Auto-Submitted : auto-generated" header. I took the advice of Adam and.created a subclass of EmailMessage named NoAutoSubmittedHeaderEmailMessage that removes the "Auto-Submitted : auto-generated" header to allow a user to opt-out of this default behavior.
comment:6 by , 5 months ago
Thanks for the patch! I am not sure if NoAutoSubmittedHeaderEmailMessage is the best option though. The way I understood Adam we should document how to create such a class, not provide it in Django itself. Either way, having to use a different Message class is a bit awkward because you can no longer use send_mail(). My proposal would be to add a setting DEFAULT_EMAIL_HEADERS which would be more flexible. I cannot think of any other headers for which this could be useful right now, but who knows. I am not sure whether we want another setting though.
comment:7 by , 5 months ago
Patch needs improvement: | set |
---|
I am also not a fan of NoAutoSubmittedHeaderEmailMessage
as a user would need to make many updates to their code (including overwriting the EmailMultiAlternatives
, mail_admins
etc).
I can see the appeal of having something like a DEFAULT_EMAIL_HEADERS
setting, however we try to avoid adding new setting to Django when we can. We'd need very strong agreement that this is the best way forward here.
I would recommend creating a discussion on the forum to try and get input from a wider audience as to what would be the best approach.
comment:8 by , 5 months ago
Summary: | Add RFC 3824 Auto-Submitted header to emails by default → Add RFC 3834 Auto-Submitted header to emails by default |
---|
comment:9 by , 5 months ago
The documentation already contains this line:
Not all features of the EmailMessage class are available through the send_mail() and related wrapper functions. If you wish to use advanced features, such as BCC’ed recipients, file attachments, or multi-part email, you’ll need to create EmailMessage instances directly.
https://docs.djangoproject.com/en/5.0/topics/email/#the-emailmessage-class
For consistency I think we should also add the Auto-Submitted
header in the wrapper functions.
The special thing here would be that dropping down to EmailMessage
allows you to remove a header rather than add one. I am not sure how best to explain that in the docs. I see that most parameters are only documented once for send_mail()
and not repeated for the other wrapper functions. So maybe it would be sufficient to add a note only to send_mail()
. The note could be something like this:
send_mail()
uses theAuto-Submitted
mail header to indicate that the mail was created by software rather than a human.
comment:10 by , 5 months ago
Thank you guys for the comments, I'll work on implementing these requests and submit an updated PR.
comment:11 by , 5 months ago
Does the header need to be removed entirely? Could we advise folk to set the header to "no" to disable it?
https://www.iana.org/assignments/auto-submitted-keywords/auto-submitted-keywords.xhtml
The current patch has this to add the header:
if "Auto-Submitted" not in self.extra_headers: # Default to adding the Auto-Submitted: auto-generated header self.extra_headers["Auto-Submitted"] = "auto-generated"
So could we documentat this by something like...
By default
EmailMessage
sets theAuto-Submitted
header toauto-generated
to indicate that the mail was created by software rather than a human. The value of theAuto-Submitted
header can be customised by theheaders
option. To disable the header the value ofAuto-Submitted
can be set to "no".
comment:12 by , 4 months ago
I don't feel good about this change. If we are to add this header by default we need a way to disable it by default as well imo (especially since one cannot patch 3rd party apps easily). This probably ties into making smtp backends configurable and for instance add options to set headers there…
comment:13 by , 3 months ago
Cc: | added |
---|
[django-anymail maintainer here]
This will break sending for many users working with an Email Service Provider.
Most ESPs restrict the headers that can be used. Some will silently strip headers they don't allow, but others will reject the message entirely. (The restrictions are sometimes different for the ESP's SMTP endpoint vs. their HTTP API. A few ESPs don't allow any extra headers.)
I would suggest, instead, documenting how users can set this header themselves when sending a message. For users who want it on all messages, one option is a "wrapper" email backend that adds the header:
# Add this to your project, and in settings.py set: # EMAIL_BACKEND="path.to.this.file.AutoSubmittedSMTPBackend" class AutoSubmittedSMTPBackend(django.core.mail.backends.smtp.EmailBackend): def send_messages(messages): for message in messages: message.extra_headers.setdefault("Auto-Submitted", "auto-generated") return super().send_messages(messages)
Long term, allowing configurable defaults for each email backend could simplify this. Looks like #35514 is starting down that path.
FWIW, django-anymail email backends already support defaults. If you're using an Anymail backend (with an ESP that allows this header), you can use:
# settings.py ANYMAIL = { ..., "SEND_DEFAULTS": { "extra_headers": { "Auto-Submitted": "auto-generated", }, }, }
comment:14 by , 3 months ago
Took a closer look at the proposed patch, and as currently implemented this will only affect email backends that call EmailMessage.message()
. Many non-SMTP email backends don't, so my earlier concern wouldn't apply to them. But this will still break anyone using an ESP's SMTP endpoint if their ESP doesn't allow the header. (And if the docs recommend setting "Auto-Submitted": "no"
in headers
to disable the feature, that will break both SMTP and non-SMTP backends for those ESPs.)
I appreciate the thought and effort that went into this change (and that it attracted a new contributor!), but I also don't feel good about it. After re-reading RFC 3834 section 5 a few times, it's not at all clear to me that Auto-Submitted: auto-generated
is an appropriate generalization for the wide variety of apps that are implemented with Django, and the emails they might send.
For example, if you're using something like django-helpdesk to reply to a user's support ticket, you are creating what the RFC calls a "manually generated message" (it's a direct reply to another message, and not automatic). The RFC says auto-generated "MUST NOT be used on manually generated messages." In practical terms, you might want to see vacation autoresponses or challenge messages necessary for your support response to be delivered. At best, this seems like it should be a django-helpdesk setting, not something Django applies by default to all apps built with the framework.
I suppose django-helpdesk might be a case where "Django is used to implement an email client" of sorts. My concern is, I'm not sure a general-purpose framework like Django can really be sure that sort of use is a "rare exception."
[Also, to be clear, django-helpdesk needs to (and does) set Auto-Submitted: auto-replied
on any automatic responses it sends, to prevent mail loops. And preventing mail loops is the primary point of RFC 3834. But "auto-replied" is always going to be app specific logic, and isn't covered by this proposed change.]
comment:15 by , 3 months ago
Resolution: | → wontfix |
---|---|
Status: | assigned → closed |
I think Mikes analysis is great. It makes sense for applications to set those headers as needed and we can audit our usage of send_mail in Django itself (ie admin error emails), but it doesn't seem to make sense for a framework. Even if it were possible to set this header manually to "no" it would be backwards incompatible enough that it would be hard to get in without a transitional setting to opt into the behavior.
comment:16 by , 3 months ago
Resolution: | wontfix |
---|---|
Status: | closed → new |
Sorry didn't mean to already close the ticket (wontfix is indeed what I am leaning towards, but that is just my opinion).
comment:17 by , 3 months ago
I am fine with closing this as wontfix as long as
- There is an option available to add the header if desired. Mike already provided a backend, but also mentioned that it does not work in all cases.
- The documentation explains how to do this. This could use more work.
follow-up: 19 comment:18 by , 3 months ago
Mike already provided a backend, but also mentioned that it does not work in all cases.
My AutoSubmittedSMTPBackend
example above is untested code, but should work in every case where this proposed change would have worked. (It won't work if your ESP doesn't allow this header, but neither would this PR. My other example above was specific to using the third-party django-anymail library, so wouldn't belong in the Django docs.)
I'd be very much in favor of Florian's "making smtp backends configurable and for instance add options to set headers there" idea. That should be a separate ticket. This Auto-Submitted header use case would make a great example for that feature's documentation.
comment:19 by , 3 months ago
Resolution: | → wontfix |
---|---|
Status: | new → closed |
Replying to Mike Edmunds:
I'd be very much in favor of Florian's "making smtp backends configurable and for instance add options to set headers there" idea. That should be a separate ticket. This Auto-Submitted header use case would make a great example for that feature's documentation.
Agreed 👍 if anyone would like to raise this ticket, I will accept it
I think we are in agreement that we will pivot from the original request here and can mark this as wontfix. Thank you all for this discussion
Hi Tobias, thank you for this!
Accepting as this sounds like the right thing to do to me. Added a couple of people in cc in case they have any concerns and can update the ticket accordingly.