Opened 3 months ago
Last modified 6 weeks ago
#35514 assigned New feature
Dictionary based EMAIL_PROVIDERS settings
Reported by: | Jacob Rief | Owned by: | Jacob Rief |
---|---|---|---|
Component: | Core (Mail) | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Mike Edmunds, Hrushikesh Vaidya | Triage Stage: | Accepted |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
As discussed in https://groups.google.com/g/django-developers/c/R8ebGynQjK0/m/Tu-o4mGeAQAJ and during the sprints at Django Con Europe 2024 (Carton Gibson, Natalia Bidart, Jacob Rief), we now have consensus that we want to add this feature, even though this proposal has been rejected in https://code.djangoproject.com/ticket/22734 10 years ago.
Reason for this change of opinion is that nowadays developers want to use different email backends and that the number of configuration settings for email providers has been steadily growing over the years.
So we want to replace all the settings starting with EMAIL_...
and replace them against a dictionary based approach such as:
EMAIL_PROVIDERS = { "default": { "BACKEND": "…", "HOST": "…", ... }, }
Change History (14)
comment:1 by , 3 months ago
Component: | Uncategorized → Core (Mail) |
---|---|
Triage Stage: | Unreviewed → Accepted |
comment:2 by , 3 months ago
comment:3 by , 3 months ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:4 by , 3 months ago
[django-anymail maintainer here]
I'm excited to see this getting traction. There are several common, real-world use cases for it.
For the setting name, I'd suggest (withdrawn)EMAIL_BACKENDS
, since it configures email backends, and a lot of existing docs and tutorials talk about "backends." Also, EMAIL_PROVIDERS
and EMAILS
could imply configuration for both sending and receiving email. (But I don't feel that strongly.)
Handful of questions:
1) Are these settings provided as kwargs to the backend's constructor (with the names lowercased)? E.g., with:
EMAIL_PROVIDERS = { "default": { "BACKEND": "django.core.mail.backends.smtp.EmailBackend", "HOST": "smtp-relay.gmail.com", "USER": "app@example.com", "PASSWORD": env["GMAIL_APP_PASSWORD"], }, "transactional": { "BACKEND": "anymail.backends.mailgun.EmailBackend", "API_KEY": env["MAILGUN_API_KEY"], "SENDER_DOMAIN": "app.example.com", }, }
am I assuming correctly that a request for the "transactional" provider would end up creating:
connection = anymail.backends.mailgun.EmailBackend( api_key="...", sender_domain="app.example.com", # (and no host, user, or password args) )
2) How does a caller request a particular provider? Is there a new argument to send_mail
and friends? (name="..."
? provider="..."
?)
3) It would be helpful if the logging AdminEmailHandler
and mail_managers
could be easily configured to use a different provider than django.contrib.auth. E.g., maybe admin/manager email uses EMAIL_PROVIDERS["admin"]
if present. (It's fairly common to want internal SMTP for admin notifications, but a transactional email service provider for password resets. And usually you want the transactional ESP to be the default.)
In general, for third-party libraries that send email, do you envision them adding a new setting to select an email provider (ALLAUTH_EMAIL_PROVIDER = "transactional"
)? Or maybe django-allauth would want to try provider="allauth"
first and fall back to provider="default"
? Or…?
4) Is DEFAULT_FROM_EMAIL
affected by this at all? There's an argument the default from_email will often need to vary by provider, but that might require changes to all email backends. The simplest answer is no, it's a global default across all providers. (Ditto SERVER_EMAIL
for admin/manager notifications.)
Again, I'm glad to see this proposal moving forward, and happy to test with django-anymail when the time comes. (Anymail backends already support constructor kwargs for their settings, so if I'm understanding the first item correctly, it should "just work.")
comment:5 by , 3 months ago
Cc: | added |
---|
comment:6 by , 3 months ago
Cc: | added |
---|
comment:7 by , 2 months ago
[Withdrawing my earlier comment about the settings name: I've warmed to EMAIL_PROVIDERS
. It's descriptive and reflects common technical jargon: "Email Service Provider" is how most providers of email sending APIs describe themselves. And it naturally leads to a terse, meaningful param name: provider="<key>"
—vs. something like name=
, backend_id=
, connection_name=
, backend=
(already used for something else), or email=
(which would practically guarantee confusion).]
comment:8 by , 2 months ago
Related to my question (1) above, is there going to be a certain set of distinguished keys that can be used at the root of a provider definition, like HOST
and USER
, while other parameters have to be put inside OPTIONS
(like with the current DATABASES
setting)? I'm, uh, "asking for a friend" who maintains a bunch of non-SMTP email backends that tend to require params like API_KEY
or SERVER_TOKEN
but couldn't care less about USER
or PASSWORD
.
I'm bringing this up because—just like DATABASES
—I suspect we'll eventually want to allow email provider configuration that isn't a backend param. For example, a hypothetical MESSAGE_DEFAULTS
feature could address both my question (4) above about provider-specific DEFAULT_FROM_EMAIL
/SERVER_EMAIL
as well as a suggestion in #36365 to allow setting header options:
EMAIL_PROVIDERS = { "admin": { "BACKEND": "django.core.mail.backends.smtp.EmailBackend", "OPTIONS": { "host": "smtp-relay.gmail.com", "user": "app@corp.example.com", "password": env["GMAIL_APP_PASSWORD"], }, "MESSAGE_DEFAULTS": { "from_email": "app-operations@corp.example.com", "reply_to": "it@corp.example.com", "headers": {"Auto-Submitted": "auto-generated"}, }, }, "default": { "BACKEND": "anymail.backends.mailersend.EmailBackend", "OPTIONS": { "api_token": env["MAILERSEND_API_TOKEN"], }, "MESSAGE_DEFAULTS": { "from_email": "noreply@app.example.com", # TODO: upgrade our MailerSend account to allow headers # "headers": {"Auto-Submitted": "auto-generated"}, }, }, }
(Here I've just pushed all backend params into OPTIONS
, to avoid relegating non-SMTP backends to second class. Also, to be clear, I'm not proposing MESSAGE_DEFAULTS
become part of this work; just asking how we leave room for configuring a feature like it.)
comment:9 by , 6 weeks ago
Created a draft pull request for this ticket: https://github.com/django/django/pull/18421
comment:10 by , 6 weeks ago
@Mike Edmunds
addressing your questions from June 10th, 2024
ad 1) yes, the provider name usually will be lowercased, jsut as with databases, caches, etc.
ad 2) I would prefer send_mail(provider="…"
, …)`.
ad 3) good point, should maybe be implemented in a separate issue.
ad 4) currently I did neither address DEFAULT_FROM_EMAIL
nor EMAIL_SUBJECT_PREFIX
.
addressing your questions from June 25th, 2024
first) On the sprints at DjangoCon Europe we (Natalia, Carlton, Jacob) brainstormed for meaningful names. The conclusion was that EMAIL_PROVIDERS
was the best choice.
second) hmm, didn't think about this yet. Currently it would behave just like using the setting EMAIL_…
but moved into a dictionary based setting.
comment:11 by , 6 weeks ago
@Mike Edmunds
in your second post on 2023-06-25 you made a good point about using a sub-dict OPTIONS
to maintain non-SMTP email backends. I now changed the code to support this.
We probably should consider use-cases, where users want to be informed through messaging apps rather than email. Signal, WhatApp, Telegram, etc. all offer an API for this purpose and in Django we should allow easy integration for them. Therefore I fully back your proposal.
comment:12 by , 6 weeks ago
@Jacob Rief Appreciate the responses.
You raise a really interesting idea about non-email messaging apps—I actually hadn't considered that case. (It probably merits its own discussion at some point.)
I know of three cases where existing third-party libraries deal with Django email and are likely to be impacted by EMAIL_PROVIDERS. I'd hope we can consider all of these in the design:
- Email backends that call ESPs' HTTP APIs to send email. (This is what I meant by "non-SMTP email backends.")
- Examples: django-ses, postmarker, django-anymail (disclosure: I maintain django-anymail)
- These backends typically have their own settings (such as
POSTMARK_API_KEY
) that are different from Django's existingEMAIL_*
settings - I think your latest PR update to support
OPTIONS
effectively handles their needs. (And in many cases, these libraries won't even need to be updated. That's great!)
- "Wrapper" email backends which add functionality (such as an asynchronous send queue) and then connect to another email backend for the actual sending
- Examples: django-mailer, django-celery-email, django-post-office
- These packages typically have their own setting to specify the "wrapped" backend and then call Django's
get_connection(backend=setting_value)
. E.g.: with django-mailer you setEMAIL_BACKEND = "mailer.backend.DbBackend"
and thenMAILER_EMAIL_BACKEND = "django.core.mail.backends.stmp.EmailBackend"
. - These libraries should continue to work for now with deprecated
EMAIL_*
settings, but will start to break as the deprecated settings are removed. We should probably deprecate thebackend
param to get_connection() now, and add a newget_connection(provider=...)
option. (Or deprecate get_connection() and add a new get_provider() to replace it.) - Wrapper email backends will need updates to properly use EMAIL_PROVIDERS. For example, django-mailer could replace its
MAILER_EMAIL_BACKEND
setting with a newprovider
option:EMAIL_PROVIDERS = { "default:" { "BACKEND": "mailer.backend.DbBackend", "OPTIONS": { # django-mailer needs a NEW option for its wrapped provider: "provider": "smtp", }, }, "smtp": { "BACKEND": "django.core.mail.backends.stmp.EmailBackend", "OPTIONS": { ... }, } }
- Or, could we think of a way django-mailer could generically wrap all defined EMAIL_PROVIDERS (except itself), without the user needing to specify a wrapped version of each?
- Third-party libraries that send email
- Examples: django-newsletter; email confirmation in django-allauth and django-user-accounts; password reset email in django.contrib.auth
- As I mentioned earlier, a common use case is wanting a transactional ESP for password resets, internal SMTP for admin messages, and a bulk ESP for newsletters
- How would we expect these libraries to allow specifying EMAIL_PROVIDER aliases for the different types of mail they send?
- (django.utils.log.AdminEmailHandler is another case of this, and already supports an email_backend option. I'd suggest it should also have an
email_provider
option, and as you say that could be a separate ticket.)
For B & C, I'm just suggesting that we think through what we might expect those packages to do, and then make sure the EMAIL_PROVIDERS feature will be compatible with those expectations. (Does that make sense?)
[btw, I agree the names EMAIL_PROVIDERS
and provider="<alias>"
are the best choice. My earlier comments on naming were in response to comment:1 from Adam Johnson, and I came to the same conclusion as you and Natalia and Carlton.]
comment:13 by , 6 weeks ago
Now, that I touched Django's mail sending code, I noticed that the current implementation has a serious flaw: get_connection(backend=…)
returns a configured email backend. This means that whenever one calls this function, the connection parameters must be provided together. The consequence is that typical configuration settings, such as hostname, user, password, etc. must be provided from inside the calling code, rather that from a configuration file, aka settings.py
. I therefore left the backend
parameter in get_connection
but made it mutually exclusive with the new parameter provider
. In my opinion we should add a deprecation warning whenever someone uses backend
. What do you think?
Anyway, the current implementation from my pull request seems to work now. I still have to add/change the documentation. Does anybody want to review this pull request?
comment:14 by , 6 weeks ago
In my opinion we should add a deprecation warning whenever someone uses backend. What do you think?
Agreed. (It's also going to cause problems for wrapper email backends after the deprecated EMAIL_*
settings are removed—see item B in comment:12.)
Great to see this get a ticket. Good luck Jacob, if you are planning on taking it up.
To bikeshed on the setting name, I think
EMAIL_PROVIDERS
is a little unclear. One might configure several backends using the same actual provider, such as for different domains. I think it would work to useEMAILS
(likeDATABASES
,CACHES
,STORAGES
). Yes, it’s a little confusing that the setting configures email backends, not actual emails, but the same could be said for the other settings.When this ticket is done, it would also be possible to add a fixer to django-upgrade to rewrite the settings, like the existing STORAGES fixer. If you feel brave, please give it a try!