Dictionary based EMAIL_PROVIDERS settings

[Jacob Rief]

As discussed in ​https://groups.google.com/g/django-developers/c/R8ebGynQjK0/m/Tu-o4mGeAQAJ and during the sprints at Django Con Europe 2024 (Carton Gibson, Natalia Bidart, Jacob Rief), we now have consensus that we want to add this feature, even though this proposal has been rejected in https://code.djangoproject.com/ticket/22734 10 years ago.

Reason for this change of opinion is that nowadays developers want to use different email backends and that the number of configuration settings for email providers has been steadily growing over the years.

So we want to replace all the settings starting with EMAIL_...and replace them against a dictionary based approach such as:

EMAIL_PROVIDERS = {
    "default": {
        "BACKEND": "…",
        "HOST":  "…",
        ...        
    },
}

[Adam Johnson]

Great to see this get a ticket. Good luck Jacob, if you are planning on taking it up.

To bikeshed on the setting name, I think EMAIL_PROVIDERS is a little unclear. One might configure several backends using the same actual provider, such as for different domains. I think it would work to use EMAILS (like DATABASES, CACHES, STORAGES). Yes, it’s a little confusing that the setting configures email backends, not actual emails, but the same could be said for the other settings.

When this ticket is done, it would also be possible to add a fixer to django-upgrade to rewrite the settings, like the existing ​STORAGES fixer. If you feel brave, please give it a try!

[Mike Edmunds]

[​django-anymail maintainer here]

I'm excited to see this getting traction. There are several common, real-world use cases for it.

For the setting name, I'd suggest EMAIL_BACKENDS, since it configures email backends, and a lot of existing docs and tutorials talk about "backends." Also, EMAIL_PROVIDERS and EMAILS could imply configuration for both sending and receiving email. (But I don't feel that strongly.) (withdrawn)

Handful of questions:

1) Are these settings provided as kwargs to the backend's constructor (with the names lowercased)? E.g., with:

EMAIL_PROVIDERS = {
    "default": {
        "BACKEND": "django.core.mail.backends.smtp.EmailBackend",
        "HOST": "smtp-relay.gmail.com",
        "USER": "app@example.com",
        "PASSWORD": env["GMAIL_APP_PASSWORD"],
    },
    "transactional": {
        "BACKEND": "anymail.backends.mailgun.EmailBackend",
        "API_KEY": env["MAILGUN_API_KEY"],
        "SENDER_DOMAIN": "app.example.com",
    },
}

am I assuming correctly that a request for the "transactional" provider would end up creating:

connection = anymail.backends.mailgun.EmailBackend(
    api_key="...", 
    sender_domain="app.example.com",
    # (and no host, user, or password args)
)

2) How does a caller request a particular provider? Is there a new argument to send_mail and friends? (name="..."? provider="..."?)

3) It would be helpful if the logging AdminEmailHandler and mail_managers could be easily configured to use a different provider than django.contrib.auth. E.g., maybe admin/manager email uses EMAIL_PROVIDERS["admin"] if present. (It's fairly common to want internal SMTP for admin notifications, but a transactional email service provider for password resets. And usually you want the transactional ESP to be the default.)

In general, for third-party libraries that send email, do you envision them adding a new setting to select an email provider (ALLAUTH_EMAIL_PROVIDER = "transactional")? Or maybe django-allauth would want to try provider="allauth" first and fall back to provider="default"? Or…?

4) Is DEFAULT_FROM_EMAIL affected by this at all? There's an argument the default from_email will often need to vary by provider, but that might require changes to all email backends. The simplest answer is no, it's a global default across all providers. (Ditto SERVER_EMAIL for admin/manager notifications.)

Again, I'm glad to see this proposal moving forward, and happy to test with django-anymail when the time comes. (Anymail backends ​already support constructor kwargs for their settings, so if I'm understanding the first item correctly, it should "just work.")

[Mike Edmunds]

[Withdrawing my earlier comment about the settings name: I've warmed to EMAIL_PROVIDERS. It's descriptive and reflects common technical jargon: "Email Service Provider" is how most providers of email sending APIs describe themselves. And it naturally leads to a terse, meaningful param name: provider="<key>"—vs. something like name=, backend_id=, connection_name=, backend= (already used for something else), or email= (which would practically guarantee confusion).]

[Mike Edmunds]

Related to my question (1) above, is there going to be a certain set of distinguished keys that can be used at the root of a provider definition, like HOST and USER, while other parameters have to be put inside OPTIONS (like with the current DATABASES setting)? I'm, uh, "asking for a friend" who maintains a bunch of non-SMTP email backends that tend to require params like API_KEY or SERVER_TOKEN but couldn't care less about USER or PASSWORD.

I'm bringing this up because—just like DATABASES—I suspect we'll eventually want to allow email provider configuration that isn't a backend param. For example, a hypothetical MESSAGE_DEFAULTS feature could address both my question (4) above about provider-specific DEFAULT_FROM_EMAIL/SERVER_EMAIL as well as a suggestion in #36365 to allow setting header options:

EMAIL_PROVIDERS = {
    "admin": {
        "BACKEND": "django.core.mail.backends.smtp.EmailBackend",
        "OPTIONS": {
            "host": "smtp-relay.gmail.com",
            "user": "app@corp.example.com",
            "password": env["GMAIL_APP_PASSWORD"],
        },
        "MESSAGE_DEFAULTS": {
            "from_email": "app-operations@corp.example.com",
            "reply_to": "it@corp.example.com",
            "headers": {"Auto-Submitted": "auto-generated"},
        },
    },
    "default": {
        "BACKEND": "anymail.backends.mailersend.EmailBackend",
        "OPTIONS": {
            "api_token": env["MAILERSEND_API_TOKEN"],
        },
        "MESSAGE_DEFAULTS": {
            "from_email": "noreply@app.example.com",
            # TODO: upgrade our MailerSend account to allow headers
            # "headers": {"Auto-Submitted": "auto-generated"},
        },
    },
}

(Here I've just pushed all backend params into OPTIONS, to avoid relegating non-SMTP backends to second class. Also, to be clear, I'm not proposing MESSAGE_DEFAULTS become part of this work; just asking how we leave room for configuring a feature like it.)

[Jacob Rief]

Created a draft pull request for this ticket: ​https://github.com/django/django/pull/18421

[Jacob Rief]

@Mike Edmunds

addressing your questions from June 10th, 2024

ad 1) yes, the provider name usually will be lowercased, jsut as with databases, caches, etc.

ad 2) I would prefer send_mail(provider="…", …)`.

ad 3) good point, should maybe be implemented in a separate issue.

ad 4) currently I did neither address DEFAULT_FROM_EMAIL nor EMAIL_SUBJECT_PREFIX.

addressing your questions from June 25th, 2024

first) On the sprints at DjangoCon Europe we (Natalia, Carlton, Jacob) brainstormed for meaningful names. The conclusion was that EMAIL_PROVIDERS was the best choice.

second) hmm, didn't think about this yet. Currently it would behave just like using the setting EMAIL_… but moved into a dictionary based setting.

[Jacob Rief]

@Mike Edmunds

in your second post on 2023-06-25 you made a good point about using a sub-dict OPTIONS to maintain non-SMTP email backends. I now changed the code to support this.

We probably should consider use-cases, where users want to be informed through messaging apps rather than email. Signal, WhatApp, Telegram, etc. all offer an API for this purpose and in Django we should allow easy integration for them. Therefore I fully back your proposal.

[Mike Edmunds]

@Jacob Rief Appreciate the responses.

You raise a really interesting idea about non-email messaging apps—I actually hadn't considered that case. (It probably merits its own discussion at some point.)

I know of three cases where existing third-party libraries deal with Django email and are likely to be impacted by EMAIL_PROVIDERS. I'd hope we can consider all of these in the design:

1. Email backends that call ESPs' HTTP APIs to send email. (This is what I meant by "non-SMTP email backends.")
   • Examples: ​django-ses, ​postmarker, ​django-anymail (disclosure: I maintain django-anymail)
   • These backends typically have their own settings (such as POSTMARK_API_KEY) that are different from Django's existing EMAIL_* settings
   • I think your latest PR update to support OPTIONS effectively handles their needs. (And in many cases, these libraries won't even need to be updated. That's great!)

2. "Wrapper" email backends which add functionality (such as an asynchronous send queue) and then connect to another email backend for the actual sending
   • Examples: ​django-mailer, ​django-celery-email, ​django-post-office
   • These packages typically have their own setting to specify the "wrapped" backend and then call Django's get_connection(backend=setting_value). E.g.: with django-mailer you set EMAIL_BACKEND = "mailer.backend.DbBackend" and then MAILER_EMAIL_BACKEND = "django.core.mail.backends.stmp.EmailBackend".
   • These libraries should continue to work for now with deprecated EMAIL_* settings, but will start to break as the deprecated settings are removed. We should probably deprecate the backend param to get_connection() now, and add a new get_connection(provider=...) option. (Or deprecate get_connection() and add a new get_provider() to replace it.)
   • Wrapper email backends will need updates to properly use EMAIL_PROVIDERS. For example, django-mailer could replace its MAILER_EMAIL_BACKEND setting with a new provider option:

     EMAIL_PROVIDERS = {
         "default:" {
             "BACKEND": "mailer.backend.DbBackend",
             "OPTIONS": {
                 # django-mailer needs a NEW option for its wrapped provider:
                 "provider": "smtp",
             },
         },
         "smtp": {
             "BACKEND": "django.core.mail.backends.stmp.EmailBackend",
             "OPTIONS": { ... },
         }
     }
     

   • Or, could we think of a way django-mailer could generically wrap all defined EMAIL_PROVIDERS (except itself), without the user needing to specify a wrapped version of each?

3. Third-party libraries that send email
   • Examples: ​django-newsletter; email confirmation in ​django-allauth and ​django-user-accounts; password reset email in django.contrib.auth
   • As I mentioned earlier, a common use case is wanting a transactional ESP for password resets, internal SMTP for admin messages, and a bulk ESP for newsletters
   • How would we expect these libraries to allow specifying EMAIL_PROVIDER aliases for the different types of mail they send?
   • (django.utils.log.AdminEmailHandler is another case of this, and already supports an ​email_backend option. I'd suggest it should also have an email_provider option, and as you say that could be a separate ticket.)

For B & C, I'm just suggesting that we think through what we might expect those packages to do, and then make sure the EMAIL_PROVIDERS feature will be compatible with those expectations. (Does that make sense?)

[btw, I agree the names EMAIL_PROVIDERS and provider="<alias>" are the best choice. My earlier comments on naming were in response to comment:1 from Adam Johnson, and I came to the same conclusion as you and Natalia and Carlton.]

[Jacob Rief]

Now, that I touched Django's mail sending code, I noticed that the current implementation has a serious flaw: get_connection(backend=…) returns a configured email backend. This means that whenever one calls this function, the connection parameters must be provided together. The consequence is that typical configuration settings, such as hostname, user, password, etc. must be provided from inside the calling code, rather that from a configuration file, aka settings.py. I therefore left the backend parameter in get_connection but made it mutually exclusive with the new parameter provider. In my opinion we should add a deprecation warning whenever someone uses backend. What do you think?

Anyway, the current implementation from my pull request seems to work now. I still have to add/change the documentation. Does anybody want to review this pull request?

[Mike Edmunds]

In my opinion we should add a deprecation warning whenever someone uses backend. What do you think?

Agreed. (It's also going to cause problems for wrapper email backends after the deprecated EMAIL_* settings are removed—see item B in comment:12.)

[Sarah Boyce]

Marking as "Patch needs improvement" following the review from Mike and that some merge conflicts need resolving

[Mike Edmunds]

(I'll try to create an updated patch once some of the other changes to mail have been merged.)

[Johanan Oppong Amoateng]

What is the updates on this? I would want to tag myself and work on it but i want to know the status whether Jacob Rief is still working on it

[Mike Edmunds]

Replying to Johanan Oppong Amoateng:

What is the updates on this? I would want to tag myself and work on it but i want to know the status whether Jacob Rief is still working on it

I believe Jacob Rief's PR is actually quite close—with the caveat that (for a feature of this scope) "quite close" will likely still involve a lot of work to get over the finish line. There are (I think) a couple of mechanical issues and at least a couple of design decisions needed to move this forward.

The mechanical issues:

• The PR needs to be rebased and conflicts resolved with the current code on main.
• The PR has exceeded the number of commits and comments where GitHub's PR review is usable. It would help to squash and open a new PR.

The design decisions:

1. To what extent do we want to support mixing new EMAIL_PROVIDERS settings with deprecated EMAIL_BACKEND and EMAIL_HOST/USER/etc. settings?
   
   My suggestion would be it's all or nothing: once a settings file contains EMAIL_PROVIDERS, the user has opted into the new approach and any attempt to set or access the old EMAIL_BACKEND, EMAIL_HOST, etc. becomes a hard error (not just a deprecation warning). One implication is users would need to wait to switch to EMAIL_PROVIDERS until all email backends they use have been updated to be providers-aware, which would impact anyone using a "wrapper" email backend like django-celery-email or django-mailer.
   
   (More detail in ​this PR comment and the ones following, plus several related suggested edits.)

2. Can we require that (Django's built-in) EmailBackend instances must be obtained by calling mail.get_connection(), or do we need to continue allowing code to construct them directly?
   
   If we can require get_connection(), it simplifies the backwards compatibility code in each EmailBackend somewhat. But it requires updating nearly all of Django's existing EmailBackend tests, which tend to call the constructor directly. (The current PR includes most of those test updates; we'd likely want to pull them into a separate "refs" commit first to prove they run correctly before this change.)
   
   The relevant docs start at ​Obtaining an instance of an email backend: "The get_connection() function in django.core.mail returns an instance of the email backend that you can use." But they go on to document available options for backends.smtp.EmailBackend() using the class constructor format.
   
   I don't know whether that means we have documented smtp.EmailBackend as a class, so the constructor must remain callable for backwards compatibility. Or if it was just a convenient docutils syntax for describing SMTP EmailBackend options that can be used as keywords to get_connection(). (I can't imagine why real-world code would want to call an EmailBackend constructor directly rather than going through get_connection().)

3. I think there was also some question about maybe deprecating the backend param to get_connection(). Or how and where to support new provider alias params alongside existing connection params in other django.core.mail APIs. (But I'm blanking on the details right now.)

(I'm hoping to have some time to look at this again more closely next year, probably late January at the earliest.)

[Jacob Rief]

Sorry for the huge delay. I just rebased the code base to the main branch, which is the current base for Django-6.1. All unit tests are working again.

Next, I have to address some issues @medmunds mentioned in GitHub's pull request. I hope to find some time during Christmas holidays.

As for the design decisions:

A:
Yes, having an all or nothing approach is the only viable option, imo.

B & C:
I have to reread the code to get some answers.

AFAIK, you can't squash commits after having merged another branch into your current branch. So creating a fresh PR probably is the only option. However, before doing so I'd like being sure that the pull requst will be merged. Otherwise I have to repeat this again and again.

[Natalia Bidart]

Hi Jacob R and Mike,

Thank you both for the work and discussion on this feature. As Django Fellows, Jacob Walls and I want to encourage you to move this forward, since we believe this work it's valuable and we'll do our best to help get it merged for the 6.1 feature freeze (set to May 20th).

About the implementation approach

I'd like to propose aligning EMAIL_PROVIDERS with Django's existing patterns for DATABASES, CACHES, and STORAGES. These all share a consistent API:

# Dict-like handler with named aliases for providers
from django.db import connections
from django.core.cache import caches
from django.core.files.storage import storages
from django.core.mail import providers  # proposed

# Access by alias
db_conn = connections['analytics']
cache = caches['session']
storage = storages['s3']
provider = providers['marketing']  # proposed

# Lazy proxy for default provider
from django.db import connection
from django.core.cache import cache
from django.core.files.storage import default_storage
from django.core.mail import provider  # proposed

This pattern would replace get_connection(backend=...) with providers["alias"], by having:

• Named provider configuration in EMAIL_PROVIDERS
• Consistent API across Django's connection systems
• Natural (?) solution for wrapper backends: configuration stays declarative storing provider "alias" names in OPTIONS, then resolve them via providers["alias"] at runtime, matching the pattern used with multiple databases/caches/storages.

I think the wrapper backend issue (django-mailer, etc.) resolves cleanly with this pattern (as far as I understand, let me know if I got that wrong!). We can deprecate get_connection(backend=...) in favor of providers["alias"] (hand-to-hand with EMAIL_* settings deprecation), while MAYBE maintaining get_connection() with no arguments for backwards compatibility. This may need further thinking.

For the ESP integration, and per Mike's feedback, I read that the OPTIONS approach solved the API_KEY issue for django-anymail and similar backends well. This appears resolved?

PR and commit structure, next steps

For the PR, we strongly recommend structuring commits as self-contained units of functionality, each complete with relevant tests and documentation. This allows:

• Reviewing each piece in isolation
• Merging incrementally when possible
• Easier identification of issues during review

For a feature this size, incremental review makes an enormous difference. Think of commits as logical building blocks rather than chronological changes.

So, concretely, next steps would be opening a fresh branch against current main with a clean, well-structured commit history. That would help move this forward efficiently. Jacob R, if you can prepare this, we'll definitely try to prioritize review. Mike, your continued feedback on the integration side would be highly appreciated. Please let us know if you have questions about this approach!

[Mike Edmunds]

Hi Natalia and Jacob W,

It's great to have some fresh eyes on this feature. And I agree it would be valuable to get it into 6.1.

From my perspective, one really helpful thing would be to get consensus on the proposed API changes and deprecation plans around the feature, before too much more effort goes into code. I can try to draft a short spec if that would be useful.

I really like Natalia's proposal to align EMAIL_PROVIDERS with the other connection types. And I also need to digest it a bit. (Natalia, were you thinking django.core.mail.providers could be built on the connection handler utilities in django.utils.connection? Or just looking for parallel APIs?)

One initial concern is that—as currently used—an EmailBackend instance is usually ephemeral. It typically gets constructed, used for a single send (or send_messages batch), and then abandoned to GC. Example in ​send_mass_mail(). There's nothing inherently wrong with keeping a backend instance around longer and reusing it: they're meant to support that. (The actual "connection" part of an EmailBackend instance is set up and torn down in its open() and close() methods.) But backend instance reuse is not the pattern in a lot of current code, so I'm not sure what problems we might uncover.

Threading is a different matter: none of Django's current EmailBackend implementations can be safely shared between threads, and I'm guessing that applies to many third-party backends too.

Maybe django.core.mail.providers["alias"] should return a new EmailBackend instance each time it's accessed, matching current get_connection() behavior? Then as a follow-on feature we could add a way for EmailBackend implementations to opt into caching (and indicate its level of threading support).

(Another, minor concern is that send_mail() and send_mass_mail() currently accept auth_user and auth_password params, which override the default EMAIL_HOST_USER and EMAIL_HOST_PASSWORD for that one send. But those params are already—and silently—broken when used together with connection, so we could just make it an error to use them with a provider.)

I'll put some thought into get_connection(). There are three use cases, and I'm not sure how common the last two are:

• get_connection()
• get_connection("path.to.some.EmailBackend")
• get_connection(..., extra="keyword args", are_passed="to the backend")

[Jacob Walls]

I can try to draft a short spec if that would be useful.

Big +1 from me. Nothing about DEPs says they have to be the length of a tome. It's definitely helpful to iterate on one document instead of appending ticket/PR comments :D