Opened 17 years ago

Last modified 4 months ago

#6989 assigned Bug

Inability to define DNS_NAME in django.core.mail results in e-mail messages being rejected or marked as spam

Reported by: Franklin Owned by: Jacob Rief
Component: Core (Mail) Version: dev
Severity: Normal Keywords: local_hostname, DNS_NAME, CachedDnsName, smtplib, SMTPConnection
Cc: jeremy+django@…, davidgrant@… Triage Stage: Accepted
Has patch: yes Needs documentation: yes
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Summary
Being the case that the application server's hostname may not necessarily be a valid fully qualified domain name (FQDN) the use of socket.getfqdn() in defining DNS_NAME may yield a number of e-mail messages either rejected or marked as spam. Rejection issues arise if settings.EMAIL_USE_TLS is True - meaning smtplib.ehlo() will be called - and if a mail server is particularly strict (or misconfigured) and rejects any ehlo command followed by an invalid FQDN. Spam issues arise with DNS_NAME being used as the local_hostname argument to smtplib.SMTP() within the SMTPConnection.open() method, meaning the initial "Received:" e-mail header may be from an invalid FQDN. Further increasing the chance of being marked as spam is DNS_NAME being used by the make_msgid() method, leaving the "Message-ID:" e-mail header ending in an invalid FQDN.

Real World Scenario
A shared server has a hostname, which you may not and can not change, of "star-wars-reference". The socket.getfqdn() method fails to find info on its IP address and by default simply returns said hostname. E-mail is then sent by Django with a Message-ID of "200804....@star-wars-reference" from a server claiming to be "star-wars-reference ([server's IP address])" and is, of course, marked as spam.

Solution
Provide an optional settings variable such as EMAIL_LOCAL_HOSTNAME where Django developers may define a FQDN which maps to their application's server. EMAIL_LOCAL_HOSTNAME will be used by the class defining DNS_NAME in lieu of socket.getfqdn() if EMAIL_LOCAL_HOSTNAME exists in the settings file (this has the added benefit of preventing a call to socket.getfqdn() - further reducing overhead on restarts).

Attachments (2)

mail-local-hostname.diff (465 bytes ) - added by Franklin 17 years ago.
mail-local-hostname2.diff (3.9 KB ) - added by Jeremy Grosser 14 years ago.

Download all attachments as: .zip

Change History (27)

by Franklin, 17 years ago

Attachment: mail-local-hostname.diff added

comment:1 by edgarsj, 16 years ago

Needs documentation: set
Patch needs improvement: set
Triage Stage: UnreviewedDesign decision needed

comment:2 by stilldodge, 15 years ago

This still appears to be an issue. Possibly this was unresolved because email servers used to accept an invalid FQDN. I can confirm that Exchange 2007 rejects the connection if the local host "calculated" name is not truly a valid FQDN

comment:3 by Malcolm Tredinnick, 14 years ago

Triage Stage: Design decision neededAccepted

by Jeremy Grosser, 14 years ago

Attachment: mail-local-hostname2.diff added

comment:4 by Jeremy Grosser, 14 years ago

I've attached an updated and documented patch (mail-local-hostname2.diff) allowing the user to specify EMAIL_LOCAL_HOSTNAME. Please review and let me know if any changes are necessary to get this merged.

comment:5 by Jeremy Grosser, 14 years ago

Cc: jeremy+django@… added

comment:6 by Adrian Holovaty, 14 years ago

I'm just skimming this, but my immediate reaction is: introducing a new setting is a non-starter. Is there a way to accomplish what you want without a setting?

comment:7 by Jeremy Grosser, 14 years ago

We could force anybody needing this to subclass the SMTP EmailBackend, but that seems a bit excessive to me.

comment:8 by Julien Phalip, 14 years ago

Severity: Normal
Type: Bug

comment:9 by Aymeric Augustin, 13 years ago

UI/UX: unset

Change UI/UX from NULL to False.

comment:10 by Aymeric Augustin, 13 years ago

Easy pickings: unset

Change Easy pickings from NULL to False.

comment:11 by Edwin, 11 years ago

Here is a workaround.

Python's socket.getfqdn() resolution process works like this on Linux:

  1. Get hostname of server.
  2. Resolve IP address for hostname.
  3. Resolve name for IP address.
  4. If 2 or 3 fail, just use the hostname.

Therefore, if you add a line like this to your /etc/hosts, socket.getfqdn() will resolve myhostname to 127.0.0.1 and resolve 127.0.0.1 to myfqdn since that is the first name in the list. You also have to make sure that there are no other entries for 127.0.0.1 on previous lines.

127.0.0.1       myfqdn myhostname

comment:12 by anonymous, 11 years ago

That workaround only makes sense if the server is hosting one django project. I have one Linux server with several django projects. Each uses a different fqdn but the same IP.

comment:13 by davidgrant@…, 11 years ago

Can we get this in? Relying on socket.get_fqdn() seems very wrong and patch looks good.

comment:14 by davidgrant@…, 11 years ago

So where do we go from here? Instead of a setting we could use an environment variable. Other than that I don't see any other way.

comment:15 by davidgrant@…, 11 years ago

Cc: davidgrant@… added

comment:16 by Russell Keith-Magee, 11 years ago

@davidgrant - Where do we go? We go the same way we do with any other ticket that has stalled: Start a discussion on django-dev. Summarize the problem. Summarize the alternatives. Propose a solution.

I'll also note that Adrian's comment about settings -- we've historically avoided adding new settings wherever possible. But if it truly isn't possible (at all, or for all practical purposes) to avoid a setting, it's not completely off the cards.

comment:17 by Tim Graham, 8 years ago

I think a setting is fine considering we added other mail settings recently. The patch needs to be updated, checked against the PatchReviewChecklist, and turned into a pull request.

comment:18 by heathervm, 8 years ago

Owner: changed from nobody to heathervm
Status: newassigned

comment:19 by Diederik van der Boor, 6 years ago

Python's smtplib.SMTP works around this problem with a fallback to the IP address::

            # RFC 2821 says we should use the fqdn in the EHLO/HELO verb, and
            # if that can't be calculated, that we should use a domain literal
            # instead (essentially an encoded IP address like [A.B.C.D]).
            fqdn = socket.getfqdn()
            if '.' in fqdn:
                self.local_hostname = fqdn
            else:
                # We can't find an fqdn hostname, so use a domain literal
                addr = '127.0.0.1'
                try:
                    addr = socket.gethostbyname(socket.gethostname())
                except socket.gaierror:
                    pass
                self.local_hostname = '[%s]' % addr

This is overwritten as Django's django.core.mail.backens.smtp.EmailBackend overwrites this to:

connection_params = {'local_hostname': DNS_NAME.get_fqdn()}

One possible extra fix would be to support the same fallback to IP-addresses when there is no dot in the hostname, just like smtplib.SMTP.open() does.

Last edited 6 years ago by Diederik van der Boor (previous) (diff)

comment:20 by Diederik van der Boor, 6 years ago

To add, this issue also happens on Kubernetes pods.

A standard Pod on Kubernetes has the following configuration:

/etc/hosts

# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.244.1.204	examplepod-77f687b9b9-5hlsg

/etc/resolv.conf

nameserver 10.96.0.10
search NAMESPACE.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Hence email is submitted using EHLO examplepod-77f687b9b9-5hlsg and thus triggering spam filtering.

comment:21 by Diederik van der Boor, 6 years ago

I bumbed into this again, in the end I had to fix this by patching:

django.core.mail.utils.DNS_NAME._fqdn = "my.desired.host"

comment:22 by Jacob Rief, 4 years ago

Owner: changed from heathervm to Jacob Rief

Pull request to add this feature:
https://github.com/django/django/pull/13728

However as a configuration directive I used EMAIL_MESSAGEID_FQDN because to me, that's more meaningful.
I'm open to discussions on the naming of this.

comment:23 by Alexander Lazarević, 10 months ago

I'm trying to understand the status of this ticket and the problem. I traced the discussion here, on the PR and google groups ...

The end of the discussion in google groups seems to be:

Jacob Rief 31 Jan 2022, 15:36:40 to Django developers (Contributions to Django itself)
On Monday, January 31, 2022 at 7:55:47 AM UTC+1 f.apo...@gmail.com wrote:

    Okay then,

    some of the things like sender reputation and different bounce hooks came to my mind as well, but it is good to hear confirmation from others. I think the next steps would be to create a new ticket to add support for *multiple* email backends and then work from that (I would only link the old ticket since it's scope was mainly putting the config into a dict as opposed to multiple backends). Given that there are plenty of +1 here already I think we already have our implementors? :)

    Some items that I like to see addressed in a PR:
     * Backwards compat
     * Similarity to Caches & Databases (ie so we don't invent yet another syntax)
     * Support for connection aliases (default/…) in send_email % friends (basically everything taking a connection now should probably take aliases as well)

OK. Since I made the proposal, it then is probably up to me to create the ticket.  


Mariusz Felisiak 1 Jun 2023, 21:48:04 to Django developers (Contributions to Django itself)

It seems we have a consensus here to support multiple email backends. Jacob, feel-free to create a new ticket.

Best, Mariusz

I couldn't find the Ticket by Jacob or anyone related to this.

A related ticket #31885 got fixed and closed. In the description it is said that "We will then recommend subclassing in the docs for more control. "

Is subclassing now also the proposed method for setting a custom DNS_NAME (or other option)? (I does not seemed to be mentioned in the docs)

Is the support for multiple email backends abandoned for now?

in reply to:  23 comment:24 by Mariusz Felisiak, 10 months ago

Replying to Alexander Lazarević:

Is subclassing now also the proposed method for setting a custom DNS_NAME (or other option)? (I does not seemed to be mentioned in the docs)

Yes, subclassing is basically a way to go for any fancy change to the EmailBackend behavior.

Is the support for multiple email backends abandoned for now?

Not really, it looks like no one had time to create a ticket. Please do if you want.

comment:25 by Mike Edmunds, 4 months ago

support for multiple email backends

See #35514 Dictionary based EMAIL_PROVIDERS settings

Note: See TracTickets for help on using tickets.
Back to Top