Opened 17 years ago
Last modified 4 months ago
#6989 assigned Bug
Inability to define DNS_NAME in django.core.mail results in e-mail messages being rejected or marked as spam
Reported by: | Franklin | Owned by: | Jacob Rief |
---|---|---|---|
Component: | Core (Mail) | Version: | dev |
Severity: | Normal | Keywords: | local_hostname, DNS_NAME, CachedDnsName, smtplib, SMTPConnection |
Cc: | jeremy+django@…, davidgrant@… | Triage Stage: | Accepted |
Has patch: | yes | Needs documentation: | yes |
Needs tests: | no | Patch needs improvement: | yes |
Easy pickings: | no | UI/UX: | no |
Description
Summary
Being the case that the application server's hostname may not necessarily be a valid fully qualified domain name (FQDN) the use of socket.getfqdn() in defining DNS_NAME may yield a number of e-mail messages either rejected or marked as spam. Rejection issues arise if settings.EMAIL_USE_TLS is True - meaning smtplib.ehlo() will be called - and if a mail server is particularly strict (or misconfigured) and rejects any ehlo command followed by an invalid FQDN. Spam issues arise with DNS_NAME being used as the local_hostname argument to smtplib.SMTP() within the SMTPConnection.open() method, meaning the initial "Received:" e-mail header may be from an invalid FQDN. Further increasing the chance of being marked as spam is DNS_NAME being used by the make_msgid() method, leaving the "Message-ID:" e-mail header ending in an invalid FQDN.
Real World Scenario
A shared server has a hostname, which you may not and can not change, of "star-wars-reference". The socket.getfqdn() method fails to find info on its IP address and by default simply returns said hostname. E-mail is then sent by Django with a Message-ID of "200804....@star-wars-reference" from a server claiming to be "star-wars-reference ([server's IP address])" and is, of course, marked as spam.
Solution
Provide an optional settings variable such as EMAIL_LOCAL_HOSTNAME where Django developers may define a FQDN which maps to their application's server. EMAIL_LOCAL_HOSTNAME will be used by the class defining DNS_NAME in lieu of socket.getfqdn() if EMAIL_LOCAL_HOSTNAME exists in the settings file (this has the added benefit of preventing a call to socket.getfqdn() - further reducing overhead on restarts).
Attachments (2)
Change History (27)
by , 17 years ago
Attachment: | mail-local-hostname.diff added |
---|
comment:1 by , 16 years ago
Needs documentation: | set |
---|---|
Patch needs improvement: | set |
Triage Stage: | Unreviewed → Design decision needed |
comment:2 by , 15 years ago
comment:3 by , 14 years ago
Triage Stage: | Design decision needed → Accepted |
---|
by , 14 years ago
Attachment: | mail-local-hostname2.diff added |
---|
comment:4 by , 14 years ago
I've attached an updated and documented patch (mail-local-hostname2.diff) allowing the user to specify EMAIL_LOCAL_HOSTNAME. Please review and let me know if any changes are necessary to get this merged.
comment:5 by , 14 years ago
Cc: | added |
---|
comment:6 by , 14 years ago
I'm just skimming this, but my immediate reaction is: introducing a new setting is a non-starter. Is there a way to accomplish what you want without a setting?
comment:7 by , 14 years ago
We could force anybody needing this to subclass the SMTP EmailBackend, but that seems a bit excessive to me.
comment:8 by , 14 years ago
Severity: | → Normal |
---|---|
Type: | → Bug |
comment:11 by , 11 years ago
Here is a workaround.
Python's socket.getfqdn() resolution process works like this on Linux:
- Get hostname of server.
- Resolve IP address for hostname.
- Resolve name for IP address.
- If 2 or 3 fail, just use the hostname.
Therefore, if you add a line like this to your /etc/hosts, socket.getfqdn() will resolve myhostname to 127.0.0.1 and resolve 127.0.0.1 to myfqdn since that is the first name in the list. You also have to make sure that there are no other entries for 127.0.0.1 on previous lines.
127.0.0.1 myfqdn myhostname
comment:12 by , 11 years ago
That workaround only makes sense if the server is hosting one django project. I have one Linux server with several django projects. Each uses a different fqdn but the same IP.
comment:13 by , 11 years ago
Can we get this in? Relying on socket.get_fqdn() seems very wrong and patch looks good.
comment:14 by , 11 years ago
So where do we go from here? Instead of a setting we could use an environment variable. Other than that I don't see any other way.
comment:15 by , 11 years ago
Cc: | added |
---|
comment:16 by , 11 years ago
@davidgrant - Where do we go? We go the same way we do with any other ticket that has stalled: Start a discussion on django-dev. Summarize the problem. Summarize the alternatives. Propose a solution.
I'll also note that Adrian's comment about settings -- we've historically avoided adding new settings wherever possible. But if it truly isn't possible (at all, or for all practical purposes) to avoid a setting, it's not completely off the cards.
comment:17 by , 8 years ago
I think a setting is fine considering we added other mail settings recently. The patch needs to be updated, checked against the PatchReviewChecklist, and turned into a pull request.
comment:18 by , 8 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:19 by , 6 years ago
Python's smtplib.SMTP
works around this problem with a fallback to the IP address::
# RFC 2821 says we should use the fqdn in the EHLO/HELO verb, and # if that can't be calculated, that we should use a domain literal # instead (essentially an encoded IP address like [A.B.C.D]). fqdn = socket.getfqdn() if '.' in fqdn: self.local_hostname = fqdn else: # We can't find an fqdn hostname, so use a domain literal addr = '127.0.0.1' try: addr = socket.gethostbyname(socket.gethostname()) except socket.gaierror: pass self.local_hostname = '[%s]' % addr
This is overwritten as Django's django.core.mail.backens.smtp.EmailBackend
overwrites this to:
connection_params = {'local_hostname': DNS_NAME.get_fqdn()}
One possible extra fix would be to support the same fallback to IP-addresses when there is no dot in the hostname, just like smtplib.SMTP.open()
does.
comment:20 by , 6 years ago
To add, this issue also happens on Kubernetes pods.
A standard Pod on Kubernetes has the following configuration:
/etc/hosts
# Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.244.1.204 examplepod-77f687b9b9-5hlsg
/etc/resolv.conf
nameserver 10.96.0.10 search NAMESPACE.svc.cluster.local svc.cluster.local cluster.local options ndots:5
Hence email is submitted using EHLO examplepod-77f687b9b9-5hlsg
and thus triggering spam filtering.
comment:21 by , 6 years ago
I bumbed into this again, in the end I had to fix this by patching:
django.core.mail.utils.DNS_NAME._fqdn = "my.desired.host"
comment:22 by , 4 years ago
Owner: | changed from | to
---|
Pull request to add this feature:
https://github.com/django/django/pull/13728
However as a configuration directive I used EMAIL_MESSAGEID_FQDN because to me, that's more meaningful.
I'm open to discussions on the naming of this.
follow-up: 24 comment:23 by , 10 months ago
I'm trying to understand the status of this ticket and the problem. I traced the discussion here, on the PR and google groups ...
The end of the discussion in google groups seems to be:
Jacob Rief 31 Jan 2022, 15:36:40 to Django developers (Contributions to Django itself) On Monday, January 31, 2022 at 7:55:47 AM UTC+1 f.apo...@gmail.com wrote: Okay then, some of the things like sender reputation and different bounce hooks came to my mind as well, but it is good to hear confirmation from others. I think the next steps would be to create a new ticket to add support for *multiple* email backends and then work from that (I would only link the old ticket since it's scope was mainly putting the config into a dict as opposed to multiple backends). Given that there are plenty of +1 here already I think we already have our implementors? :) Some items that I like to see addressed in a PR: * Backwards compat * Similarity to Caches & Databases (ie so we don't invent yet another syntax) * Support for connection aliases (default/…) in send_email % friends (basically everything taking a connection now should probably take aliases as well) OK. Since I made the proposal, it then is probably up to me to create the ticket. Mariusz Felisiak 1 Jun 2023, 21:48:04 to Django developers (Contributions to Django itself) It seems we have a consensus here to support multiple email backends. Jacob, feel-free to create a new ticket. Best, Mariusz
I couldn't find the Ticket by Jacob or anyone related to this.
A related ticket #31885 got fixed and closed. In the description it is said that "We will then recommend subclassing in the docs for more control. "
Is subclassing now also the proposed method for setting a custom DNS_NAME (or other option)? (I does not seemed to be mentioned in the docs)
Is the support for multiple email backends abandoned for now?
comment:24 by , 10 months ago
Replying to Alexander Lazarević:
Is subclassing now also the proposed method for setting a custom DNS_NAME (or other option)? (I does not seemed to be mentioned in the docs)
Yes, subclassing is basically a way to go for any fancy change to the EmailBackend
behavior.
Is the support for multiple email backends abandoned for now?
Not really, it looks like no one had time to create a ticket. Please do if you want.
comment:25 by , 4 months ago
support for multiple email backends
See #35514 Dictionary based EMAIL_PROVIDERS settings
This still appears to be an issue. Possibly this was unresolved because email servers used to accept an invalid FQDN. I can confirm that Exchange 2007 rejects the connection if the local host "calculated" name is not truly a valid FQDN